[CKAN-Security] Solr Manipulation

Wed Aug 16 10:54:48 UTC 2017

Hi Gil,

That makes sense. Sorry, in your original email you didn't mention
DataPusher and that confused me.
We'll discuss this in tomorrow's meeting and get back to you.

Adrià

On 16 August 2017 at 11:17, Gil Hilário <gil at civity.nl> wrote:

> Hi Adrià
>
>
> So there is a little misunderstanding in my explanation. What you describe
> as step 2 “2. Once the dataset is created, if someone clicks on the link
> from the resource page, the Solr index gets deleted.
>
> ” is not what I meant. The issue is not related with users clicking the
> link, because as you said, the solr is not publicly accessible to the
> world, but when the DataPusher tries to push it.
>
> We were unable to reproduce (a non-destructive version of this bug) it on
> demo.ckan.org , since you already fixed the previous issue (localfile
> usage) and therefore we could not guess the Solr port.
>
> Therefore we cannot read the the solr url in the ini file (which is good)
> and we unable to guess it without really hacking/trying
>
>
>
> CKAN is able to request the solr, by default solr has no authentication
> and it is requestable from the CKAN instance (otherwise CKAN would not work)
>
>
>
>    - Let's assume it is (using the default guide) on
>    http://localhost:8983/solr.
>    - The datapusher is running allongside CKAN website, default on port
>    8800.
>    - The datapusher is then requesting the url, which can resolve
>    anywhere in the world, but also on the server network (in most cases, on
>    localhost or 127.0.0.1)
>
>
>
> Solr is NOT accessible from the world, but Is accessible from the
> datapusher. So pushing the data to the the datastore, would result in
> executing a SOLR command (in our case, wipe the content).
>
>
>
> Anyway your suggestion of extending the default resource URL validation
> would provide a more secure approach.
>
>
>
> Best Regards,
>
> Gil Hilário
>
>
>
>
>
>
>
>
>
>
>
> [image: logo civity new]
>
>
>
> *T* +31 (0)6 24 16 07 23 | *E* gil at civity.nl
> Handelsweg 6-1 | 3707 NH Zeist
> *W* www.civity.nl
>
>
>
>
>
>
>
>
>
>
>
>
>
> *Civity is onderdeel van de Onetrail groep (www.onetrail.com
> <http://www.onetrail.com/>)*
>
> *Civity is initiatiefnemer van FIWARE LAB Nederland, de open innovatie
> omgeving voor smart cities **www.fiware-lab.nl*
> <http://www.fiware-lab.nl/>
>
> [image: cid:image005.png at 01D0D9C8.8D3B3A60]
> <https://www.linkedin.com/company/3284795?trk=tyah&trkInfo=clickedVertical%3Acompany%2CclickedEntityId%3A3284795%2Cidx%3A2-3-4%2CtarId%3A1473335093147%2Ctas%3Acivity>*[image:
> cid:image005.png at 01D20A7C.FC86C980]*
> <https://twitter.com/intent/follow?original_referer=https://about.twitter.com/resources/buttons&region=follow_link&screen_name=CivityNL&tw_p=followbutton>
>
>
>
>
>
>
>
>
>
> *From:* Adrià Mercader [mailto:adria.mercader at okfn.org]
> *Sent:* woensdag 16 augustus 2017 11:41
> *To:* Gil Hilário <gil at civity.nl>
> *Cc:* CKAN Security Alerts/Discussions <security at lists.okfn.org>
> *Subject:* Re: [CKAN-Security] Solr Manipulation
>
>
>
> Hi again Gil,
>
>
>
> Just to make sure we are discussing the same issue. The steps you describe
> are:
>
>
>
> 1. Someone creates a resource with a URL pointing to the local Solr
> instance (eg http://localhost:8983/solr/ckan-schema-2.3/update?
> commit=true&stream.body=%3cdelete%3e%3cquery%3e*:*%3c/
> query%3e%3c/delete%3e)
>
> 2. Once the dataset is created, if someone clicks on the link from the
> resource page, the Solr index gets deleted.
>
>
>
> If I understand correctly this is only an issue if you are using a CKAN
> instance in your local server / machine, where localhost:8983 gets resolved
> to the Solr instance that is locally accessible.
>
> In any other setup, the Solr instance would not publicly accessible, as
> the only port exposed publicly would be nginx and if users clicked on that
> link they would just get a not found error (Unless they have something
> running on localhost on port 8983).
>
>
>
> So there are two things described here:
>
>
>
> 1. Solr instance being accessible publicly: this should never happen on a
> production environment. You'll most likely only expose Nginx and connect to
> the internal services like Solr and Postgres inside the firewall.
>
> 2. Ability for users to link to arbitrary URLs, including administrative
> ones like the Solr example. CKAN can't filter the URLs by default as each
> CKAN site might have different requirements. It should be really easy to
> extend the default resource URL validation to customize to your own needs
> though.
>
>
>
> Hope all this makes sense, please let me know if the issue is not the one
> that I've described or if you have any other comments.
>
>
>
> Best regards,
>
>
> Adrià
>
>
>
> On 14 August 2017 at 16:04, Adrià Mercader <adria.mercader at okfn.org>
> wrote:
>
> Hi Gil,
>
>
>
> Thanks for your report. The technical team will discuss the issue and come
> back to you as soon as possible.
>
>
>
> Cheers,
>
>
>
> Adrià
>
>
>
> On 14 August 2017 at 10:40, Gil Hilário <gil at civity.nl> wrote:
>
> Hi,
>
> I recently reported a security flaw over public issue, this time I’m doing
> it through the right channel.
>
> We identified an issue that allows you to manipulate the Solr through a
> linked dataset.
>
> If you create a new dataset and you add a resource. If you give it the
> following URL, for example, http://localhost:8983/solr/
> ckan-schema-2.3/update?commit=true&stream.body=<delete><
> query>*:*</query></delete> it will simply remove all the datasets from
> the solr, and you will need to run the reindexer.
> The same way you can delete and manipulate all datasets because you have
> full access to Solr. The data itself is never touched but we though this
> was worth sharing with you because the possibilities of changing the
> perception of what’s available on the CKAN instance are limiteless.
>
> Best Regards,
>
> Gil Hilário
>
>
>
>
>
>
>
>
>
>
>
> [image: logo civity new]
>
>
>
> *T* +31 (0)6 24 16 07 23 | *E* gil at civity.nl
> Handelsweg 6-1 | 3707 NH Zeist
> *W* www.civity.nl
>
>
>
>
>
>
>
>
>
>
>
>
>
> *Civity is onderdeel van de Onetrail groep (www.onetrail.com
> <http://www.onetrail.com/>)*
>
> *Civity is initiatiefnemer van FIWARE LAB Nederland, de open innovatie
> omgeving voor smart cities **www.fiware-lab.nl*
> <http://www.fiware-lab.nl/>
>
> [image: cid:image005.png at 01D0D9C8.8D3B3A60]
> <https://www.linkedin.com/company/3284795?trk=tyah&trkInfo=clickedVertical%3Acompany%2CclickedEntityId%3A3284795%2Cidx%3A2-3-4%2CtarId%3A1473335093147%2Ctas%3Acivity>*[image:
> cid:image005.png at 01D20A7C.FC86C980]*
> <https://twitter.com/intent/follow?original_referer=https://about.twitter.com/resources/buttons&region=follow_link&screen_name=CivityNL&tw_p=followbutton>
>
>
>
>
>
>
>
> _______________________________________________
> CKAN security
> https://lists.okfn.org/mailman/listinfo/security
> https://lists.okfn.org/mailman/options/security/adria.mercader%40okfn.org
>
> Repo: https://github.com/ckan/ckan-security
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.okfn.org/mailman/private/security/attachments/20170816/d7163c81/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 3186 bytes
Desc: not available
URL: <https://lists.okfn.org/mailman/private/security/attachments/20170816/d7163c81/attachment-0001.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 1578 bytes
Desc: not available
URL: <https://lists.okfn.org/mailman/private/security/attachments/20170816/d7163c81/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 1828 bytes
Desc: not available
URL: <https://lists.okfn.org/mailman/private/security/attachments/20170816/d7163c81/attachment-0003.png>