[CKAN-Security] Solr Manipulation

Wed Aug 16 14:18:01 UTC 2017

by "off" I mean uploaded files would still work, which should be safe and
sufficient for most sites. I don't know how we would check the permission
of the user from a dataset since different users could have updated
different resources

On Wed, Aug 16, 2017 at 10:12 AM, Tyler Kennedy <tk at tkte.ch> wrote:

> Maybe instead of on or off a minimum security level? Must be user, admin
> or sysadmin?
>
> On Aug 16, 2017 9:51 AM, "Ian Ward" <ian at excess.org> wrote:
>
>> It's impossible to list all the URLs that might expose sensitive
>> information when exposed to datapusher (first we block localhost, then
>> 127.*, then ::1, then private ranges, and so on..) We'll need to make
>> datapusher fetching URLs a config option and document that it shouldn't be
>> used unless the datapusher instance is installed outside the internal
>> network.
>>
>> On Wed, Aug 16, 2017 at 6:54 AM, Adrià Mercader <adria.mercader at okfn.org>
>> wrote:
>>
>>> Hi Gil,
>>>
>>> That makes sense. Sorry, in your original email you didn't mention
>>> DataPusher and that confused me.
>>> We'll discuss this in tomorrow's meeting and get back to you.
>>>
>>> Adrià
>>>
>>>
>>> On 16 August 2017 at 11:17, Gil Hilário <gil at civity.nl> wrote:
>>>
>>>> Hi Adrià
>>>>
>>>>
>>>> So there is a little misunderstanding in my explanation. What you
>>>> describe as step 2 “2. Once the dataset is created, if someone clicks
>>>> on the link from the resource page, the Solr index gets deleted.
>>>>
>>>> ” is not what I meant. The issue is not related with users clicking the
>>>> link, because as you said, the solr is not publicly accessible to the
>>>> world, but when the DataPusher tries to push it.
>>>>
>>>> We were unable to reproduce (a non-destructive version of this bug) it
>>>> on demo.ckan.org , since you already fixed the previous issue
>>>> (localfile usage) and therefore we could not guess the Solr port.
>>>>
>>>> Therefore we cannot read the the solr url in the ini file (which is
>>>> good) and we unable to guess it without really hacking/trying
>>>>
>>>>
>>>>
>>>> CKAN is able to request the solr, by default solr has no authentication
>>>> and it is requestable from the CKAN instance (otherwise CKAN would not work)
>>>>
>>>>
>>>>
>>>>    - Let's assume it is (using the default guide) on
>>>>    http://localhost:8983/solr.
>>>>    - The datapusher is running allongside CKAN website, default on
>>>>    port 8800.
>>>>    - The datapusher is then requesting the url, which can resolve
>>>>    anywhere in the world, but also on the server network (in most cases, on
>>>>    localhost or 127.0.0.1)
>>>>
>>>>
>>>>
>>>> Solr is NOT accessible from the world, but Is accessible from the
>>>> datapusher. So pushing the data to the the datastore, would result in
>>>> executing a SOLR command (in our case, wipe the content).
>>>>
>>>>
>>>>
>>>> Anyway your suggestion of extending the default resource URL
>>>> validation would provide a more secure approach.
>>>>
>>>>
>>>>
>>>> Best Regards,
>>>>
>>>> Gil Hilário
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> [image: logo civity new]
>>>>
>>>>
>>>>
>>>> *T* +31 (0)6 24 16 07 23 | *E* gil at civity.nl
>>>> Handelsweg 6-1 | 3707 NH Zeist
>>>> *W* www.civity.nl
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *Civity is onderdeel van de Onetrail groep (www.onetrail.com
>>>> <http://www.onetrail.com/>)*
>>>>
>>>> *Civity is initiatiefnemer van FIWARE LAB Nederland, de open innovatie
>>>> omgeving voor smart cities **www.fiware-lab.nl*
>>>> <http://www.fiware-lab.nl/>
>>>>
>>>> [image: cid:image005.png at 01D0D9C8.8D3B3A60]
>>>> <https://www.linkedin.com/company/3284795?trk=tyah&trkInfo=clickedVertical%3Acompany%2CclickedEntityId%3A3284795%2Cidx%3A2-3-4%2CtarId%3A1473335093147%2Ctas%3Acivity>*[image:
>>>> cid:image005.png at 01D20A7C.FC86C980]*
>>>> <https://twitter.com/intent/follow?original_referer=https://about.twitter.com/resources/buttons&region=follow_link&screen_name=CivityNL&tw_p=followbutton>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *From:* Adrià Mercader [mailto:adria.mercader at okfn.org]
>>>> *Sent:* woensdag 16 augustus 2017 11:41
>>>> *To:* Gil Hilário <gil at civity.nl>
>>>> *Cc:* CKAN Security Alerts/Discussions <security at lists.okfn.org>
>>>> *Subject:* Re: [CKAN-Security] Solr Manipulation
>>>>
>>>>
>>>>
>>>> Hi again Gil,
>>>>
>>>>
>>>>
>>>> Just to make sure we are discussing the same issue. The steps you
>>>> describe are:
>>>>
>>>>
>>>>
>>>> 1. Someone creates a resource with a URL pointing to the local Solr
>>>> instance (eg http://localhost:8983/solr/ckan-schema-2.3/update?commit
>>>> =true&stream.body=%3cdelete%3e%3cquery%3e*:*%3c/query%3e%3c/delete%3e)
>>>>
>>>> 2. Once the dataset is created, if someone clicks on the link from the
>>>> resource page, the Solr index gets deleted.
>>>>
>>>>
>>>>
>>>> If I understand correctly this is only an issue if you are using a CKAN
>>>> instance in your local server / machine, where localhost:8983 gets resolved
>>>> to the Solr instance that is locally accessible.
>>>>
>>>> In any other setup, the Solr instance would not publicly accessible, as
>>>> the only port exposed publicly would be nginx and if users clicked on that
>>>> link they would just get a not found error (Unless they have something
>>>> running on localhost on port 8983).
>>>>
>>>>
>>>>
>>>> So there are two things described here:
>>>>
>>>>
>>>>
>>>> 1. Solr instance being accessible publicly: this should never happen on
>>>> a production environment. You'll most likely only expose Nginx and connect
>>>> to the internal services like Solr and Postgres inside the firewall.
>>>>
>>>> 2. Ability for users to link to arbitrary URLs, including
>>>> administrative ones like the Solr example. CKAN can't filter the URLs by
>>>> default as each CKAN site might have different requirements. It should be
>>>> really easy to extend the default resource URL validation to customize to
>>>> your own needs though.
>>>>
>>>>
>>>>
>>>> Hope all this makes sense, please let me know if the issue is not the
>>>> one that I've described or if you have any other comments.
>>>>
>>>>
>>>>
>>>> Best regards,
>>>>
>>>>
>>>> Adrià
>>>>
>>>>
>>>>
>>>> On 14 August 2017 at 16:04, Adrià Mercader <adria.mercader at okfn.org>
>>>> wrote:
>>>>
>>>> Hi Gil,
>>>>
>>>>
>>>>
>>>> Thanks for your report. The technical team will discuss the issue and
>>>> come back to you as soon as possible.
>>>>
>>>>
>>>>
>>>> Cheers,
>>>>
>>>>
>>>>
>>>> Adrià
>>>>
>>>>
>>>>
>>>> On 14 August 2017 at 10:40, Gil Hilário <gil at civity.nl> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I recently reported a security flaw over public issue, this time I’m
>>>> doing it through the right channel.
>>>>
>>>> We identified an issue that allows you to manipulate the Solr through a
>>>> linked dataset.
>>>>
>>>> If you create a new dataset and you add a resource. If you give it the
>>>> following URL, for example, http://localhost:8983/solr/cka
>>>> n-schema-2.3/update?commit=true&stream.body=<delete><query>*
>>>> :*</query></delete> it will simply remove all the datasets from the
>>>> solr, and you will need to run the reindexer.
>>>> The same way you can delete and manipulate all datasets because you
>>>> have full access to Solr. The data itself is never touched but we though
>>>> this was worth sharing with you because the possibilities of changing the
>>>> perception of what’s available on the CKAN instance are limiteless.
>>>>
>>>> Best Regards,
>>>>
>>>> Gil Hilário
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> [image: logo civity new]
>>>>
>>>>
>>>>
>>>> *T* +31 (0)6 24 16 07 23 | *E* gil at civity.nl
>>>> Handelsweg 6-1 | 3707 NH Zeist
>>>> *W* www.civity.nl
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *Civity is onderdeel van de Onetrail groep (www.onetrail.com
>>>> <http://www.onetrail.com/>)*
>>>>
>>>> *Civity is initiatiefnemer van FIWARE LAB Nederland, de open innovatie
>>>> omgeving voor smart cities **www.fiware-lab.nl*
>>>> <http://www.fiware-lab.nl/>
>>>>
>>>> [image: cid:image005.png at 01D0D9C8.8D3B3A60]
>>>> <https://www.linkedin.com/company/3284795?trk=tyah&trkInfo=clickedVertical%3Acompany%2CclickedEntityId%3A3284795%2Cidx%3A2-3-4%2CtarId%3A1473335093147%2Ctas%3Acivity>*[image:
>>>> cid:image005.png at 01D20A7C.FC86C980]*
>>>> <https://twitter.com/intent/follow?original_referer=https://about.twitter.com/resources/buttons&region=follow_link&screen_name=CivityNL&tw_p=followbutton>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> CKAN security
>>>> https://lists.okfn.org/mailman/listinfo/security
>>>> https://lists.okfn.org/mailman/options/security/adria.mercad
>>>> er%40okfn.org
>>>>
>>>> Repo: https://github.com/ckan/ckan-security
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> _______________________________________________
>>> CKAN security
>>> https://lists.okfn.org/mailman/listinfo/security
>>> https://lists.okfn.org/mailman/options/security/ian%40excess.org
>>>
>>> Repo: https://github.com/ckan/ckan-security
>>>
>>
>>
>> _______________________________________________
>> CKAN security
>> https://lists.okfn.org/mailman/listinfo/security
>> https://lists.okfn.org/mailman/options/security/tk%40tkte.ch
>>
>> Repo: https://github.com/ckan/ckan-security
>>
>
> _______________________________________________
> CKAN security
> https://lists.okfn.org/mailman/listinfo/security
> https://lists.okfn.org/mailman/options/security/ian%40excess.org
>
> Repo: https://github.com/ckan/ckan-security
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.okfn.org/mailman/private/security/attachments/20170816/a302d844/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 1578 bytes
Desc: not available
URL: <https://lists.okfn.org/mailman/private/security/attachments/20170816/a302d844/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 3186 bytes
Desc: not available
URL: <https://lists.okfn.org/mailman/private/security/attachments/20170816/a302d844/attachment-0001.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 1828 bytes
Desc: not available
URL: <https://lists.okfn.org/mailman/private/security/attachments/20170816/a302d844/attachment-0003.png>