[CKAN-Security] Solr Manipulation

Wed Aug 16 18:41:55 UTC 2017

It would also be sufficient to warn users that datapusher will expose any
HTTP resources available from the datapusher server, so all users that can
create resources can access all resources visible from there. http auth on
solr and redis would be a great idea regardless.

On Wed, Aug 16, 2017 at 12:42 PM, David Read <david.read at hackneyworkshop.com
> wrote:

> I struggling to see how we can tell people how to easily deploy the
> datapusher worker app, so that it is properly isolated from SOLR and
> everything else. I guess it needs a second host running outside the main
> CKAN host, so that there is a firewall in-between? Maybe the easiest way is
> to show people how to run the worker on Heroku/Google App Engine as a
> freemium thing?
>
> Alternatively we could protect SOLR using some basic auth - arguably we
> should do this anyway. Redis too. The tokens would be in our ckan config
> (and possibly environment variables), and not accessible to a malicious
> datapusher's request. So not hard to do.
>
> Ian said:
> > We'll need to make datapusher fetching URLs a config option and
> document that it shouldn't be used unless the datapusher instance is
> installed outside the internal network.
>
> I think it is pretty common to use CKAN with external URLs (rather than
> pure uploads), and want to have them pushed into datastore. So I'd be
> concerned if the implication here was that we shouldn't try too hard to
> continue to enable it by default and document how to set it up. Is that
> what's meant or did I get the wrong end of the stick here?
>
> David
>
> On 16 August 2017 at 14:51, Ian Ward <ian at excess.org> wrote:
>
>> It's impossible to list all the URLs that might expose sensitive
>> information when exposed to datapusher (first we block localhost, then
>> 127.*, then ::1, then private ranges, and so on..) We'll need to make
>> datapusher fetching URLs a config option and document that it shouldn't be
>> used unless the datapusher instance is installed outside the internal
>> network.
>>
>> On Wed, Aug 16, 2017 at 6:54 AM, Adrià Mercader <adria.mercader at okfn.org>
>> wrote:
>>
>>> Hi Gil,
>>>
>>> That makes sense. Sorry, in your original email you didn't mention
>>> DataPusher and that confused me.
>>> We'll discuss this in tomorrow's meeting and get back to you.
>>>
>>> Adrià
>>>
>>>
>>> On 16 August 2017 at 11:17, Gil Hilário <gil at civity.nl> wrote:
>>>
>>>> Hi Adrià
>>>>
>>>>
>>>> So there is a little misunderstanding in my explanation. What you
>>>> describe as step 2 “2. Once the dataset is created, if someone clicks
>>>> on the link from the resource page, the Solr index gets deleted.
>>>>
>>>> ” is not what I meant. The issue is not related with users clicking the
>>>> link, because as you said, the solr is not publicly accessible to the
>>>> world, but when the DataPusher tries to push it.
>>>>
>>>> We were unable to reproduce (a non-destructive version of this bug) it
>>>> on demo.ckan.org , since you already fixed the previous issue
>>>> (localfile usage) and therefore we could not guess the Solr port.
>>>>
>>>> Therefore we cannot read the the solr url in the ini file (which is
>>>> good) and we unable to guess it without really hacking/trying
>>>>
>>>>
>>>>
>>>> CKAN is able to request the solr, by default solr has no authentication
>>>> and it is requestable from the CKAN instance (otherwise CKAN would not work)
>>>>
>>>>
>>>>
>>>>    - Let's assume it is (using the default guide) on
>>>>    http://localhost:8983/solr.
>>>>    - The datapusher is running allongside CKAN website, default on
>>>>    port 8800.
>>>>    - The datapusher is then requesting the url, which can resolve
>>>>    anywhere in the world, but also on the server network (in most cases, on
>>>>    localhost or 127.0.0.1)
>>>>
>>>>
>>>>
>>>> Solr is NOT accessible from the world, but Is accessible from the
>>>> datapusher. So pushing the data to the the datastore, would result in
>>>> executing a SOLR command (in our case, wipe the content).
>>>>
>>>>
>>>>
>>>> Anyway your suggestion of extending the default resource URL
>>>> validation would provide a more secure approach.
>>>>
>>>>
>>>>
>>>> Best Regards,
>>>>
>>>> Gil Hilário
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> [image: logo civity new]
>>>>
>>>>
>>>>
>>>> *T* +31 (0)6 24 16 07 23 | *E* gil at civity.nl
>>>> Handelsweg 6-1 | 3707 NH Zeist
>>>> *W* www.civity.nl
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *Civity is onderdeel van de Onetrail groep (www.onetrail.com
>>>> <http://www.onetrail.com/>)*
>>>>
>>>> *Civity is initiatiefnemer van FIWARE LAB Nederland, de open innovatie
>>>> omgeving voor smart cities **www.fiware-lab.nl*
>>>> <http://www.fiware-lab.nl/>
>>>>
>>>> [image: cid:image005.png at 01D0D9C8.8D3B3A60]
>>>> <https://www.linkedin.com/company/3284795?trk=tyah&trkInfo=clickedVertical%3Acompany%2CclickedEntityId%3A3284795%2Cidx%3A2-3-4%2CtarId%3A1473335093147%2Ctas%3Acivity>*[image:
>>>> cid:image005.png at 01D20A7C.FC86C980]*
>>>> <https://twitter.com/intent/follow?original_referer=https://about.twitter.com/resources/buttons&region=follow_link&screen_name=CivityNL&tw_p=followbutton>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *From:* Adrià Mercader [mailto:adria.mercader at okfn.org]
>>>> *Sent:* woensdag 16 augustus 2017 11:41
>>>> *To:* Gil Hilário <gil at civity.nl>
>>>> *Cc:* CKAN Security Alerts/Discussions <security at lists.okfn.org>
>>>> *Subject:* Re: [CKAN-Security] Solr Manipulation
>>>>
>>>>
>>>>
>>>> Hi again Gil,
>>>>
>>>>
>>>>
>>>> Just to make sure we are discussing the same issue. The steps you
>>>> describe are:
>>>>
>>>>
>>>>
>>>> 1. Someone creates a resource with a URL pointing to the local Solr
>>>> instance (eg http://localhost:8983/solr/ckan-schema-2.3/update?commit
>>>> =true&stream.body=%3cdelete%3e%3cquery%3e*:*%3c/query%3e%3c/delete%3e)
>>>>
>>>> 2. Once the dataset is created, if someone clicks on the link from the
>>>> resource page, the Solr index gets deleted.
>>>>
>>>>
>>>>
>>>> If I understand correctly this is only an issue if you are using a CKAN
>>>> instance in your local server / machine, where localhost:8983 gets resolved
>>>> to the Solr instance that is locally accessible.
>>>>
>>>> In any other setup, the Solr instance would not publicly accessible, as
>>>> the only port exposed publicly would be nginx and if users clicked on that
>>>> link they would just get a not found error (Unless they have something
>>>> running on localhost on port 8983).
>>>>
>>>>
>>>>
>>>> So there are two things described here:
>>>>
>>>>
>>>>
>>>> 1. Solr instance being accessible publicly: this should never happen on
>>>> a production environment. You'll most likely only expose Nginx and connect
>>>> to the internal services like Solr and Postgres inside the firewall.
>>>>
>>>> 2. Ability for users to link to arbitrary URLs, including
>>>> administrative ones like the Solr example. CKAN can't filter the URLs by
>>>> default as each CKAN site might have different requirements. It should be
>>>> really easy to extend the default resource URL validation to customize to
>>>> your own needs though.
>>>>
>>>>
>>>>
>>>> Hope all this makes sense, please let me know if the issue is not the
>>>> one that I've described or if you have any other comments.
>>>>
>>>>
>>>>
>>>> Best regards,
>>>>
>>>>
>>>> Adrià
>>>>
>>>>
>>>>
>>>> On 14 August 2017 at 16:04, Adrià Mercader <adria.mercader at okfn.org>
>>>> wrote:
>>>>
>>>> Hi Gil,
>>>>
>>>>
>>>>
>>>> Thanks for your report. The technical team will discuss the issue and
>>>> come back to you as soon as possible.
>>>>
>>>>
>>>>
>>>> Cheers,
>>>>
>>>>
>>>>
>>>> Adrià
>>>>
>>>>
>>>>
>>>> On 14 August 2017 at 10:40, Gil Hilário <gil at civity.nl> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I recently reported a security flaw over public issue, this time I’m
>>>> doing it through the right channel.
>>>>
>>>> We identified an issue that allows you to manipulate the Solr through a
>>>> linked dataset.
>>>>
>>>> If you create a new dataset and you add a resource. If you give it the
>>>> following URL, for example, http://localhost:8983/solr/cka
>>>> n-schema-2.3/update?commit=true&stream.body=<delete><query>*
>>>> :*</query></delete> it will simply remove all the datasets from the
>>>> solr, and you will need to run the reindexer.
>>>> The same way you can delete and manipulate all datasets because you
>>>> have full access to Solr. The data itself is never touched but we though
>>>> this was worth sharing with you because the possibilities of changing the
>>>> perception of what’s available on the CKAN instance are limiteless.
>>>>
>>>> Best Regards,
>>>>
>>>> Gil Hilário
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> [image: logo civity new]
>>>>
>>>>
>>>>
>>>> *T* +31 (0)6 24 16 07 23 | *E* gil at civity.nl
>>>> Handelsweg 6-1 | 3707 NH Zeist
>>>> *W* www.civity.nl
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *Civity is onderdeel van de Onetrail groep (www.onetrail.com
>>>> <http://www.onetrail.com/>)*
>>>>
>>>> *Civity is initiatiefnemer van FIWARE LAB Nederland, de open innovatie
>>>> omgeving voor smart cities **www.fiware-lab.nl*
>>>> <http://www.fiware-lab.nl/>
>>>>
>>>> [image: cid:image005.png at 01D0D9C8.8D3B3A60]
>>>> <https://www.linkedin.com/company/3284795?trk=tyah&trkInfo=clickedVertical%3Acompany%2CclickedEntityId%3A3284795%2Cidx%3A2-3-4%2CtarId%3A1473335093147%2Ctas%3Acivity>*[image:
>>>> cid:image005.png at 01D20A7C.FC86C980]*
>>>> <https://twitter.com/intent/follow?original_referer=https://about.twitter.com/resources/buttons&region=follow_link&screen_name=CivityNL&tw_p=followbutton>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> CKAN security
>>>> https://lists.okfn.org/mailman/listinfo/security
>>>> https://lists.okfn.org/mailman/options/security/adria.mercad
>>>> er%40okfn.org
>>>>
>>>> Repo: https://github.com/ckan/ckan-security
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> _______________________________________________
>>> CKAN security
>>> https://lists.okfn.org/mailman/listinfo/security
>>> https://lists.okfn.org/mailman/options/security/ian%40excess.org
>>>
>>> Repo: https://github.com/ckan/ckan-security
>>>
>>
>>
>> _______________________________________________
>> CKAN security
>> https://lists.okfn.org/mailman/listinfo/security
>> https://lists.okfn.org/mailman/options/security/david.read%
>> 40hackneyworkshop.com
>>
>> Repo: https://github.com/ckan/ckan-security
>>
>
>
> _______________________________________________
> CKAN security
> https://lists.okfn.org/mailman/listinfo/security
> https://lists.okfn.org/mailman/options/security/ian%40excess.org
>
> Repo: https://github.com/ckan/ckan-security
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.okfn.org/mailman/private/security/attachments/20170816/88584796/attachment-0001.html>