[CKAN-Security] Solr Manipulation

Ian Ward ian at excess.org
Thu Aug 17 13:55:42 UTC 2017 

CORS is another way to identify resources that we should have access to.

On Thu, Aug 17, 2017 at 9:53 AM, Ian Ward <ian at excess.org> wrote:

> We can add some configuration options to blacklist ranges of IP addresses
> for datapusher. It's a more complicated approach but it would be more
> secure by default and it would allow users to block internal ranges if they
> choose, without needing a remote server or firewall configured.
>
> On Thu, Aug 17, 2017 at 2:37 AM, David Read <david.read at hackneyworkshop.
> com> wrote:
>
>> We need to avoid sounding like:
>>
>> "You've got to completely trust all your editors, and the editors from
>> all your harvester sources, because CKAN lets them hack your server. To
>> avoid this put your datapusher worker on a separate server - a setup which
>> is NOT the default in our docs."
>>
>> So I'm not convinced a warning is enough. It needs something more, like
>> the docs getting people to run the worker on heroku/appengine, or auth
>> added to solr and redis. Or any other ideas.
>>
>> Dave
>>
>>
>> On 16 August 2017 at 19:41, Ian Ward <ian at excess.org> wrote:
>>
>>> It would also be sufficient to warn users that datapusher will expose
>>> any HTTP resources available from the datapusher server, so all users that
>>> can create resources can access all resources visible from there. http auth
>>> on solr and redis would be a great idea regardless.
>>>
>>> On Wed, Aug 16, 2017 at 12:42 PM, David Read <
>>> david.read at hackneyworkshop.com> wrote:
>>>
>>>> I struggling to see how we can tell people how to easily deploy the
>>>> datapusher worker app, so that it is properly isolated from SOLR and
>>>> everything else. I guess it needs a second host running outside the main
>>>> CKAN host, so that there is a firewall in-between? Maybe the easiest way is
>>>> to show people how to run the worker on Heroku/Google App Engine as a
>>>> freemium thing?
>>>>
>>>> Alternatively we could protect SOLR using some basic auth - arguably we
>>>> should do this anyway. Redis too. The tokens would be in our ckan config
>>>> (and possibly environment variables), and not accessible to a malicious
>>>> datapusher's request. So not hard to do.
>>>>
>>>> Ian said:
>>>> > We'll need to make datapusher fetching URLs a config option and
>>>> document that it shouldn't be used unless the datapusher instance is
>>>> installed outside the internal network.
>>>>
>>>> I think it is pretty common to use CKAN with external URLs (rather than
>>>> pure uploads), and want to have them pushed into datastore. So I'd be
>>>> concerned if the implication here was that we shouldn't try too hard to
>>>> continue to enable it by default and document how to set it up. Is that
>>>> what's meant or did I get the wrong end of the stick here?
>>>>
>>>> David
>>>>
>>>> On 16 August 2017 at 14:51, Ian Ward <ian at excess.org> wrote:
>>>>
>>>>> It's impossible to list all the URLs that might expose sensitive
>>>>> information when exposed to datapusher (first we block localhost, then
>>>>> 127.*, then ::1, then private ranges, and so on..) We'll need to make
>>>>> datapusher fetching URLs a config option and document that it shouldn't be
>>>>> used unless the datapusher instance is installed outside the internal
>>>>> network.
>>>>>
>>>>> On Wed, Aug 16, 2017 at 6:54 AM, Adrià Mercader <
>>>>> adria.mercader at okfn.org> wrote:
>>>>>
>>>>>> Hi Gil,
>>>>>>
>>>>>> That makes sense. Sorry, in your original email you didn't mention
>>>>>> DataPusher and that confused me.
>>>>>> We'll discuss this in tomorrow's meeting and get back to you.
>>>>>>
>>>>>> Adrià
>>>>>>
>>>>>>
>>>>>> On 16 August 2017 at 11:17, Gil Hilário <gil at civity.nl> wrote:
>>>>>>
>>>>>>> Hi Adrià
>>>>>>>
>>>>>>>
>>>>>>> So there is a little misunderstanding in my explanation. What you
>>>>>>> describe as step 2 “2. Once the dataset is created, if someone
>>>>>>> clicks on the link from the resource page, the Solr index gets deleted.
>>>>>>>
>>>>>>> ” is not what I meant. The issue is not related with users clicking
>>>>>>> the link, because as you said, the solr is not publicly accessible to the
>>>>>>> world, but when the DataPusher tries to push it.
>>>>>>>
>>>>>>> We were unable to reproduce (a non-destructive version of this bug)
>>>>>>> it on demo.ckan.org , since you already fixed the previous issue
>>>>>>> (localfile usage) and therefore we could not guess the Solr port.
>>>>>>>
>>>>>>> Therefore we cannot read the the solr url in the ini file (which is
>>>>>>> good) and we unable to guess it without really hacking/trying
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> CKAN is able to request the solr, by default solr has no
>>>>>>> authentication and it is requestable from the CKAN instance (otherwise CKAN
>>>>>>> would not work)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>    - Let's assume it is (using the default guide) on
>>>>>>>    http://localhost:8983/solr.
>>>>>>>    - The datapusher is running allongside CKAN website, default on
>>>>>>>    port 8800.
>>>>>>>    - The datapusher is then requesting the url, which can resolve
>>>>>>>    anywhere in the world, but also on the server network (in most cases, on
>>>>>>>    localhost or 127.0.0.1)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Solr is NOT accessible from the world, but Is accessible from the
>>>>>>> datapusher. So pushing the data to the the datastore, would result in
>>>>>>> executing a SOLR command (in our case, wipe the content).
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Anyway your suggestion of extending the default resource URL
>>>>>>> validation would provide a more secure approach.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Best Regards,
>>>>>>>
>>>>>>> Gil Hilário
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> [image: logo civity new]
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> *T* +31 (0)6 24 16 07 23 | *E* gil at civity.nl
>>>>>>> Handelsweg 6-1 | 3707 NH Zeist
>>>>>>> *W* www.civity.nl
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> *Civity is onderdeel van de Onetrail groep (www.onetrail.com
>>>>>>> <http://www.onetrail.com/>)*
>>>>>>>
>>>>>>> *Civity is initiatiefnemer van FIWARE LAB Nederland, de open
>>>>>>> innovatie omgeving voor smart cities **www.fiware-lab.nl*
>>>>>>> <http://www.fiware-lab.nl/>
>>>>>>>
>>>>>>> [image: cid:image005.png at 01D0D9C8.8D3B3A60]
>>>>>>> <https://www.linkedin.com/company/3284795?trk=tyah&trkInfo=clickedVertical%3Acompany%2CclickedEntityId%3A3284795%2Cidx%3A2-3-4%2CtarId%3A1473335093147%2Ctas%3Acivity>*[image:
>>>>>>> cid:image005.png at 01D20A7C.FC86C980]*
>>>>>>> <https://twitter.com/intent/follow?original_referer=https://about.twitter.com/resources/buttons&region=follow_link&screen_name=CivityNL&tw_p=followbutton>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> *From:* Adrià Mercader [mailto:adria.mercader at okfn.org]
>>>>>>> *Sent:* woensdag 16 augustus 2017 11:41
>>>>>>> *To:* Gil Hilário <gil at civity.nl>
>>>>>>> *Cc:* CKAN Security Alerts/Discussions <security at lists.okfn.org>
>>>>>>> *Subject:* Re: [CKAN-Security] Solr Manipulation
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Hi again Gil,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Just to make sure we are discussing the same issue. The steps you
>>>>>>> describe are:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 1. Someone creates a resource with a URL pointing to the local Solr
>>>>>>> instance (eg http://localhost:8983/solr
>>>>>>> /ckan-schema-2.3/update?commit=true&stream.body=%3cdelete%3e
>>>>>>> %3cquery%3e*:*%3c/query%3e%3c/delete%3e)
>>>>>>>
>>>>>>> 2. Once the dataset is created, if someone clicks on the link from
>>>>>>> the resource page, the Solr index gets deleted.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> If I understand correctly this is only an issue if you are using a
>>>>>>> CKAN instance in your local server / machine, where localhost:8983 gets
>>>>>>> resolved to the Solr instance that is locally accessible.
>>>>>>>
>>>>>>> In any other setup, the Solr instance would not publicly accessible,
>>>>>>> as the only port exposed publicly would be nginx and if users clicked on
>>>>>>> that link they would just get a not found error (Unless they have something
>>>>>>> running on localhost on port 8983).
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> So there are two things described here:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 1. Solr instance being accessible publicly: this should never happen
>>>>>>> on a production environment. You'll most likely only expose Nginx and
>>>>>>> connect to the internal services like Solr and Postgres inside the firewall.
>>>>>>>
>>>>>>> 2. Ability for users to link to arbitrary URLs, including
>>>>>>> administrative ones like the Solr example. CKAN can't filter the URLs by
>>>>>>> default as each CKAN site might have different requirements. It should be
>>>>>>> really easy to extend the default resource URL validation to customize to
>>>>>>> your own needs though.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Hope all this makes sense, please let me know if the issue is not
>>>>>>> the one that I've described or if you have any other comments.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Best regards,
>>>>>>>
>>>>>>>
>>>>>>> Adrià
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 14 August 2017 at 16:04, Adrià Mercader <adria.mercader at okfn.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Hi Gil,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Thanks for your report. The technical team will discuss the issue
>>>>>>> and come back to you as soon as possible.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Adrià
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 14 August 2017 at 10:40, Gil Hilário <gil at civity.nl> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I recently reported a security flaw over public issue, this time I’m
>>>>>>> doing it through the right channel.
>>>>>>>
>>>>>>> We identified an issue that allows you to manipulate the Solr
>>>>>>> through a linked dataset.
>>>>>>>
>>>>>>> If you create a new dataset and you add a resource. If you give it
>>>>>>> the following URL, for example, http://localhost:8983/solr/cka
>>>>>>> n-schema-2.3/update?commit=true&stream.body=<delete><query>*
>>>>>>> :*</query></delete> it will simply remove all the datasets from the
>>>>>>> solr, and you will need to run the reindexer.
>>>>>>> The same way you can delete and manipulate all datasets because you
>>>>>>> have full access to Solr. The data itself is never touched but we though
>>>>>>> this was worth sharing with you because the possibilities of changing the
>>>>>>> perception of what’s available on the CKAN instance are limiteless.
>>>>>>>
>>>>>>> Best Regards,
>>>>>>>
>>>>>>> Gil Hilário
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> [image: logo civity new]
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> *T* +31 (0)6 24 16 07 23 | *E* gil at civity.nl
>>>>>>> Handelsweg 6-1 | 3707 NH Zeist
>>>>>>> *W* www.civity.nl
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> *Civity is onderdeel van de Onetrail groep (www.onetrail.com
>>>>>>> <http://www.onetrail.com/>)*
>>>>>>>
>>>>>>> *Civity is initiatiefnemer van FIWARE LAB Nederland, de open
>>>>>>> innovatie omgeving voor smart cities **www.fiware-lab.nl*
>>>>>>> <http://www.fiware-lab.nl/>
>>>>>>>
>>>>>>> [image: cid:image005.png at 01D0D9C8.8D3B3A60]
>>>>>>> <https://www.linkedin.com/company/3284795?trk=tyah&trkInfo=clickedVertical%3Acompany%2CclickedEntityId%3A3284795%2Cidx%3A2-3-4%2CtarId%3A1473335093147%2Ctas%3Acivity>*[image:
>>>>>>> cid:image005.png at 01D20A7C.FC86C980]*
>>>>>>> <https://twitter.com/intent/follow?original_referer=https://about.twitter.com/resources/buttons&region=follow_link&screen_name=CivityNL&tw_p=followbutton>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> CKAN security
>>>>>>> https://lists.okfn.org/mailman/listinfo/security
>>>>>>> https://lists.okfn.org/mailman/options/security/adria.mercad
>>>>>>> er%40okfn.org
>>>>>>>
>>>>>>> Repo: https://github.com/ckan/ckan-security
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> CKAN security
>>>>>> https://lists.okfn.org/mailman/listinfo/security
>>>>>> https://lists.okfn.org/mailman/options/security/ian%40excess.org
>>>>>>
>>>>>> Repo: https://github.com/ckan/ckan-security
>>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> CKAN security
>>>>> https://lists.okfn.org/mailman/listinfo/security
>>>>> https://lists.okfn.org/mailman/options/security/david.read%4
>>>>> 0hackneyworkshop.com
>>>>>
>>>>> Repo: https://github.com/ckan/ckan-security
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> CKAN security
>>>> https://lists.okfn.org/mailman/listinfo/security
>>>> https://lists.okfn.org/mailman/options/security/ian%40excess.org
>>>>
>>>> Repo: https://github.com/ckan/ckan-security
>>>>
>>>
>>>
>>> _______________________________________________
>>> CKAN security
>>> https://lists.okfn.org/mailman/listinfo/security
>>> https://lists.okfn.org/mailman/options/security/david.read%4
>>> 0hackneyworkshop.com
>>>
>>> Repo: https://github.com/ckan/ckan-security
>>>
>>
>>
>> _______________________________________________
>> CKAN security
>> https://lists.okfn.org/mailman/listinfo/security
>> https://lists.okfn.org/mailman/options/security/ian%40excess.org
>>
>> Repo: https://github.com/ckan/ckan-security
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.okfn.org/mailman/private/security/attachments/20170817/15d932c3/attachment-0001.html>

More information about the Security mailing list