[ckan-committers] Help reaching out to vulnerable CKANs
Adrià Mercader
adria.mercader at okfn.org
Wed Sep 27 09:48:02 UTC 2017
Thanks Joel.
Attached is a patch that works for CKAN 2.2 and 2.3. For any other versions
please make sure every instance runs the latest patch release (going out in
2 hours from now)
Adrià
On 26 September 2017 at 22:06, Joel Natividad <jnatividad at opengov.com>
wrote:
> Hi Adria,
>
> Thanks for the follow-up... I'll reach out to Philly and Tempe make sure
> all the sites we help with are patched in a timely manner.
>
> Best,
> Joel
>
> On Fri, Sep 22, 2017 at 6:53 AM, Adrià Mercader <adria.mercader at okfn.org>
> wrote:
>
>> Hi all,
>>
>> We are working on the patch releases for next week on this. Would be good
>> if you could check if you have contacts on these instances.
>>
>> @Joel perhaps Open Philly? http://opendataphilly.org
>>
>> @Steven lots of aussie sites there, including data.gov.au
>>
>> Thanks and have a great weekend
>>
>> On 18 September 2017 at 17:39, Sebastian Moleski <smoleski at viderum.com>
>> wrote:
>>
>>> Hi Adria,
>>>
>>>
>>>
>>> Obviously, we’re happy to publish availability of this patch through all
>>> our communication channels as well as reach out to our existing and past
>>> clients about it. For our hosted clients and trials, we’ll patch as soon as
>>> possible.
>>>
>>>
>>>
>>> I’ve also requested access to that spreadsheet for smoleski at viderum.com
>>> so that we can check for which of these portals we might already have
>>> contact information.
>>>
>>>
>>>
>>> Please let me know if there’s anything else we can do.
>>>
>>>
>>>
>>> Best,
>>>
>>>
>>>
>>> *Sebastian Moleski*
>>>
>>> Chief Executive Officer
>>>
>>>
>>>
>>> Viderum <https://www.viderum.com/> - making the world’s public data
>>> discoverable and accessible to everyone
>>>
>>> https://www.viderum.com/ | @videruminc
>>> <https://twitter.com/videruminc>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *From:* ckan-steering-group at googlegroups.com [mailto:
>>> ckan-steering-group at googlegroups.com] *On Behalf Of *Adrià Mercader
>>> *Sent:* Monday, September 18, 2017 4:57 PM
>>> *To:* CKAN Association Steering Group <ckan-steering-group at googlegro
>>> ups.com>
>>> *Cc:* ckan-committers at lists.okfn.org
>>> *Subject:* Help reaching out to vulnerable CKANs
>>>
>>>
>>>
>>> Hi all,
>>>
>>>
>>>
>>> You might be aware of this via a developer, but last week we had a
>>> public report of a vulnerability in CKAN that affects all versions since 2.0
>>>
>>>
>>>
>>> This is nothing new, we regularly patch security holes in our patch
>>> releases, but the difference is that this time the vulnerability has been
>>> reported to a bug bounty website. There were around 20 CKAN sites reported,
>>> including National instances like Australia, Singapore, etc One example:
>>>
>>>
>>>
>>> https://www.openbugbounty.org/reports/294186/
>>>
>>>
>>>
>>> The way these sites work is that they give a period of time to patch the
>>> vulnerability but then they make it public. They also notify the website
>>> owners, although I'm not clear how successful they were on that.
>>>
>>>
>>>
>>> We have decided to release a quick security patch next week and try to
>>> reach at least the sites listed on the bounty website (again, all CKANs are
>>> vulnerable, but these were the ones reported).
>>>
>>>
>>>
>>> Some of these sites are managed by vendors on the SG, some you might
>>> have a contact we can reach there.
>>>
>>>
>>>
>>> The idea is to ask them to upgrade to the latest patch release if they
>>> are running a supported version (CKAN >= 2.4) or apply the patch directly
>>> if not.
>>>
>>>
>>>
>>> Could you access this spreadsheet (please request access with whatever
>>> email you use) and fill the contact field if you manage or have a contact
>>> on that particular site?
>>>
>>>
>>>
>>> https://docs.google.com/spreadsheets/d/1qG4iBfBZwcPg2i-KbFkP
>>> V4TvpCH94hVJPSYWcg8IKCs/edit#gid=2069917099
>>>
>>>
>>>
>>>
>>>
>>> Going forward we need to address people not running the latest patch
>>> version and having their sites vulnerable, because sadly that's the most
>>> common scenario.
>>>
>>>
>>>
>>> Thanks,
>>>
>>>
>>>
>>> Adrià
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "CKAN Association Steering Group" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to ckan-steering-group+unsubscribe at googlegroups.com.
>>> To post to this group, send email to ckan-steering-group at googlegrou
>>> ps.com.
>>> To view this discussion on the web, visit https://groups.google.com/d/ms
>>> gid/ckan-steering-group/CAGJR8i%2B7kRdfUBMAO-ZQoyUzcMmzyqabM
>>> ST%3DpPScppQugNWS9w%40mail.gmail.com
>>> <https://groups.google.com/d/msgid/ckan-steering-group/CAGJR8i%2B7kRdfUBMAO-ZQoyUzcMmzyqabMST%3DpPScppQugNWS9w%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "CKAN Association Steering Group" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to ckan-steering-group+unsubscribe at googlegroups.com.
>>> To post to this group, send email to ckan-steering-group at googlegrou
>>> ps.com.
>>> To view this discussion on the web, visit https://groups.google.com/d/ms
>>> gid/ckan-steering-group/DM5PR1701MB1692F8E857FCE0320F7AECF0C
>>> 8630%40DM5PR1701MB1692.namprd17.prod.outlook.com
>>> <https://groups.google.com/d/msgid/ckan-steering-group/DM5PR1701MB1692F8E857FCE0320F7AECF0C8630%40DM5PR1701MB1692.namprd17.prod.outlook.com?utm_medium=email&utm_source=footer>
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "CKAN Association Steering Group" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to ckan-steering-group+unsubscribe at googlegroups.com.
>> To post to this group, send email to ckan-steering-group at googlegroups.com
>> .
>> To view this discussion on the web, visit https://groups.google.com/d/ms
>> gid/ckan-steering-group/CAGJR8iKPyEVKaP5oYdLJjUzJ44yMdp%
>> 2BFgEJ94UxYgDSQAwkWqA%40mail.gmail.com
>> <https://groups.google.com/d/msgid/ckan-steering-group/CAGJR8iKPyEVKaP5oYdLJjUzJ44yMdp%2BFgEJ94UxYgDSQAwkWqA%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> --
>
>
> Joel Natividad
>
> Director of Open Data
>
> OpenGov
>
> +1 (347) 565-5635 <+1%28347%29%20565-5635>
>
> jnatividad at opengov.com | @jqnatividad <http://twitter.com/jqnatividad>
>
> Silicon Valley
> <https://www.google.com/maps/place/OpenGov+Inc/@37.4859652,-122.2121292,15z/data=!4m2!3m1!1s0x0:0xb84d4c3f06ecd893>
> | Washington DC
> <https://www.google.com/maps/place/OpenGov,+Inc./@38.915617,-77.047496,17z/data=!4m6!1m3!3m2!1s0x89b7b7cf85e25661:0xf3c11f6e3f635963!2sOpenGov,+Inc.!3m1!1s0x89b7b7cf85e25661:0xf3c11f6e3f635963>
> | Portland
> <https://www.google.com/maps/place/220+NW+8th+Ave,+Portland,+OR+97209/@45.5248202,-122.6802196,17z/data=!3m1!4b1!4m2!3m1!1s0x54950a01a6fdee1f:0x7f2eb4b084a75646>
> | New York
> <https://www.google.com/maps/place/33+Irving+Pl,+New+York,+NY+10003/@40.7274072,-73.9853282,1067a,20y,338.96h,44.7t/data=!3m1!1e3!4m2!3m1!1s0x89c2599f450ace29:0xcc9ef4cf79798e7c>
> | London
>
> <https://goo.gl/maps/NFBzYXcrDcz>
>
> <https://twitter.com/opengovinc>
> <https://www.linkedin.com/company/opengov-inc>
> <https://www.facebook.com/opengovinc>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.okfn.org/mailman/private/ckan-committers/attachments/20170927/8f2e580e/attachment-0003.html>
-------------- next part --------------
commit 62fad763a7b1330a88a0e60c28cc479d8bc660e6
Author: amercader <amercadero at gmail.com>
Date: Tue Sep 26 12:15:02 2017 +0100
Standardize on url_for
diff --git a/ckan/controllers/package.py b/ckan/controllers/package.py
index ff3a37d..f744836 100644
--- a/ckan/controllers/package.py
+++ b/ckan/controllers/package.py
@@ -1131,7 +1131,7 @@ class PackageController(base.BaseController):
except KeyError:
c.package['isopen'] = False
- # TODO: find a nicer way of doing this
+ # Deprecated: c.datastore_api - use h.action_url instead
c.datastore_api = '%s/api/action' % config.get('ckan.site_url', '').rstrip('/')
c.related_count = c.pkg.related_count
diff --git a/ckan/lib/helpers.py b/ckan/lib/helpers.py
index 3288f39..007655f 100644
--- a/ckan/lib/helpers.py
+++ b/ckan/lib/helpers.py
@@ -16,6 +16,7 @@ import pprint
import copy
import urlparse
from urllib import urlencode
+import uuid
from paste.deploy.converters import asbool
from webhelpers.html import escape, HTML, literal, url_escape
@@ -117,7 +118,7 @@ def get_site_protocol_and_host():
If `ckan.site_url` is set like this::
ckan.site_url = http://example.com
-
+
Then this function would return a tuple `('http', 'example.com')`
If the setting is missing, `(None, None)` is returned instead.
@@ -2039,6 +2040,13 @@ def license_options(existing_license_id=None):
for license_id in license_ids]
+def sanitize_id(id_):
+ '''Given an id (uuid4), if it has any invalid characters it raises
+ ValueError.
+ '''
+ return str(uuid.UUID(id_))
+
+
# these are the functions that will end up in `h` template helpers
__allowed_functions__ = [
# functions defined in ckan.lib.helpers
@@ -2157,4 +2165,5 @@ __allowed_functions__ = [
'check_config_permission',
'view_resource_url',
'license_options',
+ 'sanitize_id',
]
diff --git a/ckan/templates/ajax_snippets/api_info.html b/ckan/templates/ajax_snippets/api_info.html
index daba810..6273865 100644
--- a/ckan/templates/ajax_snippets/api_info.html
+++ b/ckan/templates/ajax_snippets/api_info.html
@@ -1,18 +1,18 @@
{#
Displays information about accessing a resource via the API.
-datastore_root_url - The root API url.
resource_id - The resource id
embedded - If true will not include the "modal" classes on the snippet.
Example
- {% snippet 'ajax_snippets/api_info.html', datastore_root_url=datastore_root_url, resource_id=resource_id, embedded=true %}
+ {% snippet 'ajax_snippets/api_info.html', resource_id=resource_id, embedded=true %}
#}
-{% set sql_example_url = datastore_root_url + '/datastore_search_sql?sql=SELECT * from "' + resource_id + '" WHERE title LIKE \'jones\'' %}
-
+{% set resource_id = h.sanitize_id(resource_id) %}
+{% set sql_example_url = h.url_for(controller='api', action='action', ver=3, logic_function='datastore_search_sql', qualified=True) + '?sql=SELECT * from "' + resource_id + '" WHERE title LIKE \'jones\'' %}
+{# not urlencoding the sql because its clearer #}
<div{% if not embedded %} class="modal"{% endif %}>
<div class="modal-header">
<h3>
@@ -40,19 +40,19 @@ Example
<tbody>
<tr>
<th scope="row">{{ _('Create') }}</th>
- <td><code>{{ datastore_root_url }}/datastore_create</code></td>
+ <td><code>{{ h.url_for(controller='api', action='action', ver=3, logic_function='datastore_create', qualified=True) }}</code></td>
</tr>
<tr>
<th scope="row">{{ _('Update / Insert') }}</th>
- <td><code>{{ datastore_root_url }}/datastore_upsert</code></td>
+ <td><code>{{ h.url_for(controller='api', action='action', ver=3, logic_function='datastore_upsert', qualified=True) }}</code></td>
</tr>
<tr>
<th scope="row">{{ _('Query') }}</th>
- <td><code>{{ datastore_root_url }}/datastore_search</code></td>
+ <td><code>{{ h.url_for(controller='api', action='action', ver=3, logic_function='datastore_search', qualified=True) }}</code></td>
</tr>
<tr>
<th scope="row">{{ _('Query (via SQL)') }}</th>
- <td><code>{{ datastore_root_url }}/datastore_search_sql</code></td>
+ <td><code>{{ h.url_for(controller='api', action='action', ver=3, logic_function='datastore_search_sql', qualified=True) }}</code></td>
</tr>
</tbody>
@@ -69,13 +69,12 @@ Example
<div class="accordion-inner">
<strong>{{ _('Query example (first 5 results)') }}</strong>
<p>
- <code><a href="{{ datastore_root_url }}/datastore_search?resource_id={{resource_id}}&limit=5" target="_blank">{{ datastore_root_url }}/datastore_search?resource_id={{resource_id}}&limit=5</a></code>
+ <code><a href="{{ h.url_for(controller='api', action='action', ver=3, logic_function='datastore_search', resource_id=resource_id, limit=5, qualified=True) }}" target="_blank">{{ h.url_for(controller='api', action='action', ver=3, logic_function='datastore_search', resource_id=resource_id, limit=5, qualified=True) }}</a></code>
</p>
<strong>{{ _('Query example (results containing \'jones\')') }}</strong>
<p>
- <code><a href="{{ datastore_root_url }}/datastore_search?resource_id={{resource_id}}&q=jones"
- target="_blank">{{ datastore_root_url }}/datastore_search?resource_id={{resource_id}}&q=jones</a></code>
+ <code><a href="{{ h.url_for(controller='api', action='action', ver=3, logic_function='datastore_search', resource_id=resource_id, q='jones', qualified=True) }}" target="_blank">{{ h.url_for(controller='api', action='action', ver=3, logic_function='datastore_search', resource_id=resource_id, q='jones', qualified=True) }}</a></code>
</p>
<strong>{{ _('Query example (via SQL statement)') }}</strong>
@@ -102,7 +101,7 @@ Example
q: 'jones' // query for 'jones'
};
$.ajax({
- url: '{{ datastore_root_url }}/datastore_search',
+ url: '{{ h.url_for(controller='api', action='action', ver=3, logic_function='datastore_search', qualified=True) }}',
data: data,
dataType: 'jsonp',
success: function(data) {
@@ -121,7 +120,7 @@ Example
<div class="accordion-inner">
<pre>
import urllib
-url = '{{ datastore_root_url }}/datastore_search?resource_id={{resource_id}}&limit=5&q=title:jones'
+url = '{{ h.url_for(qualified=True, controller='api', action='action', ver=3, logic_function='datastore_search', resource_id=resource_id, limit=5) + '&q=title:jones' }}' {# not urlencoding the ":" because its clearer #}
fileobj = urllib.urlopen(url)
print fileobj.read()
</pre>
diff --git a/ckan/templates/package/resource_read.html b/ckan/templates/package/resource_read.html
index 613df9f..b0339fe 100644
--- a/ckan/templates/package/resource_read.html
+++ b/ckan/templates/package/resource_read.html
@@ -46,7 +46,7 @@
</li>
{% endif %}
{% if 'datastore' in g.plugins %}
- <li>{% snippet 'package/snippets/data_api_button.html', resource=res, datastore_root_url=c.datastore_api %}</li>
+ <li>{% snippet 'package/snippets/data_api_button.html', resource=res %}</li>
{% endif %}
{% endblock %}
</ul>
diff --git a/ckan/templates/package/snippets/data_api_button.html b/ckan/templates/package/snippets/data_api_button.html
index 3681c00..46e5d28 100644
--- a/ckan/templates/package/snippets/data_api_button.html
+++ b/ckan/templates/package/snippets/data_api_button.html
@@ -6,6 +6,6 @@ datastore_root_url: the root url of the datastore
#}
{% if resource.datastore_active %}
{% set loading_text = _('Loading...') %}
- {% set api_info_url = h.url_for(controller='api', action='snippet', ver=1, snippet_path='api_info.html', datastore_root_url=datastore_root_url, resource_id=resource.id) %}
+ {% set api_info_url = h.url_for(controller='api', action='snippet', ver=1, snippet_path='api_info.html', resource_id=resource.id) %}
<a class="btn btn-success" href="{{ api_info_url }}" data-module="api-info" data-module-template="{{ api_info_url }}" data-loading-text="{{ loading_text }}"><i class="icon-beaker icon-large"></i> {{ _('Data API') }}</a>
{% endif %}
More information about the ckan-committers
mailing list