[ckan-dev] API abuse

Thu Apr 18 15:14:27 UTC 2013

Hello

We just blocked api/* for this case but serving from a different web server
is possible too.

The config looks like the below, including some caching.

proxy_cache_path /tmp/nginx_cache levels=1:2 keys_zone=cache:30m
max_size=250m;
proxy_temp_path /tmp/nginx_proxy 1 2;
limit_req_zone  $binary_remote_addr  zone=one:10m   rate=1r/s;

server {
    location / {
        proxy_pass http://127.0.0.1:8080/;
        proxy_set_header Host $host;
        proxy_cache cache;
        proxy_cache_bypass $cookie_auth_tkt;
        proxy_no_cache $cookie_auth_tkt;
        proxy_cache_valid 30m;
        proxy_cache_key $host$scheme$proxy_host$request_uri;
        # In emergency comment out line to force caching
        # proxy_ignore_headers X-Accel-Expires Expires Cache-Control;
    }

    location /api/ {
        proxy_pass http://127.0.0.1:8080/api/;
        proxy_set_header Host $host;
        proxy_cache cache;
        proxy_cache_bypass $cookie_auth_tkt;
        proxy_no_cache $cookie_auth_tkt;
        proxy_cache_valid 30m;
        proxy_cache_key $host$scheme$proxy_host$request_uri;
#        proxy_ignore_headers X-Accel-Expires Expires Cache-Control;
        limit_req  zone=one burst=10;
    }
}

On Thu, Apr 18, 2013 at 3:06 PM, Ian Ward <ian at excess.org> wrote:

> Hi David,
>
> Is there anything else you're doing that you have found helpful? Do you
> aserve your api responses from a separate web server, or just apply the
> rate limiting to requests to api/*?
>
> On Apr 18, 2013 9:42 AM, "David Raznick" <david.raznick at okfn.org> wrote:
> >
> > Hello,
> >
> > The simplest approach to this, and we have done this on the datahub, was
> to limit api requests to 1 req/s per ip.  This was done with nginx though.
> http://wiki.nginx.org/HttpLimitReqModule
> >
> > Thanks
> >
> > David
> >
> >
> > On Thu, Apr 18, 2013 at 1:25 PM, Toby Dacre <toby.okfn at gmail.com> wrote:
> >>
> >>
> >>
> >> On 18 April 2013 11:13, David Read <david.read at hackneyworkshop.com>
> wrote:
> >>>
> >>> We had an incident yesterday caused by a java web bot making
> >>> simultaneous connections to our CKAN API. Averaging 10 requests per
> >>> second, it caused serious server problems - postgres filling the CPU
> >>> use, Apache spawning lots of processes. Normally big loads are not a
> >>> problem for us because of using a cache in front of CKAN, but because
> >>> the API v3 is not easily cached, it caused the problems.
> >>>
> >>> The user was POSTing requests to package_show, without api key. Nagios
> >>> alerted us to the slowing server and we banned their IP manually
> >>> within a few minutes to take it back to normal. But it has become a
> >>> concern.
> >>>
> >>> Does anyone have any thoughts on how the CKAN community might deal
> >>> with this sort of behaviour better, either in the design of CKAN or
> >>> with server software?
> >>
> >>
> >> Hmmm,
> >>
> >> there are a few approaches.
> >>
> >> 1) we could look at rate limiting api calls but that might be painful
> have overheads and block some legitimate use
> >>
> >> 2) we could insist on a valid API key much earlier in the process (do
> we allow anon api access?  or add a config option for this)
> >>
> >> 3) package_show is hard because we get the package from the db even if
> we then find we won't show it - this is hard to avoid though maybe we could
> look at pre auth checking before even hitting the db but I'm not sure that
> would help here
> >>
> >> I think 2 is the easiest/quickest approach but it might be too blunt
> >>
> >> Toby
> >>
> >>
> >>>
> >>> David
> >>>
> >>> _______________________________________________
> >>> ckan-dev mailing list
> >>> ckan-dev at lists.okfn.org
> >>> http://lists.okfn.org/mailman/listinfo/ckan-dev
> >>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
> >>
> >>
> >>
> >> _______________________________________________
> >> ckan-dev mailing list
> >> ckan-dev at lists.okfn.org
> >> http://lists.okfn.org/mailman/listinfo/ckan-dev
> >> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
> >>
> >
> >
> > _______________________________________________
> > ckan-dev mailing list
> > ckan-dev at lists.okfn.org
> > http://lists.okfn.org/mailman/listinfo/ckan-dev
> > Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
> >
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20130418/673d49c6/attachment-0001.html>