[ckan-dev] API abuse

Thu Apr 18 12:25:58 UTC 2013

On 18 April 2013 11:13, David Read <david.read at hackneyworkshop.com> wrote:

> We had an incident yesterday caused by a java web bot making
> simultaneous connections to our CKAN API. Averaging 10 requests per
> second, it caused serious server problems - postgres filling the CPU
> use, Apache spawning lots of processes. Normally big loads are not a
> problem for us because of using a cache in front of CKAN, but because
> the API v3 is not easily cached, it caused the problems.
>
> The user was POSTing requests to package_show, without api key. Nagios
> alerted us to the slowing server and we banned their IP manually
> within a few minutes to take it back to normal. But it has become a
> concern.
>
> Does anyone have any thoughts on how the CKAN community might deal
> with this sort of behaviour better, either in the design of CKAN or
> with server software?
>

Hmmm,

there are a few approaches.

1) we could look at rate limiting api calls but that might be painful have
overheads and block some legitimate use

2) we could insist on a valid API key much earlier in the process (do we
allow anon api access?  or add a config option for this)

3) package_show is hard because we get the package from the db even if we
then find we won't show it - this is hard to avoid though maybe we could
look at pre auth checking before even hitting the db but I'm not sure that
would help here

I think 2 is the easiest/quickest approach but it might be too blunt

Toby

> David
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20130418/6c43099b/attachment-0001.html>