[ckan-dev] API abuse

Thu Apr 18 14:06:52 UTC 2013

Hi David,

Is there anything else you're doing that you have found helpful? Do you
aserve your api responses from a separate web server, or just apply the
rate limiting to requests to api/*?

On Apr 18, 2013 9:42 AM, "David Raznick" <david.raznick at okfn.org> wrote:
>
> Hello,
>
> The simplest approach to this, and we have done this on the datahub, was
to limit api requests to 1 req/s per ip.  This was done with nginx though.
http://wiki.nginx.org/HttpLimitReqModule
>
> Thanks
>
> David
>
>
> On Thu, Apr 18, 2013 at 1:25 PM, Toby Dacre <toby.okfn at gmail.com> wrote:
>>
>>
>>
>> On 18 April 2013 11:13, David Read <david.read at hackneyworkshop.com>
wrote:
>>>
>>> We had an incident yesterday caused by a java web bot making
>>> simultaneous connections to our CKAN API. Averaging 10 requests per
>>> second, it caused serious server problems - postgres filling the CPU
>>> use, Apache spawning lots of processes. Normally big loads are not a
>>> problem for us because of using a cache in front of CKAN, but because
>>> the API v3 is not easily cached, it caused the problems.
>>>
>>> The user was POSTing requests to package_show, without api key. Nagios
>>> alerted us to the slowing server and we banned their IP manually
>>> within a few minutes to take it back to normal. But it has become a
>>> concern.
>>>
>>> Does anyone have any thoughts on how the CKAN community might deal
>>> with this sort of behaviour better, either in the design of CKAN or
>>> with server software?
>>
>>
>> Hmmm,
>>
>> there are a few approaches.
>>
>> 1) we could look at rate limiting api calls but that might be painful
have overheads and block some legitimate use
>>
>> 2) we could insist on a valid API key much earlier in the process (do we
allow anon api access?  or add a config option for this)
>>
>> 3) package_show is hard because we get the package from the db even if
we then find we won't show it - this is hard to avoid though maybe we could
look at pre auth checking before even hitting the db but I'm not sure that
would help here
>>
>> I think 2 is the easiest/quickest approach but it might be too blunt
>>
>> Toby
>>
>>
>>>
>>> David
>>>
>>> _______________________________________________
>>> ckan-dev mailing list
>>> ckan-dev at lists.okfn.org
>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>
>>
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>
>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20130418/a8056dab/attachment-0001.html>