[ckan-dev] API abuse

Thu Apr 18 10:13:36 UTC 2013

We had an incident yesterday caused by a java web bot making
simultaneous connections to our CKAN API. Averaging 10 requests per
second, it caused serious server problems - postgres filling the CPU
use, Apache spawning lots of processes. Normally big loads are not a
problem for us because of using a cache in front of CKAN, but because
the API v3 is not easily cached, it caused the problems.

The user was POSTing requests to package_show, without api key. Nagios
alerted us to the slowing server and we banned their IP manually
within a few minutes to take it back to normal. But it has become a
concern.

Does anyone have any thoughts on how the CKAN community might deal
with this sort of behaviour better, either in the design of CKAN or
with server software?

David