[ckan-dev] API abuse

[Timothy Lebo]

CKAN,

As a developer that occationally uses CKAN and now has tooling build around it, it is *drastically* easier to develop read operations without the overhead of a key to read.
Because I'm building a system that others use, I can't hard code a key in. Besides, I wouldn't want my key shared as part of my code.

The current requirement for a key to write makes sense, but from my experience it is a sharper overhead.

So, if you move towards using an API key for everything, please also make the effort to make it easier to develop with keys, e.g. by accessing shell environment variables, using canonical (and configurable) file paths for where to find keys (PER CKAN instance - for cases where one works against multiple CKANs.)

A related issue is https://github.com/okfn/ckanclient/issues/15 which remains open.

I'm sure none of my points are new, but I wanted to express a "developer user" perspective.

Thanks!
Tim Lebo


On Apr 18, 2013, at 10:06 AM, Ian Ward <ian at excess.org> wrote:

> Hi David,
> 
> Is there anything else you're doing that you have found helpful? Do you aserve your api responses from a separate web server, or just apply the rate limiting to requests to api/*?
> 
> On Apr 18, 2013 9:42 AM, "David Raznick" <david.raznick at okfn.org> wrote:
> >
> > Hello,
> >
> > The simplest approach to this, and we have done this on the datahub, was to limit api requests to 1 req/s per ip.  This was done with nginx though.  http://wiki.nginx.org/HttpLimitReqModule
> >
> > Thanks
> >
> > David
> >
> >
> > On Thu, Apr 18, 2013 at 1:25 PM, Toby Dacre <toby.okfn at gmail.com> wrote:
> >>
> >>
> >>
> >> On 18 April 2013 11:13, David Read <david.read at hackneyworkshop.com> wrote:
> >>>
> >>> We had an incident yesterday caused by a java web bot making
> >>> simultaneous connections to our CKAN API. Averaging 10 requests per
> >>> second, it caused serious server problems - postgres filling the CPU
> >>> use, Apache spawning lots of processes. Normally big loads are not a
> >>> problem for us because of using a cache in front of CKAN, but because
> >>> the API v3 is not easily cached, it caused the problems.
> >>>
> >>> The user was POSTing requests to package_show, without api key. Nagios
> >>> alerted us to the slowing server and we banned their IP manually
> >>> within a few minutes to take it back to normal. But it has become a
> >>> concern.
> >>>
> >>> Does anyone have any thoughts on how the CKAN community might deal
> >>> with this sort of behaviour better, either in the design of CKAN or
> >>> with server software?
> >>
> >>
> >> Hmmm,
> >>
> >> there are a few approaches.
> >>
> >> 1) we could look at rate limiting api calls but that might be painful have overheads and block some legitimate use
> >>
> >> 2) we could insist on a valid API key much earlier in the process (do we allow anon api access?  or add a config option for this)
> >>
> >> 3) package_show is hard because we get the package from the db even if we then find we won't show it - this is hard to avoid though maybe we could look at pre auth checking before even hitting the db but I'm not sure that would help here 
> >>
> >> I think 2 is the easiest/quickest approach but it might be too blunt
> >>
> >> Toby
> >>
> >>
> >>>
> >>> David
> >>>
> >>> _______________________________________________
> >>> ckan-dev mailing list
> >>> ckan-dev at lists.okfn.org
> >>> http://lists.okfn.org/mailman/listinfo/ckan-dev
> >>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
> >>
> >>
> >>
> >> _______________________________________________
> >> ckan-dev mailing list
> >> ckan-dev at lists.okfn.org
> >> http://lists.okfn.org/mailman/listinfo/ckan-dev
> >> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
> >>
> >
> >
> > _______________________________________________
> > ckan-dev mailing list
> > ckan-dev at lists.okfn.org
> > http://lists.okfn.org/mailman/listinfo/ckan-dev
> > Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
> >
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20130418/ced0a318/attachment-0001.html>