[ckan-dev] API Performance

[Toby Dacre]

On 25 June 2013 10:50, Ross Jones <ross at servercode.co.uk> wrote:
> Hi Toby,
>
> As Ross (T) raised API performance in another thread, I thought I'd ask - is anything happening about API performance generally? Read as well as write.
>
> The APIController can be quite memory hungry when returning large results and I've had some small success in using a generator instead of passing around large strings. I imagine that there might be an efficient way of using a generator to return each item for the results (as it is encoded) rather than directly encoding a blob as json in memory.

This does sound like something we should be looking at.  Maybe it is a
paging issue as we shouldn't be returning so much stuff that the in
memory stuff is an issue.  Are there particular api calls that are
really bad?

Much of the performance problem as I see it is that often we get data
and then throw it away eg getting a list of items will get the whole
items and then do `return [x['id'] for x in items]`

>
> Ross (J).
>
>
> On 24 June 2013 16:58, Ross Thompson <ross.thompson.ca at gmail.com> wrote:
>> 2. Performance issues when dealing with 180000+ datasets. Loading and
>> updating larges numbers of datasets through the API can take a weekend.
>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev



-- 
Toby Dacre

The Open Knowledge Foundation

Empowering through Open Knowledge
http://okfn.org/  |  @okfn