[ckan-discuss] Recent Updates ATOM feed

[Rufus Pollock]

On 7 April 2010 18:33, Colin Calnan <colin at raisedeyebrow.com> wrote:
> Hi,
>
> I'm building a cron task for pulling in the recent changes ATOM feed.
> However my task is timing out as it's taking a long time to get the feed
> back from http://ca.ckan.net/revision?format=atom. In some cases it's taking
> 30-60 seconds. Is there anything we can do to speed this up or any way to
> limit the changes to the last 5.

Easy to see what the problem is :)

Basically the atom feed does 2 relatively dumb things (though they
haven't been a problem before).

a) it loads all revisions to construct the feed

b) it puts all changes from the last 30 days without any limit on the
number of items.

a) is actually not that  bad for performance actually (as you can see
from ckan.net behaviour) but b) is a killer for you because you've had
a lot of edits in a short space of time.

I think I can put in a fix for you guys in next 24h which can then be
merged to head.

> Basically, with the help of Luke Closs, I've built a function that is
> checking the md5sum of the feed and if it has changed I'm running all my
> processes to clear cache etc on our Drupal site. So I really only need 1
> recent item, and if that's different to the one I got previously, then I can
> clear my caches.

Couldn't you just get one item and checks its timestamp or is the
md5sum more reliable. I also note we're about to have implemented
<http://knowledgeforge.net/ckan/trac/ticket/265> though obviously like
other changes this won't necesssarily get deployed immediately.

Rufus