[ckan-discuss] Recent Updates ATOM feed

Wed Apr 7 20:39:49 BST 2010

Rufus,

I found this

just found this http://ca.ckan.net/revision/list?format=atom&days=1

Which limits to the past day. That's perfect really, in most cases it 
will be a relatively small set of data, and i will be running cron more 
than once a day so this cover all the bases perfectly.

Col.

*Colin Calnan | Front End Developer*
Raised Eyebrow Web Studio
www.raisedeyebrow.com <http://www.raisedeyebrow.com>
T. 604.684.2498, ext. 5 | E. colin at raisedeyebrow.com
UFCW Local 120B

Rufus Pollock wrote:
> On 7 April 2010 18:33, Colin Calnan <colin at raisedeyebrow.com> wrote:
>   
>> Hi,
>>
>> I'm building a cron task for pulling in the recent changes ATOM feed.
>> However my task is timing out as it's taking a long time to get the feed
>> back from http://ca.ckan.net/revision?format=atom. In some cases it's taking
>> 30-60 seconds. Is there anything we can do to speed this up or any way to
>> limit the changes to the last 5.
>>     
>
> Easy to see what the problem is :)
>
> Basically the atom feed does 2 relatively dumb things (though they
> haven't been a problem before).
>
> a) it loads all revisions to construct the feed
>
> b) it puts all changes from the last 30 days without any limit on the
> number of items.
>
> a) is actually not that  bad for performance actually (as you can see
> from ckan.net behaviour) but b) is a killer for you because you've had
> a lot of edits in a short space of time.
>
> I think I can put in a fix for you guys in next 24h which can then be
> merged to head.
>
>   
>> Basically, with the help of Luke Closs, I've built a function that is
>> checking the md5sum of the feed and if it has changed I'm running all my
>> processes to clear cache etc on our Drupal site. So I really only need 1
>> recent item, and if that's different to the one I got previously, then I can
>> clear my caches.
>>     
>
> Couldn't you just get one item and checks its timestamp or is the
> md5sum more reliable. I also note we're about to have implemented
> <http://knowledgeforge.net/ckan/trac/ticket/265> though obviously like
> other changes this won't necesssarily get deployed immediately.
>
> Rufus
>
>
>   