[ckan-discuss] Mining other catalogues for packages

Mon May 23 10:12:07 BST 2011

Tim,

Using Scraperwiki for scrapers is definitely recommended.

For the CKAN end of things, we use the API to edit packages.

I can think of one automated script we have for DGU which we have for
processing ONS data from an XML feed, and that just runs as a cron as
you suggest.

David

On 22 May 2011 07:26, Tim McNamara <paperless at timmcnamara.co.nz> wrote:
> I apologise if this is documented somewhere. What is the process for
> creating a script which can be run as a cron job, which will automatically
> update current packages as they are amended or create new ones as they are
> found in external catalogues?
> I think it might be sensible to use a public service like ScraperWiki to
> extract the content, then grab the content from there in a structured
> manner. That would allow other people to contribute to the scrapers when
> they break.
>
>
> Tim
>
> _______________________________________________
> ckan-discuss mailing list
> ckan-discuss at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-discuss
>
>