[ckan-discuss] DataPusher, going from DataStore to CSV.

Fri Apr 4 07:08:52 UTC 2014

I would like to use the DataStore via the API as primary data-source. This works without a problem already.

However, if people wants to download the entire resource as a CSV, via /dump/, it only downloads 100K records (this is hardcoded into CKAN).
It also takes quite a long time to generate the CSV file.

I have resources with over 10+ mio. Rows and would like to offer a complete download via CSV. But changing the hardcoded 100K row limit puts a lot of pressure on the system.
It would be very nice to have a feature where, using the API for the DataStore, would update a corresponding CSV-file for download. So download wouldn't need to generate the file.

Fra: Rufus Pollock [mailto:rufus.pollock at okfn.org]
Sendt: 3. april 2014 15:10
Til: Henrik Aagaard Sørensen
Cc: ckan-discuss at lists.okfn.org
Emne: Re: [ckan-discuss] DataPusher, going from DataStore to CSV.

On 1 April 2014 07:45, Henrik Aagaard Sørensen <BU1G at tmf.kk.dk<mailto:BU1G at tmf.kk.dk>> wrote:
I'm running CKAN 2.1.1, if things have changed in 2.2 I do apologize.

If I'm having a DataStore and using the API to append records.
It's only possible to download 100.000 records as CSV via /dump/.

Is it possible to have the DataPusher or other extension adding new records to a corresponding CSV file for quick download of the entire resource?

Not sure quite what you mean by this question. Do you mean add them to the DataStore Table?

Rufus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.okfn.org/pipermail/ckan-discuss/attachments/20140404/ab2ae889/attachment.html>