[ckan-discuss] DataPusher, going from DataStore to CSV.

Tue Apr 8 06:56:08 UTC 2014

https://github.com/ckan/ideas-and-roadmap/issues/34

Done :)

Fra: Rufus Pollock [mailto:rufus.pollock at okfn.org]
Sendt: 4. april 2014 19:20
Til: Henrik Aagaard Sørensen
Cc: ckan-discuss at lists.okfn.org
Emne: Re: [ckan-discuss] DataPusher, going from DataStore to CSV.

On 4 April 2014 08:08, Henrik Aagaard Sørensen <BU1G at tmf.kk.dk<mailto:BU1G at tmf.kk.dk>> wrote:
I would like to use the DataStore via the API as primary data-source. This works without a problem already.

However, if people wants to download the entire resource as a CSV, via /dump/, it only downloads 100K records (this is hardcoded into CKAN).
It also takes quite a long time to generate the CSV file.

I have resources with over 10+ mio. Rows and would like to offer a complete download via CSV. But changing the hardcoded 100K row limit puts a lot of pressure on the system.
It would be very nice to have a feature where, using the API for the DataStore, would update a corresponding CSV-file for download. So download wouldn't need to generate the file.

completely understand and makes a lot of sense. Can I suggest you open an "idea" here (and include your excellent explanation above):

https://github.com/ckan/ideas-and-roadmap/issues

This is essentially a request for a new feature or an improvement in an existing one.

Rufus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.okfn.org/pipermail/ckan-discuss/attachments/20140408/7a19d9ce/attachment.html>