[ckan-discuss] DataPusher, going from DataStore to CSV.

[Rufus Pollock]

On 4 April 2014 08:08, Henrik Aagaard Sørensen <BU1G at tmf.kk.dk> wrote:

> I would like to use the DataStore via the API as primary data-source. This
> works without a problem already.
>
>
>
> However, if people wants to download the entire resource as a CSV, via
> /dump/, it only downloads 100K records (this is hardcoded into CKAN).
>
> It also takes quite a long time to generate the CSV file.
>
>
>
> I have resources with over 10+ mio. Rows and would like to offer a
> complete download via CSV. But changing the hardcoded 100K row limit puts a
> lot of pressure on the system.
>
> It would be very nice to have a feature where, using the API for the
> DataStore, would update a corresponding CSV-file for download. So download
> wouldn't need to generate the file.
>

completely understand and makes a lot of sense. Can I suggest you open an
"idea" here (and include your excellent explanation above):

https://github.com/ckan/ideas-and-roadmap/issues

This is essentially a request for a new feature or an improvement in an
existing one.

Rufus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.okfn.org/pipermail/ckan-discuss/attachments/20140404/55f9d000/attachment.html>