[ckan-dev] Large data file upload

[Prashant Gupta]

Hi there,

I have been trying to upload large resources (5GB-50GB) via CKAN API, and facing lot of issues. CKAN currently doesn’t support chunking and appending the data when uploading data via API, which causes all sort of issues – memory problem on the client side, temp directory storage issue, etc. Has anyone faced this issue, and, if yes, how did you sort it out?

There are two solutions that come to my mind: 1) create a plugin to add a new API call (if CKAN allows it) which may allow a client to send data in chunks and then append it on the server side; 2) this one is more of a hack, where I create an empty resource and add the file directly to the filesystem (using resource id details) and update the CKAN database. I would be keen to know what the community thinks.

Cheers,
Prashant Gupta
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20171219/546f2c87/attachment-0002.html>