[ckan-dev] Large data file upload

[Harald von Waldow]

Hi Prashant

I found the problem to upload large files via API was that it doesn't
support streaming uploads.

This has been fixed here:
https://github.com/ckan/ckanapi/pull/109

Works well for me. Also maybe you want to have a look at a client which
makes use of that:
https://github.com/eawag-rdm/resup

Cheers

On 2017-12-19 23:39, Prashant Gupta wrote:
> Hi there,
> 
>  
> 
> I have been trying to upload large resources (5GB-50GB) via CKAN API,
> and facing lot of issues. CKAN currently doesn’t support chunking and
> appending the data when uploading data via API, which causes all sort of
> issues – memory problem on the client side, temp directory storage
> issue, etc. Has anyone faced this issue, and, if yes, how did you sort
> it out?
> 
>  
> 
> There are two solutions that come to my mind: 1) create a plugin to add
> a new API call (if CKAN allows it) which may allow a client to send data
> in chunks and then append it on the server side; 2) this one is more of
> a hack, where I create an empty resource and add the file directly to
> the filesystem (using resource id details) and update the CKAN database.
> I would be keen to know what the community thinks.
> 
>  
> 
> Cheers,
> 
> Prashant Gupta
> 
> 
> 
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
> 

-- 
Harald von Waldow
Eawag
ICT Services
Research Data Management
Ueberlandstrasse 133
8600 Duebendorf
http://www.eawag.ch
https://opendata.eawag.ch

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20171220/2e8e8c34/attachment-0003.sig>