[ckan-dev] ckan-dev Digest, Vol 86, Issue 16

Wed Dec 20 21:34:31 UTC 2017

Hi Harold,

That looks perfect for now.

Have a Merry Christmas and a Happy New Year (

Cheers,
Prashant

From: Harald von Waldow <harald.vonwaldow at eawag.ch>
    To: <ckan-dev at lists.okfn.org>
    Subject: Re: [ckan-dev] Large data file upload
    Message-ID: <c06d6891-7945-f7ba-5bb5-7dc0230960d2 at eawag.ch>
    Content-Type: text/plain; charset="utf-8"

    Hi Prashant

    I found the problem to upload large files via API was that it doesn't
    support streaming uploads.

    This has been fixed here:
    https://github.com/ckan/ckanapi/pull/109

    Works well for me. Also maybe you want to have a look at a client which
    makes use of that:
    https://github.com/eawag-rdm/resup

    Cheers

    On 2017-12-19 23:39, Prashant Gupta wrote:
    > Hi there,
    > 
    > ?
    > 
    > I have been trying to upload large resources (5GB-50GB) via CKAN API,
    > and facing lot of issues. CKAN currently doesn?t support chunking and
    > appending the data when uploading data via API, which causes all sort of
    > issues ? memory problem on the client side, temp directory storage
    > issue, etc. Has anyone faced this issue, and, if yes, how did you sort
    > it out?
    > 
    > ?
    > 
    > There are two solutions that come to my mind: 1) create a plugin to add
    > a new API call (if CKAN allows it) which may allow a client to send data
    > in chunks and then append it on the server side; 2) this one is more of
    > a hack, where I create an empty resource and add the file directly to
    > the filesystem (using resource id details) and update the CKAN database.
    > I would be keen to know what the community thinks.
    > 
    > ?
    > 
    > Cheers,
    > 
    > Prashant Gupta
    > 
    > 
    > 
    > _______________________________________________
    > ckan-dev mailing list
    > ckan-dev at lists.okfn.org
    > https://lists.okfn.org/mailman/listinfo/ckan-dev
    > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
    > 

    -- 
    Harald von Waldow
    Eawag
    ICT Services
    Research Data Management
    Ueberlandstrasse 133
    8600 Duebendorf
    http://www.eawag.ch
    https://opendata.eawag.ch

    -------------- next part --------------
    A non-text attachment was scrubbed...
    Name: signature.asc
    Type: application/pgp-signature
    Size: 833 bytes
    Desc: OpenPGP digital signature
    URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20171220/2e8e8c34/attachment-0001.sig>

    ------------------------------

    Subject: Digest Footer

    _______________________________________________
    ckan-dev mailing list
    ckan-dev at lists.okfn.org
    https://lists.okfn.org/mailman/listinfo/ckan-dev
    Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev

    ------------------------------

    End of ckan-dev Digest, Vol 86, Issue 16
    ****************************************