[ckan-dev] Batch upload

[Stefan Oderbolz]

Hi Aaron,

what you describe is a use case that we often have and usually we write
semi-customized harvesters to fetch the data. An example that comes close
to what you want would be this harvester:
https://github.com/opendatazurich/ckanext-stadtzh-harvest

Our client provides a folder with datasets to import. Each contains
metadata as an XML-File ("meta.xml"). We use the power of ckanext-harvest
to process all the datasets. The code is not very generic, as we had to
cover some special cases, but I think it gets you an idea how it could be
done.

- Stefan

On Thu, Sep 3, 2015 at 4:23 PM, Denis Zgonjanin <deniszgonjanin at gmail.com>
wrote:

> There is a project we did a few months ago, to batch export all CKAN data
> and metadata into the Data Packages (
> http://dataprotocols.org/data-packages/) format. The code we wrote is
> built into the ckanapi:
> https://github.com/ckan/ckanapi#bulk-dataset-and-resource-export---datapackagejson-format
>
> Now what's left is the same functionality, the other way. A batch import.
>
> If you have any appetite to take it on, it might be a good addition for
> ckanapi as well.
>
>
>
> On Thu, Sep 3, 2015 at 12:44 AM, Aaron McGlinchy <
> McGlinchyA at landcareresearch.co.nz> wrote:
>
>> Hi,
>>    Has anyone done any work on batch upload of data e.g. point to a
>> folder containing numerous resources, enter dataset level metadata, point
>> to the folder and load up all the files in the folder under that dataset.
>> Could have a manifest file that contained a list of the file names and the
>> description to be associated with that file when it was uploaded as a
>> resource...
>>
>> Could consider zipping e.g. if the main folder contained subfolders which
>> contained several files, maybe zip each folder.
>>
>> Not wedded to anything, just wondering what if any work been done around
>> such things.
>>
>> Thanks
>> Aaron
>>
>> ________________________________
>>
>> Please consider the environment before printing this email
>> Warning: This electronic message together with any attachments is
>> confidential. If you receive it in error: (i) you must not read, use,
>> disclose, copy or retain it; (ii) please contact the sender immediately by
>> reply email and then delete the emails.
>> The views expressed in this email may not be those of Landcare Research
>> New Zealand Limited. http://www.landcareresearch.co.nz
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>
>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>
>


-- 
Liip AG  // Limmatstrasse 183 //  CH-8005 Zürich
Tel +41 43 500 39 80 // GnuPG 0x7B588C67 // www.liip.ch
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20150904/47e95993/attachment-0003.html>