[ckan-dev] Import datasets from remote CKAN with ckanext-harvest and Datastore

[Steven De Costa]

For the larger resources people should check out the express loader  :)

https://ckan.org/2017/11/20/express-loader-for-ckan-datastore/

It’s fresh!

On Wed, Nov 22, 2017 at 10:48 AM Stefan Oderbolz <stefan.oderbolz at liip.ch>
wrote:

> Hi Rafa,
>
> we have the same use case and use DataPusher for this purpose (
> http://docs.ckan.org/projects/datapusher/en/latest/).
>
> 1. We harvest the datasets using a ckanext-harvest from the source to the
> target instance
> 2. DataPusher is installed on the target CKAN and starts importing the
> datasets to the DataStore
>
> Depending on the size of the ressources and the amount of ressources this
> can lead to performance problems. We currently haven't figured out a stable
> setup to replicate this for larger ressources (> 50MB), therefore we limit
> datapusher to only import smaller ressources.
>
> - Stefan
>
> On Mon, Nov 20, 2017 at 12:16 PM, Rafa <rafsalrob at gmail.com> wrote:
>
>> Hey Everyone,
>>
>> I'm involved in a project in which I need to import datasets from a
>> remote CKAN which uses Datastore for store the data. I have configured
>> DataStore and Haverst extensions in the target CKAN but when I run the
>> harvest job, the process only copy the resources metadata (title, url,
>> name, format) and the datastore tables are not created. Is there any way to
>> copy the files from the FileStore and the data from the DataStore (tables)?
>>
>> Cheers,
>> Rafa.
>>
>> --
>> --------------------------------------------------
>> Rafael Salas Robledo
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>
>>
>
>
> --
> Liip AG  // Limmatstrasse 183 //  CH-8005 Zürich
> Tel +41 43 500 39 80 // GnuPG 0x7B588C67 // www.liip.ch
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>
-- 
Please note: I’m based in Berlin up to 27 November and my email response
times will reflect that timezone

*STEVEN DE COSTA *|
*EXECUTIVE DIRECTOR*www.linkdigital.com.au

Want to be updated with news from Link Digital? Sign up
<http://linkdigital.com.au/content/newsletter?utm_source=email_signature&utm_medium=email&utm_campaign=newsletter_signup>
to
receive our Newsletter!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20171122/355a9c99/attachment-0003.html>