[datahub-discuss] Harvesting datasets into datahub

Mon Sep 28 12:23:07 UTC 2015

Hi Adrià,

Ah, well, that's useful to know - I wasn't sure anyone was doing anything with it (it was 503ing when I first looked).

Thanks for the update, I'll leave it alone from now on then :)

Cheers

Ross.

> On 28 Sep 2015, at 13:20, Adrià Mercader <adria.mercader at okfn.org> wrote:
> 
> Hi Ross,
> 
> Obviously this hasn't been communicated very well but OK is working to
> migrate datahub.io to a new cloud based infrastructure to hopefully
> make it more performant and cheaper to run. I've been working on the
> initial version of this (based on Docker) and didn't include
> harvesting as it wasn't yet enabled. We'll have to see how to set up
> redis, queues, etc on the cloud env, but I guess we will be able to
> set it up on at a later stage.
> 
> On 26 September 2015 at 15:48, Ross Jones <ross at servercode.co.uk> wrote:
>> Hi,
>> 
>> I've just installed and enabled the CKAN Harvester on datahub.io, but currently just with the CKAN harvester.  Because the machine that datahub is on isn't ideal with the extra work of harvesting (it's on the ever-increasing task list) there's no immediate/easy way to get to the harvesting page so that it doesn't get abused ... obviously it's not hard to find if you want it *cough*http://datahub.io/harvest*cough* but right now it's best not to make it too obvious.
>> 
>> If you want to harvest somewhere, or particularly want installation of another harvester type (I think there's one for data.json, and some geo stuff) then please shout and I'll help you get it set up.  This should make it a lot easier for importing any of the old ckan.net sites that are still out there.
>> 
>> Cheers
>> 
>> Ross
>> 
>> 
>> 
>> _______________________________________________
>> datahub-discuss mailing list
>> datahub-discuss at lists.okfn.org
>> https://lists.okfn.org/mailman/listinfo/datahub-discuss