[datahub-discuss] Harvesting datasets into datahub

Mon Sep 28 12:20:57 UTC 2015

Hi Ross,

Obviously this hasn't been communicated very well but OK is working to
migrate datahub.io to a new cloud based infrastructure to hopefully
make it more performant and cheaper to run. I've been working on the
initial version of this (based on Docker) and didn't include
harvesting as it wasn't yet enabled. We'll have to see how to set up
redis, queues, etc on the cloud env, but I guess we will be able to
set it up on at a later stage.

On 26 September 2015 at 15:48, Ross Jones <ross at servercode.co.uk> wrote:
> Hi,
>
> I've just installed and enabled the CKAN Harvester on datahub.io, but currently just with the CKAN harvester.  Because the machine that datahub is on isn't ideal with the extra work of harvesting (it's on the ever-increasing task list) there's no immediate/easy way to get to the harvesting page so that it doesn't get abused ... obviously it's not hard to find if you want it *cough*http://datahub.io/harvest*cough* but right now it's best not to make it too obvious.
>
> If you want to harvest somewhere, or particularly want installation of another harvester type (I think there's one for data.json, and some geo stuff) then please shout and I'll help you get it set up.  This should make it a lot easier for importing any of the old ckan.net sites that are still out there.
>
> Cheers
>
> Ross
>
>
>
> _______________________________________________
> datahub-discuss mailing list
> datahub-discuss at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/datahub-discuss