[ckan-dev] Middleware harvesting

Mon Feb 11 12:22:05 UTC 2013

Hi Lasse,

On 8 February 2013 20:58, Lasse Vestergaard <ibbernik at gmail.com> wrote:
> I first put this mail under a wrong title. Sorry.
>
> Hi CKAN devs.
>
> I have been looking at the harvester extensions and how to create your own.
> I know it is asking for a lot, but would it be possible to expand the
> documentation on this part? I think I got the conceptual idea about how
> harvesters work, but I can't figure out how to get started. I don't have any
> concrete problems, my issues relates more to the following questions:
>
> Isn't it possible to harvest everything your heart want?
No, you will need to write a harvester for a specific source type if
it doesn't already exist

> I could harvest actual data and not only metadata?
Only metadata is harvested, although it should be possible to write
custom extensions that downloaded files, upload them to the datastore,
etc depending on your needs. Again, current functionality does not
support it.

> Couldn't I harvest directly from an Oracle database client that has some
> kind of accessible API?
> Or could I harvest data directly form a web service?
See above

>
> When I read the documentation it seems like I can only harvest metadata from
> data resources that conform to CSW.
ckanext-harvest provides the backend to build harvesters in CKAN.
ckanext-spatial implements this interface for providing harvesters for
spatial metadada (eg CSW), but that is just a specific implementation.
For instance ckanext-harvest, has also the CKAN harvester that
harvests metadata from other CKAN instances.

Hope this helps,

Adrià

> My concrete issue is that I need to
> write middleware when ever I want to access data from a new proprietary data
> source. Wouldn't it be interesting to extend CKAN with some kind of generic
> middleware hook? That way one could program a new data source extension on
> top of the generic one. This might be related to the datastore, I guess.
>
> Regards
>
> Lasse Vestergaard
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>