[ckan-dev] Middleware harvesting

Mon Feb 11 16:23:52 UTC 2013

Thanks for the response. I was just making sure that I didn't miss
interpret the havesting mechanism.

Lasse Vestergaard

2013/2/11 Adrià Mercader <adria.mercader at okfn.org>

> Hi Lasse,
>
> On 8 February 2013 20:58, Lasse Vestergaard <ibbernik at gmail.com> wrote:
> > I first put this mail under a wrong title. Sorry.
> >
> > Hi CKAN devs.
> >
> > I have been looking at the harvester extensions and how to create your
> own.
> > I know it is asking for a lot, but would it be possible to expand the
> > documentation on this part? I think I got the conceptual idea about how
> > harvesters work, but I can't figure out how to get started. I don't have
> any
> > concrete problems, my issues relates more to the following questions:
> >
> > Isn't it possible to harvest everything your heart want?
> No, you will need to write a harvester for a specific source type if
> it doesn't already exist
>
> > I could harvest actual data and not only metadata?
> Only metadata is harvested, although it should be possible to write
> custom extensions that downloaded files, upload them to the datastore,
> etc depending on your needs. Again, current functionality does not
> support it.
>
> > Couldn't I harvest directly from an Oracle database client that has some
> > kind of accessible API?
> > Or could I harvest data directly form a web service?
> See above
>
> >
> > When I read the documentation it seems like I can only harvest metadata
> from
> > data resources that conform to CSW.
> ckanext-harvest provides the backend to build harvesters in CKAN.
> ckanext-spatial implements this interface for providing harvesters for
> spatial metadada (eg CSW), but that is just a specific implementation.
> For instance ckanext-harvest, has also the CKAN harvester that
> harvests metadata from other CKAN instances.
>
>
> Hope this helps,
>
> Adrià
>
> > My concrete issue is that I need to
> > write middleware when ever I want to access data from a new proprietary
> data
> > source. Wouldn't it be interesting to extend CKAN with some kind of
> generic
> > middleware hook? That way one could program a new data source extension
> on
> > top of the generic one. This might be related to the datastore, I guess.
> >
> > Regards
> >
> > Lasse Vestergaard
> >
> > _______________________________________________
> > ckan-dev mailing list
> > ckan-dev at lists.okfn.org
> > http://lists.okfn.org/mailman/listinfo/ckan-dev
> > Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
> >
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20130211/555f15d8/attachment-0001.html>