[ckan-dev] CKAN harvester harvests datasets with type "harvest"

Thu Aug 15 13:59:54 UTC 2013

Hi Stefan,

You are right, this should be handled somehow. I'n not keen in
modifying the API to not return them, as this has wider implications
(and they are datasets after all, so you still want them on the API).
I think that the easiest way to fix for the time being is ignoring
harvest sources on the CKAN harvester, I added a small patch that does
this [1].
The whole CKAN harvester is a bit dated and would need a bit of a refactor.

Thanks for your other PR, I also merged it.

Adrià

[1] https://github.com/okfn/ckanext-harvest/commit/01ca5c0d

On 14 August 2013 12:10, Stefan Oderbolz <stefan.oderbolz at liip.ch> wrote:
> Hi there,
>
> On my CKAN instance I was unable to reach the /harvest page to manage the
> harvest sources. I ran across a strange error in the logfile:
>
> [error] [client XX.XX.XX.XX] Error - <class
> 'jinja2.exceptions.UndefinedError'>: 'dict object' has no attribute 'status'
>
> When I investigated the templates of the ckanext-harvest extension, I found
> some occurances of "status" and I was finally able to find the ones causing
> this exception. I added checks to the templates to make them robust against
> this it, here is my pull request:
> https://github.com/okfn/ckanext-harvest/pull/60
>
> When I was finally able to see the harvest page again, I saw what likely
> caused the error: I run the harvester against another CKAN instance. This
> other instance has itself some customer harvesters of mine to load data.
> Now it seems, that the CKAN-CKAN harvester also harvests the harvester
> sources (instead of ignoring them as I expected). These "new" harvest
> sources took down the harvest page because they didn't have a status,
> because they actually only exist on my 2nd instance.
>
> I think this is a bug, or am I missing a flag to configure whether or not to
> harvest the "harvest" datasets?
>
> If someone can point me to the right direction, I willing to provide a pull
> request to fix it. My guess would be, that the API should not return harvest
> sources or that we should introduce a new configuration option for this.
>
> Best Regards
> Stefan
>
> --
> Liip AG  //  Feldstrasse 133 //  CH-8004 Zurich
> Tel +41 43 500 39 80 // GnuPG 0x7B588C67 // www.liip.ch
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>