[ckan-dev] DCAT JSON harvester license property not importing

Fri Feb 17 14:01:18 UTC 2017

The entry point on setup.py for the datajson_harvester plugin points to:

datajson_harvest=ckanext.datajson.plugin:DataJsonHarvester

But that class is not present on plugin.py.

It should be

datajson_harvest=ckanext.datajson.harvester_datajson:DataJsonHarvester

or actually import DataJsonHarvester on plugin.py

Hope this helps,
Adrià

On 17 February 2017 at 08:15, Cam Findlay <cam at camfindlay.com> wrote:
> Managed to get the module installed - the data.json endpoint works a treat,
> however the harvester is a different story - error log is showing
> "ImportError: 'module' object has no attribute 'DataJsonHarvester'" and
> Apache is throwing a 500 response (I'm running via an Apache WSGI).
>
> I note the module mentions a quirk requiring an "import ckanext" in wsgi.py
> (unsure where to find this, is this meant to mean the apache.wsgi file in
> /etc/ckan/default?).
>
> Scratching my head a bit but keen to keep working through the problem to get
> this harvester working. Any further insight would be appreciated. I'm
> running CKAN 2.6.x.
>
> Thanks.
>
> C.
>
>
>
>
> On 17 February 2017 at 08:58, Cam Findlay <cam at camfindlay.com> wrote:
>>
>> Nice one, I'll check out that extension Adrià.
>>
>> Epic high fives for all the help everyone.
>>
>> +1 open source community :D
>>
>>
>>
>>
>> On 16 February 2017 at 21:48, Adrià Mercader <adria.mercader at okfn.org>
>> wrote:
>>>
>>> Hi Cam,
>>>
>>> GSA maintain an extension to harvest data.json files:
>>> https://github.com/GSA/ckanext-datajson
>>>
>>> It might be worth investigating.
>>>
>>> Best,
>>>
>>> Adrià
>>>
>>> On 15 February 2017 at 20:41, Cam Findlay <cam at camfindlay.com> wrote:
>>> > Thanks for all the responses :)
>>> >
>>> > The json we are looking to harvest conform to the US Project Open Data
>>> > schema https://project-open-data.cio.gov/v1.1/schema/ (this seems to
>>> > have a
>>> > license property at the dataset level).
>>> >
>>> > We've found that ESRI GIS open data portal have this baked in as a
>>> > standard
>>> > endpoint if you simply add data.json to the end of the portal URL. Eg.
>>> > http://aucklandopendata-aucklandcouncil.opendata.arcgis.com/data.json
>>> >
>>> > Has anyone put together a custom profile for this schema we could
>>> > reuse?
>>> >
>>> >
>>> > Many thanks,
>>> >
>>> > Cam.
>>> >
>>> >
>>> >
>>> >
>>> > On 15 February 2017 at 23:29, Adrià Mercader <adria.mercader at okfn.org>
>>> > wrote:
>>> >>
>>> >> That's a really interesting approach!
>>> >>
>>> >> I guess it's possible because a very clearly limited number of options
>>> >> which are easily ordered. It would be difficult to apply to a more
>>> >> generic list like the ones listed here
>>> >> http://opendefinition.org/licenses/
>>> >>
>>> >> But well worth a try if you have limited licensing options
>>> >>
>>> >> Adrià
>>> >>
>>> >> On 15 February 2017 at 10:15, Stefan Oderbolz
>>> >> <stefan.oderbolz at liip.ch>
>>> >> wrote:
>>> >> > For our portal we followed the approach of defining a "hierarchy of
>>> >> > openess"
>>> >> > of our terms of use (for legal reasons we don't have licenses, but
>>> >> > terms
>>> >> > of
>>> >> > use, but it's basically the same).
>>> >> >
>>> >> > So the "terms of use" of a dataset is defined as the "least open
>>> >> > terms
>>> >> > of
>>> >> > use of one it's resources", you can find the code here:
>>> >> >
>>> >> >
>>> >> > https://github.com/opendata-swiss/ckanext-switzerland/blob/74fc50090d5e4e27af9b44802bb8d693e37d4894/ckanext/switzerland/logic.py#L52-L88
>>> >> >
>>> >> >
>>> >> >
>>> >> > On Wed, Feb 15, 2017 at 10:32 AM, Adrià Mercader
>>> >> > <adria.mercader at okfn.org>
>>> >> > wrote:
>>> >> >>
>>> >> >> Hi,
>>> >> >>
>>> >> >> Yes, annoyingly DCAT defines license at the resource level and
>>> >> >> there
>>> >> >> isn't one at the dataset level that resources can inherit from (but
>>> >> >> there is one in catalog!)
>>> >> >>
>>> >> >> https://github.com/ckan/ckanext-dcat/issues/42
>>> >> >>
>>> >> >> Anyway, I think that the use case of grabbing the first license
>>> >> >> found
>>> >> >> on a resource and applying it to the dataset is a common enough one
>>> >> >> to
>>> >> >> implement it on the default profile, and people can override it if
>>> >> >> not
>>> >> >> happy with it (it will be better than no license at all).
>>> >> >>
>>> >> >> I'd be more than happy to review and merge a contribution if
>>> >> >> someone
>>> >> >> is up for it.
>>> >> >>
>>> >> >> Adrià
>>> >> >>
>>> >> >>
>>> >> >> On 15 February 2017 at 08:16, Stefan Oderbolz
>>> >> >> <stefan.oderbolz at liip.ch>
>>> >> >> wrote:
>>> >> >> > Hi Cam,
>>> >> >> >
>>> >> >> > I guess you are not using a custom DCAT profile, because then the
>>> >> >> > default
>>> >> >> > profile is being used which is "euro_dcat_ap", which is defined
>>> >> >> > here:
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> > https://github.com/ckan/ckanext-dcat/blob/e1245cdb5e29194787c33430db637be74e5bc2e1/ckanext/dcat/profiles.py#L577
>>> >> >> > This profile does not define licenses on a dataset level, but
>>> >> >> > only in
>>> >> >> > a
>>> >> >> > level of a distribution (a.k.a. resources). Which DCAT-AP
>>> >> >> > standard do
>>> >> >> > you
>>> >> >> > want to harvest?
>>> >> >> >
>>> >> >> > It might be needed to create your own profile to define exactly
>>> >> >> > this
>>> >> >> > specifities. The process to do so is described in the README:
>>> >> >> > https://github.com/ckan/ckanext-dcat#profiles
>>> >> >> >
>>> >> >> > I hope this helps.
>>> >> >> >
>>> >> >> > Best regards Stefan
>>> >> >> >
>>> >> >> > On Tue, Feb 14, 2017 at 7:57 PM, Cam Findlay <cam at camfindlay.com>
>>> >> >> > wrote:
>>> >> >> >>
>>> >> >> >> Hi all.
>>> >> >> >>
>>> >> >> >> Has anyone else had trouble with the DCAT JSON harvester in
>>> >> >> >> particular
>>> >> >> >> the
>>> >> >> >> license property not being set on the harvested dataset?
>>> >> >> >>
>>> >> >> >> Our JSON source includes at the dataset level:
>>> >> >> >>
>>> >> >> >> "license": "https://creativecommons.org/licenses/by/4.0/",
>>> >> >> >>
>>> >> >> >> However the final result is no license gets set for the dataset
>>> >> >> >> post
>>> >> >> >> harvest. I'd expect "Creative Commons Attribution" to show up
>>> >> >> >> but we
>>> >> >> >> get
>>> >> >> >> nothing. If I edit the dataset and manually set the license then
>>> >> >> >> it
>>> >> >> >> shows
>>> >> >> >> fine. So it seems to be lost on import for some reason.
>>> >> >> >>
>>> >> >> >> No problems with the CKAN to CKAN harvester, we get the license
>>> >> >> >> correctly
>>> >> >> >> so just checking if there is any known issues around DCAT JSON
>>> >> >> >> and
>>> >> >> >> license
>>> >> >> >> properties.
>>> >> >> >>
>>> >> >> >> Trying to determine if it's extension related or a problem at
>>> >> >> >> our
>>> >> >> >> end.
>>> >> >> >>
>>> >> >> >> (Running CKAN 2.6.X, Ubuntu 14.04 LTS)
>>> >> >> >>
>>> >> >> >> Thanks in advance :)
>>> >> >> >>
>>> >> >> >> Cam.
>>> >> >> >>
>>> >> >> >> _______________________________________________
>>> >> >> >> ckan-dev mailing list
>>> >> >> >> ckan-dev at lists.okfn.org
>>> >> >> >> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>> >> >> >> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>> >> >> >>
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> > --
>>> >> >> > Liip AG  // Limmatstrasse 183 //  CH-8005 Zürich
>>> >> >> > Tel +41 43 500 39 80 // GnuPG 0x7B588C67 // www.liip.ch
>>> >> >> >
>>> >> >> > _______________________________________________
>>> >> >> > ckan-dev mailing list
>>> >> >> > ckan-dev at lists.okfn.org
>>> >> >> > https://lists.okfn.org/mailman/listinfo/ckan-dev
>>> >> >> > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>> >> >> >
>>> >> >> _______________________________________________
>>> >> >> ckan-dev mailing list
>>> >> >> ckan-dev at lists.okfn.org
>>> >> >> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>> >> >> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>> >> >
>>> >> >
>>> >> >
>>> >> >
>>> >> > --
>>> >> > Liip AG  // Limmatstrasse 183 //  CH-8005 Zürich
>>> >> > Tel +41 43 500 39 80 // GnuPG 0x7B588C67 // www.liip.ch
>>> >> >
>>> >> > _______________________________________________
>>> >> > ckan-dev mailing list
>>> >> > ckan-dev at lists.okfn.org
>>> >> > https://lists.okfn.org/mailman/listinfo/ckan-dev
>>> >> > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>> >> >
>>> >> _______________________________________________
>>> >> ckan-dev mailing list
>>> >> ckan-dev at lists.okfn.org
>>> >> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>> >> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>> >
>>> >
>>> >
>>> > _______________________________________________
>>> > ckan-dev mailing list
>>> > ckan-dev at lists.okfn.org
>>> > https://lists.okfn.org/mailman/listinfo/ckan-dev
>>> > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>> >
>>> _______________________________________________
>>> ckan-dev mailing list
>>> ckan-dev at lists.okfn.org
>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>
>>
>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>