[ckan-dev] DCAT JSON harvester license property not importing

Fri Feb 17 08:15:41 UTC 2017

Managed to get the module installed - the data.json endpoint works a treat,
however the harvester is a different story - error log is showing
"ImportError: 'module' object has no attribute 'DataJsonHarvester'" and
Apache is throwing a 500 response (I'm running via an Apache WSGI).

I note the module mentions a quirk requiring an "import ckanext" in wsgi.py
(unsure where to find this, is this meant to mean the apache.wsgi file in
/etc/ckan/default?).

Scratching my head a bit but keen to keep working through the problem to
get this harvester working. Any further insight would be appreciated. I'm
running CKAN 2.6.x.

Thanks.

C.

On 17 February 2017 at 08:58, Cam Findlay <cam at camfindlay.com> wrote:

> Nice one, I'll check out that extension Adrià.
>
> Epic high fives for all the help everyone.
>
> +1 open source community :D
>
>
>
>
> On 16 February 2017 at 21:48, Adrià Mercader <adria.mercader at okfn.org>
> wrote:
>
>> Hi Cam,
>>
>> GSA maintain an extension to harvest data.json files:
>> https://github.com/GSA/ckanext-datajson
>>
>> It might be worth investigating.
>>
>> Best,
>>
>> Adrià
>>
>> On 15 February 2017 at 20:41, Cam Findlay <cam at camfindlay.com> wrote:
>> > Thanks for all the responses :)
>> >
>> > The json we are looking to harvest conform to the US Project Open Data
>> > schema https://project-open-data.cio.gov/v1.1/schema/ (this seems to
>> have a
>> > license property at the dataset level).
>> >
>> > We've found that ESRI GIS open data portal have this baked in as a
>> standard
>> > endpoint if you simply add data.json to the end of the portal URL. Eg.
>> > http://aucklandopendata-aucklandcouncil.opendata.arcgis.com/data.json
>> >
>> > Has anyone put together a custom profile for this schema we could reuse?
>> >
>> >
>> > Many thanks,
>> >
>> > Cam.
>> >
>> >
>> >
>> >
>> > On 15 February 2017 at 23:29, Adrià Mercader <adria.mercader at okfn.org>
>> > wrote:
>> >>
>> >> That's a really interesting approach!
>> >>
>> >> I guess it's possible because a very clearly limited number of options
>> >> which are easily ordered. It would be difficult to apply to a more
>> >> generic list like the ones listed here
>> >> http://opendefinition.org/licenses/
>> >>
>> >> But well worth a try if you have limited licensing options
>> >>
>> >> Adrià
>> >>
>> >> On 15 February 2017 at 10:15, Stefan Oderbolz <stefan.oderbolz at liip.ch
>> >
>> >> wrote:
>> >> > For our portal we followed the approach of defining a "hierarchy of
>> >> > openess"
>> >> > of our terms of use (for legal reasons we don't have licenses, but
>> terms
>> >> > of
>> >> > use, but it's basically the same).
>> >> >
>> >> > So the "terms of use" of a dataset is defined as the "least open
>> terms
>> >> > of
>> >> > use of one it's resources", you can find the code here:
>> >> >
>> >> > https://github.com/opendata-swiss/ckanext-switzerland/blob/7
>> 4fc50090d5e4e27af9b44802bb8d693e37d4894/ckanext/switzerland/
>> logic.py#L52-L88
>> >> >
>> >> >
>> >> >
>> >> > On Wed, Feb 15, 2017 at 10:32 AM, Adrià Mercader
>> >> > <adria.mercader at okfn.org>
>> >> > wrote:
>> >> >>
>> >> >> Hi,
>> >> >>
>> >> >> Yes, annoyingly DCAT defines license at the resource level and there
>> >> >> isn't one at the dataset level that resources can inherit from (but
>> >> >> there is one in catalog!)
>> >> >>
>> >> >> https://github.com/ckan/ckanext-dcat/issues/42
>> >> >>
>> >> >> Anyway, I think that the use case of grabbing the first license
>> found
>> >> >> on a resource and applying it to the dataset is a common enough one
>> to
>> >> >> implement it on the default profile, and people can override it if
>> not
>> >> >> happy with it (it will be better than no license at all).
>> >> >>
>> >> >> I'd be more than happy to review and merge a contribution if someone
>> >> >> is up for it.
>> >> >>
>> >> >> Adrià
>> >> >>
>> >> >>
>> >> >> On 15 February 2017 at 08:16, Stefan Oderbolz <
>> stefan.oderbolz at liip.ch>
>> >> >> wrote:
>> >> >> > Hi Cam,
>> >> >> >
>> >> >> > I guess you are not using a custom DCAT profile, because then the
>> >> >> > default
>> >> >> > profile is being used which is "euro_dcat_ap", which is defined
>> here:
>> >> >> >
>> >> >> >
>> >> >> > https://github.com/ckan/ckanext-dcat/blob/e1245cdb5e29194787
>> c33430db637be74e5bc2e1/ckanext/dcat/profiles.py#L577
>> >> >> > This profile does not define licenses on a dataset level, but
>> only in
>> >> >> > a
>> >> >> > level of a distribution (a.k.a. resources). Which DCAT-AP
>> standard do
>> >> >> > you
>> >> >> > want to harvest?
>> >> >> >
>> >> >> > It might be needed to create your own profile to define exactly
>> this
>> >> >> > specifities. The process to do so is described in the README:
>> >> >> > https://github.com/ckan/ckanext-dcat#profiles
>> >> >> >
>> >> >> > I hope this helps.
>> >> >> >
>> >> >> > Best regards Stefan
>> >> >> >
>> >> >> > On Tue, Feb 14, 2017 at 7:57 PM, Cam Findlay <cam at camfindlay.com>
>> >> >> > wrote:
>> >> >> >>
>> >> >> >> Hi all.
>> >> >> >>
>> >> >> >> Has anyone else had trouble with the DCAT JSON harvester in
>> >> >> >> particular
>> >> >> >> the
>> >> >> >> license property not being set on the harvested dataset?
>> >> >> >>
>> >> >> >> Our JSON source includes at the dataset level:
>> >> >> >>
>> >> >> >> "license": "https://creativecommons.org/licenses/by/4.0/",
>> >> >> >>
>> >> >> >> However the final result is no license gets set for the dataset
>> post
>> >> >> >> harvest. I'd expect "Creative Commons Attribution" to show up
>> but we
>> >> >> >> get
>> >> >> >> nothing. If I edit the dataset and manually set the license then
>> it
>> >> >> >> shows
>> >> >> >> fine. So it seems to be lost on import for some reason.
>> >> >> >>
>> >> >> >> No problems with the CKAN to CKAN harvester, we get the license
>> >> >> >> correctly
>> >> >> >> so just checking if there is any known issues around DCAT JSON
>> and
>> >> >> >> license
>> >> >> >> properties.
>> >> >> >>
>> >> >> >> Trying to determine if it's extension related or a problem at our
>> >> >> >> end.
>> >> >> >>
>> >> >> >> (Running CKAN 2.6.X, Ubuntu 14.04 LTS)
>> >> >> >>
>> >> >> >> Thanks in advance :)
>> >> >> >>
>> >> >> >> Cam.
>> >> >> >>
>> >> >> >> _______________________________________________
>> >> >> >> ckan-dev mailing list
>> >> >> >> ckan-dev at lists.okfn.org
>> >> >> >> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> >> >> >> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>> >> >> >>
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > --
>> >> >> > Liip AG  // Limmatstrasse 183 //  CH-8005 Zürich
>> >> >> > Tel +41 43 500 39 80 // GnuPG 0x7B588C67 // www.liip.ch
>> >> >> >
>> >> >> > _______________________________________________
>> >> >> > ckan-dev mailing list
>> >> >> > ckan-dev at lists.okfn.org
>> >> >> > https://lists.okfn.org/mailman/listinfo/ckan-dev
>> >> >> > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>> >> >> >
>> >> >> _______________________________________________
>> >> >> ckan-dev mailing list
>> >> >> ckan-dev at lists.okfn.org
>> >> >> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> >> >> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Liip AG  // Limmatstrasse 183 //  CH-8005 Zürich
>> >> > Tel +41 43 500 39 80 // GnuPG 0x7B588C67 // www.liip.ch
>> >> >
>> >> > _______________________________________________
>> >> > ckan-dev mailing list
>> >> > ckan-dev at lists.okfn.org
>> >> > https://lists.okfn.org/mailman/listinfo/ckan-dev
>> >> > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>> >> >
>> >> _______________________________________________
>> >> ckan-dev mailing list
>> >> ckan-dev at lists.okfn.org
>> >> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> >> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>> >
>> >
>> >
>> > _______________________________________________
>> > ckan-dev mailing list
>> > ckan-dev at lists.okfn.org
>> > https://lists.okfn.org/mailman/listinfo/ckan-dev
>> > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>> >
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20170217/a138d55a/attachment-0003.html>