[ckan-dev] DCAT JSON harvester license property not importing

Tue Mar 7 20:03:29 UTC 2017

Pull request raised for this at
https://github.com/GSA/ckanext-datajson/pull/35 if anyone wants to chime in
and provide more context that would be great.

On 18 February 2017 at 22:25, Cam Findlay <cam at camfindlay.com> wrote:

> That fixed it. Will look to raise a PR for this fix.
>
> C.
>
>
>
> On Sat, 18 Feb 2017 at 3:01 AM, Adrià Mercader <adria.mercader at okfn.org>
> wrote:
>
>> The entry point on setup.py for the datajson_harvester plugin points to:
>>
>> datajson_harvest=ckanext.datajson.plugin:DataJsonHarvester
>>
>> But that class is not present on plugin.py.
>>
>> It should be
>>
>> datajson_harvest=ckanext.datajson.harvester_datajson:DataJsonHarvester
>>
>> or actually import DataJsonHarvester on plugin.py
>>
>>
>> Hope this helps,
>> Adrià
>>
>> On 17 February 2017 at 08:15, Cam Findlay <cam at camfindlay.com> wrote:
>> > Managed to get the module installed - the data.json endpoint works a
>> treat,
>> > however the harvester is a different story - error log is showing
>> > "ImportError: 'module' object has no attribute 'DataJsonHarvester'" and
>> > Apache is throwing a 500 response (I'm running via an Apache WSGI).
>> >
>> > I note the module mentions a quirk requiring an "import ckanext" in
>> wsgi.py
>> > (unsure where to find this, is this meant to mean the apache.wsgi file
>> in
>> > /etc/ckan/default?).
>> >
>> > Scratching my head a bit but keen to keep working through the problem
>> to get
>> > this harvester working. Any further insight would be appreciated. I'm
>> > running CKAN 2.6.x.
>> >
>> > Thanks.
>> >
>> > C.
>> >
>> >
>> >
>> >
>> > On 17 February 2017 at 08:58, Cam Findlay <cam at camfindlay.com> wrote:
>> >>
>> >> Nice one, I'll check out that extension Adrià.
>> >>
>> >> Epic high fives for all the help everyone.
>> >>
>> >> +1 open source community :D
>> >>
>> >>
>> >>
>> >>
>> >> On 16 February 2017 at 21:48, Adrià Mercader <adria.mercader at okfn.org>
>> >> wrote:
>> >>>
>> >>> Hi Cam,
>> >>>
>> >>> GSA maintain an extension to harvest data.json files:
>> >>> https://github.com/GSA/ckanext-datajson
>> >>>
>> >>> It might be worth investigating.
>> >>>
>> >>> Best,
>> >>>
>> >>> Adrià
>> >>>
>> >>> On 15 February 2017 at 20:41, Cam Findlay <cam at camfindlay.com> wrote:
>> >>> > Thanks for all the responses :)
>> >>> >
>> >>> > The json we are looking to harvest conform to the US Project Open
>> Data
>> >>> > schema https://project-open-data.cio.gov/v1.1/schema/ (this seems
>> to
>> >>> > have a
>> >>> > license property at the dataset level).
>> >>> >
>> >>> > We've found that ESRI GIS open data portal have this baked in as a
>> >>> > standard
>> >>> > endpoint if you simply add data.json to the end of the portal URL.
>> Eg.
>> >>> > http://aucklandopendata-aucklandcouncil.opendata.
>> arcgis.com/data.json
>> >>> >
>> >>> > Has anyone put together a custom profile for this schema we could
>> >>> > reuse?
>> >>> >
>> >>> >
>> >>> > Many thanks,
>> >>> >
>> >>> > Cam.
>> >>> >
>> >>> >
>> >>> >
>> >>> >
>> >>> > On 15 February 2017 at 23:29, Adrià Mercader <
>> adria.mercader at okfn.org>
>> >>> > wrote:
>> >>> >>
>> >>> >> That's a really interesting approach!
>> >>> >>
>> >>> >> I guess it's possible because a very clearly limited number of
>> options
>> >>> >> which are easily ordered. It would be difficult to apply to a more
>> >>> >> generic list like the ones listed here
>> >>> >> http://opendefinition.org/licenses/
>> >>> >>
>> >>> >> But well worth a try if you have limited licensing options
>> >>> >>
>> >>> >> Adrià
>> >>> >>
>> >>> >> On 15 February 2017 at 10:15, Stefan Oderbolz
>> >>> >> <stefan.oderbolz at liip.ch>
>> >>> >> wrote:
>> >>> >> > For our portal we followed the approach of defining a "hierarchy
>> of
>> >>> >> > openess"
>> >>> >> > of our terms of use (for legal reasons we don't have licenses,
>> but
>> >>> >> > terms
>> >>> >> > of
>> >>> >> > use, but it's basically the same).
>> >>> >> >
>> >>> >> > So the "terms of use" of a dataset is defined as the "least open
>> >>> >> > terms
>> >>> >> > of
>> >>> >> > use of one it's resources", you can find the code here:
>> >>> >> >
>> >>> >> >
>> >>> >> > https://github.com/opendata-swiss/ckanext-switzerland/blob/
>> 74fc50090d5e4e27af9b44802bb8d693e37d4894/ckanext/
>> switzerland/logic.py#L52-L88
>> >>> >> >
>> >>> >> >
>> >>> >> >
>> >>> >> > On Wed, Feb 15, 2017 at 10:32 AM, Adrià Mercader
>> >>> >> > <adria.mercader at okfn.org>
>> >>> >> > wrote:
>> >>> >> >>
>> >>> >> >> Hi,
>> >>> >> >>
>> >>> >> >> Yes, annoyingly DCAT defines license at the resource level and
>> >>> >> >> there
>> >>> >> >> isn't one at the dataset level that resources can inherit from
>> (but
>> >>> >> >> there is one in catalog!)
>> >>> >> >>
>> >>> >> >> https://github.com/ckan/ckanext-dcat/issues/42
>> >>> >> >>
>> >>> >> >> Anyway, I think that the use case of grabbing the first license
>> >>> >> >> found
>> >>> >> >> on a resource and applying it to the dataset is a common enough
>> one
>> >>> >> >> to
>> >>> >> >> implement it on the default profile, and people can override it
>> if
>> >>> >> >> not
>> >>> >> >> happy with it (it will be better than no license at all).
>> >>> >> >>
>> >>> >> >> I'd be more than happy to review and merge a contribution if
>> >>> >> >> someone
>> >>> >> >> is up for it.
>> >>> >> >>
>> >>> >> >> Adrià
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> On 15 February 2017 at 08:16, Stefan Oderbolz
>> >>> >> >> <stefan.oderbolz at liip.ch>
>> >>> >> >> wrote:
>> >>> >> >> > Hi Cam,
>> >>> >> >> >
>> >>> >> >> > I guess you are not using a custom DCAT profile, because then
>> the
>> >>> >> >> > default
>> >>> >> >> > profile is being used which is "euro_dcat_ap", which is
>> defined
>> >>> >> >> > here:
>> >>> >> >> >
>> >>> >> >> >
>> >>> >> >> >
>> >>> >> >> > https://github.com/ckan/ckanext-dcat/blob/
>> e1245cdb5e29194787c33430db637be74e5bc2e1/ckanext/dcat/profiles.py#L577
>> >>> >> >> > This profile does not define licenses on a dataset level, but
>> >>> >> >> > only in
>> >>> >> >> > a
>> >>> >> >> > level of a distribution (a.k.a. resources). Which DCAT-AP
>> >>> >> >> > standard do
>> >>> >> >> > you
>> >>> >> >> > want to harvest?
>> >>> >> >> >
>> >>> >> >> > It might be needed to create your own profile to define
>> exactly
>> >>> >> >> > this
>> >>> >> >> > specifities. The process to do so is described in the README:
>> >>> >> >> > https://github.com/ckan/ckanext-dcat#profiles
>> >>> >> >> >
>> >>> >> >> > I hope this helps.
>> >>> >> >> >
>> >>> >> >> > Best regards Stefan
>> >>> >> >> >
>> >>> >> >> > On Tue, Feb 14, 2017 at 7:57 PM, Cam Findlay <
>> cam at camfindlay.com>
>> >>> >> >> > wrote:
>> >>> >> >> >>
>> >>> >> >> >> Hi all.
>> >>> >> >> >>
>> >>> >> >> >> Has anyone else had trouble with the DCAT JSON harvester in
>> >>> >> >> >> particular
>> >>> >> >> >> the
>> >>> >> >> >> license property not being set on the harvested dataset?
>> >>> >> >> >>
>> >>> >> >> >> Our JSON source includes at the dataset level:
>> >>> >> >> >>
>> >>> >> >> >> "license": "https://creativecommons.org/licenses/by/4.0/",
>> >>> >> >> >>
>> >>> >> >> >> However the final result is no license gets set for the
>> dataset
>> >>> >> >> >> post
>> >>> >> >> >> harvest. I'd expect "Creative Commons Attribution" to show up
>> >>> >> >> >> but we
>> >>> >> >> >> get
>> >>> >> >> >> nothing. If I edit the dataset and manually set the license
>> then
>> >>> >> >> >> it
>> >>> >> >> >> shows
>> >>> >> >> >> fine. So it seems to be lost on import for some reason.
>> >>> >> >> >>
>> >>> >> >> >> No problems with the CKAN to CKAN harvester, we get the
>> license
>> >>> >> >> >> correctly
>> >>> >> >> >> so just checking if there is any known issues around DCAT
>> JSON
>> >>> >> >> >> and
>> >>> >> >> >> license
>> >>> >> >> >> properties.
>> >>> >> >> >>
>> >>> >> >> >> Trying to determine if it's extension related or a problem at
>> >>> >> >> >> our
>> >>> >> >> >> end.
>> >>> >> >> >>
>> >>> >> >> >> (Running CKAN 2.6.X, Ubuntu 14.04 LTS)
>> >>> >> >> >>
>> >>> >> >> >> Thanks in advance :)
>> >>> >> >> >>
>> >>> >> >> >> Cam.
>> >>> >> >> >>
>> >>> >> >> >> _______________________________________________
>> >>> >> >> >> ckan-dev mailing list
>> >>> >> >> >> ckan-dev at lists.okfn.org
>> >>> >> >> >> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> >>> >> >> >> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>> >>> >> >> >>
>> >>> >> >> >
>> >>> >> >> >
>> >>> >> >> >
>> >>> >> >> > --
>> >>> >> >> > Liip AG  // Limmatstrasse 183 //  CH-8005 Zürich
>> >>> >> >> > Tel +41 43 500 39 80 // GnuPG 0x7B588C67 // www.liip.ch
>> >>> >> >> >
>> >>> >> >> > _______________________________________________
>> >>> >> >> > ckan-dev mailing list
>> >>> >> >> > ckan-dev at lists.okfn.org
>> >>> >> >> > https://lists.okfn.org/mailman/listinfo/ckan-dev
>> >>> >> >> > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>> >>> >> >> >
>> >>> >> >> _______________________________________________
>> >>> >> >> ckan-dev mailing list
>> >>> >> >> ckan-dev at lists.okfn.org
>> >>> >> >> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> >>> >> >> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>> >>> >> >
>> >>> >> >
>> >>> >> >
>> >>> >> >
>> >>> >> > --
>> >>> >> > Liip AG  // Limmatstrasse 183 //  CH-8005 Zürich
>> >>> >> > Tel +41 43 500 39 80 // GnuPG 0x7B588C67 // www.liip.ch
>> >>> >> >
>> >>> >> > _______________________________________________
>> >>> >> > ckan-dev mailing list
>> >>> >> > ckan-dev at lists.okfn.org
>> >>> >> > https://lists.okfn.org/mailman/listinfo/ckan-dev
>> >>> >> > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>> >>> >> >
>> >>> >> _______________________________________________
>> >>> >> ckan-dev mailing list
>> >>> >> ckan-dev at lists.okfn.org
>> >>> >> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> >>> >> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>> >>> >
>> >>> >
>> >>> >
>> >>> > _______________________________________________
>> >>> > ckan-dev mailing list
>> >>> > ckan-dev at lists.okfn.org
>> >>> > https://lists.okfn.org/mailman/listinfo/ckan-dev
>> >>> > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>> >>> >
>> >>> _______________________________________________
>> >>> ckan-dev mailing list
>> >>> ckan-dev at lists.okfn.org
>> >>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> >>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>> >>
>> >>
>> >
>> >
>> > _______________________________________________
>> > ckan-dev mailing list
>> > ckan-dev at lists.okfn.org
>> > https://lists.okfn.org/mailman/listinfo/ckan-dev
>> > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>> >
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20170308/bde87402/attachment-0002.html>