[ckan-dev] DCAT JSON harvester license property not importing

Cam Findlay cam at camfindlay.com
Sat Feb 18 09:25:41 UTC 2017 

That fixed it. Will look to raise a PR for this fix.

C.



On Sat, 18 Feb 2017 at 3:01 AM, Adrià Mercader <adria.mercader at okfn.org>
wrote:

> The entry point on setup.py for the datajson_harvester plugin points to:
>
> datajson_harvest=ckanext.datajson.plugin:DataJsonHarvester
>
> But that class is not present on plugin.py.
>
> It should be
>
> datajson_harvest=ckanext.datajson.harvester_datajson:DataJsonHarvester
>
> or actually import DataJsonHarvester on plugin.py
>
>
> Hope this helps,
> Adrià
>
> On 17 February 2017 at 08:15, Cam Findlay <cam at camfindlay.com> wrote:
> > Managed to get the module installed - the data.json endpoint works a
> treat,
> > however the harvester is a different story - error log is showing
> > "ImportError: 'module' object has no attribute 'DataJsonHarvester'" and
> > Apache is throwing a 500 response (I'm running via an Apache WSGI).
> >
> > I note the module mentions a quirk requiring an "import ckanext" in
> wsgi.py
> > (unsure where to find this, is this meant to mean the apache.wsgi file in
> > /etc/ckan/default?).
> >
> > Scratching my head a bit but keen to keep working through the problem to
> get
> > this harvester working. Any further insight would be appreciated. I'm
> > running CKAN 2.6.x.
> >
> > Thanks.
> >
> > C.
> >
> >
> >
> >
> > On 17 February 2017 at 08:58, Cam Findlay <cam at camfindlay.com> wrote:
> >>
> >> Nice one, I'll check out that extension Adrià.
> >>
> >> Epic high fives for all the help everyone.
> >>
> >> +1 open source community :D
> >>
> >>
> >>
> >>
> >> On 16 February 2017 at 21:48, Adrià Mercader <adria.mercader at okfn.org>
> >> wrote:
> >>>
> >>> Hi Cam,
> >>>
> >>> GSA maintain an extension to harvest data.json files:
> >>> https://github.com/GSA/ckanext-datajson
> >>>
> >>> It might be worth investigating.
> >>>
> >>> Best,
> >>>
> >>> Adrià
> >>>
> >>> On 15 February 2017 at 20:41, Cam Findlay <cam at camfindlay.com> wrote:
> >>> > Thanks for all the responses :)
> >>> >
> >>> > The json we are looking to harvest conform to the US Project Open
> Data
> >>> > schema https://project-open-data.cio.gov/v1.1/schema/ (this seems to
> >>> > have a
> >>> > license property at the dataset level).
> >>> >
> >>> > We've found that ESRI GIS open data portal have this baked in as a
> >>> > standard
> >>> > endpoint if you simply add data.json to the end of the portal URL.
> Eg.
> >>> >
> http://aucklandopendata-aucklandcouncil.opendata.arcgis.com/data.json
> >>> >
> >>> > Has anyone put together a custom profile for this schema we could
> >>> > reuse?
> >>> >
> >>> >
> >>> > Many thanks,
> >>> >
> >>> > Cam.
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > On 15 February 2017 at 23:29, Adrià Mercader <
> adria.mercader at okfn.org>
> >>> > wrote:
> >>> >>
> >>> >> That's a really interesting approach!
> >>> >>
> >>> >> I guess it's possible because a very clearly limited number of
> options
> >>> >> which are easily ordered. It would be difficult to apply to a more
> >>> >> generic list like the ones listed here
> >>> >> http://opendefinition.org/licenses/
> >>> >>
> >>> >> But well worth a try if you have limited licensing options
> >>> >>
> >>> >> Adrià
> >>> >>
> >>> >> On 15 February 2017 at 10:15, Stefan Oderbolz
> >>> >> <stefan.oderbolz at liip.ch>
> >>> >> wrote:
> >>> >> > For our portal we followed the approach of defining a "hierarchy
> of
> >>> >> > openess"
> >>> >> > of our terms of use (for legal reasons we don't have licenses, but
> >>> >> > terms
> >>> >> > of
> >>> >> > use, but it's basically the same).
> >>> >> >
> >>> >> > So the "terms of use" of a dataset is defined as the "least open
> >>> >> > terms
> >>> >> > of
> >>> >> > use of one it's resources", you can find the code here:
> >>> >> >
> >>> >> >
> >>> >> >
> https://github.com/opendata-swiss/ckanext-switzerland/blob/74fc50090d5e4e27af9b44802bb8d693e37d4894/ckanext/switzerland/logic.py#L52-L88
> >>> >> >
> >>> >> >
> >>> >> >
> >>> >> > On Wed, Feb 15, 2017 at 10:32 AM, Adrià Mercader
> >>> >> > <adria.mercader at okfn.org>
> >>> >> > wrote:
> >>> >> >>
> >>> >> >> Hi,
> >>> >> >>
> >>> >> >> Yes, annoyingly DCAT defines license at the resource level and
> >>> >> >> there
> >>> >> >> isn't one at the dataset level that resources can inherit from
> (but
> >>> >> >> there is one in catalog!)
> >>> >> >>
> >>> >> >> https://github.com/ckan/ckanext-dcat/issues/42
> >>> >> >>
> >>> >> >> Anyway, I think that the use case of grabbing the first license
> >>> >> >> found
> >>> >> >> on a resource and applying it to the dataset is a common enough
> one
> >>> >> >> to
> >>> >> >> implement it on the default profile, and people can override it
> if
> >>> >> >> not
> >>> >> >> happy with it (it will be better than no license at all).
> >>> >> >>
> >>> >> >> I'd be more than happy to review and merge a contribution if
> >>> >> >> someone
> >>> >> >> is up for it.
> >>> >> >>
> >>> >> >> Adrià
> >>> >> >>
> >>> >> >>
> >>> >> >> On 15 February 2017 at 08:16, Stefan Oderbolz
> >>> >> >> <stefan.oderbolz at liip.ch>
> >>> >> >> wrote:
> >>> >> >> > Hi Cam,
> >>> >> >> >
> >>> >> >> > I guess you are not using a custom DCAT profile, because then
> the
> >>> >> >> > default
> >>> >> >> > profile is being used which is "euro_dcat_ap", which is defined
> >>> >> >> > here:
> >>> >> >> >
> >>> >> >> >
> >>> >> >> >
> >>> >> >> >
> https://github.com/ckan/ckanext-dcat/blob/e1245cdb5e29194787c33430db637be74e5bc2e1/ckanext/dcat/profiles.py#L577
> >>> >> >> > This profile does not define licenses on a dataset level, but
> >>> >> >> > only in
> >>> >> >> > a
> >>> >> >> > level of a distribution (a.k.a. resources). Which DCAT-AP
> >>> >> >> > standard do
> >>> >> >> > you
> >>> >> >> > want to harvest?
> >>> >> >> >
> >>> >> >> > It might be needed to create your own profile to define exactly
> >>> >> >> > this
> >>> >> >> > specifities. The process to do so is described in the README:
> >>> >> >> > https://github.com/ckan/ckanext-dcat#profiles
> >>> >> >> >
> >>> >> >> > I hope this helps.
> >>> >> >> >
> >>> >> >> > Best regards Stefan
> >>> >> >> >
> >>> >> >> > On Tue, Feb 14, 2017 at 7:57 PM, Cam Findlay <
> cam at camfindlay.com>
> >>> >> >> > wrote:
> >>> >> >> >>
> >>> >> >> >> Hi all.
> >>> >> >> >>
> >>> >> >> >> Has anyone else had trouble with the DCAT JSON harvester in
> >>> >> >> >> particular
> >>> >> >> >> the
> >>> >> >> >> license property not being set on the harvested dataset?
> >>> >> >> >>
> >>> >> >> >> Our JSON source includes at the dataset level:
> >>> >> >> >>
> >>> >> >> >> "license": "https://creativecommons.org/licenses/by/4.0/",
> >>> >> >> >>
> >>> >> >> >> However the final result is no license gets set for the
> dataset
> >>> >> >> >> post
> >>> >> >> >> harvest. I'd expect "Creative Commons Attribution" to show up
> >>> >> >> >> but we
> >>> >> >> >> get
> >>> >> >> >> nothing. If I edit the dataset and manually set the license
> then
> >>> >> >> >> it
> >>> >> >> >> shows
> >>> >> >> >> fine. So it seems to be lost on import for some reason.
> >>> >> >> >>
> >>> >> >> >> No problems with the CKAN to CKAN harvester, we get the
> license
> >>> >> >> >> correctly
> >>> >> >> >> so just checking if there is any known issues around DCAT JSON
> >>> >> >> >> and
> >>> >> >> >> license
> >>> >> >> >> properties.
> >>> >> >> >>
> >>> >> >> >> Trying to determine if it's extension related or a problem at
> >>> >> >> >> our
> >>> >> >> >> end.
> >>> >> >> >>
> >>> >> >> >> (Running CKAN 2.6.X, Ubuntu 14.04 LTS)
> >>> >> >> >>
> >>> >> >> >> Thanks in advance :)
> >>> >> >> >>
> >>> >> >> >> Cam.
> >>> >> >> >>
> >>> >> >> >> _______________________________________________
> >>> >> >> >> ckan-dev mailing list
> >>> >> >> >> ckan-dev at lists.okfn.org
> >>> >> >> >> https://lists.okfn.org/mailman/listinfo/ckan-dev
> >>> >> >> >> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
> >>> >> >> >>
> >>> >> >> >
> >>> >> >> >
> >>> >> >> >
> >>> >> >> > --
> >>> >> >> > Liip AG  // Limmatstrasse 183 //  CH-8005 Zürich
> >>> >> >> > Tel +41 43 500 39 80 // GnuPG 0x7B588C67 // www.liip.ch
> >>> >> >> >
> >>> >> >> > _______________________________________________
> >>> >> >> > ckan-dev mailing list
> >>> >> >> > ckan-dev at lists.okfn.org
> >>> >> >> > https://lists.okfn.org/mailman/listinfo/ckan-dev
> >>> >> >> > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
> >>> >> >> >
> >>> >> >> _______________________________________________
> >>> >> >> ckan-dev mailing list
> >>> >> >> ckan-dev at lists.okfn.org
> >>> >> >> https://lists.okfn.org/mailman/listinfo/ckan-dev
> >>> >> >> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
> >>> >> >
> >>> >> >
> >>> >> >
> >>> >> >
> >>> >> > --
> >>> >> > Liip AG  // Limmatstrasse 183 //  CH-8005 Zürich
> >>> >> > Tel +41 43 500 39 80 // GnuPG 0x7B588C67 // www.liip.ch
> >>> >> >
> >>> >> > _______________________________________________
> >>> >> > ckan-dev mailing list
> >>> >> > ckan-dev at lists.okfn.org
> >>> >> > https://lists.okfn.org/mailman/listinfo/ckan-dev
> >>> >> > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
> >>> >> >
> >>> >> _______________________________________________
> >>> >> ckan-dev mailing list
> >>> >> ckan-dev at lists.okfn.org
> >>> >> https://lists.okfn.org/mailman/listinfo/ckan-dev
> >>> >> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
> >>> >
> >>> >
> >>> >
> >>> > _______________________________________________
> >>> > ckan-dev mailing list
> >>> > ckan-dev at lists.okfn.org
> >>> > https://lists.okfn.org/mailman/listinfo/ckan-dev
> >>> > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
> >>> >
> >>> _______________________________________________
> >>> ckan-dev mailing list
> >>> ckan-dev at lists.okfn.org
> >>> https://lists.okfn.org/mailman/listinfo/ckan-dev
> >>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
> >>
> >>
> >
> >
> > _______________________________________________
> > ckan-dev mailing list
> > ckan-dev at lists.okfn.org
> > https://lists.okfn.org/mailman/listinfo/ckan-dev
> > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
> >
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20170218/774c9544/attachment-0003.html>

More information about the ckan-dev mailing list