[ckan-discuss] Questions for "RDF/XML" of CKAN page

[David Read]

Pablo,

I see - part of the superb LOD cloud effort. I'll feed this use case
into our discussions about multiple CKAN instances and federation vs
DVCS.

In the meantime, I agree that it seems a reasonable plan to do a
simple copy of the 160 RDF packages from data.gov.uk into
thedatahub.org. A very simple script with calling ckanclient would do
the trick. I can give you a hand if you need it.

David

On 15 July 2011 16:47, Pablo Mendes <pablomendes at gmail.com> wrote:
>
> David,
> Thanks for the prompt answer.
> We are in the process of cataloging all datasets following the Linked Data
> principles (linkeddata.org). Besides what is already in data.gov.uk, we also
> collect some provenance, accessibility, interpretability, availability,
> among other quality indicators.
> http://www4.wiwiss.fu-berlin.de/lodcloud/ckan/validator/levels.html
> We could have pulled them into our own in-house database, but we thought it
> would be a much useful effort if we contributed our metadata to CKAN (which
> also allows dataset owners to add some more information they might have, and
> potential users to rate the content somehow).
> We currently have some crowdsourcing going on:
> http://lists.w3.org/Archives/Public/public-lod/2011Jul/0059.html
> And in parallel I am trying to ingest other catalogs to have the info all in
> one place.
> As a result of this effort, we will release a new version of the LOD cloud
> diagram:
> http://www4.wiwiss.fu-berlin.de/lodcloud/state/
> We could discuss better ways to do these things (e.g. we could keep the
> quality metadata as voID files in our own servers, and the federated CKAN
> search attaches them to the search results on-the-fly). However, we need a
> short-term solution.
> We'd like to get this nice overview of how open data is interconnected on
> the Web by next week. And, to the extend of my knowledge, pulling the
> metadata for these catalogs into CKAN would be the best (temporary)
> solution.
> What do you think?
> Cheers,
> Pablo
> On Fri, Jul 15, 2011 at 5:23 PM, David Read <david.read at okfn.org> wrote:
>>
>> On 15 July 2011 15:39, Pablo Mendes <pablomendes at gmail.com> wrote:
>> > David,
>> > very helpful message for all of us! I have a similar need. I want to
>> > pull
>> > all of the metadata in data.gov.uk about datasets that contain RDF into
>> > CKAN.net. There are about 160 of them.
>> >
>> > http://data.gov.uk/search/apachesolr_search?filters=sm_resource_formats%3ARDF
>> > Would the same advice apply?
>>
>> The data.gov.uk datasets are in CKAN format already. e.g.
>>
>> http://catalogue.data.gov.uk/api/rest/package/ni-115-substance-misuse-by-young-people
>> so it would be relatively simple to pull data.gov.uk datasets into
>> another CKAN, such as thedatahub.org. We've been considering this, but
>> not sure if the 7000 of those would flood thedatahub.org somewhat. And
>> since you want to stay in step, how do you deal with edits at both
>> ends?
>>
>> Our current thinking is to aim at:
>> a) allowing the public to edit/correction metadata on data.gov.uk
>> (with moderation)
>> b) use federated search across all CKAN instances
>>
>> I'd be interested to hear why you suggested pulling the RDF datasets
>> from data.gov.uk into thedatahub.org/ckan.net - do let us know your
>> thoughts.
>>
>> David
>>
>> > Cheers,
>> > Pablo
>> > On Fri, Jul 15, 2011 at 4:16 PM, David Read <david.read at okfn.org> wrote:
>> >>
>> >> On 15 July 2011 12:18, Tetsuro TOYODA <toyoda at base.riken.jp> wrote:
>> >> > Dear David,
>> >> >
>> >> > Thank you for giving us your quick reply.
>> >> >
>> >> > http://biolod.org is the download site of biological databases funded
>> >> > by
>> >> > the National Database Integration Project of Japan.
>> >> >
>> >> > RDF files are already generated and supplied from the site.
>> >> >
>> >> > Could you advise us how to register the data files including RDFs
>> >> > into
>> >> > CKAN.
>> >>
>> >> Tetsuro
>> >> Basically we need to write a little code that transforms the biolod
>> >> RDF metadata to the CKAN JSON format.
>> >>
>> >> To do a one-off import then you should use the CKAN API [1]. You could
>> >> do it in Python and use the ckanclient library [2].
>> >>
>> >> Or to do a daily automatic import you would write a harvester. Derive
>> >> from the base class [3]
>> >> and write a custom "_create_or_update_package" method.
>> >>
>> >> This way has the advantage that the original RDF is stored alongside
>> >> the CKAN version. We'd be happy to install this harvester in
>> >> thedatahub.org or another CKAN instance.
>> >>
>> >> David
>> >>
>> >> [1] http://packages.python.org/ckan/api.html
>> >> [2] https://bitbucket.org/okfn/ckanclient
>> >> [3]
>> >>
>> >> https://bitbucket.org/okfn/ckanext-harvest/src/61844c8d2374/ckanext/harvest/harvesters/base.py
>> >>
>> >> >
>> >> > Best regards,
>> >> >
>> >> > Tetsuro Toyoda.
>> >> >
>> >> > -----Original Message-----
>> >> > From: d.t.read at gmail.com [mailto:d.t.read at gmail.com] On Behalf Of
>> >> > David
>> >> > Read
>> >> > Sent: Friday, July 15, 2011 7:52 PM
>> >> > To: Koro NISHIKATA
>> >> > Cc: ckan-discuss at lists.okfn.org
>> >> > Subject: Re: [ckan-discuss] Questions for "RDF/XML" of CKAN page
>> >> >
>> >> > 2011/7/15 Koro NISHIKATA <koro at base.riken.jp>:
>> >> >> Dear all of CKAN discuss mailing list,
>> >> >>
>> >> >> Hello, nice to meet you.
>> >> >> I am a new member of this mailing list, "Koro NISHIKATA".
>> >> >
>> >> > Welcome Koro!
>> >> >
>> >> >> I have a question about "the Data Hub API / datapkg" section of CKAN
>> >> >> page.
>> >> >>　 http://ckan.net/package/biolod-pdb
>> >> >>
>> >> >> Can a user register data to "RDF/XML" ?
>> >> >> Or automatically produced after register or editing ?
>> >> >
>> >> > CKAN accepts data via a number of ways:
>> >> > * the Web page form
>> >> > * API (JSON format)
>> >> > * harvesters (INSPIRE, custom formats, etc)
>> >> >
>> >> > So you are correct - we don't accept RDF/XML. This format is one of
>> >> > the exported formats, derived from the raw data.
>> >> >
>> >> > David
>> >> >
>> >> >>
>> >> >> [RDF/XML]
>> >>
>> >> >>
>> >> >> >> >>　 http://semantic.ckan.net/record//be684dad-b24a-4f59-b8e2-969229200c63.rdf
>> >> >>
>> >> >> Best regard,
>> >> >> Koro NISHIKATA
>> >> >> E-mail: koro at base.riken.jp
>> >> >>
>> >
>> >
>
>