[ckan-dev] Resource text search

Tue May 8 15:32:58 UTC 2012

On 5/8/12 9:54 AM, "Eduardo Santos" <eduardo.edusantos at gmail.com> wrote:

>Hi Salman,
>
>Thank you for your comment. I saw your message and Rufus response. It
>seems like we are doing the same thing in different approaches.
>
>> Since your data is in XML/RDF format and ElasticSearch is a JSON
>>document
>> store, you will have to convert your source data appropriately.
>
>I found a simple RDF->Json script that should fulfill my purpose:
>http://docs.api.talis.com/platform-api/output-types/rdf-json
>
>The only challenge here will be to keep elastic search and json
>identifiers the same.
>
>Talking about my work, what I've been done so far is to create two
>extensions:
>
>1 - Datadaemon extension that goes to RDF source, retrieves the file and
>store it and its hash in database. It's just to keep control of
>downloaded files and very similar to datastore-client

Does the daemon upload the file via the filestore API? If so, I've been
working adding a uploading a dataset resource via the Ckan Python Client.
You might find it useful:

See 'add_package_resource':
https://github.com/slmnhq/ckanclient/commit/d233fcb3f73644b8b505839bedc16d3
52434e946

I'm still working on adding unit tests for these changes but am awaiting
direction from the other developers.

>
>2 - My extension that will have to:
>
>- Parse the downloaded RDF document
>- Insert it as a Ckan resource (PostgreSQL) and elastic search document

For my use case, I'm thinking of creating a new extension based off
ckanext-datastorer. Whereas the latter converts CSV and XLS files into
JSON and puts them in ES, my plugin will convert my custom XML document
into JSON and put it in ES.

>- Extend ResourceSearchQuery class to query elastic search for resource
>contents
>- Create an UI to show the documents in a more user friendly way.

I also have this requirement but haven't made any progress towards it yet.

>
>Perharps we could share work somehow?

Yes, please. Let me know your thoughts how to go about this.

>
>
>-- 
>Eduardo Santos
>Analista de Sistemas
>
>http://www.eduardosan.com
>http://twitter.com/eduardosan
>
>
>_______________________________________________
>ckan-dev mailing list
>ckan-dev at lists.okfn.org
>http://lists.okfn.org/mailman/listinfo/ckan-dev