[ckan-dev] Create Package programmatically

Tue Mar 12 18:03:12 UTC 2013

I tried the release-v2.0 on ckanext-harvest and ckanext-spatial yesterday. There is a little difference in the aims, I think, between what I want to do and what OKFN goals are. To be clear, I want to:
Create packages using the web-interface. 
Harvest records from other CSW servers - ingest ISO XML to build packages
All packages (harvested and created) can be edited
All packages are exposed via CSW as ISO XML -- built through something like a Package.to_iso_xml() kind of method.

I think where my situation diverges is that all packages can be edited, and even those that were not harvested are exposed via CSW. It looks to me like harvested packages cannot be edited, and it still looks like the CSW only exposes harvested records and not manually generated packages.

The harvesting seems really solid, and if I'm not mistaken it looks like there's configurability in what get mined out of XML and stored as package.extras. This is awesome. What I'd still love to see though is:
pycsw used for the CSW server
ALL packages exposed through the CSW

I've written code to accomplish these two things, and would love to see it worked into the OKFN extensions, if the functionality seems relevant.

p.s. Peder -- this stuff worked pretty much out-of-the-box for me. My setup was something like:
install ckan from the 2.0 branch -- pip install -e git+git://github.com/okfn/ckan.git@release-v2.0#egg=ckan
install ckanext-harvest from 2.0 branch -- pip install -e git+github.com/okfn/ckanext-harvest.git at release-v2.0#egg=ckanext-harvest
install ckanext-spatial from 2.0 branch -- pip install -e git+github.com/okfn/ckanext-spatial.get at release-v2.0#egg=ckanext-spatial
Install dependencies: pip install -r src/ckanext-harvest/pip-requirements.txt
More dependencies: pip install -r src/ckanext-spatial/pip-requirements.txt
Install what seem to be missing dependencies: pip install pyparsing==1.5.7
Make sure you have limbxml2 v2.9+ installed, and then pip install lxml==3.0.1
Put these plugins into your development.ini: harvest spatial_metadata spatial_query cswserver spatial_harvest_metadata_api csw_harvester
Remove datastorer extension if you have it installed as it seems to conflict with the harvesters (https://github.com/okfn/ckanext-harvest/issues/11)
Install harvester tables: paster --plugin=ckanext-harvest harvester initdb --config=development.ini
Install spatial tables: paster --plugin=ckanext-spatial spatial initdb --config=development.ini
Install RabbitMQ (brew install rabbitmq on Mac -- sudo apt-get install rabbitmq, I might guess?)
Run Solr, RabbitMQ, CKAN, and two harvester processes
paster --plugin=ckanext-harvest harvester gather_consumer --config=development.ini
paster --plugin=ckanext-harvest harvester fetch_consumer --config=development.ini
Go to http://localhost:5000/harvest and try out the harvesting. Mine is right now processing 13,000 records and going strong!
____________________

Ryan Clark
ryan.clark at azgs.az.gov
(520) 302-4871

On Mar 11, 2013, at 12:18 PM, Peder Jakobsen <pjakobsen at gmail.com> wrote:

> 
> On 2013-03-11, at 9:41 AM, Adrià Mercader <adria.mercader at okfn.org> wrote:
> 
>> 
>> Just to clarify that the CSW harvester on ckanext-spatial does exactly
>> what you need (ie, gets all iso19139 documents from a remote CSW
>> server, parses them and creates CKAN packages with the metadata), plus
>> you get all nice things from the harvesting framework like UI, job
>> reports, validation errors, etc.
>> Both "release-v2.0" branches of ckanext-spatial and ckanext-harvest,
>> although not finished for release, should be in a good state for
>> testing this out.
> 
> Any examples of what a configuration object might look like for harvesting Spatial ISO 19139, or any other settings required to make this work?
> 
> Thanks,
> 
> Peder 
> 
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20130312/0be7c25c/attachment-0001.html>