[ckan-dev] Create Package programmatically

Mon Mar 11 17:59:56 UTC 2013

Good to hear that the spatial development is coming along. I'm trying this out right now -- just a couple of comments / concerns:

My first harvest attempt gathered IDs, but failed validation as soon as fetching started: lxml had trouble parsing the schemas. What versions of lxml - libxml2 - libxslt do I need to have installed? libxml2 2.9.0 I assume? Setting that up on OSX is difficult.

I wonder how the pycsw integration effort is going? I worry a bit about the "lightweight" csw service that is built at https://github.com/okfn/ckanext-spatial/blob/release-v2.0/ckanext/spatial/controllers/csw.py. I built pycsw integration into my own custom extension last week before I knew that ckanext-spatial had developed as far as it has. Is there any way that I can help bring pycsw into the spatial extension more quickly?

Thanks!
____________________

Ryan Clark
ryan.clark at azgs.az.gov
(520) 302-4871

On Mar 11, 2013, at 6:41 AM, Adrià Mercader <adria.mercader at okfn.org> wrote:

> Hi,
> 
> Just to clarify that the CSW harvester on ckanext-spatial does exactly
> what you need (ie, gets all iso19139 documents from a remote CSW
> server, parses them and creates CKAN packages with the metadata), plus
> you get all nice things from the harvesting framework like UI, job
> reports, validation errors, etc.
> Both "release-v2.0" branches of ckanext-spatial and ckanext-harvest,
> although not finished for release, should be in a good state for
> testing this out.
> 
> If for some reason you don't want or can't use the harvester per se
> you can have a look at the source to see how the documents are parsed
> and packages created
> 
> https://github.com/okfn/ckanext-spatial/blob/release-v2.0/ckanext/spatial/harvesters/csw.py
> https://github.com/okfn/ckanext-spatial/blob/release-v2.0/ckanext/spatial/harvesters/base.py
> 
> These lines on base.py are probably the ones that are more relevant to you:
> 
> iso_values = ISODocument(harvest_object.content).read_values()
> [...]
> package_dict = self.get_package_dict(iso_values, harvest_object)
> [...]
> package_id = p.toolkit.get_action('package_create')(context, package_dict)
> 
> Hope this helps
> 
> On 8 March 2013 18:49, Ryan Clark <ryan.clark at azgs.az.gov> wrote:
>> This is starting to make sense to me. If I understood Toby correctly, I
>> should
>> 
>> ckan.logic.get_action("package_create")(None, { "package": "data" })
>> 
>> I realize that it'll make up the missing context variable for me, but I'll
>> deal with that later. Now the question is, can I use this approach to attach
>> whatever "extras" I want to the package? For my project its really important
>> that I be able to attach a specific set of "extras".
>> 
>> Thanks!
>> ____________________
>> 
>> Ryan Clark
>> ryan.clark at azgs.az.gov
>> (520) 302-4871
>> 
>> 
>> 
>> 
>> 
>> 
>> On Mar 8, 2013, at 7:28 AM, Ryan Clark <ryan.clark at azgs.az.gov> wrote:
>> 
>> I'm building a CSW-based harvesting mechanism. I know there are existing
>> harvesting tools, but they don't quite get to what I need, and my project's
>> deadline means I can't wait...
>> 
>> The harvesting itself will need to happen at fixed intervals, without user
>> input. When an XML record is received, it will be parsed and a package will
>> be created. After setting up a job, no user input should be required. This
>> is why I say I won't have a "context" when I get to the point of wanting to
>> generate a package.
>> 
>> Thanks!
>> Ryan
>> 
>> 
>> 
>> On Mar 8, 2013, at 6:52 AM, Nigel Babu <nigel.babu at okfn.org> wrote:
>> 
>> Hi Ryan,
>> 
>> `ckan.logic.action.package_create` is probably the right way to do it. Can
>> you explain more about there not being an HTTP call associated with your
>> code? You could use the action API via the paster post command, but you'd
>> need to be on the server for that.  Or from Python via the plugins
>> toolkit[1] (again, you have to be on the server).
>> 
>> [1] http://docs.ckan.org/en/latest/toolkit.html
>> 
>> Nigel
>> 
>> 
>> On 7 March 2013 04:21, Ryan Clark <ryan.clark at azgs.az.gov> wrote:
>>> 
>>> I'm trying to generate Packages from ISO 19139 documents. What is the best
>>> way to create a new package programmatically? I was thinking that
>>> ckan.logic.action.package_create looked like it might be the best place, but
>>> I'd rather not have to spoof the context object, since there won't be an
>>> HTTP request associated with the code I'm writing.
>>> 
>>> Thanks!!
>>> ____________________
>>> 
>>> Ryan Clark
>>> ryan.clark at azgs.az.gov
>>> (520) 302-4871
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> ckan-dev mailing list
>>> ckan-dev at lists.okfn.org
>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>> 
>> 
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>> 
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>> 
>> 
>> 
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>> 
> 
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20130311/56115399/attachment-0001.html>