[ckan-dev] Getting resources from packages via the ckanapi

Fri Feb 24 20:24:28 UTC 2017

Hmm. Try rebuilding your search index.

paster search-index rebuild

On Feb 24, 2017 2:52 PM, "Koebrick, Andrew (MNIT)" <
andrew.koebrick at state.mn.us> wrote:

Same output after the simplejson downgrade.  Here are my versions:

ckanapi==4.0

docopt==0.6.2

pprint==0.1

requests==2.13.0

simplejson==3.3.1

wheel==0.24.0

*From:* ckan-dev [mailto:ckan-dev-bounces at lists.okfn.org] *On Behalf Of *Ian
Ward
*Sent:* Friday, February 24, 2017 12:56 PM

*To:* CKAN Development Discussions <ckan-dev at lists.okfn.org>
*Subject:* Re: [ckan-dev] Getting resources from packages via the ckanapi

Try using the version specified in the requirements.txt file:

pip install simplejson==3.3.1

On Fri, Feb 24, 2017 at 1:49 PM, Koebrick, Andrew (MNIT) <
andrew.koebrick at state.mn.us> wrote:

Ian,

I was at 3.8 but upgraded 3.10.  No change.

I also tried another tack- in my testing I had been using a break so that
it only attempted to get the packages for one organization (to make
debugging quicker).  But when I took that out and ran the script over all
the organizations  (94) and attempted to get the resources for all
datasets/packages (993) a total of 8 actually came back as expected- as
full dictionaries with the resources.

I looked over the 8 packages that were successfully returned as dicts with
resources and see nothing setting them apart from those that return as -1.

As an experiment I will rewrite to do straight http requests and try taking
ckanapi out of the mix.  In the meanwhile, here is the current state of my
script (with try/excepts pulled out for brevity).  If anyone sees a python
or ckan problems I would be grateful for assistance:

*from *pprint *import **
*from *ckanapi *import **

key=
*"xxxxxxxxxxxx" *ua =
*'harvestScript/1.0 (+https://gisdata.mn.gov <https://gisdata.mn.gov>)' *ckan
= RemoteCKAN(*'https://ng911.gisdata.mn.gov/
<https://ng911.gisdata.mn.gov/>'*, user_agent=ua, apikey=key)
packageCount = 0
dictCount = 0
orgCount = 0
organizations = ckan.action.organization_list(include_datasets =*'True'*)
*for *organization *in *organizations:
       orgCount += 1
       orgDetails = ckan.action.organization_show(id=organization,
include_datasets=*'True'*)
       packages= orgDetails[*'packages'*]
       *for *package *in *packages:
           packageCount += 1
           packageObj = ckan.action.package_show(id=package[*'id'*])
           *if *isinstance(packageObj,dict):
               pprint(packageObj)
               dictCount += 1
*print **"Total packages: " *+ str(packageCount)
*print **"Total dictionaries: " *+ str(dictCount)
*print **"Total orgs: " *+ str(orgCount)

For counts, this is what gets kicked out:

Total packages: 993

Total dictionaries: 8

Total orgs: 94

Thanks,

Andrew

*From:* ckan-dev [mailto:ckan-dev-bounces at lists.okfn.org] *On Behalf Of *Ian
Ward
*Sent:* Thursday, February 23, 2017 3:53 PM

*To:* CKAN Development Discussions <ckan-dev at lists.okfn.org>
*Subject:* Re: [ckan-dev] Getting resources from packages via the ckanapi

What version of simplejson is installed in the virtualenv?

On Feb 23, 2017 4:01 PM, "Koebrick, Andrew (MNIT)" <
andrew.koebrick at state.mn.us> wrote:

Ian,

thanks for the assistance, unfortunately I get:

    pprint dict(packageObj)

              ^

SyntaxError: invalid syntax

In terms of versions I am attempting to pull from ckan 2.5.2 which does not
seem that old; and I am using RemoteCKAN; I just did not bother adding the
initialization to the code I sent.  Here is the excluded part of my script:

key=
*"REDACTED" *ua = *'harvestScript/1.0 (+**https://gisdata.mn.gov*
<https://gisdata.mn.gov>
*)' *ckan = RemoteCKAN(*'**https://ng911.gisdata.mn.gov/*
<https://ng911.gisdata.mn.gov/>*'*, user_agent=ua, apikey=key)

Unfortunately this server is firewalled off from public access, but if you
want to test ckanapi against it, I can add your IP address to the whitelist
and send you a key.

Andrew

*From:* ckan-dev [mailto:ckan-dev-bounces at lists.okfn.org] *On Behalf Of *Ian
Ward
*Sent:* Thursday, February 23, 2017 2:29 PM
*To:* CKAN Development Discussions <ckan-dev at lists.okfn.org>
*Subject:* Re: [ckan-dev] Getting resources from packages via the ckanapi

On Thu, Feb 23, 2017 at 3:14 PM, Koebrick, Andrew (MNIT) <
andrew.koebrick at state.mn.us> wrote:

Tim,

I had tried that approach earlier with this syntax which seems to match
what works for the organization_show:

packageObj = ckan.action.package_show(id=package[*'id'*])

But when I attempt to see what I retrieved like so:

pprint (packageObj)

All that I get is

-1

Oh.

Try pprint dict(packageObj)

That -1 repr is due to an old performance hack (when using the actions
internally you're getting a LazyJSON object instead of a normal dict) We've
since reverted that hack. You wouldn't see it if you were using RemoteCKAN
or a newer version of CKAN.

Ian

I thought perhaps it was a problem with my install of ckanapi so I upgraded
from 3.6 to 4.0 but that did not fix the issue.

The syntax you suggested seems to have a problem but I also tried this
variation:

packageObj = ckan.action.package_show(‘*id:' +* package[*'id'*]) #note
addition of plus sign…

which throws:

TypeError: action() takes exactly 0 arguments (1 given)

Still plunking away…

Andrew

*From:* ckan-dev [mailto:ckan-dev-bounces at lists.okfn.org] *On Behalf
Of *Timothy
Giles
*Sent:* Thursday, February 23, 2017 1:21 PM
*To:* CKAN Development Discussions <ckan-dev at lists.okfn.org>
*Subject:* Re: [ckan-dev] Getting resources from packages via the ckanapi

Hi Andrew,

Once you have the package id, you can :

                 packageObj = ckan.action.package_show(‘*id:'* package[*'id'*])

This will return you the package object with all the resources
included. You can loop through the resources if you want but I believe
all the resource metadata is provided.

Regards Tim

*From:* ckan-dev [mailto:ckan-dev-bounces at lists.okfn.org
<ckan-dev-bounces at lists.okfn.org>] *On Behalf Of *Koebrick, Andrew (MNIT)
*Sent:* den 23 februari 2017 19:44
*To:* ckan-dev at lists.okfn.org
*Subject:* [ckan-dev] Getting resources from packages via the ckanapi

Hoping somebody can assist:  I am attempting to create a report for one of
my ckan instances that lists all the organizations, and then all of their
packages along with the id of the packages’ resource (we have only one
resource per package).  However, I do not seem to be able to link to the
resource id via API calls.

Here is what I have so far:

organizations = ckan.action.organization_list(include_datasets =*'True'*)
*for *organization *in *organizations:

         packages= orgDetails[*'packages'*]

*         for *package *in *packages:

                 #And here is where it breaks down…. Here are a few of
the many searches / shows I have tried:

                 resourceObj =
ckan.action.resource_search(query=*'package_id:'* +package[*'id'*])

                    #which returns: {u'query': u'Field "package_id"
not recognised in resource_search.', u'__type': u'Validation Error'}

                 revisionObj =
ckan.action.revision_show(id=package[*'revision_id'*])

                    #which just brings back information already knows
about the revision.

While iterating through the package information I seem unable to get
the related resources.  I have tried all sorts of queries using what
info I have from the package to get the related resources.

What can I use to link from the package list found in the return
object from organization_show to get resources?

Many thanks,

*Andrew Koebrick  | * MINNESOTA GEOSPATIAL INFORMATION OFFICE

Web Coordinator / Systems administrator / Librarian

MN.IT Services @ MNGEO

651-201-2465 <(651)%20201-2465> (w)  *|  * 651-296-6398 <(651)%20296-6398>
(f) *|*  andrew.koebrick at state.mn.us

658 Cedar St., Room 300, St. Paul, MN 55155, www.mngeo.state.mn.us

*[image: cid:image002.jpg at 01CE61F8.52552AE0]* <http://www.mn.gov/oet>

Information Technology for Minnesota Government   *|*   mn.gov/mnit
<http://www.mn.gov/oet>

_______________________________________________
ckan-dev mailing list
ckan-dev at lists.okfn.org
https://lists.okfn.org/mailman/listinfo/ckan-dev
Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev

_______________________________________________
ckan-dev mailing list
ckan-dev at lists.okfn.org
https://lists.okfn.org/mailman/listinfo/ckan-dev
Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev

_______________________________________________
ckan-dev mailing list
ckan-dev at lists.okfn.org
https://lists.okfn.org/mailman/listinfo/ckan-dev
Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev

_______________________________________________
ckan-dev mailing list
ckan-dev at lists.okfn.org
https://lists.okfn.org/mailman/listinfo/ckan-dev
Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20170224/fc065158/attachment-0003.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 1624 bytes
Desc: not available
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20170224/fc065158/attachment-0003.jpg>