[ckan-dev] Getting resources from packages via the ckanapi
Koebrick, Andrew (MNIT)
andrew.koebrick at state.mn.us
Tue Feb 28 17:59:12 UTC 2017
Ian, thanks for your continued assistance. I rebuilt the search index and also rewrote my script to do direct calls against CKAN, without use of your ckanapi library. I get the exact same results. I tried to dump the results of my package_show call, and for the all the bad requests it is coming back from CKAN as:
'{"help": "https://ng911.gisdata.mn.gov/api/3/action/help_show?name=package_show", "success": true, "result": -1}'
For what it is worth, here is my non-ckanapi script:
import json
import urllib2
import urllib
from pprint import pprint
key= "xxxxxxxxxxx"
packageCount = 0
dictCount = 0
orgCount = 0
response = urllib2.urlopen('https://ng911.gisdata.mn.gov/api/action/organization_list')
assert response.code == 200
response = json.loads(response.read())
orgDict = response['result']
for organization in orgDict:
print organization
orgCount += 1
data = {'id': organization, 'include_datasets': True}
data_string = urllib.quote(json.dumps(data))
request = urllib2.Request('https://ng911.gisdata.mn.gov/api/action/organization_show')
request.add_header('Authorization', key)
response = urllib2.urlopen(request, data_string)
assert response.code == 200
response_dict = json.loads(response.read())
assert response_dict['success'] is True
org_dict= response_dict['result']
#pprint(org_dict)
packages= org_dict['packages']
for package in packages:
packageCount += 1
data = {'id': package['name']}
pprint (data)
data_string = urllib.quote(json.dumps(data))
request = urllib2.Request('https://ng911.gisdata.mn.gov/api/action/package_show')
request.add_header('Authorization', key)
response = urllib2.urlopen(request, data_string)
assert response.code == 200
test = response.read()
pprint (test)
response_dict = json.loads(response.read())
assert response_dict['success'] is True
package_dict= response_dict['result']
if isinstance(package_dict,dict):
pprint(package_dict)
dictCount += 1
print "\n\nTotal packages: " + str(packageCount)
print "\n\nTotal dictionaries: " + str(dictCount)
print "\n\nTotal orgs: " + str(orgCount)
From: ckan-dev [mailto:ckan-dev-bounces at lists.okfn.org] On Behalf Of Ian Ward
Sent: Friday, February 24, 2017 2:24 PM
To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
Subject: Re: [ckan-dev] Getting resources from packages via the ckanapi
Hmm. Try rebuilding your search index.
paster search-index rebuild
On Feb 24, 2017 2:52 PM, "Koebrick, Andrew (MNIT)" <andrew.koebrick at state.mn.us<mailto:andrew.koebrick at state.mn.us>> wrote:
Same output after the simplejson downgrade. Here are my versions:
ckanapi==4.0
docopt==0.6.2
pprint==0.1
requests==2.13.0
simplejson==3.3.1
wheel==0.24.0
From: ckan-dev [mailto:ckan-dev-bounces at lists.okfn.org<mailto:ckan-dev-bounces at lists.okfn.org>] On Behalf Of Ian Ward
Sent: Friday, February 24, 2017 12:56 PM
To: CKAN Development Discussions <ckan-dev at lists.okfn.org<mailto:ckan-dev at lists.okfn.org>>
Subject: Re: [ckan-dev] Getting resources from packages via the ckanapi
Try using the version specified in the requirements.txt file:
pip install simplejson==3.3.1
On Fri, Feb 24, 2017 at 1:49 PM, Koebrick, Andrew (MNIT) <andrew.koebrick at state.mn.us<mailto:andrew.koebrick at state.mn.us>> wrote:
Ian,
I was at 3.8 but upgraded 3.10. No change.
I also tried another tack- in my testing I had been using a break so that it only attempted to get the packages for one organization (to make debugging quicker). But when I took that out and ran the script over all the organizations (94) and attempted to get the resources for all datasets/packages (993) a total of 8 actually came back as expected- as full dictionaries with the resources.
I looked over the 8 packages that were successfully returned as dicts with resources and see nothing setting them apart from those that return as -1.
As an experiment I will rewrite to do straight http requests and try taking ckanapi out of the mix. In the meanwhile, here is the current state of my script (with try/excepts pulled out for brevity). If anyone sees a python or ckan problems I would be grateful for assistance:
from pprint import *
from ckanapi import *
key= "xxxxxxxxxxxx"
ua = 'harvestScript/1.0 (+https://gisdata.mn.gov)'
ckan = RemoteCKAN('https://ng911.gisdata.mn.gov/', user_agent=ua, apikey=key)
packageCount = 0
dictCount = 0
orgCount = 0
organizations = ckan.action.organization_list(include_datasets ='True')
for organization in organizations:
orgCount += 1
orgDetails = ckan.action.organization_show(id=organization, include_datasets='True')
packages= orgDetails['packages']
for package in packages:
packageCount += 1
packageObj = ckan.action.package_show(id=package['id'])
if isinstance(packageObj,dict):
pprint(packageObj)
dictCount += 1
print "Total packages: " + str(packageCount)
print "Total dictionaries: " + str(dictCount)
print "Total orgs: " + str(orgCount)
For counts, this is what gets kicked out:
Total packages: 993
Total dictionaries: 8
Total orgs: 94
Thanks,
Andrew
From: ckan-dev [mailto:ckan-dev-bounces at lists.okfn.org<mailto:ckan-dev-bounces at lists.okfn.org>] On Behalf Of Ian Ward
Sent: Thursday, February 23, 2017 3:53 PM
To: CKAN Development Discussions <ckan-dev at lists.okfn.org<mailto:ckan-dev at lists.okfn.org>>
Subject: Re: [ckan-dev] Getting resources from packages via the ckanapi
What version of simplejson is installed in the virtualenv?
On Feb 23, 2017 4:01 PM, "Koebrick, Andrew (MNIT)" <andrew.koebrick at state.mn.us<mailto:andrew.koebrick at state.mn.us>> wrote:
Ian,
thanks for the assistance, unfortunately I get:
pprint dict(packageObj)
^
SyntaxError: invalid syntax
In terms of versions I am attempting to pull from ckan 2.5.2 which does not seem that old; and I am using RemoteCKAN; I just did not bother adding the initialization to the code I sent. Here is the excluded part of my script:
key= "REDACTED"
ua = 'harvestScript/1.0 (+https://gisdata.mn.gov)'
ckan = RemoteCKAN('https://ng911.gisdata.mn.gov/', user_agent=ua, apikey=key)
Unfortunately this server is firewalled off from public access, but if you want to test ckanapi against it, I can add your IP address to the whitelist and send you a key.
Andrew
From: ckan-dev [mailto:ckan-dev-bounces at lists.okfn.org<mailto:ckan-dev-bounces at lists.okfn.org>] On Behalf Of Ian Ward
Sent: Thursday, February 23, 2017 2:29 PM
To: CKAN Development Discussions <ckan-dev at lists.okfn.org<mailto:ckan-dev at lists.okfn.org>>
Subject: Re: [ckan-dev] Getting resources from packages via the ckanapi
On Thu, Feb 23, 2017 at 3:14 PM, Koebrick, Andrew (MNIT) <andrew.koebrick at state.mn.us<mailto:andrew.koebrick at state.mn.us>> wrote:
Tim,
I had tried that approach earlier with this syntax which seems to match what works for the organization_show:
packageObj = ckan.action.package_show(id=package['id'])
But when I attempt to see what I retrieved like so:
pprint (packageObj)
All that I get is
-1
Oh.
Try pprint dict(packageObj)
That -1 repr is due to an old performance hack (when using the actions internally you're getting a LazyJSON object instead of a normal dict) We've since reverted that hack. You wouldn't see it if you were using RemoteCKAN or a newer version of CKAN.
Ian
I thought perhaps it was a problem with my install of ckanapi so I upgraded from 3.6 to 4.0 but that did not fix the issue.
The syntax you suggested seems to have a problem but I also tried this variation:
packageObj = ckan.action.package_show(‘id:' + package['id']) #note addition of plus sign…
which throws:
TypeError: action() takes exactly 0 arguments (1 given)
Still plunking away…
Andrew
From: ckan-dev [mailto:ckan-dev-bounces at lists.okfn.org<mailto:ckan-dev-bounces at lists.okfn.org>] On Behalf Of Timothy Giles
Sent: Thursday, February 23, 2017 1:21 PM
To: CKAN Development Discussions <ckan-dev at lists.okfn.org<mailto:ckan-dev at lists.okfn.org>>
Subject: Re: [ckan-dev] Getting resources from packages via the ckanapi
Hi Andrew,
Once you have the package id, you can :
packageObj = ckan.action.package_show(‘id:' package['id'])
This will return you the package object with all the resources included. You can loop through the resources if you want but I believe all the resource metadata is provided.
Regards Tim
From: ckan-dev [mailto:ckan-dev-bounces at lists.okfn.org] On Behalf Of Koebrick, Andrew (MNIT)
Sent: den 23 februari 2017 19:44
To: ckan-dev at lists.okfn.org<mailto:ckan-dev at lists.okfn.org>
Subject: [ckan-dev] Getting resources from packages via the ckanapi
Hoping somebody can assist: I am attempting to create a report for one of my ckan instances that lists all the organizations, and then all of their packages along with the id of the packages’ resource (we have only one resource per package). However, I do not seem to be able to link to the resource id via API calls.
Here is what I have so far:
organizations = ckan.action.organization_list(include_datasets ='True')
for organization in organizations:
packages= orgDetails['packages']
for package in packages:
#And here is where it breaks down…. Here are a few of the many searches / shows I have tried:
resourceObj = ckan.action.resource_search(query='package_id:' +package['id'])
#which returns: {u'query': u'Field "package_id" not recognised in resource_search.', u'__type': u'Validation Error'}
revisionObj = ckan.action.revision_show(id=package['revision_id'])
#which just brings back information already knows about the revision.
While iterating through the package information I seem unable to get the related resources. I have tried all sorts of queries using what info I have from the package to get the related resources.
What can I use to link from the package list found in the return object from organization_show to get resources?
Many thanks,
Andrew Koebrick | MINNESOTA GEOSPATIAL INFORMATION OFFICE
Web Coordinator / Systems administrator / Librarian
MN.IT<http://MN.IT> Services @ MNGEO
651-201-2465<tel:(651)%20201-2465> (w) | 651-296-6398<tel:(651)%20296-6398> (f) | andrew.koebrick at state.mn.us<mailto:andrew.koebrick at state.mn.us>
658 Cedar St., Room 300, St. Paul, MN 55155, www.mngeo.state.mn.us<http://www.mngeo.state.mn.us/>
[cid:image002.jpg at 01CE61F8.52552AE0]<http://www.mn.gov/oet>
Information Technology for Minnesota Government | mn.gov/mnit<http://www.mn.gov/oet>
_______________________________________________
ckan-dev mailing list
ckan-dev at lists.okfn.org<mailto:ckan-dev at lists.okfn.org>
https://lists.okfn.org/mailman/listinfo/ckan-dev
Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
_______________________________________________
ckan-dev mailing list
ckan-dev at lists.okfn.org<mailto:ckan-dev at lists.okfn.org>
https://lists.okfn.org/mailman/listinfo/ckan-dev
Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
_______________________________________________
ckan-dev mailing list
ckan-dev at lists.okfn.org<mailto:ckan-dev at lists.okfn.org>
https://lists.okfn.org/mailman/listinfo/ckan-dev
Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
_______________________________________________
ckan-dev mailing list
ckan-dev at lists.okfn.org<mailto:ckan-dev at lists.okfn.org>
https://lists.okfn.org/mailman/listinfo/ckan-dev
Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20170228/7c0dca38/attachment-0003.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 1624 bytes
Desc: image001.jpg
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20170228/7c0dca38/attachment-0003.jpg>
More information about the ckan-dev
mailing list