[ckan-dev] Getting resources from packages via the ckanapi

• Previous message: [ckan-dev] Getting resources from packages via the ckanapi
• Next message: [ckan-dev] Getting resources from packages via the ckanapi
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

Try using the version specified in the requirements.txt file:

pip install simplejson==3.3.1


On Fri, Feb 24, 2017 at 1:49 PM, Koebrick, Andrew (MNIT) <
andrew.koebrick at state.mn.us> wrote:

> Ian,
>
>
>
> I was at 3.8 but upgraded 3.10.  No change.
>
>
>
> I also tried another tack- in my testing I had been using a break so that
> it only attempted to get the packages for one organization (to make
> debugging quicker).  But when I took that out and ran the script over all
> the organizations  (94) and attempted to get the resources for all
> datasets/packages (993) a total of 8 actually came back as expected- as
> full dictionaries with the resources.
>
>
>
> I looked over the 8 packages that were successfully returned as dicts with
> resources and see nothing setting them apart from those that return as -1.
>
>
>
> As an experiment I will rewrite to do straight http requests and try
> taking ckanapi out of the mix.  In the meanwhile, here is the current state
> of my script (with try/excepts pulled out for brevity).  If anyone sees a
> python or ckan problems I would be grateful for assistance:
>
>
>
> *from *pprint *import **
> *from *ckanapi *import **
>
> key=
> *"xxxxxxxxxxxx" *ua =
> *'harvestScript/1.0 (+https://gisdata.mn.gov <https://gisdata.mn.gov>)' *ckan
> = RemoteCKAN(*'https://ng911.gisdata.mn.gov/
> <https://ng911.gisdata.mn.gov/>'*, user_agent=ua, apikey=key)
> packageCount = 0
> dictCount = 0
> orgCount = 0
> organizations = ckan.action.organization_list(include_datasets =*'True'*)
> *for *organization *in *organizations:
>        orgCount += 1
>        orgDetails = ckan.action.organization_show(id=organization,
> include_datasets=*'True'*)
>        packages= orgDetails[*'packages'*]
>        *for *package *in *packages:
>            packageCount += 1
>            packageObj = ckan.action.package_show(id=package[*'id'*])
>            *if *isinstance(packageObj,dict):
>                pprint(packageObj)
>                dictCount += 1
> *print **"Total packages: " *+ str(packageCount)
> *print **"Total dictionaries: " *+ str(dictCount)
> *print **"Total orgs: " *+ str(orgCount)
>
>
>
> For counts, this is what gets kicked out:
>
> Total packages: 993
>
> Total dictionaries: 8
>
> Total orgs: 94
>
>
>
>
>
> Thanks,
>
>
>
> Andrew
>
>
>
>
>
>
>
>
>
> *From:* ckan-dev [mailto:ckan-dev-bounces at lists.okfn.org] *On Behalf Of *Ian
> Ward
> *Sent:* Thursday, February 23, 2017 3:53 PM
>
> *To:* CKAN Development Discussions <ckan-dev at lists.okfn.org>
> *Subject:* Re: [ckan-dev] Getting resources from packages via the ckanapi
>
>
>
> What version of simplejson is installed in the virtualenv?
>
>
>
> On Feb 23, 2017 4:01 PM, "Koebrick, Andrew (MNIT)" <
> andrew.koebrick at state.mn.us> wrote:
>
> Ian,
>
>
>
> thanks for the assistance, unfortunately I get:
>
>
>
>     pprint dict(packageObj)
>
>               ^
>
> SyntaxError: invalid syntax
>
>
>
> In terms of versions I am attempting to pull from ckan 2.5.2 which does
> not seem that old; and I am using RemoteCKAN; I just did not bother adding
> the initialization to the code I sent.  Here is the excluded part of my
> script:
>
>
>
> key=
> *"REDACTED" *ua = *'harvestScript/1.0 (+**https://gisdata.mn.gov*
> <https://gisdata.mn.gov>
> *)' *ckan = RemoteCKAN(*'**https://ng911.gisdata.mn.gov/*
> <https://ng911.gisdata.mn.gov/>*'*, user_agent=ua, apikey=key)
>
>
>
> Unfortunately this server is firewalled off from public access, but if you
> want to test ckanapi against it, I can add your IP address to the whitelist
> and send you a key.
>
>
>
> Andrew
>
>
>
> *From:* ckan-dev [mailto:ckan-dev-bounces at lists.okfn.org] *On Behalf Of *Ian
> Ward
> *Sent:* Thursday, February 23, 2017 2:29 PM
> *To:* CKAN Development Discussions <ckan-dev at lists.okfn.org>
> *Subject:* Re: [ckan-dev] Getting resources from packages via the ckanapi
>
>
>
>
>
>
>
> On Thu, Feb 23, 2017 at 3:14 PM, Koebrick, Andrew (MNIT) <
> andrew.koebrick at state.mn.us> wrote:
>
> Tim,
>
>
>
> I had tried that approach earlier with this syntax which seems to match
> what works for the organization_show:
>
> packageObj = ckan.action.package_show(id=package[*'id'*])
>
> But when I attempt to see what I retrieved like so:
>
> pprint (packageObj)
>
> All that I get is
>
> -1
>
>
>
> Oh.
>
>
>
> Try pprint dict(packageObj)
>
>
>
> That -1 repr is due to an old performance hack (when using the actions
> internally you're getting a LazyJSON object instead of a normal dict) We've
> since reverted that hack. You wouldn't see it if you were using RemoteCKAN
> or a newer version of CKAN.
>
>
>
> Ian
>
>
>
>
>
> I thought perhaps it was a problem with my install of ckanapi so I
> upgraded from 3.6 to 4.0 but that did not fix the issue.
>
>
>
> The syntax you suggested seems to have a problem but I also tried this
> variation:
>
> packageObj = ckan.action.package_show(‘*id:' +* package[*'id'*]) #note addition of plus sign…
>
>
>
> which throws:
>
> TypeError: action() takes exactly 0 arguments (1 given)
>
>
>
> Still plunking away…
>
>
>
> Andrew
>
>
>
> *From:* ckan-dev [mailto:ckan-dev-bounces at lists.okfn.org] *On Behalf Of *Timothy
> Giles
> *Sent:* Thursday, February 23, 2017 1:21 PM
> *To:* CKAN Development Discussions <ckan-dev at lists.okfn.org>
> *Subject:* Re: [ckan-dev] Getting resources from packages via the ckanapi
>
>
>
> Hi Andrew,
>
>
>
> Once you have the package id, you can :
>
>
>
>                  packageObj = ckan.action.package_show(‘*id:'* package[*'id'*])
>
>
>
> This will return you the package object with all the resources included. You can loop through the resources if you want but I believe all the resource metadata is provided.
>
>
>
> Regards Tim
>
>
>
>
>
> *From:* ckan-dev [mailto:ckan-dev-bounces at lists.okfn.org
> <ckan-dev-bounces at lists.okfn.org>] *On Behalf Of *Koebrick, Andrew (MNIT)
> *Sent:* den 23 februari 2017 19:44
> *To:* ckan-dev at lists.okfn.org
> *Subject:* [ckan-dev] Getting resources from packages via the ckanapi
>
>
>
> Hoping somebody can assist:  I am attempting to create a report for one of
> my ckan instances that lists all the organizations, and then all of their
> packages along with the id of the packages’ resource (we have only one
> resource per package).  However, I do not seem to be able to link to the
> resource id via API calls.
>
>
>
> Here is what I have so far:
>
>
> organizations = ckan.action.organization_list(include_datasets =*'True'*)
> *for *organization *in *organizations:
>
>          packages= orgDetails[*'packages'*]
>
> *         for *package *in *packages:
>
>                  #And here is where it breaks down…. Here are a few of the many searches / shows I have tried:
>
>                  resourceObj = ckan.action.resource_search(query=*'package_id:'* +package[*'id'*])
>
>                     #which returns: {u'query': u'Field "package_id" not recognised in resource_search.', u'__type': u'Validation Error'}
>
>                  revisionObj = ckan.action.revision_show(id=package[*'revision_id'*])
>
>                     #which just brings back information already knows about the revision.
>
>
>
>
>
> While iterating through the package information I seem unable to get the related resources.  I have tried all sorts of queries using what info I have from the package to get the related resources.
>
>
>
> What can I use to link from the package list found in the return object from organization_show to get resources?
>
>
>
> Many thanks,
>
>
>
> *Andrew Koebrick  | * MINNESOTA GEOSPATIAL INFORMATION OFFICE
>
> Web Coordinator / Systems administrator / Librarian
>
> MN.IT Services @ MNGEO
>
> 651-201-2465 <(651)%20201-2465> (w)  *|  * 651-296-6398 <(651)%20296-6398>
> (f) *|*  andrew.koebrick at state.mn.us
>
> 658 Cedar St., Room 300, St. Paul, MN 55155, www.mngeo.state.mn.us
>
> *[image: cid:image002.jpg at 01CE61F8.52552AE0]* <http://www.mn.gov/oet>
>
> Information Technology for Minnesota Government   *|*   mn.gov/mnit
> <http://www.mn.gov/oet>
>
>
>
>
>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>
>
>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20170224/7d8be05e/attachment-0003.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 1624 bytes
Desc: not available
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20170224/7d8be05e/attachment-0003.jpg>

• Previous message: [ckan-dev] Getting resources from packages via the ckanapi
• Next message: [ckan-dev] Getting resources from packages via the ckanapi
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]