[ckan-dev] CKAN API Clients

Wed May 2 19:26:28 UTC 2012

On 4/30/12 4:05 PM, "Haq, Salman" <Salman.Haq at neustar.biz> wrote:

>
>
>On 4/30/12 3:23 PM, "Rufus Pollock" <rufus.pollock at okfn.org> wrote:
>
>>On 30 April 2012 18:46, Haq, Salman <Salman.Haq at neustar.biz> wrote:
>>>
>>>
>>> On 4/30/12 10:08 AM, "Sean Hammond" <sean.hammond at okfn.org> wrote:
>>>
>>>>> > And this is a the new ckanclient.loader.util module:
>>>>> > http://git.io/zYk8NA
>>>>> >
>>>>> > 'upload_file' and 'add_package_resource' are two high-level utility
>>>>> > functions it defines.
>>>>> >
>>>>> > To actually upload the file, we have to perform a multi-part form
>>>>>upload
>>>>> > which urllib2 does not handle by default so I've used httplib for
>>>>>that
>>>>> > purpose. The 'poster' module has a special urllib2 opener for doing
>>>>> > multi-part uploads but I didn't want to introduce a new dependency.
>>>>> >
>>>>> > Test cases to follow soon.
>>>>>
>>>>> Sorry I didn't have time to look at this this week. I'll get back to
>>>>>you
>>>>> about it next week (unless someone else can do it sooner?) but I
>>>>>think
>>>>> you can go ahead and add test cases, and maybe also integrate it into
>>>>> ckanclient/__init__.py? I think it's great work and will surely be
>>>>> useful
>>>>
>>>>This code looks good to me so I'd like to merge it once it's integrated
>>>>into ckanclient/__init__.py and maybe has some test cases. I did have
>>>>one problem testing it locally here, line 87 of util.py is using
>>>>u.hostname but that was None, when it should be '127.0.0.1:5000', so
>>>>the
>>>>upload was failing. I think the problem is that auth_dict['action'] on
>>>>line 84 may be just a path like /storage/upload that does not contain
>>>>the hostname.
>>>
>>> Sean,
>>>
>>> I'll fix the issue with localhost issue that you¹ve highlight.
>>>
>>> Should the functions `upload_file` and `add_package_resource` become
>>> methods of the CkanClient class while the helpers remain the `util`
>>> module? Or do you want to not have a `util` module at all?
>>
>>I'd suggest adding to main class. I'm also happy to review the
>>functions once done (this could substitute for getting you an upload
>>example).
>
>Okay, I've moved all code from the util module into the CkanClient class.
>ckanclient.loaders.util has been deleted.
>
>See the changeset here:
>
>https://github.com/slmnhq/ckanclient/commit/d233fcb3f73644b8b505839bedc16d
>3
>52434e946
>
>Thanks,
>Salman
>
>
>
>>
>>I also wonder if now (or soon) is when we do the mass rename package ->
>>dataset.
>>
>>> I still haven't had a chance to work on adding test case, but will do
>>>so
>>> later this week hopefully.
>>
>>That would be great. For upload we almost certainly need to mock ...
>>
>>Rufus

Rufus,

Do you have any pointers about how I should go about mock testing the
upload functionality?

Currently test_ckanclient.py starts a CKAN server instance, loads it with
test data, and performs tests by making API calls assuming the existence
of that test data.

Ideally, if the test server was configured to use local storage, then the
upload functionality could be tested quite easily without any mocking. An
example test would upload a dummy resource to the `annakarenina` dataset
and run assertions against the returned package entity dict. Of course,
the test server isn't configured to use local storage so this isn't
currently possible.

Let me know if anyone has any ideas.

Thanks,
Salman