[ckan-dev] CKAN API Clients

Wed Apr 25 20:06:14 UTC 2012

On 4/25/12 3:49 PM, "Rufus Pollock" <rufus.pollock at okfn.org> wrote:

>On 25 April 2012 17:09, Sean Hammond <sean.hammond at okfn.org> wrote:
>>
>> > > Ok, I read and re-read the filestore documentation. The following
>> > > snippet from [2]:
>> > >
>> > > "Provides credentials for doing operations on storage directly from
>>a
>> > > client (using web form style POSTs)."
>> > >
>> > > Suggests that it should be possible to upload a file. However there
>> > > aren't many details and no working examples.
>
>I will add a working example. Upload over the API is definitely
>possible (though when using the S3 or GStorage backends it is a little
>more complex).

Working examples will be great.

I've also added some working code in ckanclient to upload files and add
resources (see email from earlier).

>
>In essence there are 2 parts:
>
>* Getting credentials to do an upload:
>http://docs.ckan.org/en/latest/filestore.html#form-authentication
>* Uploading to the storage endpoint (in a standard file post format)
>
>When the backend is local disk the latter will be :
>
>{your-site}/storage/upload_handle

I had to reverse engineer the js upload behavior to understand all of this.

Furthermore, the documentation uses the term 'label' without elaborating
whether this is the internal id or user-specified name.

>
>See 
>https://github.com/okfn/ckan/blob/master/ckan/controllers/storage.py#L128
>
>For S3 and Google Storage this will be the S3 or Google Storage
>endpoint (you get the relevant info from the call to get credentials)
>
>> > > The ckanclient lib also has some methods related to the storage api.
>> > > However they too lack any documentation or examples.  The test
>>classes
>> > > in [3] also don't seem to test file uploading.
>> >
>> > Hmm. I'll take a look into this soon and see if I can figure it out
>>and
>> > maybe improve the docs
>>
>> On looking into this, it does look as if there's some code in CKAN for
>> uploading files to the filestore via the API, but this code isn't
>> documented or tested very well, so I'd consider it unsupported for now.
>
>You don't upload "through the API" per se but I guess you could think
>of /storage/upload_file as an API and obviously you do upload to S3 or
>GStorage APIs (after getting credentials from CKAN API).

Uploading to /storage/upload_file is a matter of doing a multi-part form
submission. 
But the documentation doesn't mention this.

>
>> I recommend you go forward with the side-loading via scp/rsync approach
>> that we discussed.
>
>Testing is non-trivial because it requires actual access to backend
>(or *lots* of mocking). As such key tests are disabled in normal test
>runs. However, I developed and ran those tests and they do generally
>work.

Do you recommend that I try to extend the ckanclient nose tests to test my
file/resource upload functions?

>
>> A number of people have asked about uploading to the filestore via the
>> api recently, so we may allocate some time to get this code fixed up
>> with proper tests and docs at some point in the next few weeks.
>
>Code is fixed up and functional :-) We use it in production every day
>-- our javascript file uploader in the interface is a client of the
>storage API!

Yes, the web ui works very well in 1.6.

Salman