[ckan-dev] CKAN API Clients

Wed Apr 25 19:49:59 UTC 2012

On 25 April 2012 17:09, Sean Hammond <sean.hammond at okfn.org> wrote:
>
> > > Ok, I read and re-read the filestore documentation. The following
> > > snippet from [2]:
> > >
> > > "Provides credentials for doing operations on storage directly from a
> > > client (using web form style POSTs)."
> > >
> > > Suggests that it should be possible to upload a file. However there
> > > aren't many details and no working examples.

I will add a working example. Upload over the API is definitely
possible (though when using the S3 or GStorage backends it is a little
more complex).

In essence there are 2 parts:

* Getting credentials to do an upload:
http://docs.ckan.org/en/latest/filestore.html#form-authentication
* Uploading to the storage endpoint (in a standard file post format)

When the backend is local disk the latter will be :

{your-site}/storage/upload_handle

See https://github.com/okfn/ckan/blob/master/ckan/controllers/storage.py#L128

For S3 and Google Storage this will be the S3 or Google Storage
endpoint (you get the relevant info from the call to get credentials)

> > > The ckanclient lib also has some methods related to the storage api.
> > > However they too lack any documentation or examples.  The test classes
> > > in [3] also don't seem to test file uploading.
> >
> > Hmm. I'll take a look into this soon and see if I can figure it out and
> > maybe improve the docs
>
> On looking into this, it does look as if there's some code in CKAN for
> uploading files to the filestore via the API, but this code isn't
> documented or tested very well, so I'd consider it unsupported for now.

You don't upload "through the API" per se but I guess you could think
of /storage/upload_file as an API and obviously you do upload to S3 or
GStorage APIs (after getting credentials from CKAN API).

> I recommend you go forward with the side-loading via scp/rsync approach
> that we discussed.

Testing is non-trivial because it requires actual access to backend
(or *lots* of mocking). As such key tests are disabled in normal test
runs. However, I developed and ran those tests and they do generally
work.

> A number of people have asked about uploading to the filestore via the
> api recently, so we may allocate some time to get this code fixed up
> with proper tests and docs at some point in the next few weeks.

Code is fixed up and functional :-) We use it in production every day
-- our javascript file uploader in the interface is a client of the
storage API!

Rufus