[ckan-dev] OFS uploader ref: storage.ckan.net

Thu Dec 2 23:27:12 UTC 2010

So I've just committed the first bit of the OFS uploader. Rufus had
asked me to look at how a client that talked to the rest store could
be told how to upload directly to goostor/s3 without hairpinning the
traffic through our servers.

The different pieces of the puzzle are now separated out. There is an
experimental uploader program called ofs_upload that takes a remote
path and a local file, goes through the creating bucket and key dance,
and then constructs the authentication headers it needs to upload
directly. It then copies some code from boto (difficult to use without
cut-and-paste) and performs the upload.

There needs to be a method added to the rest store that takes a
dictionary of headers and returns a hostname (goostor) and a
dictionary of headers with authentication information added. The
authentication information is basically a token that must be used
within 15 minutes.

Code is here: https://bitbucket.org/ww/ofs in particular the
ofs/command.py -- also OFS hackers, would like your opinion of the 
make_label() method, it seems to need to use some internal OFS things
which maybe should be exposed better since it doesn't seem an
unreasonable thing to want to do to make a bucket/label without
putting anything in (actually it puts a zero-length file in
temporarily). 

I'll look at fleshing this out and extending the rest server from
tomorrow and shortly all the dots should be joined. Not the prettiest
code in the world (owing to the cut-and-paste job from boto) but it
should work well enough.

Cheers,
-w
-- 
William Waites
http://eris.okfn.org/ww/foaf#i
9C7E F636 52F6 1004 E40A  E565 98E3 BBF3 8320 7664