[ckan-dev] Upcoming Filestore Improvements

Wed Aug 21 05:45:13 UTC 2013

Hello,

The CKAN Filestore has been giving us a few issues because of how it's
implemented.  From last week, I've been working on improving this in a
branch<https://github.com/okfn/ckan/compare/880-filestore-relative-urls?expand=1>
.

Here's what I'm roughly hoping to do:
1. Resources stored in the filestore will have url_type set to "filestore".

2. For resources stored in the filestore, url will be stored in the
resource table with path relative to CKAN's site_url. Here are some
examples:
Example 1: CKAN installed on http://example.com
Current URL:
http://example.com/storage/f/2013-07-25T10%3A59%3A23.632Z/file.csv
New URL: /storage/f/2013-07-25T10%3A59%3A23.632Z/file.csv.
Example 2: CKAN installed on http://example.com/ckan
Current URL:
http://example.com/ckan/storage/f/2013-07-25T10%3A59%3A23.632Z/file.csv
New URL: /storage/f/2013-07-25T10%3A59%3A23.632Z/file.csv

3. Local filestore will use a multipart form and the file will only be able
to be uploaded when submitting the form. This lets eliminates duplicate
files being uploaded. I'm still working on how much the UI will change, but
it will have to change.

4. Migration script to convert current CKAN instances to fix existing
entries in the resource table to the new format.

5. A script to delete resources in the filesystem that don't exist in the
filestore. This step may be part of this pull request or a follow up pull
request.

Note: While the filestore supports using Google Cloud Storage, Amazon S3,
and local storage with pairtree, my current work will particularly focus on
local storage with pairtree. I'm hoping that in the future, we can work on
improving Google Storage and S3.

Although I've already started some work on this, I'd welcome comments and
suggestions.

Nigel Babu

Developer  |  @nigelbabu <https://twitter.com/nigelbabu>

The Open Knowledge Foundation <http://okfn.org/>

Empowering through Open Knowledge

http://okfn.org/  |  @okfn <http://twitter.com/OKFN>  |  OKF on
Facebook<https://www.facebook.com/OKFNetwork> |
Blog <http://blog.okfn.org/>  |  Newsletter<http://okfn.org/about/newsletter>

 CKAN | http://ckan.org/ | @CKANproject
<http://twitter.com/CKANproject> |the world’s leading open-source data
portal platform
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20130821/40c2a9eb/attachment.html>