[ckan-dev] Upcoming Filestore Improvements

Wed Aug 21 16:11:55 UTC 2013

Hi Nigel,

Haven't had a chance to check out the branch (but looking forward to it), but I did have a thought about the abstraction over the storage engine.  It's great that CKAN will (again) support Amazon and Google for storage, but might they better be implemented in an extension? Perhaps ckanext-storage-extras or something?   You could implement pairtree as the default in core, and remove the others.

The work in making the plugin for this public would make it much easier to write more extensions such as for dropbox, my owncloud installation or whatever else people want to use for storage.  It would have the added benefit of being able to use the new storage plugin for ckanext-archiver.

Apologies if your branch already takes this approach, I will read through the code, promise...

Ross

On 21 Aug 2013, at 06:45, Nigel Babu <nigel.babu at okfn.org> wrote:

> Hello,
> 
> The CKAN Filestore has been giving us a few issues because of how it's implemented.  From last week, I've been working on improving this in a branch.
> 
> Here's what I'm roughly hoping to do:
> 1. Resources stored in the filestore will have url_type set to "filestore".
> 
> 2. For resources stored in the filestore, url will be stored in the resource table with path relative to CKAN's site_url. Here are some examples:
> Example 1: CKAN installed on http://example.com
> Current URL: http://example.com/storage/f/2013-07-25T10%3A59%3A23.632Z/file.csv
> New URL: /storage/f/2013-07-25T10%3A59%3A23.632Z/file.csv.
> Example 2: CKAN installed on http://example.com/ckan
> Current URL: http://example.com/ckan/storage/f/2013-07-25T10%3A59%3A23.632Z/file.csv
> New URL: /storage/f/2013-07-25T10%3A59%3A23.632Z/file.csv
> 
> 3. Local filestore will use a multipart form and the file will only be able to be uploaded when submitting the form. This lets eliminates duplicate files being uploaded. I'm still working on how much the UI will change, but it will have to change.
> 
> 4. Migration script to convert current CKAN instances to fix existing entries in the resource table to the new format.
> 
> 5. A script to delete resources in the filesystem that don't exist in the filestore. This step may be part of this pull request or a follow up pull request.
> 
> Note: While the filestore supports using Google Cloud Storage, Amazon S3, and local storage with pairtree, my current work will particularly focus on local storage with pairtree. I'm hoping that in the future, we can work on improving Google Storage and S3.
> 
> Although I've already started some work on this, I'd welcome comments and suggestions.
> 
> Nigel Babu
> Developer  |  @nigelbabu
> The Open Knowledge Foundation
> Empowering through Open Knowledge
> http://okfn.org/  |  @okfn  |  OKF on Facebook  |  Blog  |  Newsletter
> 
> CKAN | http://ckan.org/ | @CKANproject | the world’s leading open-source data portal platform
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20130821/3484da4f/attachment-0001.html>