[ckan-dev] Upcoming Filestore Improvements

Thu Aug 22 05:57:17 UTC 2013

Hi Ross,

I really like this idea. I'd want pairtree to be the core so we can really
focus on making it work. Right now, I'm neutral on this. I'm just improving
pairtree support. I'm a little scared of making too many changes in the
storage controller for fear of breaking external clients and people's
scripts.

Nigel Babu

Developer  |  @nigelbabu <https://twitter.com/nigelbabu>

The Open Knowledge Foundation <http://okfn.org/>

Empowering through Open Knowledge

http://okfn.org/  |  @okfn <http://twitter.com/OKFN>  |  OKF on
Facebook<https://www.facebook.com/OKFNetwork> |
Blog <http://blog.okfn.org/>  |  Newsletter<http://okfn.org/about/newsletter>

 CKAN | http://ckan.org/ | @CKANproject
<http://twitter.com/CKANproject> |the world’s leading open-source data
portal platform

On 21 August 2013 21:41, Ross Jones <ross at servercode.co.uk> wrote:

> Hi Nigel,
>
> Haven't had a chance to check out the branch (but looking forward to it),
> but I did have a thought about the abstraction over the storage engine.
>  It's great that CKAN will (again) support Amazon and Google for storage,
> but might they better be implemented in an extension? Perhaps
> ckanext-storage-extras or something?   You could implement pairtree as the
> default in core, and remove the others.
>
> The work in making the plugin for this public would make it much easier to
> write more extensions such as for dropbox, my owncloud installation or
> whatever else people want to use for storage.  It would have the added
> benefit of being able to use the new storage plugin for ckanext-archiver.
>
> Apologies if your branch already takes this approach, I will read through
> the code, promise...
>
> Ross
>
>
>
> On 21 Aug 2013, at 06:45, Nigel Babu <nigel.babu at okfn.org> wrote:
>
> Hello,
>
> The CKAN Filestore has been giving us a few issues because of how it's
> implemented.  From last week, I've been working on improving this in a
> branch<https://github.com/okfn/ckan/compare/880-filestore-relative-urls?expand=1>
> .
>
> Here's what I'm roughly hoping to do:
> 1. Resources stored in the filestore will have url_type set to "filestore".
>
> 2. For resources stored in the filestore, url will be stored in the
> resource table with path relative to CKAN's site_url. Here are some
> examples:
> Example 1: CKAN installed on http://example.com
> Current URL:
> http://example.com/storage/f/2013-07-25T10%3A59%3A23.632Z/file.csv
> New URL: /storage/f/2013-07-25T10%3A59%3A23.632Z/file.csv.
> Example 2: CKAN installed on http://example.com/ckan
> Current URL:
> http://example.com/ckan/storage/f/2013-07-25T10%3A59%3A23.632Z/file.csv
> New URL: /storage/f/2013-07-25T10%3A59%3A23.632Z/file.csv
>
> 3. Local filestore will use a multipart form and the file will only be
> able to be uploaded when submitting the form. This lets eliminates
> duplicate files being uploaded. I'm still working on how much the UI will
> change, but it will have to change.
>
> 4. Migration script to convert current CKAN instances to fix existing
> entries in the resource table to the new format.
>
> 5. A script to delete resources in the filesystem that don't exist in the
> filestore. This step may be part of this pull request or a follow up pull
> request.
>
> Note: While the filestore supports using Google Cloud Storage, Amazon S3,
> and local storage with pairtree, my current work will particularly focus on
> local storage with pairtree. I'm hoping that in the future, we can work on
> improving Google Storage and S3.
>
> Although I've already started some work on this, I'd welcome comments and
> suggestions.
>
> Nigel Babu
> Developer  |  @nigelbabu <https://twitter.com/nigelbabu>
> The Open Knowledge Foundation <http://okfn.org/>
> Empowering through Open Knowledge
> http://okfn.org/  |  @okfn <http://twitter.com/OKFN>  |  OKF on Facebook<https://www.facebook.com/OKFNetwork> |
> Blog <http://blog.okfn.org/>  |  Newsletter<http://okfn.org/about/newsletter>
>
>  CKAN | http://ckan.org/ | @CKANproject <http://twitter.com/CKANproject> |the world’s leading open-source data portal platform
>  _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>
>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20130822/1432a747/attachment-0001.html>