[ckan-dev] CKAN with local file storage issue

Sun Jan 19 19:33:33 UTC 2014

Hi Milica,

We've removed re-worked filestore for 2.2 and this problem is better
handled.  Each resource is stored with folder structure that's derived from
the id of the resource. All ids are generated with uuid.uuid4(), and a
resource with an id of 8545553f-1f38-4b29-bb93-0e86048376af is stored in
resources/854/555/3f-1f38-4b29-bb93-0e86048376af.

In future releases, this should solve the problem.

Nigel Babu

Developer  |  @nigelbabu <https://twitter.com/nigelbabu>

The Open Knowledge Foundation <http://okfn.org/>

Empowering through Open Knowledge

http://okfn.org/  |  @okfn <http://twitter.com/OKFN>  |  OKF on
Facebook<https://www.facebook.com/OKFNetwork> |
Blog <http://blog.okfn.org/>  |  Newsletter<http://okfn.org/about/newsletter>

 CKAN | http://ckan.org/ | @CKANproject
<http://twitter.com/CKANproject> |the world’s leading open-source data
portal platform

On 16 January 2014 09:23, Milica Knezevic <knezevic.milica at gmail.com> wrote:

> Hi all,
>
> I have installed CKAN 2.1.1 with local file storage configuration.
> Recently, I encountered a problem when the number of uploaded files
> exceeded 32.000 because of the limitation on ext3 file system. The problem
> is solved by converting to ext4, but what concerns me are potential
> performance issues caused by current organization of uploaded files. My
> CKAN instance is configured with the following storage settings:
>
> ofs.impl = pairtree
> ofs.storage_dir = /var/lib/ckan/default/data
>
> Each uploaded file is stored in its own directory named after the
> timestamp and the storage looks like this:
> /var/lib/ckan/default/data/de/fa/ul/t/obj/<timestamp1>/<filename1>
> ...
> /var/lib/ckan/default/data/de/fa/ul/t/obj/<timestampN>/<filenameN>
>
> So we ended up with a large number of subdirs in obj directory. I'm
> concerned about potential performance issues caused by this kind of
> organization. Isn't the idea of pairtree to "spread the load of storing
> high numbers of objects, while retaining the ability to treat each object
> distinctly"?
>
> Best,
> Milica
>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20140119/8a4397db/attachment-0003.html>