[ckan-dev] CKAN with local file storage issue

Sun Jan 19 20:10:41 UTC 2014

Thanks for the reply.

Best,
Milica

On Sun, Jan 19, 2014 at 8:33 PM, Nigel Babu <nigel.babu at okfn.org> wrote:

> Hi Milica,
>
> We've removed re-worked filestore for 2.2 and this problem is better
> handled.  Each resource is stored with folder structure that's derived from
> the id of the resource. All ids are generated with uuid.uuid4(), and a
> resource with an id of 8545553f-1f38-4b29-bb93-0e86048376af is stored in
> resources/854/555/3f-1f38-4b29-bb93-0e86048376af.
>
> In future releases, this should solve the problem.
>
> Nigel Babu
>
> Developer  |  @nigelbabu <https://twitter.com/nigelbabu>
>
> The Open Knowledge Foundation <http://okfn.org/>
>
> Empowering through Open Knowledge
>
> http://okfn.org/  |  @okfn <http://twitter.com/OKFN>  |  OKF on Facebook<https://www.facebook.com/OKFNetwork> |
> Blog <http://blog.okfn.org/>  |  Newsletter<http://okfn.org/about/newsletter>
>
>  CKAN | http://ckan.org/ | @CKANproject <http://twitter.com/CKANproject> |the world’s leading open-source data portal platform
>
>
> On 16 January 2014 09:23, Milica Knezevic <knezevic.milica at gmail.com>wrote:
>
>> Hi all,
>>
>> I have installed CKAN 2.1.1 with local file storage configuration.
>> Recently, I encountered a problem when the number of uploaded files
>> exceeded 32.000 because of the limitation on ext3 file system. The problem
>> is solved by converting to ext4, but what concerns me are potential
>> performance issues caused by current organization of uploaded files. My
>> CKAN instance is configured with the following storage settings:
>>
>> ofs.impl = pairtree
>> ofs.storage_dir = /var/lib/ckan/default/data
>>
>> Each uploaded file is stored in its own directory named after the
>> timestamp and the storage looks like this:
>> /var/lib/ckan/default/data/de/fa/ul/t/obj/<timestamp1>/<filename1>
>> ...
>> /var/lib/ckan/default/data/de/fa/ul/t/obj/<timestampN>/<filenameN>
>>
>> So we ended up with a large number of subdirs in obj directory. I'm
>> concerned about potential performance issues caused by this kind of
>> organization. Isn't the idea of pairtree to "spread the load of storing
>> high numbers of objects, while retaining the ability to treat each object
>> distinctly"?
>>
>> Best,
>> Milica
>>
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>
>>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20140119/e06d492d/attachment-0003.html>