[ckan-dev] CKAN with local file storage issue

Thu Jan 16 09:23:09 UTC 2014

Hi all,

I have installed CKAN 2.1.1 with local file storage configuration.
Recently, I encountered a problem when the number of uploaded files
exceeded 32.000 because of the limitation on ext3 file system. The problem
is solved by converting to ext4, but what concerns me are potential
performance issues caused by current organization of uploaded files. My
CKAN instance is configured with the following storage settings:

ofs.impl = pairtree
ofs.storage_dir = /var/lib/ckan/default/data

Each uploaded file is stored in its own directory named after the timestamp
and the storage looks like this:
/var/lib/ckan/default/data/de/fa/ul/t/obj/<timestamp1>/<filename1>
...
/var/lib/ckan/default/data/de/fa/ul/t/obj/<timestampN>/<filenameN>

So we ended up with a large number of subdirs in obj directory. I'm
concerned about potential performance issues caused by this kind of
organization. Isn't the idea of pairtree to "spread the load of storing
high numbers of objects, while retaining the ability to treat each object
distinctly"?

Best,
Milica
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20140116/dfc759eb/attachment-0002.html>