[ckan-dev] file storage setup in 1.7

Rufus Pollock rufus.pollock at okfn.org
Thu May 24 16:57:13 UTC 2012


Yes! As Salman correctly says, any "main" settings need to go above
all the logging stuff in the [app:main] section (sections in the ini
file are separated by [...] section headers)!

Thinking about it now, this may be rather non-obvious and we need to
perhaps think about making this clearer in the config file.

Rufus

On 24 May 2012 16:54, Haq, Salman <Salman.Haq at neustar.biz> wrote:
> Move the following lines from the end of the file to end of the [app:main]
> section:
>
>
> #storage stuff
> ckan.storage.bucket = /srv/data
> ## OFS configuration
> ofs.impl = pairtree
> # directory on disk for data storage (should be empty)
> ofs.storage_dir = /srv/data
>
>
>
> On 5/24/12 11:32 AM, "Jim Craner" <jim at codeforamerica.org> wrote:
>
>>Hi everyone,
>>
>>Thanks for the suggestions...  Here were the steps I took this morning
>>based on feedback from Salman and Ross:
>>
>>1) Removed "storage" from .ini file's list of plugins  //no effect
>>
>>2) python-argparse was NOT installed, but installing it had  //no effect
>>
>>3) I created the following directory/file structure in my
>>{ofs.storage_dir} (which is /srv/data):
>>
>>root at dataserver:/srv/data# ls -al
>>drwxr-xr-x 3 www-data www-data 4096 2012-05-24 08:10 .
>>drwxr-xr-x 3 root     root     4096 2012-05-23 21:55 ..
>>drwxr-xr-x 2 www-data www-data 4096 2012-05-24 08:10 pairtree_root
>>-rw-r--r-- 1 www-data www-data    0 2012-05-24 08:03 pairtree_version0_1
>>
>>//no effect
>>
>>I pasted my .ini file to pastebin -- storage settings are at the very
>>bottom:
>>
>>http://pastebin.com/EgQV6vZE
>>
>>Thanks for any additional suggestions!
>>-Jim
>>
>>
>>On Thu, May 24, 2012 at 6:43 AM, Haq, Salman <Salman.Haq at neustar.biz>
>>wrote:
>>>
>>> On 5/24/12 4:51 AM, "Ross Jones" <ross.jones at okfn.org> wrote:
>>>
>>>>Hi Jim
>>>>
>>>>If you're using 1.7 you shouldn't need ckanext-storage at all, it might
>>>>just
>>>>confuse the situation so probably best if you disable it (remove it from
>>>>your .ini).
>>>>
>>>>Could you post your .ini to pastebin with any sensitive (database) info
>>>>removed?
>>>>Everytime this happens to me it is because I've forgotten to install
>>>>pairtree
>>>>or have set ofs.storage_dir to a folder where I don't have permissions.
>>>>
>>>>I'm assuming you are using local file storage, in which case it might be
>>>>worth seeing
>>>>if CKAN has created a pairtree_version0_1 file (empty) in the folder
>>>>specified in
>>>>ofs.storage_dir. If not create one and see if that makes a difference.
>>>>
>>>>Ross
>>>
>>>
>>>
>>> Jim,
>>>
>>> Building on Ross's suggestions, there may be several things that are
>>> broken and need to be reported as bugs:
>>>
>>> 1. python-argparse debian package dependency may be missing.
>>> 2. python-pairtree debian package dependency may be missing.
>>> 3. {ofs.storage_dir}/pairtree_root/ directory has to be explicitly
>>>created
>>> with www-data:www-data ownership and mode 755.
>>> 4. {ofs.storage_dir}/pairtree_version0_1 file has to be explicitly
>>>touched
>>> with www-data:www-data ownership and mode 644.
>>>
>>>
>>> The mailing list has similar threads from the March/April time frame
>>>where
>>> the same problem was reported.
>>>
>>> Hope this helps.
>>>
>>> Salman
>>>
>>>
>>>>
>>>>
>>>>On 24 May 2012, at 06:27, Jim Craner wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> I'm trying to configure CKAN 1.7 (package install) on an Ubuntu 10.04
>>>>> appliance and I'm having some problems with the file storage
>>>>> functionality.  I'm trying to use local file storage based on these
>>>>> instructions:
>>>>>
>>>>> http://docs.ckan.org/en/ckan-1.7/filestore.html
>>>>>
>>>>> In addition to the settings recommended in that doc, I also did a "pip
>>>>> install ckanext-storage" and added "storage" to my .ini file's list of
>>>>> extensions.
>>>>>
>>>>> 1) When I try to upload a file as a resource, I am able to select the
>>>>> file but when I click the "upload" button I get the "Failed to get
>>>>> credentials for storage upload. Upload cannot proceed" error message
>>>>> (I am logged in as the admin user).
>>>>>
>>>>> 2) When I try to upload a file via the /storage/upload method, I am
>>>>> again able to select the file but attempting to upload returns a
>>>>> generic "Server Error / an internal error occurred" message.  In the
>>>>> /var/log/apache2/INSTANCENAME.error.log, I can see dozens of error
>>>>> messages related to "/storage/upload" and "/storage/upload_handle" but
>>>>> I'm not really sure what I'm looking for.
>>>>>
>>>>> Any suggestions?
>>>>>
>>>>> Thanks!
>>>>> Jim
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> -----
>>>>> Jim Craner, 2012 Fellow
>>>>> Code for America
>>>>> jim at codeforamerica.org
>>>>> (+1) 773-809-4546
>>>>> http://codeforamerica.org
>>>>>
>>>>> _______________________________________________
>>>>> ckan-dev mailing list
>>>>> ckan-dev at lists.okfn.org
>>>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>>>
>>>>
>>>>_______________________________________________
>>>>ckan-dev mailing list
>>>>ckan-dev at lists.okfn.org
>>>>http://lists.okfn.org/mailman/listinfo/ckan-dev
>>>
>>>
>>> _______________________________________________
>>> ckan-dev mailing list
>>> ckan-dev at lists.okfn.org
>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>
>>
>>
>>--
>>-----
>>Jim Craner, 2012 Fellow
>>Code for America
>>jim at codeforamerica.org
>>(+1) 773-809-4546
>>http://codeforamerica.org
>>
>>_______________________________________________
>>ckan-dev mailing list
>>ckan-dev at lists.okfn.org
>>http://lists.okfn.org/mailman/listinfo/ckan-dev
>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev



-- 
Co-Founder, Open Knowledge Foundation
Promoting Open Knowledge in a Digital Age
http://www.okfn.org/ - http://blog.okfn.org/




More information about the ckan-dev mailing list