[ckan-dev] DataStorer extension location question

Fri Jun 15 12:57:35 UTC 2012

On 6/14/12 6:27 PM, "Jim Craner" <jim at codeforamerica.org> wrote:

>Hi Salman,
>
>~/pyenv/lib/python2.6/site-packages/ckanext-datasource.egg-link does
>NOT exist.  (Nor does that filename exist in any other folder on the
>system.)
>
>Is that useful info?  Was there a problem with the datastorer
>extension installation or something else?

Yes. It means that the package did not install successfully in your pyenv.
That is consistent with the error in your log file about not being able to
find the plugin because import of the module fails.

Let's try installing the plugin again.

Try this command:

sudo -u CKANUSERNAME pip install -I -U -E CKANPYENVHOME -e
git+https://github.com/okfn/ckanext-datastorer.git#egg=ckanext-datastorer
-f 
http://github.com/okfn/ckanext-archiver/tarball/master#egg=ckanext-archiver

You don't need to activate your virtualenv to run this command.
-e tells pip to install from a the given git repo.
-f tells pip to install dependencies from the given path(s).
-E tells pip where to install the packages.
-I tells pip to ignore the already installed version of the package.
-U tells pip to upgrade the package if it is already installed.
sudo -u CKANUSERNAME runs the pip command as CKANUSERNAME.

Salman

>
>Thanks!
>Jim
>
>
>
>
>On Thu, Jun 14, 2012 at 9:59 AM, Haq, Salman <Salman.Haq at neustar.biz>
>wrote:
>> The log file suggests that the extension has not been properly
>>installed.
>>
>> Does the following file exist?
>>
>> 
>>YOURCKANHOME/pyenv/lib/python2.6/site-packages/ckanext-datasource.egg-lin
>>k
>>
>> And if does exist, what is the output of `cat`ting it?
>>
>> Salman
>>
>> On 6/14/12 12:21 PM, "Jim Craner" <jim at codeforamerica.org> wrote:
>>
>>>Hi Adrià,
>>>
>>>Here is a pastebin of the /var/log/apache/MYCKAN.error.ini when I try
>>>to load the home page and get the 500 error:
>>>
>>>http://pastebin.com/h6a6jjVM
>>>
>>>The datastorer extension installation procedure you suggested
>>>yesterday resulted in this path:
>>>
>>>/home/backadmin/pyenv/src/ckanext-datastorer/ckanext/datastorer
>>>
>>>I also pasted my datastorer nosetests results to the list yesterday;
>>>not sure if that helps at all either.
>>>
>>>Please let me know if there is any other info I can provide that might
>>>help you or other list members suggest troubleshooting steps.
>>>
>>>Thanks again!
>>>Jim
>>>
>>>On Thu, Jun 14, 2012 at 2:00 AM, Adrià Mercader <amercadero at gmail.com>
>>>wrote:
>>>> Hi Jim,
>>>>
>>>> Yes, the traceback would be great to have, can you post it somewhere?
>>>>
>>>> Thanks,
>>>>
>>>> Adrià
>>>>
>>>> On 14 June 2012 00:02, Jim Craner <jim at codeforamerica.org> wrote:
>>>>> Hi Adrià,
>>>>>
>>>>> Thanks for the suggestions -- every command ran successfully until I
>>>>> restarted Apache.  I'm still getting a 500 error when I try to access
>>>>> CKAN, and the "PluginNotFoundException: datastorer" in the error
>>>>>logs.
>>>>>
>>>>> Would the rest of the Traceback help you troubleshoot?  Any other
>>>>>suggestions?
>>>>>
>>>>> Thanks again,
>>>>> Jim
>>>>>
>>>>> [Wed Jun 13 00:20:54 2012] [error] [client 127.0.0.1]
>>>>> PluginNotFoundException: datastorer
>>>>>
>>>>>
>>>>> On Wed, Jun 13, 2012 at 8:53 AM, Adrià Mercader
>>>>><amercadero at gmail.com>
>>>>>wrote:
>>>>>> Hi Jim,
>>>>>>
>>>>>> Sorry to hear that you are having trouble with the datastorer. Trust
>>>>>> me, it definitely works on different projects :)
>>>>>>
>>>>>> I've improved the install instructions on the README [1], which I
>>>>>> agree could be confusing, so I suggest we start all over again.
>>>>>>
>>>>>> 1. Remove any datastorer folder that you have right now
>>>>>> 2. Activate your virtualenv
>>>>>> 3. Run pip install -e
>>>>>>
>>>>>>git+git://github.com/okfn/ckanext-datastorer.git#egg=ckanext-datastor
>>>>>>er
>>>>>> 4. Run pip install -r ckanext-datastorer/requires.txt
>>>>>>
>>>>>> At this point you should have a ~/pyenv/src/ckanext-datastorer
>>>>>>folder
>>>>>> and running this command should not return any errors or prompts:
>>>>>>
>>>>>> python -c "import ckanext.datastorer"
>>>>>>
>>>>>> After this point you need to add this to your ini file:
>>>>>>
>>>>>> ckan.plugins = datastorer <rest of plugins>...
>>>>>>
>>>>>>
>>>>>> Let me know if this works for you.
>>>>>>
>>>>>> Adrià
>>>>>>
>>>>>>
>>>>>> [1] 
>>>>>>https://github.com/okfn/ckanext-datastorer/blob/master/README.rst
>>>>>>
>>>>>>
>>>>>> On 13 June 2012 16:30, Jim Craner <jim at codeforamerica.org> wrote:
>>>>>>> Hi Sean,
>>>>>>>
>>>>>>> I really appreciate the suggestions, but I think at this point I
>>>>>>>need
>>>>>>> explicit "You should do X, Y, and Z" instructions.
>>>>>>>
>>>>>>> I'm pretty sure I've tried a hundred variations on everything you
>>>>>>> suggested over the past couple of weeks, I *have* been referring to
>>>>>>> that "how to install an extension" doc, I've tried every
>>>>>>>combination
>>>>>>> of file structure for the extensions folder and I'm still getting
>>>>>>> nothing.
>>>>>>>
>>>>>>> Since this (allegedly :-) works on the Data Hub, could someone with
>>>>>>> access to that server just go in and tell me how the "datastorer"
>>>>>>> extension is set up in the CKAN source folder structure?  And what
>>>>>>>the
>>>>>>> plugins line is?  (I've seen "datastore archiver" and "datastorer
>>>>>>> archive" but never know which was correct).
>>>>>>>
>>>>>>> My offer to help improve this part of the documentation --
>>>>>>>assuming I
>>>>>>> can ever get this figured out -- still stands as well :-)
>>>>>>>
>>>>>>> Thanks again!
>>>>>>> Jim
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Jun 13, 2012 at 5:01 AM, Sean Hammond
>>>>>>><sean.hammond at okfn.org> wrote:
>>>>>>>>> I'm (still) trying to get the DataStorer functionality working.
>>>>>>>>>I
>>>>>>>>> have a fresh install of CKAN 1.7 from source running on a local
>>>>>>>>>Ubuntu
>>>>>>>>> 10.04 LTS virtual environment.
>>>>>>>>>
>>>>>>>>> I got CKAN running, the FileStore running, everything is looking
>>>>>>>>>good.
>>>>>>>>>
>>>>>>>>> I really need some help clarifying the instructions found here:
>>>>>>>>>
>>>>>>>>> https://github.com/okfn/ckanext-datastorer
>>>>>>>>>
>>>>>>>>> These instructions *seem* to work, as in no errors are generated
>>>>>>>>> during the installation, but I'm really not sure where the files
>>>>>>>>>are
>>>>>>>>> supposed to end up.  When I try to access CKAN after installing
>>>>>>>>>this,
>>>>>>>>> I get a 500 error and the logs report "PluginNotFoundException:
>>>>>>>>> datastore."
>>>>>>>>>
>>>>>>>>> My CKAN folder is in my server's user's home folder,
>>>>>>>>>~/pyenv/src/ckan.
>>>>>>>>>  The config file is ~/pyenv/src/ckan/datasc.ini.  The
>>>>>>>>>installation
>>>>>>>>> instructions instruct me to download a zip file which becomes an
>>>>>>>>> okfn-ckanext-datastorer folder (which I've tried putting in ~ and
>>>>>>>>>in
>>>>>>>>> ~/pyenv/src/ckan).  When I run the installation, it generates a
>>>>>>>>> subfolder "ckanext" which has its own folder within it called
>>>>>>>>> "datastorer".  I tried putting that final "datastorer" folder
>>>>>>>>>directly
>>>>>>>>> in my existing ~/pyenv/src/ckan/ckanext folder and got the same
>>>>>>>>> results.  So at that point I'm completely lost.
>>>>>>>>>
>>>>>>>>> Can someone who has DataStorer running successfully tell me a)
>>>>>>>>>what to
>>>>>>>>> put in my CONFIG.ini file's plugins line (e.g., "datastore
>>>>>>>>>archiver"
>>>>>>>>> or "datastorer archiver" or something else)?  And b) what should
>>>>>>>>>the
>>>>>>>>> correct file structure be for the datastorer extension folder(s)
>>>>>>>>> relative to the root of my CKAN installation?
>>>>>>>>
>>>>>>>> Hi Jim,
>>>>>>>>
>>>>>>>> I haven't installed datastorer before myself, but I would try
>>>>>>>>reading
>>>>>>>> the 'Installing an Extension' docs here:
>>>>>>>>
>>>>>>>>
>>>>>>>>http://docs.ckan.org/en/ckan-1.7/extensions.html#installing-an-exte
>>>>>>>>ns
>>>>>>>>ion
>>>>>>>>
>>>>>>>> and then you might understand the ckanext-datastorer README a
>>>>>>>>little
>>>>>>>> better. For example, I would probably download ckanext-datastorer
>>>>>>>>by
>>>>>>>> activating my virtualenv and running a command like:
>>>>>>>>
>>>>>>>>    pip install -e
>>>>>>>>git+https://github.com/okfn/ckanext-datastorer.git#egg=ckanext-data
>>>>>>>>st
>>>>>>>>orer
>>>>>>>>
>>>>>>>> this should download and install ckanext-datastorer in a directory
>>>>>>>>like:
>>>>>>>>
>>>>>>>>    pyenv/src/ckanext-datastorer
>>>>>>>>
>>>>>>>> You might still need to run pip install -r on the requires.txt
>>>>>>>>file
>>>>>>>>in
>>>>>>>> the ckanext-datastorer directory.
>>>>>>>>
>>>>>>>> In the plugins line of your CKAN ini file you should add something
>>>>>>>>like:
>>>>>>>>
>>>>>>>>    plugins = datastorer
>>>>>>>>
>>>>>>>> A couple of things you can do to test whether datastorer is
>>>>>>>>working
>>>>>>>> properly are to run CKAN with `paster serve development.ini` and
>>>>>>>>see if
>>>>>>>> it crashes with something like 'Plugin not found: datastorer', run
>>>>>>>>the
>>>>>>>> datastorer tests, something like:
>>>>>>>>
>>>>>>>>    nosetests tests
>>>>>>>>
>>>>>>>> from the ckanext-datastorer directory, and to try out the
>>>>>>>> `paster datastorer` command (from the pyenv/src/ckan directory).
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> ckan-dev mailing list
>>>>>>>> ckan-dev at lists.okfn.org
>>>>>>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> -----
>>>>>>> Jim Craner, 2012 Fellow
>>>>>>> Code for America
>>>>>>> jim at codeforamerica.org
>>>>>>> (+1) 773-809-4546
>>>>>>> http://codeforamerica.org
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> ckan-dev mailing list
>>>>>>> ckan-dev at lists.okfn.org
>>>>>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>>>>>
>>>>>> _______________________________________________
>>>>>> ckan-dev mailing list
>>>>>> ckan-dev at lists.okfn.org
>>>>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> -----
>>>>> Jim Craner, 2012 Fellow
>>>>> Code for America
>>>>> jim at codeforamerica.org
>>>>> (+1) 773-809-4546
>>>>> http://codeforamerica.org
>>>>>
>>>>> _______________________________________________
>>>>> ckan-dev mailing list
>>>>> ckan-dev at lists.okfn.org
>>>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>>>
>>>> _______________________________________________
>>>> ckan-dev mailing list
>>>> ckan-dev at lists.okfn.org
>>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>>
>>>
>>>
>>>--
>>>-----
>>>Jim Craner, 2012 Fellow
>>>Code for America
>>>jim at codeforamerica.org
>>>(+1) 773-809-4546
>>>http://codeforamerica.org
>>>
>>>_______________________________________________
>>>ckan-dev mailing list
>>>ckan-dev at lists.okfn.org
>>>http://lists.okfn.org/mailman/listinfo/ckan-dev
>>
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>
>
>
>-- 
>-----
>Jim Craner, 2012 Fellow
>Code for America
>jim at codeforamerica.org
>(+1) 773-809-4546
>http://codeforamerica.org
>
>_______________________________________________
>ckan-dev mailing list
>ckan-dev at lists.okfn.org
>http://lists.okfn.org/mailman/listinfo/ckan-dev