[ckan-dev] Datastore setup for ckan 2.0
Dominik Moritz
dominik.moritz at okfn.org
Thu Mar 21 20:50:41 UTC 2013
You can try to change to the branch if you have installed the datastore via github. If not, you will need to patch `plugin.py`. There is no need to change the tasks.py since you do not need the download retry code.
On 21 Mar 2013, at 17:22, Stefán Þór Finnsson <stefan.thor.finnsson at reykjavik.is> wrote:
> Hi Dominik
>
> No worries, thanks for this?
>
> Do you want me to replace the plugin.py and make the tasks.py with the one from the pull request?
>
> -----Original Message-----
> From: ckan-dev-bounces at lists.okfn.org [mailto:ckan-dev-bounces at lists.okfn.org] On Behalf Of Dominik Moritz
> Sent: 21. mars 2013 08:47
> To: CKAN Development Discussions
> Subject: Re: [ckan-dev] Datastore setup for ckan 2.0
>
> Hi,
>
> sorry that my answer took so long. Can you try the version from the pull request at https://github.com/okfn/ckanext-datastorer/pull/21? It looks like it could fix the problem.
>
> If not, could you send me your .ini file (without the passwords for the database or any secret keys ;-)).
>
> Cheers,
> Dominik
>
> On 14 Mar 2013, at 16:37, Stefán Þór Finnsson <stefan.thor.finnsson at reykjavik.is> wrote:
>
>> Still having this problem, did try to edit deployment.ini, but now
>> luck, if I leave the ckan.site_url = empty I see when I try to upload
>> a file that it does not add the ckan-test but then I get some authentication error, it ask me to login agin and goes in to a loop.
>>
>> Would be nice if someone could help me see the light here :)
>>
>> -----Original Message-----
>> From: ckan-dev-bounces at lists.okfn.org
>> [mailto:ckan-dev-bounces at lists.okfn.org] On Behalf Of Stefán Þór
>> Finnsson
>> Sent: 11. mars 2013 13:36
>> To: CKAN Development Discussions
>> Subject: Re: [ckan-dev] Datastore setup for ckan 2.0
>>
>> Hi,
>>
>> I did see error´s now when the celery is running:
>>
>> [2013-03-11 10:36:40,829: ERROR/MainProcess] Task datastorer.upload[41e99198-a0be-43a0-b4d6-54eb4a9b9d80] raised exception: LinkHeadRequestError('URL unobtainable: Server returned HTTP 404',) Traceback (most recent call last):
>> File "/usr/local/ckan-test/pyenv/lib/python2.6/site-packages/celery/execute/trace.py", line 47, in trace
>> return cls(states.SUCCESS, retval=fun(*args, **kwargs)) File
>> "/usr/local/ckan-test/pyenv/lib/python2.6/site-packages/celery/app/task/__init__.py", line 247, in __call__
>> return self.run(*args, **kwargs)
>> File "/usr/local/ckan-test/pyenv/lib/python2.6/site-packages/celery/app/__init__.py", line 175, in run
>> return fun(*args, **kwargs)
>> File "/usr/local/ckan-test/pyenv/src/ckanext-datastorer/ckanext/datastorer/tasks.py", line 108, in datastorer_upload
>> return _datastorer_upload(context, data, logger) File
>> "/usr/local/ckan-test/pyenv/src/ckanext-datastorer/ckanext/datastorer/tasks.py", line 123, in _datastorer_upload
>> result = download(context, resource, data_formats=DATA_FORMATS)
>> File "/usr/local/ckan-test/pyenv/src/ckanext-archiver/ckanext/archiver/tasks.py", line 103, in download
>> headers = json.loads(link_checker(link_context, link_data)) File
>> "/usr/local/ckan-test/pyenv/lib/python2.6/site-packages/celery/app/task/__init__.py", line 247, in __call__
>> return self.run(*args, **kwargs)
>> File "/usr/local/ckan-test/pyenv/lib/python2.6/site-packages/celery/app/__init__.py", line 175, in run
>> return fun(*args, **kwargs)
>> File "/usr/local/ckan-test/pyenv/src/ckanext-archiver/ckanext/archiver/tasks.py", line 361, in link_checker
>> raise LinkHeadRequestError(error_message)
>> LinkHeadRequestError: URL unobtainable: Server returned HTTP 404
>>
>> When I go to download for the document I get this path:
>> http://ckan-test/ckan-test/storage/f/2013-03-11T10%3A36%3A28.446Z/ckan
>> test.csv
>>
>> and get 404 error, But if I take the extra ckan-test I can download the document, not sure where it gets the parameter to add this ckan-test in the url, is it from the ini file or some other config?
>>
>>
>>
>> -----Original Message-----
>> From: ckan-dev-bounces at lists.okfn.org
>> [mailto:ckan-dev-bounces at lists.okfn.org] On Behalf Of Dominik Moritz
>> Sent: 8. mars 2013 15:26
>> To: CKAN Development Discussions
>> Subject: Re: [ckan-dev] Datastore setup for ckan 2.0
>>
>> Hi,
>>
>> could you go into a little bit more detail what you've done? Have you installed the datastorer properly (see https://github.com/okfn/ckanext-datastorer)? Did you wait for the datastorer to finish its work? Is the recline preview enabled in your .ini file?
>>
>> Dominik
>>
>> On 7 Mar 2013, at 15:59, Stefán Þór Finnsson <stefan.thor.finnsson at reykjavik.is> wrote:
>>
>>> The fun goes on :) When I click preview for csv or xlsx file it get this:
>>>
>>> This resource cannot be previewed. More details...
>>> No handler defined for data type: xls.
>>>
>>> When filestore is in place I can upload files and download them, that
>>> works
>>>
>>> When I have added datastore to the party and running celeryd I should see preview of the file and be able to do something with it on the page? Or not? I´m in the dark with the functions?
>>>
>>> -----Original Message-----
>>> From: ckan-dev-bounces at lists.okfn.org
>>> [mailto:ckan-dev-bounces at lists.okfn.org] On Behalf Of Stefán Þór
>>> Finnsson
>>> Sent: 6. mars 2013 14:19
>>> To: CKAN Development Discussions
>>> Subject: Re: [ckan-dev] Datastore setup for ckan 2.0
>>>
>>> Great, somehow I did not see this document :) thanks :)
>>>
>>> -----Original Message-----
>>> From: ckan-dev-bounces at lists.okfn.org
>>> [mailto:ckan-dev-bounces at lists.okfn.org] On Behalf Of Dominik Moritz
>>> Sent: 5. mars 2013 17:28
>>> To: CKAN Development Discussions
>>> Subject: Re: [ckan-dev] Datastore setup for ckan 2.0
>>>
>>> I hope this answers all your questions:
>>> http://docs.ckan.org/en/latest/datastore.html ;-)
>>>
>>> On 5 Mar 2013, at 16:18, Stefán Þór Finnsson <stefan.thor.finnsson at reykjavik.is> wrote:
>>>
>>>> Hi
>>>>
>>>> Thanks for this :) - I have change the ckanext/datastore/plugin.py file and now the system starts and the ckan page does work.
>>>>
>>>> But, I can find any good info about how to use the datastore compared to use the filestore, do I have to have both enabled? How do I put data to the datastore? Do I need the datastorer plugin or is there another way to use it?
>>>>
>>>> Would be nice if someone could clear this out for me :)
>>>>
>>>> -----Original Message-----
>>>> From: ckan-dev-bounces at lists.okfn.org
>>>> [mailto:ckan-dev-bounces at lists.okfn.org] On Behalf Of Dominik Moritz
>>>> Sent: 28. febrúar 2013 18:18
>>>> To: CKAN Development Discussions
>>>> Subject: Re: [ckan-dev] Datastore setup for ckan 2.0
>>>>
>>>> Hi,
>>>>
>>>> looks like your postgres version does not have support for the concat function. I submitted a pull request [1] that replaces the concat function. You could either cherry-pick the commit, get the patch or wait for it to be in master.
>>>>
>>>> I'm sorry that we did'n catch that during our tests.
>>>>
>>>> Cheers,
>>>> Dominik
>>>>
>>>> [1] https://github.com/okfn/ckan/pull/514
>>>>
>>>> On 28 Feb 2013, at 17:53, Stefán Þór Finnsson <stefan.thor.finnsson at reykjavik.is> wrote:
>>>>
>>>>> Hello :)
>>>>>
>>>>> When I have added the datastore extension and done the config for
>>>>> it I get this error in my ckan log and the system does not work.
>>>>> Without the datastore it works fine. I´m lost in what could be
>>>>> causing this error, anyone here with a quick answer for me :)
>>>>>
>>>>> [Thu Feb 28 16:46:48 2013] [info] [client 82.112.70.95] mod_wsgi (pid=28703, process='', application='ckan-test|'): Loading WSGI script '/usr/local/ckan-test/pyenv/bin/ckan-test.py'.
>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] mod_wsgi (pid=28703): Target WSGI script '/usr/local/ckan-test/pyenv/bin/ckan-test.py' cannot be loaded as Python module.
>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] mod_wsgi (pid=28703): Exception occurred processing WSGI script '/usr/local/ckan-test/pyenv/bin/ckan-test.py'.
>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] Traceback (most recent call last):
>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] File "/usr/local/ckan-test/pyenv/bin/ckan-test.py", line 11, in <module>
>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] application = loadapp('config:%s' % config_filepath)
>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] File "/usr/local/ckan-test/pyenv/lib/python2.6/site-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] return loadobj(APP, uri, name=name, **kw)
>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] File "/usr/local/ckan-test/pyenv/lib/python2.6/site-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] return context.create()
>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] File "/usr/local/ckan-test/pyenv/lib/python2.6/site-packages/paste/deploy/loadwsgi.py", line 710, in create
>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] return self.object_type.invoke(self)
>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] File "/usr/local/ckan-test/pyenv/lib/python2.6/site-packages/paste/deploy/loadwsgi.py", line 146, in invoke
>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] return fix_call(context.object, context.global_conf, **context.local_conf)
>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] File "/usr/local/ckan-test/pyenv/lib/python2.6/site-packages/paste/deploy/util.py", line 56, in fix_call
>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] val = callable(*args, **kw)
>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] File "/usr/local/ckan-test/pyenv/src/ckan/ckan/config/middleware.py", line 54, in make_app
>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] load_environment(global_conf, app_conf)
>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] File "/usr/local/ckan-test/pyenv/src/ckan/ckan/config/environment.py", line 349, in load_environment
>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] plugin.configure(config)
>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] File "/usr/local/ckan-test/pyenv/src/ckan/ckanext/datastore/plugin.py", line 68, in configure
>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] self._create_alias_table()
>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] File "/usr/local/ckan-test/pyenv/src/ckan/ckanext/datastore/plugin.py", line 206, in _create_alias_table
>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] connection.execute(create_alias_table_sql)
>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] File "/usr/local/ckan-test/pyenv/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 1449, in execute
>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] params)
>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] File "/usr/local/ckan-test/pyenv/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 1628, in _execute_text
>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] statement, parameters
>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] File "/usr/local/ckan-test/pyenv/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 1698, in _execute_context
>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] context)
>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] File "/usr/local/ckan-test/pyenv/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 1691, in _execute_context
>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] context)
>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] File "/usr/local/ckan-test/pyenv/lib/python2.6/site-packages/sqlalchemy/engine/default.py", line 331, in do_execute
>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] cursor.execute(statement, parameters)
>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] ProgrammingError: (ProgrammingError) function concat(name, name) does not exist
>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] LINE 3: substr(md5(concat(dependee.relname, dependen...
>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] ^
>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] HINT: No function matches the given name and argument types. You might need to add explicit type casts.
>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] 'CREATE OR REPLACE VIEW "_table_metadata" AS \\n SELECT DISTINCT\\n substr(md5(concat(dependee.relname, dependent.relname)), 0, 17) AS "_id",\\n dependee.relname AS name,\\n dependee.oid AS oid,\\n dependent.relname AS alias_of\\n -- dependent.oid AS oid\\n FROM\\n pg_class AS dependee\\n LEFT OUTER JOIN pg_rewrite AS r ON r.ev_class = dependee.oid\\n LEFT OUTER JOIN pg_depend AS d ON d.objid = r.oid\\n LEFT OUTER JOIN pg_class AS dependent ON d.refobjid = dependent.oid\\n WHERE\\n (dependee.oid != dependent.oid OR dependent.oid IS NULL) AND\\n (dependee.relname IN (SELECT tablename FROM pg_catalog.pg_tables)\\n OR dependee.relname IN (SELECT viewname FROM pg_catalog.pg_views)) AND\\n dependee.relnamespace = (SELECT oid FROM pg_namespace WHERE nspname=\\'public\\')\\n ORDER BY dependee.oid DESC;\\n ' {}
>>>>>
>>>>> __________________________________
>>>>> Reglur um trúnað í tölvupóstsamskiptum:
>>>>> http://www.reykjavik.is/trunadur
>>>>>
>>>>> Leiddu hugann að umhverfinu áður en þú prentar út þennan tölvupóst. Prentaðu á báðar hliðar í svarthvítu ef nauðsyn krefur.
>>>>> _______________________________________________
>>>>> ckan-dev mailing list
>>>>> ckan-dev at lists.okfn.org
>>>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>>>
>>>>
>>>> _______________________________________________
>>>> ckan-dev mailing list
>>>> ckan-dev at lists.okfn.org
>>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>>>
>>>> _______________________________________________
>>>> ckan-dev mailing list
>>>> ckan-dev at lists.okfn.org
>>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>>
>>>
>>> _______________________________________________
>>> ckan-dev mailing list
>>> ckan-dev at lists.okfn.org
>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>>
>>> _______________________________________________
>>> ckan-dev mailing list
>>> ckan-dev at lists.okfn.org
>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>>
>>> _______________________________________________
>>> ckan-dev mailing list
>>> ckan-dev at lists.okfn.org
>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
More information about the ckan-dev
mailing list