[ckan-dev] Datastore setup for ckan 2.0
Dominik Moritz
dominik.moritz at okfn.org
Thu Apr 11 18:44:37 UTC 2013
Hey,
so the first error says that the resource cannot be fetched. The server returns a 404.
The second error is because you need to specify a full url in your ini file. So, instead of `ckan.site_url = ckan-test`, use your ckan domain. For example `ckan.site_url = https://www.ckan.org`.
I think there should be a better error message than the one you got but at the moment we are building a replacement for the datastorer which will make everything much easier.
Hope that helps,
Dominik
On 10 Apr 2013, at 18:53, Stefán Þór Finnsson <stefan.thor.finnsson at reykjavik.is> wrote:
> Sending this again, dont see the attached file I sent last time with the post, the ini config is after this log error.
>
> Here is what the log files says:
>
> Here is what the log says:
>
> [2013-03-22 12:47:19,014: INFO/MainProcess] Got task from broker: datastorer.upload[3922d5da-88bc-4d67-9348-8e3c11646415]
> [2013-03-22 12:47:19,031: INFO/PoolWorker-1] Starting new HTTP connection (1): ckan-test
> [2013-03-22 12:47:19,060: ERROR/PoolWorker-1] datastorer.upload[3922d5da-88bc-4d67-9348-8e3c11646415]: Error while performing download: LinkHeadRequestError('URL unobtainable: Server returned HTTP 404',). Retrying...
> [2013-03-22 12:47:19,062: INFO/PoolWorker-1] Starting new HTTP connection (1): ckan-test
> [2013-03-22 12:47:19,090: ERROR/PoolWorker-1] datastorer.upload[3922d5da-88bc-4d67-9348-8e3c11646415]: Error while performing download: LinkHeadRequestError('URL unobtainable: Server returned HTTP 404',). Retrying...
> [2013-03-22 12:47:24,096: INFO/PoolWorker-1] Starting new HTTP connection (1): ckan-test
> [2013-03-22 12:47:24,925: ERROR/PoolWorker-1] datastorer.upload[3922d5da-88bc-4d67-9348-8e3c11646415]: Error while performing download: LinkHeadRequestError('URL unobtainable: Server returned HTTP 404',). Retrying...
> [2013-03-22 12:47:34,928: INFO/PoolWorker-1] Starting new HTTP connection (1): ckan-test
> [2013-03-22 12:47:34,966: ERROR/PoolWorker-1] datastorer.upload[3922d5da-88bc-4d67-9348-8e3c11646415]: Error while performing download: LinkHeadRequestError('URL unobtainable: Server returned HTTP 404',). Retrying...
> [2013-03-22 12:47:49,983: INFO/PoolWorker-1] Starting new HTTP connection (1): ckan-test
> [2013-03-22 12:47:50,016: ERROR/PoolWorker-1] datastorer.upload[3922d5da-88bc-4d67-9348-8e3c11646415]: Error while performing download: LinkHeadRequestError('URL unobtainable: Server returned HTTP 404',). Retrying...
> [2013-03-22 12:48:10,099: ERROR/MainProcess] Task datastorer.upload[3922d5da-88bc-4d67-9348-8e3c11646415] raised exception: MissingSchema("Invalid URL u'api/action/task_status_update': No schema supplied",) Traceback (most recent call last):
> File "/usr/local/ckan-test/pyenv/lib/python2.6/site-packages/celery/execute/trace.py", line 47, in trace
> return cls(states.SUCCESS, retval=fun(*args, **kwargs))
> File "/usr/local/ckan-test/pyenv/lib/python2.6/site-packages/celery/app/task/__init__.py", line 247, in __call__
> return self.run(*args, **kwargs)
> File "/usr/local/ckan-test/pyenv/lib/python2.6/site-packages/celery/app/__init__.py", line 175, in run
> return fun(*args, **kwargs)
> File "/usr/local/ckan-test/pyenv/src/ckanext-datastorer/ckanext/datastorer/tasks.py", line 118, in datastorer_upload
> }, logger)
> File "/usr/local/ckan-test/pyenv/src/ckanext-archiver/ckanext/archiver/tasks.py", line 474, in update_task_status
> 'Content-Type': 'application/json'}
> File "/usr/local/ckan-test/pyenv/lib/python2.6/site-packages/requests/api.py", line 98, in post
> return request('post', url, data=data, **kwargs)
> File "/usr/local/ckan-test/pyenv/lib/python2.6/site-packages/requests/safe_mode.py", line 39, in wrapped
> return function(method, url, **kwargs)
> File "/usr/local/ckan-test/pyenv/lib/python2.6/site-packages/requests/api.py", line 51, in request
> return session.request(method=method, url=url, **kwargs)
> File "/usr/local/ckan-test/pyenv/lib/python2.6/site-packages/requests/sessions.py", line 252, in request
> r.send(prefetch=prefetch)
> File "/usr/local/ckan-test/pyenv/lib/python2.6/site-packages/requests/models.py", line 490, in send
> url = self.full_url
> File "/usr/local/ckan-test/pyenv/lib/python2.6/site-packages/requests/models.py", line 393, in full_url
> raise MissingSchema("Invalid URL %r: No schema supplied" % url)
> MissingSchema: Invalid URL u'api/action/task_status_update': No schema supplied
>
>
> And here is the ini file without users for postgres:
>
> #
> # ckan - Pylons configuration
> #
> # The %(here)s variable will be replaced with the parent directory of this file
> #
> [DEFAULT]
>
> # Change debug to true when doing CKAN development, it enables Pylons'
> # interactive debugging tool, makes Fanstatic serve unminified JS and CSS
> # files, and enables CKAN templates' debugging features.
> #
> # WARNING: *THIS SETTING MUST BE SET TO FALSE ON A PRODUCTION ENVIRONMENT*
> # Debug mode will enable the interactive debugging tool, allowing ANYONE to
> # execute malicious code after an exception is raised.
> debug = false
>
> email_to = you at yourdomain.com
> error_email_from = paste at localhost
>
> # The SMTP server to connect to when sending emails to users, with optional
> # port (default: 25). For example: lavabit.com, or smtp.gmail.com:587. To
> # connect to a local sendmail process enter 'localhost'.
> smtp.server = localhost
>
> # Whether or not to use STARTTLS when connecting to the SMTP server.
> smtp.starttls = False
>
> # The username and password to use to authenticate with the SMTP server.
> # Optional, if these are left commented-out then CKAN will try to send email
> # to the SMTP server without logging in first.
> #smtp.user = your_username at gmail.com
> #smtp.password = your_password
>
> # The email address that emails sent by CKAN to users should appear to come
> # from. Optional (default: None, note that the SMTP server may insert its own
> # from address).
> #smtp.mail_from =
>
> [server:main]
> use = egg:Paste#http
> host = ckan-test
> port = 5000
>
> [app:main]
> # Required
> ## 'Bucket' (subdirectory for file based storage) to use for file storage
> ckan.storage.bucket = ck
>
> ## Optional
> ## maximum content size for uploads in bytes, defaults to 1Gb
> # ckanext.storage.max_content_length = 1000000000
>
> ## OFS configuration
> ofs.impl = pairtree
> # directory on disk for data storage (should be empty)
> ofs.storage_dir = /var/local/ckan
>
> use = egg:ckan
> full_stack = true
> cache_dir = %(here)s/data
> beaker.session.key = ckan
> beaker.session.secret =
> app_instance_uuid = {af3de861-f18b-48b5-99a7-b9c50a7d179b}
>
> # List the names of CKAN extensions to activate.
> # Note: This line is required to be here for packaging, even if it is empty.
> # Note: Add ``pdf_preview`` to enable the resource preview for PDFs
> # Add the ``resource_proxy`` plugin to enable resorce proxying and get around the same origin policy
> # Add ``datastore`` to enable the CKAN DataStore extension
> ckan.plugins = stats synchronous_search datastore datastorer pdf_preview
>
> # If you'd like to fine-tune the individual locations of the cache data dirs
> # for the Cache data, or the Session saves, un-comment the desired settings
> # here:
> #beaker.cache.data_dir = %(here)s/data/cache
> #beaker.session.data_dir = %(here)s/data/sessions
>
> # Specify the database for SQLAlchemy to use:
> # * Postgres is currently required for a production CKAN deployment
> # * Sqlite (memory or file) can be used as a quick alternative for testing
> sqlalchemy.url = postgresql://:@localhost/ckantest
> #sqlalchemy.url = sqlite:///
> #sqlalchemy.url = sqlite:///%(here)s/somedb.db
>
> # Un-comment and specify the URLs for the DataStore database.
> # * Postgres is required
> ckan.datastore.write_url = postgresql://:@localhost/datastore
> ckan.datastore.read_url = postgresql://:@localhost/datastore
>
> # repoze.who config
> who.config_file = %(here)s/who.ini
> who.log_level = warning
> who.log_file = %(cache_dir)s/who_log.ini
>
> # Location of RDF versions of datasets
> #rdf_packages = http://semantic.ckan.net/record/
>
> # Location of licenses group (defaults to cached local version of ckan group)
> #licenses_group_url = http://licenses.opendefinition.org/licenses/groups/ckan.json
>
> # Dataset form to use
> package_form = standard
>
> # Hide certain extras fields from dataset read form:
> # package_hide_extras = for_search_index_only
>
> # API configuration
> #apikey_header_name = X-CKAN-API-Key
>
> ## extra places to look for templates and public files (comma separated lists)
> ## any templates/files found will override correspondingly named ones in
> ## ckan/templates/ and ckan/public
> ## (e.g. to override main layout template layout.html or add extra css files)
> # extra_template_paths = %(here)s/my-templates
> # extra_public_paths = %(here)s/my-public
>
> # Dataset form integration
> #package_edit_return_url = http://another.frontend/dataset/<NAME>
> #package_new_return_url = http://another.frontend/dataset/<NAME>
>
>
> # Turn on messaging with carrot, default to false
> #ckan.async_notifier = true
> # Messaging module used by carrot:
> # * pyamqplib - AMQP (e.g. for RabbitMQ)
> # * queue - native Python Queue (debugging and tests only)
> #carrot_messaging_library = pyamqplib
>
> ## Perform search just using database (rather than use e.g. solr).
> ## In this setup search is crude and limited .e.g no full-text search, no faceting ...
> ## However, very useful for getting up and running quickly with CKAN
> # ckan.simple_search = 1
>
> ## Title of site (using in several places including templates and <title> tag
> ckan.site_title = Opin gogn hja
>
> ## Logo image to use on the home page
> ckan.site_logo = /base/images/merki.jpg
>
> ## Site tagline / description (used on front page)
> ckan.site_description = Opin gogn fyrir alla, til notkunar eda dreifingar
>
> ## Used in creating some absolute urls (such as rss feeds, css files) and
> ## dump filenames
> ckan.site_url = ckan-test
> ## Favicon (default is the CKAN software favicon)
> ckan.favicon = /images/icons/ckan.ico
>
> ## The gravatar default to use. This can be any of the pre-defined strings
> ## as defined on http://en.gravatar.com/site/implement/images/ (e.g. "identicon"
> ## or "mm"). Or it can be a url, e.g. "http://example.com/images/avatar.jpg"
> ckan.gravatar_default = identicon
>
> ## Solr support
> solr_url = http://127.0.0.1:8080/solr/ckan-schema-1.4
>
> ## Automatic indexing. Make all changes immediately available via the search
> ## after editing or creating a dataset. Default is true. If for some reason
> ## you need the indexing to occur asynchronously, set this option to 0.
> # ckan.search.automatic_indexing = 1
>
> ## An 'id' for the site (using, for example, when creating entries in a common search index)
> ## If not specified derived from the site_url
> ckan.site_id = ckan-test
>
> ## API url to use (e.g. in AJAX callbacks)
> ## Enable if the API is at a different domain
> # ckan.api_url = http://www.ckan.net
>
> ## html content to be inserted just before </head> tag (e.g. extra stylesheet)
> ## NB: can use html e.g. <strong>blah</strong>
> ## NB: can have multiline strings just indent following lines
> # ckan.template_head_end = <link rel="stylesheet" href="http://mysite.org/css/custom.css" type="text/css">
>
> ## html content to be inserted just before </body> tag (e.g. google analytics code)
> ## NB: can use html e.g. <strong>blah</strong>
> ## NB: can have multiline strings just indent following lines
> # ckan.template_footer_end =
>
> # These three settings (ckan.log_dir, ckan.dump_dir and ckan.backup_dir) are
> # all used in cron jobs, not in CKAN itself. CKAN logging is configured
> # in the logging configuration below
> # Directory for logs (produced by cron scripts associated with ckan)
> ckan.log_dir = %(here)s/log
> # Directory for JSON/CSV dumps (must match setting in apache config)
> ckan.dump_dir = %(here)s/dump
> # Directory for SQL database backups
> ckan.backup_dir = %(here)s/backup
>
> # Default authorizations for new domain objects
> #ckan.default_roles.Package = {"visitor": ["reader"], "logged_in": ["reader"]}
> #ckan.default_roles.Group = {"visitor": ["reader"], "logged_in": ["reader"]}
> #ckan.default_roles.System = {"visitor": ["reader"], "logged_in": ["editor"]}
> #ckan.default_roles.AuthorizationGroup = {"visitor": ["reader"], "logged_in": ["reader"]}
>
> ## Ckan public and private recaptcha keys [localhost]
> #ckan.recaptcha.publickey =
> #ckan.recaptcha.privatekey =
>
> # Locale/languages
> ckan.locale_default = en
> #ckan.locales_offered =
> # Languages are grouped by percentage of strings in CKAN 1.8 translated
> # (those with 100% first, then those with >=80%, then >=50%, then <50%) and
> # within these groups roughly sorted by number of worldwide native speakers
> # according to Wikipedia.
> ckan.locale_order = en pt_BR ja it cs_CZ ca es fr el sv sr sr at latin no sk fi ru de pl nl bg ko_KR hu sa sl lv
> ckan.locales_filtered_out =
>
> ## Atom Feeds
> #
> # Settings for customising the metadata provided in
> # atom feeds.
> #
> # These settings are used to generate the <id> tags for both feeds
> # and entries. The unique <id>s are created following the method
> # outlined in http://www.taguri.org/ ie - they generate tagURIs, as specified
> # in http://tools.ietf.org/html/rfc4151#section-2.1 :
> #
> # <id>tag:thedatahub.org,2012:/feeds/group/933f3857-79fd-4beb-a835-c0349e31ce76</id>
> #
> # Each component has the corresponding settings:
> #
> # "thedatahub.org" is ckan.feeds.authority_name
> # "2012" is ckan.feeds.date
> #
>
> # Leave blank to use the ckan.site_url config value, otherwise set to a
> # domain or email address that you own. e.g. thedatahub.org or
> # admin at thedatahub.org
> ckan.feeds.authority_name =
>
> # Pick a date of the form "yyyy[-mm[-dd]]" during which the above domain was
> # owned by you.
> ckan.feeds.date = 2012
>
> # If not set, then the value in `ckan.site_id` is used.
> ckan.feeds.author_name =
>
> # If not set, then the value in `ckan.site_url` is used.
> ckan.feeds.author_link =
>
> ## File Store
> #
> # CKAN allows users to upload files directly to file storage either on the local
> # file system or to online âcloudâ storage like Amazon S3 or Google Storage.
> #
> # If you are using local file storage, remember to set ckan.site_url.
> #
> # To enable cloud storage (Google or S3), first run: pip install boto
> #
> # @see http://docs.ckan.org/en/latest/filestore.html
>
> # 'Bucket' to use for file storage
> #ckan.storage.bucket = my-bucket-name
>
> # To enable local file storage:
> #ofs.impl = pairtree
> #ofs.storage_dir = /my/path/to/storage/root/directory
>
> # To enable Google cloud storage:
> #ofs.impl = google
> #ofs.gs_access_key_id =
> #ofs.gs_secret_access_key =
>
> # To enable S3 cloud storage:
> #ofs.impl = s3
> #ofs.aws_access_key_id = ....
> #ofs.aws_secret_access_key = ....
>
>
> ## Previews
> #
> # Set the file types that should be previewed inline (e.g. images) or directly in an iframe.
>
> ckan.preview.direct = png jpg gif
> ckan.preview.loadable = html htm rdf+xml owl+xml xml n3 n-triples turtle plain atom csv tsv rss txt json
>
>
> # Activity Streams
> #
> # Default maximum number of activities to show in an activity stream.
> # ckan.activity_list_limit = 31
>
>
> # Activity Streams Email Notifications
> #
> # Uncomment this line to enable activity streams email notifications.
> # You also need to setup a cron job to send the emails, see the documentation.
> #ckan.activity_streams_email_notifications = True
>
> # Email notifications for events older than this time delta will not be sent.
> # Accepted formats: '2 days', '14 days', '4:35:00' (hours, minutes, seconds),
> # '7 days, 3:23:34', etc.
> # ckan.email_notifications_since = 2 days
>
>
> # DEBUGGING
>
> # ckan.debug_supress_header This option can be set to suppress the debug
> # information showing the controller and action recieving the request being
> # shown in the header. Note: This info only shows if debug is set to true.
> ckan.debug_supress_header = false
>
> ## ===================================
> ## Extensions
>
> ## Config option to enable the (1 day) cache for stats
> ## Default (if not defined) is True as stats computations are intensive
> # ckanext.stats.cache_enabled = True
>
> # Logging configuration
> [loggers]
> keys = root, ckan, ckanext
>
> [handlers]
> keys = console, file
>
> [formatters]
> keys = generic
>
> [logger_root]
> level = WARNING
> handlers = console, file
>
> [logger_ckan]
> level = INFO
> handlers = console, file
> qualname = ckan
> propagate = 0
>
> [logger_ckanext]
> level = DEBUG
> handlers = console, file
> qualname = ckanext
> propagate = 0
>
> [handler_console]
> class = StreamHandler
> args = (sys.stderr,)
> level = NOTSET
> formatter = generic
>
> [handler_file]
> class = logging.handlers.RotatingFileHandler
> formatter = generic
> level = NOTSET
> args = ("/var/log/ckan/ckan-test/ckan.log", "a", 20000000, 9)
>
> [formatter_generic]
> format = %(asctime)s %(levelname)-5.5s [%(name)s] %(message)s
>
> -----Original Message-----
> From: Stefán Þór Finnsson
> Sent: 22. mars 2013 16:43
> To: 'CKAN Development Discussions'
> Subject: RE: [ckan-dev] Datastore setup for ckan 2.0
>
> Ok - changed the branch and see that the plugin.py has changed. But still get this error.
>
> Here is what the log says:
>
> [2013-03-22 12:47:19,014: INFO/MainProcess] Got task from broker: datastorer.upload[3922d5da-88bc-4d67-9348-8e3c11646415]
> [2013-03-22 12:47:19,031: INFO/PoolWorker-1] Starting new HTTP connection (1): ckan-test
> [2013-03-22 12:47:19,060: ERROR/PoolWorker-1] datastorer.upload[3922d5da-88bc-4d67-9348-8e3c11646415]: Error while performing download: LinkHeadRequestError('URL unobtainable: Server returned HTTP 404',). Retrying...
> [2013-03-22 12:47:19,062: INFO/PoolWorker-1] Starting new HTTP connection (1): ckan-test
> [2013-03-22 12:47:19,090: ERROR/PoolWorker-1] datastorer.upload[3922d5da-88bc-4d67-9348-8e3c11646415]: Error while performing download: LinkHeadRequestError('URL unobtainable: Server returned HTTP 404',). Retrying...
> [2013-03-22 12:47:24,096: INFO/PoolWorker-1] Starting new HTTP connection (1): ckan-test
> [2013-03-22 12:47:24,925: ERROR/PoolWorker-1] datastorer.upload[3922d5da-88bc-4d67-9348-8e3c11646415]: Error while performing download: LinkHeadRequestError('URL unobtainable: Server returned HTTP 404',). Retrying...
> [2013-03-22 12:47:34,928: INFO/PoolWorker-1] Starting new HTTP connection (1): ckan-test
> [2013-03-22 12:47:34,966: ERROR/PoolWorker-1] datastorer.upload[3922d5da-88bc-4d67-9348-8e3c11646415]: Error while performing download: LinkHeadRequestError('URL unobtainable: Server returned HTTP 404',). Retrying...
> [2013-03-22 12:47:49,983: INFO/PoolWorker-1] Starting new HTTP connection (1): ckan-test
> [2013-03-22 12:47:50,016: ERROR/PoolWorker-1] datastorer.upload[3922d5da-88bc-4d67-9348-8e3c11646415]: Error while performing download: LinkHeadRequestError('URL unobtainable: Server returned HTTP 404',). Retrying...
> [2013-03-22 12:48:10,099: ERROR/MainProcess] Task datastorer.upload[3922d5da-88bc-4d67-9348-8e3c11646415] raised exception: MissingSchema("Invalid URL u'api/action/task_status_update': No schema supplied",) Traceback (most recent call last):
> File "/usr/local/ckan-test/pyenv/lib/python2.6/site-packages/celery/execute/trace.py", line 47, in trace
> return cls(states.SUCCESS, retval=fun(*args, **kwargs))
> File "/usr/local/ckan-test/pyenv/lib/python2.6/site-packages/celery/app/task/__init__.py", line 247, in __call__
> return self.run(*args, **kwargs)
> File "/usr/local/ckan-test/pyenv/lib/python2.6/site-packages/celery/app/__init__.py", line 175, in run
> return fun(*args, **kwargs)
> File "/usr/local/ckan-test/pyenv/src/ckanext-datastorer/ckanext/datastorer/tasks.py", line 118, in datastorer_upload
> }, logger)
> File "/usr/local/ckan-test/pyenv/src/ckanext-archiver/ckanext/archiver/tasks.py", line 474, in update_task_status
> 'Content-Type': 'application/json'}
> File "/usr/local/ckan-test/pyenv/lib/python2.6/site-packages/requests/api.py", line 98, in post
> return request('post', url, data=data, **kwargs)
> File "/usr/local/ckan-test/pyenv/lib/python2.6/site-packages/requests/safe_mode.py", line 39, in wrapped
> return function(method, url, **kwargs)
> File "/usr/local/ckan-test/pyenv/lib/python2.6/site-packages/requests/api.py", line 51, in request
> return session.request(method=method, url=url, **kwargs)
> File "/usr/local/ckan-test/pyenv/lib/python2.6/site-packages/requests/sessions.py", line 252, in request
> r.send(prefetch=prefetch)
> File "/usr/local/ckan-test/pyenv/lib/python2.6/site-packages/requests/models.py", line 490, in send
> url = self.full_url
> File "/usr/local/ckan-test/pyenv/lib/python2.6/site-packages/requests/models.py", line 393, in full_url
> raise MissingSchema("Invalid URL %r: No schema supplied" % url)
> MissingSchema: Invalid URL u'api/action/task_status_update': No schema supplied
>
> The ini is as attachment.
>
>
>
>
>
>
>
>
> -----Original Message-----
> From: ckan-dev-bounces at lists.okfn.org [mailto:ckan-dev-bounces at lists.okfn.org] On Behalf Of Dominik Moritz
> Sent: 21. mars 2013 20:51
> To: CKAN Development Discussions
> Subject: Re: [ckan-dev] Datastore setup for ckan 2.0
>
> You can try to change to the branch if you have installed the datastore via github. If not, you will need to patch `plugin.py`. There is no need to change the tasks.py since you do not need the download retry code.
>
> On 21 Mar 2013, at 17:22, Stefán Þór Finnsson <stefan.thor.finnsson at reykjavik.is> wrote:
>
>> Hi Dominik
>>
>> No worries, thanks for this?
>>
>> Do you want me to replace the plugin.py and make the tasks.py with the one from the pull request?
>>
>> -----Original Message-----
>> From: ckan-dev-bounces at lists.okfn.org
>> [mailto:ckan-dev-bounces at lists.okfn.org] On Behalf Of Dominik Moritz
>> Sent: 21. mars 2013 08:47
>> To: CKAN Development Discussions
>> Subject: Re: [ckan-dev] Datastore setup for ckan 2.0
>>
>> Hi,
>>
>> sorry that my answer took so long. Can you try the version from the pull request at https://github.com/okfn/ckanext-datastorer/pull/21? It looks like it could fix the problem.
>>
>> If not, could you send me your .ini file (without the passwords for the database or any secret keys ;-)).
>>
>> Cheers,
>> Dominik
>>
>> On 14 Mar 2013, at 16:37, Stefán Þór Finnsson <stefan.thor.finnsson at reykjavik.is> wrote:
>>
>>> Still having this problem, did try to edit deployment.ini, but now
>>> luck, if I leave the ckan.site_url = empty I see when I try to upload
>>> a file that it does not add the ckan-test but then I get some authentication error, it ask me to login agin and goes in to a loop.
>>>
>>> Would be nice if someone could help me see the light here :)
>>>
>>> -----Original Message-----
>>> From: ckan-dev-bounces at lists.okfn.org
>>> [mailto:ckan-dev-bounces at lists.okfn.org] On Behalf Of Stefán Þór
>>> Finnsson
>>> Sent: 11. mars 2013 13:36
>>> To: CKAN Development Discussions
>>> Subject: Re: [ckan-dev] Datastore setup for ckan 2.0
>>>
>>> Hi,
>>>
>>> I did see error´s now when the celery is running:
>>>
>>> [2013-03-11 10:36:40,829: ERROR/MainProcess] Task datastorer.upload[41e99198-a0be-43a0-b4d6-54eb4a9b9d80] raised exception: LinkHeadRequestError('URL unobtainable: Server returned HTTP 404',) Traceback (most recent call last):
>>> File "/usr/local/ckan-test/pyenv/lib/python2.6/site-packages/celery/execute/trace.py", line 47, in trace
>>> return cls(states.SUCCESS, retval=fun(*args, **kwargs)) File
>>> "/usr/local/ckan-test/pyenv/lib/python2.6/site-packages/celery/app/task/__init__.py", line 247, in __call__
>>> return self.run(*args, **kwargs)
>>> File "/usr/local/ckan-test/pyenv/lib/python2.6/site-packages/celery/app/__init__.py", line 175, in run
>>> return fun(*args, **kwargs)
>>> File "/usr/local/ckan-test/pyenv/src/ckanext-datastorer/ckanext/datastorer/tasks.py", line 108, in datastorer_upload
>>> return _datastorer_upload(context, data, logger) File
>>> "/usr/local/ckan-test/pyenv/src/ckanext-datastorer/ckanext/datastorer/tasks.py", line 123, in _datastorer_upload
>>> result = download(context, resource, data_formats=DATA_FORMATS)
>>> File "/usr/local/ckan-test/pyenv/src/ckanext-archiver/ckanext/archiver/tasks.py", line 103, in download
>>> headers = json.loads(link_checker(link_context, link_data)) File
>>> "/usr/local/ckan-test/pyenv/lib/python2.6/site-packages/celery/app/task/__init__.py", line 247, in __call__
>>> return self.run(*args, **kwargs)
>>> File "/usr/local/ckan-test/pyenv/lib/python2.6/site-packages/celery/app/__init__.py", line 175, in run
>>> return fun(*args, **kwargs)
>>> File "/usr/local/ckan-test/pyenv/src/ckanext-archiver/ckanext/archiver/tasks.py", line 361, in link_checker
>>> raise LinkHeadRequestError(error_message)
>>> LinkHeadRequestError: URL unobtainable: Server returned HTTP 404
>>>
>>> When I go to download for the document I get this path:
>>> http://ckan-test/ckan-test/storage/f/2013-03-11T10%3A36%3A28.446Z/cka
>>> n
>>> test.csv
>>>
>>> and get 404 error, But if I take the extra ckan-test I can download the document, not sure where it gets the parameter to add this ckan-test in the url, is it from the ini file or some other config?
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: ckan-dev-bounces at lists.okfn.org
>>> [mailto:ckan-dev-bounces at lists.okfn.org] On Behalf Of Dominik Moritz
>>> Sent: 8. mars 2013 15:26
>>> To: CKAN Development Discussions
>>> Subject: Re: [ckan-dev] Datastore setup for ckan 2.0
>>>
>>> Hi,
>>>
>>> could you go into a little bit more detail what you've done? Have you installed the datastorer properly (see https://github.com/okfn/ckanext-datastorer)? Did you wait for the datastorer to finish its work? Is the recline preview enabled in your .ini file?
>>>
>>> Dominik
>>>
>>> On 7 Mar 2013, at 15:59, Stefán Þór Finnsson <stefan.thor.finnsson at reykjavik.is> wrote:
>>>
>>>> The fun goes on :) When I click preview for csv or xlsx file it get this:
>>>>
>>>> This resource cannot be previewed. More details...
>>>> No handler defined for data type: xls.
>>>>
>>>> When filestore is in place I can upload files and download them,
>>>> that works
>>>>
>>>> When I have added datastore to the party and running celeryd I should see preview of the file and be able to do something with it on the page? Or not? I´m in the dark with the functions?
>>>>
>>>> -----Original Message-----
>>>> From: ckan-dev-bounces at lists.okfn.org
>>>> [mailto:ckan-dev-bounces at lists.okfn.org] On Behalf Of Stefán Þór
>>>> Finnsson
>>>> Sent: 6. mars 2013 14:19
>>>> To: CKAN Development Discussions
>>>> Subject: Re: [ckan-dev] Datastore setup for ckan 2.0
>>>>
>>>> Great, somehow I did not see this document :) thanks :)
>>>>
>>>> -----Original Message-----
>>>> From: ckan-dev-bounces at lists.okfn.org
>>>> [mailto:ckan-dev-bounces at lists.okfn.org] On Behalf Of Dominik Moritz
>>>> Sent: 5. mars 2013 17:28
>>>> To: CKAN Development Discussions
>>>> Subject: Re: [ckan-dev] Datastore setup for ckan 2.0
>>>>
>>>> I hope this answers all your questions:
>>>> http://docs.ckan.org/en/latest/datastore.html ;-)
>>>>
>>>> On 5 Mar 2013, at 16:18, Stefán Þór Finnsson <stefan.thor.finnsson at reykjavik.is> wrote:
>>>>
>>>>> Hi
>>>>>
>>>>> Thanks for this :) - I have change the ckanext/datastore/plugin.py file and now the system starts and the ckan page does work.
>>>>>
>>>>> But, I can find any good info about how to use the datastore compared to use the filestore, do I have to have both enabled? How do I put data to the datastore? Do I need the datastorer plugin or is there another way to use it?
>>>>>
>>>>> Would be nice if someone could clear this out for me :)
>>>>>
>>>>> -----Original Message-----
>>>>> From: ckan-dev-bounces at lists.okfn.org
>>>>> [mailto:ckan-dev-bounces at lists.okfn.org] On Behalf Of Dominik
>>>>> Moritz
>>>>> Sent: 28. febrúar 2013 18:18
>>>>> To: CKAN Development Discussions
>>>>> Subject: Re: [ckan-dev] Datastore setup for ckan 2.0
>>>>>
>>>>> Hi,
>>>>>
>>>>> looks like your postgres version does not have support for the concat function. I submitted a pull request [1] that replaces the concat function. You could either cherry-pick the commit, get the patch or wait for it to be in master.
>>>>>
>>>>> I'm sorry that we did'n catch that during our tests.
>>>>>
>>>>> Cheers,
>>>>> Dominik
>>>>>
>>>>> [1] https://github.com/okfn/ckan/pull/514
>>>>>
>>>>> On 28 Feb 2013, at 17:53, Stefán Þór Finnsson <stefan.thor.finnsson at reykjavik.is> wrote:
>>>>>
>>>>>> Hello :)
>>>>>>
>>>>>> When I have added the datastore extension and done the config for
>>>>>> it I get this error in my ckan log and the system does not work.
>>>>>> Without the datastore it works fine. I´m lost in what could be
>>>>>> causing this error, anyone here with a quick answer for me :)
>>>>>>
>>>>>> [Thu Feb 28 16:46:48 2013] [info] [client 82.112.70.95] mod_wsgi (pid=28703, process='', application='ckan-test|'): Loading WSGI script '/usr/local/ckan-test/pyenv/bin/ckan-test.py'.
>>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] mod_wsgi (pid=28703): Target WSGI script '/usr/local/ckan-test/pyenv/bin/ckan-test.py' cannot be loaded as Python module.
>>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] mod_wsgi (pid=28703): Exception occurred processing WSGI script '/usr/local/ckan-test/pyenv/bin/ckan-test.py'.
>>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] Traceback (most recent call last):
>>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] File "/usr/local/ckan-test/pyenv/bin/ckan-test.py", line 11, in <module>
>>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] application = loadapp('config:%s' % config_filepath)
>>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] File "/usr/local/ckan-test/pyenv/lib/python2.6/site-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
>>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] return loadobj(APP, uri, name=name, **kw)
>>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] File "/usr/local/ckan-test/pyenv/lib/python2.6/site-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
>>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] return context.create()
>>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] File "/usr/local/ckan-test/pyenv/lib/python2.6/site-packages/paste/deploy/loadwsgi.py", line 710, in create
>>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] return self.object_type.invoke(self)
>>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] File "/usr/local/ckan-test/pyenv/lib/python2.6/site-packages/paste/deploy/loadwsgi.py", line 146, in invoke
>>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] return fix_call(context.object, context.global_conf, **context.local_conf)
>>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] File "/usr/local/ckan-test/pyenv/lib/python2.6/site-packages/paste/deploy/util.py", line 56, in fix_call
>>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] val = callable(*args, **kw)
>>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] File "/usr/local/ckan-test/pyenv/src/ckan/ckan/config/middleware.py", line 54, in make_app
>>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] load_environment(global_conf, app_conf)
>>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] File "/usr/local/ckan-test/pyenv/src/ckan/ckan/config/environment.py", line 349, in load_environment
>>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] plugin.configure(config)
>>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] File "/usr/local/ckan-test/pyenv/src/ckan/ckanext/datastore/plugin.py", line 68, in configure
>>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] self._create_alias_table()
>>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] File "/usr/local/ckan-test/pyenv/src/ckan/ckanext/datastore/plugin.py", line 206, in _create_alias_table
>>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] connection.execute(create_alias_table_sql)
>>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] File "/usr/local/ckan-test/pyenv/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 1449, in execute
>>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] params)
>>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] File "/usr/local/ckan-test/pyenv/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 1628, in _execute_text
>>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] statement, parameters
>>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] File "/usr/local/ckan-test/pyenv/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 1698, in _execute_context
>>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] context)
>>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] File "/usr/local/ckan-test/pyenv/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 1691, in _execute_context
>>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] context)
>>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] File "/usr/local/ckan-test/pyenv/lib/python2.6/site-packages/sqlalchemy/engine/default.py", line 331, in do_execute
>>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] cursor.execute(statement, parameters)
>>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] ProgrammingError: (ProgrammingError) function concat(name, name) does not exist
>>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] LINE 3: substr(md5(concat(dependee.relname, dependen...
>>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] ^
>>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] HINT: No function matches the given name and argument types. You might need to add explicit type casts.
>>>>>> [Thu Feb 28 16:46:49 2013] [error] [client 82.112.70.95] 'CREATE OR REPLACE VIEW "_table_metadata" AS \\n SELECT DISTINCT\\n substr(md5(concat(dependee.relname, dependent.relname)), 0, 17) AS "_id",\\n dependee.relname AS name,\\n dependee.oid AS oid,\\n dependent.relname AS alias_of\\n -- dependent.oid AS oid\\n FROM\\n pg_class AS dependee\\n LEFT OUTER JOIN pg_rewrite AS r ON r.ev_class = dependee.oid\\n LEFT OUTER JOIN pg_depend AS d ON d.objid = r.oid\\n LEFT OUTER JOIN pg_class AS dependent ON d.refobjid = dependent.oid\\n WHERE\\n (dependee.oid != dependent.oid OR dependent.oid IS NULL) AND\\n (dependee.relname IN (SELECT tablename FROM pg_catalog.pg_tables)\\n OR dependee.relname IN (SELECT viewname FROM pg_catalog.pg_views)) AND\\n dependee.relnamespace = (SELECT oid FROM pg_namespace WHERE nspname=\\'public\\')\\n ORDER BY dependee.oid DESC;\\n ' {}
>>>>>>
>>>>>> __________________________________
>>>>>> Reglur um trúnað í tölvupóstsamskiptum:
>>>>>> http://www.reykjavik.is/trunadur
>>>>>>
>>>>>> Leiddu hugann að umhverfinu áður en þú prentar út þennan tölvupóst. Prentaðu á báðar hliðar í svarthvítu ef nauðsyn krefur.
>>>>>> _______________________________________________
>>>>>> ckan-dev mailing list
>>>>>> ckan-dev at lists.okfn.org
>>>>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>>>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> ckan-dev mailing list
>>>>> ckan-dev at lists.okfn.org
>>>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>>>>
>>>>> _______________________________________________
>>>>> ckan-dev mailing list
>>>>> ckan-dev at lists.okfn.org
>>>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>>>
>>>>
>>>> _______________________________________________
>>>> ckan-dev mailing list
>>>> ckan-dev at lists.okfn.org
>>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>>>
>>>> _______________________________________________
>>>> ckan-dev mailing list
>>>> ckan-dev at lists.okfn.org
>>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>>>
>>>> _______________________________________________
>>>> ckan-dev mailing list
>>>> ckan-dev at lists.okfn.org
>>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>>
>>>
>>> _______________________________________________
>>> ckan-dev mailing list
>>> ckan-dev at lists.okfn.org
>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>>
>>> _______________________________________________
>>> ckan-dev mailing list
>>> ckan-dev at lists.okfn.org
>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>>
>>> _______________________________________________
>>> ckan-dev mailing list
>>> ckan-dev at lists.okfn.org
>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
More information about the ckan-dev
mailing list