[ckan-dev] Proxy server crashed and now static files fails

Ross Jones ross at servercode.co.uk
Tue Mar 25 13:37:12 UTC 2014


Hi Henrik,

The persisted_state.json* file is missing. I haven’t tested it, but apparently https://github.com/ckan/ofs-hero is there to fix it.

If you have the option you should upgrade to a newer CKAN so that you no longer need to use the old (dangerous) file storage.

Ross

*This wasn’t intended for production use but apparently snuck in at some point in the past.


On 25 Mar 2014, at 13:30, Henrik Aagaard Sørensen <BU1G at tmf.kk.dk> wrote:

> I’ve been away from my CKAN installation and when returning I saw the proxy server had crashed.
> Restarting the server completely seemed to fix the issue.
>  
> However, now it seems that all the static linked files in /var/lib/ckan/default/…. Cannot be accessed, even though the files are there.
>  
> Error from apache states the following (very long copy-paste):
>  
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1] Error - <class 'pairtree.storage_exceptions.FileNotFoundException'>:
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1] URL: http://data.kk.dk/storage/f/2013-08-22T09%3A26%3A43.308Z/vilkar-parkeringspladser.pdf
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1] File '/usr/lib/ckan/default/lib/python2.7/site-packages/weberror/errormiddleware.py', line 162 in __call__
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   app_iter = self.application(environ, sr_checker)
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1] File '/usr/lib/ckan/default/lib/python2.7/site-packages/webob/dec.py', line 147 in __call__
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   resp = self.call_func(req, *args, **self.kwargs)
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1] File '/usr/lib/ckan/default/lib/python2.7/site-packages/webob/dec.py', line 208 in call_func
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   return self.func(req, *args, **kwargs)
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1] File '/usr/lib/ckan/default/lib/python2.7/site-packages/fanstatic/publisher.py', line 234 in __call__
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   return request.get_response(self.app)
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1] File '/usr/lib/ckan/default/lib/python2.7/site-packages/webob/request.py', line 1053 in get_response
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   application, catch_exc_info=False)
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1] File '/usr/lib/ckan/default/lib/python2.7/site-packages/webob/request.py', line 1022 in call_application
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   app_iter = application(self.environ, start_response)
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1] File '/usr/lib/ckan/default/lib/python2.7/site-packages/webob/dec.py', line 147 in __call__
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   resp = self.call_func(req, *args, **self.kwargs)
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1] File '/usr/lib/ckan/default/lib/python2.7/site-packages/webob/dec.py', line 208 in call_func
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   return self.func(req, *args, **kwargs)
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1] File '/usr/lib/ckan/default/lib/python2.7/site-packages/fanstatic/injector.py', line 54 in __call__
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   response = request.get_response(self.app)
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1] File '/usr/lib/ckan/default/lib/python2.7/site-packages/webob/request.py', line 1053 in get_response
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   application, catch_exc_info=False)
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1] File '/usr/lib/ckan/default/lib/python2.7/site-packages/webob/request.py', line 1022 in call_application
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   app_iter = application(self.environ, start_response)
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1] File '/usr/lib/ckan/default/lib/python2.7/site-packages/beaker/middleware.py', line 73 in __call__
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   return self.app(environ, start_response)
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1] File '/usr/lib/ckan/default/lib/python2.7/site-packages/beaker/middleware.py', line 155 in __call__
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   return self.wrap_app(environ, session_start_response)
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1] File '/usr/lib/ckan/default/lib/python2.7/site-packages/routes/middleware.py', line 131 in __call__
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   response = self.app(environ, start_response)
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1] File '/usr/lib/ckan/default/lib/python2.7/site-packages/pylons/wsgiapp.py', line 125 in __call__
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   response = self.dispatch(controller, environ, start_response)
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1] File '/usr/lib/ckan/default/lib/python2.7/site-packages/pylons/wsgiapp.py', line 324 in dispatch
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   return controller(environ, start_response)
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1] File '/usr/lib/ckan/default/src/ckan/ckan/lib/base.py', line 325 in __call__
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   res = WSGIController.__call__(self, environ, start_response)
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1] File '/usr/lib/ckan/default/lib/python2.7/site-packages/pylons/controllers/core.py', line 221 in __call__
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   response = self._dispatch_call()
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1] File '/usr/lib/ckan/default/lib/python2.7/site-packages/pylons/controllers/core.py', line 172 in _dispatch_call
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   response = self._inspect_call(func)
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1] File '/usr/lib/ckan/default/lib/python2.7/site-packages/pylons/controllers/core.py', line 107 in _inspect_call
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   result = self._perform_call(func, args)
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1] File '/usr/lib/ckan/default/lib/python2.7/site-packages/pylons/controllers/core.py', line 60 in _perform_call
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   return func(**args)
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1] File '/usr/lib/ckan/default/src/ckan/ckan/controllers/storage.py', line 182 in file
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   metadata = self.ofs.get_metadata(BUCKET, label)
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1] File '/usr/lib/ckan/default/lib/python2.7/site-packages/ofs/local/pairtreestore.py', line 128 in get_metadata
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   raise FileNotFoundException
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1] FileNotFoundException:
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1] CGI Variables
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1] -------------
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   CKAN_CURRENT_URL: '/storage/f/2013-08-22T09%3A26%3A43.308Z/vilkar-parkeringspladser.pdf'
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   CKAN_LANG: 'da'
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   CKAN_LANG_IS_DEFAULT: True
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   CONTENT_TYPE: '; charset=utf-8'
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   DOCUMENT_ROOT: '/etc/apache2/htdocs'
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   GATEWAY_INTERFACE: 'CGI/1.1'
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   HTTP_ACCEPT: 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   HTTP_ACCEPT_ENCODING: 'gzip, deflate'
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   HTTP_ACCEPT_LANGUAGE: 'da,en-us;q=0.7,en;q=0.3'
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   HTTP_CONNECTION: 'close'
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   HTTP_HOST: 'data.kk.dk'
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   HTTP_USER_AGENT: 'Mozilla/5.0 (Windows NT 5.1; rv:27.0) Gecko/20100101 Firefox/27.0'
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   PATH_INFO: '/storage/f/2013-08-22T09:26:43.308Z/vilkar-parkeringspladser.pdf'
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   PATH_TRANSLATED: '/etc/ckan/default/apache.wsgi/storage/f/2013-08-22T09:26:43.308Z/vilkar-parkeringspladser.pdf'
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   REMOTE_ADDR: '127.0.0.1'
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   REMOTE_PORT: '42118'
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   REQUEST_METHOD: 'GET'
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   REQUEST_URI: '/storage/f/2013-08-22T09:26:43.308Z/vilkar-parkeringspladser.pdf'
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   SCRIPT_FILENAME: '/etc/ckan/default/apache.wsgi'
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   SERVER_ADDR: '127.0.0.1'
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   SERVER_ADMIN: '[no address given]'
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   SERVER_NAME: 'data.kk.dk'
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   SERVER_PORT: '80'
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   SERVER_PROTOCOL: 'HTTP/1.0'
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   SERVER_SIGNATURE: '<address>Apache/2.2.22 (Ubuntu) Server at data.kk.dkPort 80</address>\\n'
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   SERVER_SOFTWARE: 'Apache/2.2.22 (Ubuntu)'
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1] WSGI Variables
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1] --------------
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   application: <fanstatic.publisher.Delegator object at 0x7ffe03187910>
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   beaker.cache: <beaker.cache.CacheManager object at 0x7ffe03187d90>
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   beaker.get_session: <bound method SessionMiddleware._get_session of <beaker.middleware.SessionMiddleware object at 0x7ffe02dbab50>>
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   beaker.session: {'_accessed_time': 1395754161.459972, '_creation_time': 1395754161.459972}
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   fanstatic.needed: <fanstatic.core.NeededResources object at 0x7ffdeefddbd0>
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   mod_wsgi.application_group: 'default.ckanhosted.com|'
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   mod_wsgi.callable_object: 'application'
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   mod_wsgi.handler_script: ''
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   mod_wsgi.input_chunked: '0'
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   mod_wsgi.listener_host: ''
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   mod_wsgi.listener_port: '8080'
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   mod_wsgi.process_group: 'ckan_default'
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   mod_wsgi.request_handler: 'wsgi-script'
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   mod_wsgi.script_reloading: '1'
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   mod_wsgi.version: (3, 3)
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   paste.cookies: (<SimpleCookie: >, '')
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   paste.registry: <paste.registry.Registry object at 0x7ffdf49ac710>
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   paste.throw_errors: True
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   pylons.action_method: <bound method StorageController.file of <ckan.controllers.storage.StorageController object at 0x7ffdeefdd550>>
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   pylons.controller: <ckan.controllers.storage.StorageController object at 0x7ffdeefdd550>
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   pylons.environ_config: {'session': 'beaker.session', 'cache': 'beaker.cache'}
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   pylons.pylons: <pylons.util.PylonsContext object at 0x7ffdeefdd1d0>
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   pylons.routes_dict: {'action': u'file', 'controller': u'ckan.controllers.storage:StorageController', 'label': u'2013-08-22T09:26:43.308Z/vilkar-parkeringspladser.pdf'}
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   repoze.who.logger: <logging.Logger object at 0x7ffe0318c910>
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   repoze.who.plugins: {'openid': <OpenIdIdentificationPlugin 140728950378192>, 'friendlyform': <FriendlyFormPlugin 140728947127184>, 'ckan.lib.authenticator:UsernamePasswordAuthenticator': <ckan.lib.authenticator.UsernamePasswordAuthenticator object at 0x7ffe0334d690>, 'auth_tkt': <AuthTktCookiePlugin 140728947127312>, 'ckan.lib.authenticator:OpenIDAuthenticator': <ckan.lib.authenticator.OpenIDAuthenticator object at 0x7ffe0334d610>}
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   routes.route: <routes.route.Route object at 0x7ffe02fae890>
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   routes.url: <routes.util.URLGenerator object at 0x7ffdeefddb10>
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   webob._parsed_query_vars: (GET([]), '')
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   webob.adhoc_attrs: {'response': <Response at 0x7ffdeefdd610 200 OK>, 'language': 'en-us'}
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   wsgi process: 'Multi process AND threads (?)'
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   wsgi.file_wrapper: <built-in method file_wrapper of mod_wsgi.Adapter object at 0x7ffde495caf8>
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   wsgi.version: (1, 1)
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   wsgiorg.routing_args: (<routes.util.URLGenerator object at 0x7ffdeefddb10>, {'action': u'file', 'controller': u'ckan.controllers.storage:StorageController', 'label': u'2013-08-22T09:26:43.308Z/vilkar-parkeringspladser.pdf'})
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1] ------------------------------------------------------------
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]             <p>Additionally an error occurred while sending the <weberror.reporter.EmailReporter object at 0x7ffe039d1a90> report:
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]             <pre>Traceback (most recent call last):
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   File "/usr/lib/ckan/default/lib/python2.7/site-packages/weberror/errormiddleware.py", line 450, in send_report
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]     rep.report(exc_data)
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   File "/usr/lib/ckan/default/lib/python2.7/site-packages/weberror/reporter.py", line 45, in report
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]     server = smtplib.SMTP(self.smtp_server)
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   File "/usr/lib/python2.7/smtplib.py", line 249, in __init__
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]     (code, msg) = self.connect(host, port)
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   File "/usr/lib/python2.7/smtplib.py", line 309, in connect
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]     self.sock = self._get_socket(host, port, self.timeout)
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   File "/usr/lib/python2.7/smtplib.py", line 284, in _get_socket
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]     return socket.create_connection((port, host), timeout)
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]   File "/usr/lib/python2.7/socket.py", line 571, in create_connection
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]     raise err
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1] error: [Errno 111] Connection refused
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1] </pre>
> [Tue Mar 25 14:29:21 2014] [error] [client 127.0.0.1]             </p>
> [Tue Mar 25 14:29:24 2014] [error] 2014-03-25 14:29:24,966 WARNI [ckan.lib.maintain] Use `c.search_facets` instead.
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20140325/476ce8fe/attachment-0003.html>


More information about the ckan-dev mailing list