[ckan-dev] Error with DataPusher uploading to DataStore - HTTP connection timeout.

Mon Aug 8 12:59:22 UTC 2016

Hello list,

I have installed CKAN 2.5.2 on an Ubuntu 14.04 64bit server (WEB1) and PostgreSQL on another server (DB1, same config).  The site is currently only accessible over the web from our external IP address.  Ports 80 and 443 are open.

I am getting an error message when trying to upload a CSV file to the DataStore:

Error: File "/usr/lib/ckan/datapusher/lib/python2.7/site-packages/apscheduler/scheduler.py", line 512, in _run_job retval = job.func(*job.args, **job.kwargs) File "/usr/lib/ckan/datapusher/src/datapusher/datapusher/jobs.py", line 300, in push_to_datastore resource = get_resource(resource_id, ckan_url, api_key) File "/usr/lib/ckan/datapusher/src/datapusher/datapusher/jobs.py", line 250, in get_resource 'Authorization': api_key} File "/usr/lib/ckan/datapusher/lib/python2.7/site-packages/requests/api.py", line 107, in post return request('post', url, data=data, json=json, **kwargs) File "/usr/lib/ckan/datapusher/lib/python2.7/site-packages/requests/api.py", line 53, in request return session.request(method=method, url=url, **kwargs) File "/usr/lib/ckan/datapusher/lib/python2.7/site-packages/requests/sessions.py", line 468, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/ckan/datapusher/lib/python2.7/site-packages/requests/sessions.py", line 576, in send r = adapter.send(request, **kwargs) File "/usr/lib/ckan/datapusher/lib/python2.7/site-packages/requests/adapters.py", line 437, in send raise ConnectionError(e, request=request) ConnectionError(MaxRetryError("HTTPConnectionPool(host='opendata.angus.gov.uk', port=80): Max retries exceeded with url: /api/3/action/resource_show (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fb3f805d410>: Failed to establish a new connection: [Errno 110] Connection timed out',))",),)

I installed CKAN from package following the latest docs on WEB1.

I set up the PostgreSQL databases and Solr search on DB1 following the docs and making sure the various URLs were correct.  Solr search works correctly.

I set up the FileStore correctly following the docs - updating production.ini with storage_path and setting permissions on the folder.  Files can be uploaded and downed from my CKAN site. When I try to preview the CSV file with DataEXplorer I get a Loading.... And then "resource if not available at the moment."  And:

Could not load view: DataProxy returned an error (Data transformation failed. HTTPException: Deadline exceeded while waiting for HTTP response from URL: http://opendata.angus.gov.uk/dataset/ae3ea4af-cefd-4e08-8f21-849449033619/resource/e5d753b6-cb21-44ce-af41-e1fbc455350a/download/mobilelibraries.csv)

I set up the DataStore following the docs (enabling plugin on WEB1, setting up database with permissions on DB1, updated datastore read and write URLs in the ini file

If I test the DataStore with curl -X GET "http://127.0.0.1:8800/api/3/action/datastore_search?resource_id=_table_metadata"
I get a 404 from the server saying the requested URL was not found on the server.  If I drop the :8800 from the URL I get some JSON back.

I set up the DataPusher with ckan.datapusher.url = http://0.0.0.0:8800/  in the ini file and my site url set to http://opendata.angus.gov.uk. The datapusher plugin is enabled.  When I try to upload data on the Manage Resource page I don't get any info in the upload log (no nice green ticks) but just the error above about the HTTP connection timing out.  The datapusher error log has the following:

[Mon Aug 08 13:27:31.688789 2016] [:error] [pid 26950:tid 140410783409920]     return request('post', url, data=data, json=json, **kwargs)

[Mon Aug 08 13:27:31.688791 2016] [:error] [pid 26950:tid 140410783409920]   File "/usr/lib/ckan/datapusher/lib/python2.7/site-packages/requests/api.py", line 53, in request

[Mon Aug 08 13:27:31.688794 2016] [:error] [pid 26950:tid 140410783409920]     return session.request(method=method, url=url, **kwargs)

[Mon Aug 08 13:27:31.688796 2016] [:error] [pid 26950:tid 140410783409920]   File "/usr/lib/ckan/datapusher/lib/python2.7/site-packages/requests/sessions.py", line 468, in request

[Mon Aug 08 13:27:31.688799 2016] [:error] [pid 26950:tid 140410783409920]     resp = self.send(prep, **send_kwargs)

[Mon Aug 08 13:27:31.688801 2016] [:error] [pid 26950:tid 140410783409920]   File "/usr/lib/ckan/datapusher/lib/python2.7/site-packages/requests/sessions.py", line 576, in send

[Mon Aug 08 13:27:31.688804 2016] [:error] [pid 26950:tid 140410783409920]     r = adapter.send(request, **kwargs)

[Mon Aug 08 13:27:31.688806 2016] [:error] [pid 26950:tid 140410783409920]   File "/usr/lib/ckan/datapusher/lib/python2.7/site-packages/requests/adapters.py", line 437, in send

[Mon Aug 08 13:27:31.688809 2016] [:error] [pid 26950:tid 140410783409920]     raise ConnectionError(e, request=request)

[Mon Aug 08 13:27:31.688812 2016] [:error] [pid 26950:tid 140410783409920] ConnectionError: HTTPConnectionPool(host='opendata.angus.gov.uk', port=80): Max retries exceeded with url: /api/3/action/resource_show (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fb3f805d410>: Failed to establish a new connection: [Errno 110] Connection timed out',))

If I visit http://opendata.angus.gov.uk/api/3/action/resource_show I get the follow JSON:

{"help": "http://opendata.angus.gov.uk/api/3/action/help_show?name=resource_show", "success": false, "error": {"__type": "Validation Error", "id": ["Missing value"]}}

Any ideas where this can be corrected?

Thanks in advance

Ross

Ross McDonald | GIS Data Coordinator | Resources Department, IT Division | Angus Council, Angus House, Orchardbank Business Park, Forfar, DD8 1AT
T: 01307 476419 | F: 01307 476401 | E: mcdonaldr at angus.gov.uk<mailto:mcdonaldr at angus.gov.uk>

This message is strictly confidential. If you have received this in error, please inform the sender and remove it from your system. If received in error you may not copy, print, forward or use it or any attachment in any way. This message is not capable of creating a legal contract or a binding representation and does not represent the views of Angus Council. Emails may be monitored for security and network management reasons. Messages containing inappropriate content may be intercepted. Angus Council does not accept any liability for any harm that may be caused to the recipient system or data on it by this message or any attachment.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20160808/6fce4093/attachment-0002.html>