[ckan-dev] Problem viewing data from datastore in data-resource preview / using docker set-up on EC2

Thu Sep 25 11:52:41 UTC 2014

I have the exact same set-up described below working locally, but on an EC2
instance, for some reason it is causing problems.  I have extended the
docker set-up below manually to enable the datastore extension, including
manually setting up the default_datastore database and pulling across
(pg_restore) data from a legacy system.  In the preview of a data-resource
(where ordinarily it connects to the default_datastore database), I'm
getting "the server refused the connection" error, eventhough I have
triple-checked that the credentials in ckan.ini are set correctly and have a
basic password policy (same as login) for testing.  The 172.17.0.83 is the
docker container IP of 'db'.

## Database Settings

sqlalchemy.url = postgresql://ckan:ckan@172.17.0.83:5432/ckan

ckan.datastore.write_url =
postgresql://ckan_default:ckan_default@172.17.0.83:5432/datastore_default

ckan.datastore.read_url =
postgresql://datastore_default:datastore_default@172.17.0.83:5432/datastore_
default

Docker containers:

CONTAINER ID        IMAGE                    COMMAND                CREATED
STATUS              PORTS                    NAMES

3213a9880a08        cygri/ckan:2.2           "/sbin/my_init"        17 hours
ago        Up 10 hours         0.0.0.0:5001->80/tcp     ckan2.2

b6ee66cd95a6        ckan/postgresql:latest   "/usr/local/bin/run"   24 hours
ago        Up 10 hours         0.0.0.0:5432->5432/tcp   ckan2.2/db,db

a64b455c920d        ckan/solr:latest         "java -jar start.jar   41 hours
ago        Up 11 hours         8983/tcp                 ckan2.2/solr,solr2.2

Note: I exposed port 5432 to the host for db since I wanted to be able to
push data into the new database from a legacy database.

I'm using a 2.2 version of the CKAN docker container (described here:
https://registry.hub.docker.com/u/cygri/ckan/) , since I was having real
problems in getting any of the paster scripts for upgrading to version 2.3a
(db clean, db load, db upgrade etc.) working in a docker set-up.  I have
been  using the approach described under "Running maintenance commands" here
(http://docs.ckan.org/en/latest/maintaining/installing/install-using-docker.
html?highlight=docker), but in most cases, these scripts don't run properly
or hang.  I ended up dropping the ckan database manually from the db
container in order to pg_restore my old data properly . and repeating the
process to bring my legacy datastore data on-stream too.

>From habit, anytime I change settings, I run 'sudo service apache2 restart'
and 'sudo service nginx restart' on the main 'ckan2.2' container.  In each
case, although it says 'fail', I can see from the apache logs that it
appears to work, but as you can see below, it resumes with errors each time.

>From apache2 error.log:

[Thu Sep 25 09:25:31.357884 2014] [mpm_event:notice] [pid 36:tid
139634909292416] AH00491: caught SIGTERM, shutting down

AH00558: apache2: Could not reliably determine the server's fully qualified
domain name, using 172.17.0.84. Set the 'ServerName' directive globally to
suppress this message

[Thu Sep 25 09:25:31.464529 2014] [mpm_event:notice] [pid 383:tid
139953733179264] AH00489: Apache/2.4.7 (Ubuntu) mod_wsgi/3.4 Python/2.7.6
configured -- resuming normal operations

[Thu Sep 25 09:25:31.464796 2014] [core:notice] [pid 383:tid
139953733179264] AH00094: Command line: '/usr/sbin/apache2 -D FOREGROUND'

[Thu Sep 25 09:25:52.443664 2014] [:error] [pid 384:tid 139953522603776]
2014-09-25 09:25:52,443 WARNI [ckan.lib.maintain] Use `c.search_facets`
instead.

[Thu Sep 25 09:25:55.409718 2014] [:error] [pid 384:tid 139953522603776]
2014-09-25 09:25:55,409 INFO  [ckan.lib.base]
/organization/international-monetary-fund-imf render time 3.314 seconds

Postgres logs suggest a problem with the app forcing a connection to close:

2014-09-25 09:55:13 UTC ERROR:  database "ckan" already exists

2014-09-25 09:55:13 UTC STATEMENT:  CREATE DATABASE ckan OWNER ckan TEMPLATE
DEFAULT;

2014-09-25 09:55:14 UTC LOG:  database system was shut down at 2014-09-25
09:55:13 UTC

2014-09-25 09:55:14 UTC LOG:  database system is ready to accept connections

2014-09-25 09:55:14 UTC LOG:  autovacuum launcher started

2014-09-25 10:13:32 UTC LOG:  could not receive data from client: Connection
reset by peer

2014-09-25 10:13:32 UTC LOG:  unexpected EOF on client connection

Thanks,

Colum

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20140925/68f5fa88/attachment-0002.html>