[ckan-dev] CKAN 1.5 .deb Packaging Update

Mon Nov 14 13:10:16 UTC 2011

Hi all,

The release of CKAN 1.5 brought with it a new approach to Debian 
packaging involving virtual environments based on the feedback from the 
community meeting. The new docs for installing CKAN 1.5 from packages 
are here:

http://docs.ckan.org/docs/ckan/en/ckan-1.5/install-from-package.html#run-the-package-installer

In case you don't want to read the full docs I'll just take you through 
the key steps here. As always, this only works on Ubuntu 10.04.

Let's say we have created a CKAN instance called "test" like this:

     sudo apt-get update
     sudo apt-get install -y wget
     echo "deb http://apt.okfn.org/ckan-1.5 lucid universe" | sudo tee 
/etc/apt/sources.list.d/okfn.list
     wget -qO- "http://apt.okfn.org/packages_public.key" | sudo apt-key 
add -
     sudo apt-get update
     sudo apt-get install -y ckan postgresql-8.4 solr-jetty
     sudo ckan-setup-solr
     sudo ckan-create-instance test test.ckan.net yes

In order to run paster commands that use the test instance's virtualenv 
you'll need to install pastescript into the virtualenv too:

     sudo -u ckantest /var/lib/ckan/dguuklp/pyenv/bin/pip install 
--ignore-installed pastescript

You can now install custom extensions for this instance like this:

     sudo -u ckantest /var/lib/ckan/test/pyenv/bin/pip install 
<name-of-extension>

Notice that you have to run everything as the ckantest user that was 
created for you and that each command must use the full paths to the 
scripts in the test instance's virtual environment.

Any time you make a change you can let Apache know like this:

     sudo touch /var/lib/ckan/std/wsgi.py

Then you must *force* refresh your browser to see changes (since CKAN 
tells it to cache its pages by default).

One other thing that is worth pointing out (and that isn't covered in 
the docs) is that because multiple CKAN instances on the same server are 
now supported you have a different database and database user for each 
instance. If you want to load one database dump into another database, 
you therefore have to change the ownership of the tables to match the 
instance. I've written some simple SQL to help. Here's a brief tutorial 
to show how to restore a database dump to a different instance...

Let's copy a database dump from the "live" database to our new instance 
called "test" for testing.

1. First stop Apache so there are no active database connections:

     sudo /etc/init.d/apache2 stop

2. Drop the old database and restore the new one (setting up the plpgsql 
language as you do):

     sudo -u postgres dropdb test
     sudo -u postgres createdb test
     sudo -u postgres createlang plpgsql test
     sudo -u postgres psql test -f to_restore.pg_dump

3. Change the table permissions (in this case the old instance name was 
called live, the new one is test)@

     sudo -u postgres psql test

Then from the psql command line interface run:

begin;
CREATE OR REPLACE FUNCTION changeowner(text, text)
RETURNS TEXT STRICT VOLATILE AS '
DECLARE
     old ALIAS FOR $1;
     new ALIAS FOR $2;
     rel record;
     sql text;
BEGIN
     FOR rel IN select * from pg_tables where tableowner=old
     LOOP sql :=  ''ALTER TABLE "'' || rel.tablename || ''" OWNER to '' 
|| new;
         RAISE NOTICE ''%'', sql;
         EXECUTE sql;
     END LOOP;
     RETURN ''OK'';
END;
' LANGUAGE 'plpgsql';
SELECT  changeowner('live', 'test');
commit;
\q

You will need to change the call to changeowner('live', 'test'); to 
reflect the old and new names of the database owner. Now your test 
instance has the correct permissions.

Once you've changed the database, you'll also need to re-build the Solr 
search index (this can take an hour or two if there is a lot of data):

     sudo -u ckantest  /var/lib/ckan/test/pyenv/bin/paster --plugin=ckan 
search-index rebuild --config=/etc/ckan/test/test.ini

Notice that you are using the full path to the paster within the virtual 
environment (so that it has access to the extensions you installed) and 
that you are running as the ckantest user.

As well as being able to use the new CKAN packaging, you might be 
interested to know that with this release, the work of building the CKAN 
packages is no longer soley in the domain of the OKF. I've written a 
quick blog post here about how the newly released packages were built 
with buildkit automatically from Python code here:

      http://ckan.org/2011/11/14/building-ckan-debian-packages/

The script that does the actual building is linked below:

     https://bitbucket.org/okfn/ckan/src/release-v1.5/build.sh

Using this script it should be possible for anyone who fancies it to 
build unofficial CKAN packages, you don't have to use the official OKF 
ones. There is also some documentation for those of you who are 
interested here:

http://packages.python.org/buildkit/manual.html#example-building-and-testing-the-ckan-package-install

You can get hold of buildkit here:

     http://pypi.python.org/pypi/buildkit/0.2.0
     git clone git://git.3aims.com/buildkit

Hope that's a useful introduction. You should read the docs linked to 
for full information, but do let me know if you have any questions.

Cheers,

James