[ckan-dev] [ckan-discuss] How to install extensions for a package installation?

Thu Sep 8 16:58:02 UTC 2011

Hi Max and Flo,

[cc'ing Jan and David so that you know where I am with packaging and can 
decide if you want to come to the community meetup proposed for the 22nd 
September at the end of this email to discuss packaging].

>> I did everything what is said under "Prepare to Use Extensions". What
>> I don't understand: I installed the package (the DEB from your source
>> using apt-get). It runs very fine using apache+wsgi (I didn't even
>> have to deploy it). And now I have to create a python environment
>> somewhere else (in my case: /home/max/ckan)? How is that supposed to
>> work? I mean, how can ckan (installed using DEB package and lying
>> somewhere in /usr/share and /usr/lib and having its config files in
>> /etc/ckan) find these plugins installed in a virtual environment owned
>> by my user?

Perhaps I can chip in here too with some background, a explanation of 
two possible solutions and a quick chat about our plans for the future.

Historically everyone deployed CKAN manually using virtual environments 
but this involved many manual steps with opportunities for errors so we 
are transitioned to a package-based approach. This works well for core 
CKAN but we haven't packaged all the CKAN extensions yet which is why 
you have to still manually install them. I'm working on a set of tools 
that will make it easy to package up your own CKAN extensions too (more 
on this later) but for the time-being we are left with this hybrid 
approach where most of the work is done with packages but extensions 
that aren't packaged yet require manual installation.

There are broadly two approaches you can take if you want a CKAN 
installation with extra packages:

1. Install them into the system Python as Flo has done
2. Create a virtual environment and serve CKAN from that

The advantage of the first approach is that you don't need to change any 
settings. Apache, postgreSQL etc can all stay the same. If you don't 
want to package your extension, and you have an entire machine dedicated 
for a CKAN install, this is actually the approach I'd recommend. At 
OKCon in Berlin, David Read and I gave a workshop which took people 
though this process. The notes are still online at the URL below:

http://ckan.okfnpad.org/sprint-okcon-2011-wishlist

If you look at line 110 onward you'll see how we install the 
ckanext-exampletheme <https://bitbucket.org/okfn/ckanext-exampletheme> 
extension.

The advantage of the second approach is that your new CKAN installation 
is completely isolated from the system CKAN but the disadvantage is that 
you effectively have to know everything a developer would about 
installing CKAN, re-configuring Apache etc. The current documentation is 
slightly misleading in that it appears to suggest you can easily use the 
system CKAN from a virtual environment without changing any other 
settings when actually you can't. You'd need to change Apache to use the 
Python instance in your virtual environment at the very least so we 
should write that in the docs.

> I just installed ckan from source on the same machine and tried to add
> the admin extension but it threw errors that it can not find such a
> module. So I tried another extension, disqus. This seemed to work. So
> I believe the admin extension is broken. I went on with the disqus
> extension and set up a disqus account and so on. When I finally ran
> the server (no errors) and went on a data package page, there was no
> disqus comment box. I don't know what's wrong there. I tried the
> wordpress extension, too. Same result. Nothing. I can't provide any
> error messages or stuff because there is none.
>
> I'm really lost here.

@Max, it sounds like the problems are mounting up one after the other! 
Let's try to resolve the initial problem of how to install extensions 
properly first and then take things from there?

>> No. In step 1 it says /home/ubuntu/pyenv for the virtual environment
>> but in step 2 its ~/var/srvc/ckan.net/pyenv. I don't understand.

It doesn't really matter where you put your virtualenv but if you use 
one, you need to change your Apache settings so it can find it.

>> Yes. But when I did this, the server crashes giving me a
>> "PluginNotFoundException: admin".

I suspect you haven't re-configured Apache that which is why you are 
getting this error. Instead, if your setup allows it and you are 
comfortable doing so, install your extnesion into the system Python as 
described in option 1 above and everything should work.

>  But for developing and testing it would be nice to have a separate ckan
>  instances on the same machine, but honestly - and I think this is Max's
>  problem, too - I dont understand how all the bits and pieces are bolted
>  together to form one instance:

@Flo, let me just try to briefly describe how everything is bolted together:

> - ckan-python-code

Each CKAN instance is called something like ckan-std, ckan-dgu etc and 
is treated as an *instance package*, that is, CKAN is installed as an 
application with Apache, PostgreSQL etc all set up as part of the 
installation. Multiple different CKAN instances can all be installed on 
the same server as long as they use the same versions of other 
dependencies such as the Python libraries.

> - python libraries

Python libraries are packaged as python-name so we'll get things like 
python-ckan, python-formalchemy, python-ckanext-harvest etc. These are 
all installed into /usr/lib as dependencies of the ckan instance package.

> - .ini

There is a different .ini file for each CKAN instance installed. The 
ckan-std package's INI file is in /etc/ckan/std/std.ini, other ones are 
named similarly.

> - apache

Apache is installed as a dependency of ckan-std too. But in the latest 
version of the packaging it isn't set up until you run the 
`ckan-std-install' script which is also installed as part of ckan-std.

> - ckan-extensions

These are just standard Python libraries as above, and get installed 
along with any other Python libraries that are dependencies of the 
ckan-std package. The only difference is that the `ckan-std-install' 
script will automatically add the extension name to the std.ini file it 
creates if it wants those extensions enabled.

> Maybe this is clear to people who installed ckan in the early days, for
> us APT-kiddies it is confusing.

No, I recognize it isn't really that clear, but rest assured we're 
working on ways to make it simpler.

In the future we aim to provide tools that make creating packages and 
testing they work much simpler so that you can package extensions 
yourself and just have them installed via apt-get the same way as 
everything else. To that end, I'd like to invite you to a packaging 
community meet up on the 22nd September to discuss packaging going 
forward. Max, Jan, Flo, can you make it at some point during the day? 
Perhaps 4pm UK time? I'll announce it properly with a separate email to 
the list nearer the time.

> Apart from that, thanks for the great work, it's a pleasure to use and
> hack CKAN.

Thank you!

Cheers,

James

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20110908/4dd6ab96/attachment-0001.html>