[ckan-dev] [ckan-discuss] How to install extensions for a package installation?

[Florian Marienfeld]

Hi,

Thank you for this detailed reply! glad that attending that workshop got
me on the right track from the start :-)

ckan-dgu etc. are debian-packages, right? where do I find these? did not
find them at first sight in bitbucket or apt. I would try to mimic them
to setup a non-packaged instance.

I'm definitely interested in the whole packaging business, so please
keep me posted on the respective community meetup, I fear that I am only
available 8am-2pm uk time on 22nd.

@Max, for me installing ckan-admin worked recently using the method
described in the workshop. are you interested in meeting up in berlin at
some point? so we can pool our ignorance and suck out some expertise
from Friedrich :-) it would also be nice to talk about metadata for
German portals.

Best regards, keep the good support up!

FLO





On 08.09.11 18:58, James Gardner wrote:
> Hi Max and Flo,
> 
> [cc'ing Jan and David so that you know where I am with packaging and can
> decide if you want to come to the community meetup proposed for the 22nd
> September at the end of this email to discuss packaging].
> 
>>> I did everything what is said under "Prepare to Use Extensions". What
>>> I don't understand: I installed the package (the DEB from your source
>>> using apt-get). It runs very fine using apache+wsgi (I didn't even
>>> have to deploy it). And now I have to create a python environment
>>> somewhere else (in my case: /home/max/ckan)? How is that supposed to
>>> work? I mean, how can ckan (installed using DEB package and lying
>>> somewhere in /usr/share and /usr/lib and having its config files in
>>> /etc/ckan) find these plugins installed in a virtual environment owned
>>> by my user?
> 
> Perhaps I can chip in here too with some background, a explanation of
> two possible solutions and a quick chat about our plans for the future.
> 
> Historically everyone deployed CKAN manually using virtual environments
> but this involved many manual steps with opportunities for errors so we
> are transitioned to a package-based approach. This works well for core
> CKAN but we haven't packaged all the CKAN extensions yet which is why
> you have to still manually install them. I'm working on a set of tools
> that will make it easy to package up your own CKAN extensions too (more
> on this later) but for the time-being we are left with this hybrid
> approach where most of the work is done with packages but extensions
> that aren't packaged yet require manual installation.
> 
> There are broadly two approaches you can take if you want a CKAN
> installation with extra packages:
> 
> 1. Install them into the system Python as Flo has done
> 2. Create a virtual environment and serve CKAN from that
> 
> The advantage of the first approach is that you don't need to change any
> settings. Apache, postgreSQL etc can all stay the same. If you don't
> want to package your extension, and you have an entire machine dedicated
> for a CKAN install, this is actually the approach I'd recommend. At
> OKCon in Berlin, David Read and I gave a workshop which took people
> though this process. The notes are still online at the URL below:
> 
> http://ckan.okfnpad.org/sprint-okcon-2011-wishlist
> 
> If you look at line 110 onward you'll see how we install the
> ckanext-exampletheme <https://bitbucket.org/okfn/ckanext-exampletheme>
> extension.
> 
> The advantage of the second approach is that your new CKAN installation
> is completely isolated from the system CKAN but the disadvantage is that
> you effectively have to know everything a developer would about
> installing CKAN, re-configuring Apache etc. The current documentation is
> slightly misleading in that it appears to suggest you can easily use the
> system CKAN from a virtual environment without changing any other
> settings when actually you can't. You'd need to change Apache to use the
> Python instance in your virtual environment at the very least so we
> should write that in the docs.
> 
>> I just installed ckan from source on the same machine and tried to add
>> the admin extension but it threw errors that it can not find such a
>> module. So I tried another extension, disqus. This seemed to work. So
>> I believe the admin extension is broken. I went on with the disqus
>> extension and set up a disqus account and so on. When I finally ran
>> the server (no errors) and went on a data package page, there was no
>> disqus comment box. I don't know what's wrong there. I tried the
>> wordpress extension, too. Same result. Nothing. I can't provide any
>> error messages or stuff because there is none.
>>
>> I'm really lost here.
> 
> @Max, it sounds like the problems are mounting up one after the other!
> Let's try to resolve the initial problem of how to install extensions
> properly first and then take things from there?
> 
>>> No. In step 1 it says /home/ubuntu/pyenv for the virtual environment
>>> but in step 2 its ~/var/srvc/ckan.net/pyenv. I don't understand.
> 
> It doesn't really matter where you put your virtualenv but if you use
> one, you need to change your Apache settings so it can find it.
> 
>>> Yes. But when I did this, the server crashes giving me a
>>> "PluginNotFoundException: admin".
> 
> I suspect you haven't re-configured Apache that which is why you are
> getting this error. Instead, if your setup allows it and you are
> comfortable doing so, install your extnesion into the system Python as
> described in option 1 above and everything should work.
> 
>> But for developing and testing it would be nice to have a separate ckan
>> instances on the same machine, but honestly - and I think this is Max's
>> problem, too - I dont understand how all the bits and pieces are bolted
>> together to form one instance:
> 
> 
> @Flo, let me just try to briefly describe how everything is bolted together:
> 
>> - ckan-python-code
> 
> Each CKAN instance is called something like ckan-std, ckan-dgu etc and
> is treated as an *instance package*, that is, CKAN is installed as an
> application with Apache, PostgreSQL etc all set up as part of the
> installation. Multiple different CKAN instances can all be installed on
> the same server as long as they use the same versions of other
> dependencies such as the Python libraries.
> 
>> - python libraries
> 
> Python libraries are packaged as python-name so we'll get things like
> python-ckan, python-formalchemy, python-ckanext-harvest etc. These are
> all installed into /usr/lib as dependencies of the ckan instance package.
> 
>> - .ini
> 
> There is a different .ini file for each CKAN instance installed. The
> ckan-std package's INI file is in /etc/ckan/std/std.ini, other ones are
> named similarly.
> 
>> - apache
> 
> Apache is installed as a dependency of ckan-std too. But in the latest
> version of the packaging it isn't set up until you run the
> `ckan-std-install' script which is also installed as part of ckan-std.
> 
>> - ckan-extensions
> 
> These are just standard Python libraries as above, and get installed
> along with any other Python libraries that are dependencies of the
> ckan-std package. The only difference is that the `ckan-std-install'
> script will automatically add the extension name to the std.ini file it
> creates if it wants those extensions enabled.
> 
>> Maybe this is clear to people who installed ckan in the early days, for
>> us APT-kiddies it is confusing.
> 
> No, I recognize it isn't really that clear, but rest assured we're
> working on ways to make it simpler.
> 
> In the future we aim to provide tools that make creating packages and
> testing they work much simpler so that you can package extensions
> yourself and just have them installed via apt-get the same way as
> everything else. To that end, I'd like to invite you to a packaging
> community meet up on the 22nd September to discuss packaging going
> forward. Max, Jan, Flo, can you make it at some point during the day?
> Perhaps 4pm UK time? I'll announce it properly with a separate email to
> the list nearer the time.
> 
>> Apart from that, thanks for the great work, it's a pleasure to use and
>> hack CKAN.
> 
> Thank you!
> 
> Cheers,
> 
> James
> 
>