[ckan-discuss] installation from source changed in git master

[Sean Hammond]

Does your script create a virtualenv with the --no-site-packages option?
It should, or you might end up with inconsistent results in your pip
freeze.

TLDR: I don't think there are any mistakes in the new requirements and
install instructions, but it'd be worthwhile going through all our
requirements and seeing which ones we can update to the latest versions
(including MarkupSafe and WebHelpers), and also seeing if we can only
install the part of pyutilib that we need and thereby avoid a lot of
modules getting pulled in by pyutilib.

Details:

Keep in mind, the recent changes to the requirements file and source
install instructions do more than just upgrade and remove dependencies.
We also changed to using a virtualenv with site packages disabled and
installing all of the Python modules we depend on into the virtualenv.
This means you should get a consistent set of Python modules across
different operating systems and OS versions. Previously if you did a
source install and then a pip freeze, you would get quite different
output on different OS's and even on different versions of Ubuntu, this
is the reason for some of the differences you're seeing in your pip
freeze.

> 7c7
> < MarkupSafe==0.15
> ---
> > MarkupSafe==0.9.2

Our pip-requirements file has actually always specified
MarkupSafe==0.9.2, that the previous install instructions ended up with
a different version of MarkupSafe on some systems seems to be have been
a bug with those instructions. (But it probably would be a good idea to
upgrade our MarkupSafe dependency to the latest version.)

> < PasteScript==1.7.3
> ---
> > PasteScript==1.7.5

I deliberately upgraded this. The old requirements file used to
specifically list version 1.7.3, but I can't see any reason for that
other than that's what version Ubuntu 10.04 has. With the new
requirements file 1.7.5 actually gets installed as a dependency of
something else, and is not explicitly given in our requirements file
anymore, if you explictly give 1.7.3 in the new requirements file pip
will crash.

> > PyUtilib==4.0.2848

We had previously been installing pyutilib from the sandia.gov
subversion server which was problematic, we now install it from pypi.
4.0.2848 is the only version that is on pypi so that's what you get (and
we had to update parts of the ckan code to work with this new version).

< Routes==1.11
---
> Routes==1.12

We upgraded this deliberately.

< WebHelpers==1.3
---
> WebHelpers==1.2

Same thing as MarkupSafe, WebHelpers==1.2 was always specified in our
requirements file, but with the previous install instructions if you
weren't on Ubuntu 10.04 you would actually end up with the wrong
version. (Again, it'd be worth updating our WebHelpers dependency but
that's a different job.)

> pyutilib.R==3.1
> pyutilib.autotest==2.0
> pyutilib.common==3.0.7
> pyutilib.component.app==3.2
> pyutilib.component.config==3.4
> pyutilib.component.core==4.5.3
> pyutilib.component.doc==1.0.1
> pyutilib.component.executables==3.5
> pyutilib.component.loader==3.4
> pyutilib.dev==2.0
> pyutilib.enum==1.1
> pyutilib.excel==3.1.1
> pyutilib.math==3.3
> pyutilib.misc==5.3.1
> pyutilib.ply==3.0.7
> pyutilib.pyro==3.5.2
> pyutilib.services==3.4
> pyutilib.subprocess==3.5.2
> pyutilib.svn==1.3.1
> pyutilib.th==5.3
> pyutilib.virtualenv==3.0
> pyutilib.workflow==3.2

These subpackages get installed when pyutilib is installed from pypi
(rather than from sandia.gov's svn server as we did before). We only use
pyutilib.component.core, I guess we could try specifying just that
subpackage in the requirements file rather than PyUtilib itself.

< simplejson==2.5.2
---
> simplejson==2.6.0

We don't specify simplejson it gets installed as a dependency. On Ubuntu
12.04 if you follow either the old or the new install instructions you
get simplejson 2.6.0. With the new instructions you should get this
version on any system. I guess with the old instructions, you somehow
ended up with 2.5.2 instead on your system.

> unittest2==0.5.1

It gets installed as a dependency of something we require, don't know
what. Possibly PyUtilib.

> virtualenv==1.7.2

This looks like it's also a result of installing PyUtilib from pypi
instead of svn, I think PyUtilib depends on virtualenv.