[ckan-discuss] installation from source changed in git master

Thu Jun 28 13:56:38 BST 2012

On 28 June 2012 13:40, Uwe Geuder <2hsfka7m5g at snkmail.com> wrote:

> Hi!
>
> I wonder whether some dependency changes recently made, especially
> downgrades, were really intended.
>
> We have a script that installs CKAN from source on CentOS 6. We are
> currently in an internal development phase, so we just install from git
> master and we re-install from scratch frequently.
>
> Today I noticed that installation has changed:
>

yes we are trying to improve the from source installation - removing
dependencies such as svn/hg etc and simplifying getting the right python
packages.

>
> commit db7481921ba37e7b044ea59360d90cc934149b74
> Author: Sean Hammond <seanhammond at lavabit.com>
> Date:   Sat Jun 23 11:11:02 2012 +0200
>
>    Update and simplify install instructions
>
>
> That's all fine, the change was easy to make in our script:
>
>
>  diff --git a/rpms/kata-ckan-dev/src/02getpythonpackages.sh
> b/rpms/kata-ckan-dev/src/02getpythonpackages.sh
> index 5470438..976f013 100644
> --- a/rpms/kata-ckan-dev/src/02getpythonpackages.sh
> +++ b/rpms/kata-ckan-dev/src/02getpythonpackages.sh
> @@ -9,9 +9,7 @@ fi
>  instloc=$1
>  cd $instloc
>  source pyenv/bin/activate
> -pip install --ignore-installed -e git+
> https://github.com/okfn/ckan.git#egg=ckan
> -pip install --ignore-installed -r
> pyenv/src/ckan/requires/lucid_missing.txt -r
> pyenv/src/ckan/requires/lucid_conflict.txt
> -pip install webob==1.0.8
> -pip install --ignore-installed -r
> pyenv/src/ckan/requires/lucid_present.txt
> +pip install -e git+https://github.com/okfn/ckan.git#egg=ckan
> +pip install -r pyenv/src/ckan/pip-requirements.txt
>
>
> This works all fine.
>
> After the installation the script runs a "pip freeze" and compares to a
> previously known result. That's where I started wondering:
>
> (The following diff goes from old to new)
>
>
> 7c7
> < MarkupSafe==0.15
> ---
> > MarkupSafe==0.9.2
>
> Looks like quite a big downgrade. And without looking at the code I
> could at least guess that MarkupSafe could be even
> security-relevant. Was that really the intention?
>
> 11c11,12
> < PasteScript==1.7.3
> ---
> > PasteScript==1.7.5
> > PyUtilib==4.0.2848
> 14c15
> < Routes==1.11
> ---
> > Routes==1.12
>
> 2 upgrades and one new dependency. Have not checked where the new one is
> needed.
>

This upgrade allows us to remove some compat code in config/routes.py but
that code has not yet been removed as it risk breaking ckan for people who
don't upgrade their requirements

>
> 18c19
> < WebHelpers==1.3
> ---
> > WebHelpers==1.2
>
> Another downgrade.
>
>
> 23c24
> < -e git+
> https://github.com/okfn/ckan.git@f573a3a8d70f269c659f44282f798c6db08d9ece#egg=ckan-dev
> ---
> > -e git+
> https://github.com/okfn/ckan.git@fcc52c04adc14276e99e66cb5adb63674f8d3fbe#egg=ckan-dev
> 25d25
> < -e hg+
> http://hg.saddi.com/flup@301a58656bfbce7b77d538112b6348f67ce1162a#egg=flup-dev
>
> My old list is already 2 weeks or more old, the flup removal has appeared
> before the installation change. OK.
>
>
> 30c30,51
> < -e svn+
> https://software.sandia.gov/svn/public/pyutilib/pyutilib.component.core/trunk@1886#egg=pyutilib.component.core-4.1-py2.6-dev_r1886
> ---
> > pyutilib.R==3.1
> > pyutilib.autotest==2.0
> > pyutilib.common==3.0.7
> > pyutilib.component.app==3.2
> > pyutilib.component.config==3.4
> > pyutilib.component.core==4.5.3
> > pyutilib.component.doc==1.0.1
> > pyutilib.component.executables==3.5
> > pyutilib.component.loader==3.4
> > pyutilib.dev==2.0
> > pyutilib.enum==1.1
> > pyutilib.excel==3.1.1
> > pyutilib.math==3.3
> > pyutilib.misc==5.3.1
> > pyutilib.ply==3.0.7
> > pyutilib.pyro==3.5.2
> > pyutilib.services==3.4
> > pyutilib.subprocess==3.5.2
> > pyutilib.svn==1.3.1
> > pyutilib.th==5.3
> > pyutilib.virtualenv==3.0
> > pyutilib.workflow==3.2
>
> That's again one I do not understand. Do we really need that many new
> dependencies? I shortly looked into the previously used svn and it did
> not seem to contain all that stuff.
>
> Pyutillib was part of the reason for these changes as we were getting from
svn and this was unreliable.  Currently we only use a small part of
pyutillib but it is hard to just get the bits we need from pypi (we'd like
all our python dependencies from here).  We are looking at a better
solution to this maybe taking some of the code in tree - no decision has
yet been reached.

again currently not upgrading the requirements allows ckan to run as normal
but does break 2 core tests

> 34c55
> < simplejson==2.5.2
> ---
> > simplejson==2.6.0
>
> OK.
>
> 36a58
> > unittest2==0.5.1
>
> Do we need this for a "normal" installation? I understand there is an
> extra list pip-requirements-test.txt.
>
> 37a60
> > virtualenv==1.7.2
>
> Weird that this was not listed earlier. Should be OK, though.
>
>
> Thanks for this
I suspect some of the downgrades maybe due to system python packages - we
are looking to have all these in the pyenv for better stability
Feedback is always welcome let us know any problems you have or how we can
improve the documentation.

@sean do you want to look at the other issues - seems like we should avoid
the downgrades

Toby

> Regards,
>
> Uwe Geuder
> Nomovok Ltd.
> Tampere, Finland
> uwe.gxuder at nomovok.com (bot check: humans correct 1 obvious spelling
> error)
>
> _______________________________________________
> ckan-discuss mailing list
> ckan-discuss at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-discuss/attachments/20120628/98d32909/attachment.htm>