[wdmmg-dev] wdmmg dev setup

Thu Jun 9 21:50:59 UTC 2011

Hi Jackson,

--On Donnerstag, Juni 09, 2011 14:46:57 +0100 Jackson Pauls
<jacksonpauls at gmail.com> wrote:

> Hi,
>
> I'm trying to set up wdmmg on my development machine to allow me to
> contribute some code/effort to the project. I'm following the
> instructions here:
> https://bitbucket.org/okfn/wdmmg/src/5d60b4d8f5d1/doc/install.rst
>
> It all seems ok until line 135. Running the nostests, I get 2 errors
> and 1 failure. The failure seems related to this:
> http://trac.openspending.org/ticket/170. The errors are as follow:

The failing test is intentional and you can ignore it.
It's a specific missing feature that does not affect your installation.

======================================================================
> ERROR: test suite for test_search.TestSearch
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> [...]
>   File "/home/j/src/wdmmgenv/lib/python2.7/site-packages/solr/core.py",
> line 639, in _post
>     return check_response_status(self.conn.getresponse())
>   File "/home/j/src/wdmmgenv/lib/python2.7/site-packages/solr/core.py",
> line 1096, in check_response_status
>     raise ex
> SolrException: HTTP code=404, reason=Not Found

I think our install instructions are not complete here.
For complete tests, you need to copy the test.example.ini to test.ini
and configure mongodb and solr in test.ini.
The solr to run the tests should be different from the solr you use
during development. It used to be possible to run the test
with "sorl.url = stub" in test.ini, but that does not work anymore.

A second solr could be a second core in a multi core solr
installation or a dedicated solr server (I'm not that familar with solr,
so the terminology maybe isn't completely correct).
For my development setup, I downloaded a solr tarball. It contains a
directory "example" from which you can start a solr with
`java -jar start.jar`. I just linked  solr/wdmmg_schema.xml from the
wdmmg repository as example/solr/conf/schema.xml and start solr from
the command line when I want to run the tests and use
http://localhost:8983/solr as the solr url.

> ======================================================================
> ERROR: Checks that the CRA datapkg is available and has all the files
> we expect it
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> [...]
>   File "/home/j/src/wdmmgenv/src/datapkg/datapkg/package.py", line 161,
> in load     dist = datapkg.distribution.load(path)
>   File
> "/home/j/src/wdmmgenv/src/datapkg/datapkg/distribution/__init__.py", line
> 44, in load
>     raise DatapkgException(msg)
> DatapkgException: Failed to load distribution from
> /home/j/src/wdmmgenv/wdmmg/pylons_data/getdata/ukgov-finances-cra
> ['File/Directory at
> /home/j/src/wdmmgenv/wdmmg/pylons_data/getdata/ukgov-finances-cra does
> not look like a Distribution', "[Errno 2] No such file or directory:
> '/home/j/src/wdmmgenv/wdmmg/pylons_data/getdata/ukgov-finances-cra/packag
> e.json'"]
>
> ======================================================================
>
>
> If I skip past that, I have a file that fails to download if I run the
> command at line 159 (all the others download ok):
> (wdmmgenv)[j at lianli wdmmg]$ datapkg download ckan://ukgov-finances-cra
> ./pylons_data/getdata
> [...]
> Downloading package resource: http://bitbucket.org/okfn/ukgov-finances-cra
> Total size: -1b
> [...]

I had a similar problem when I installed wdmmg first. I got the files
manually from a working wdmmg installation I think.

Maybe somebody familar with ckan knows the reason. I've temporarily upload
a tarball to <http://rehfisch.de/okfn/cofog_cra.tar.gz> which should
contain the necessary files to load cofog, cra and cra2010. Unpack it to
.../getdata.

> And if I ignore that I get the following related error when trying to
> load the data into the database:
> (wdmmgenv)[j at lianli wdmmg]$ paster load cra2010
> [...]
> IOError: [Errno 2] No such file or directory:
> '/home/j/src/wdmmgenv/wdmmg/pylons_data/getdata/ukgov-finances-cra/cofog_
> map.json'

This should go away then.

> Am in relatively unfamiliar territory with this software stack, would
> be very grateful for any pointers to what changes are needed, or where
> to look for answers.

If you run the tests check that it runs all wdmmg tests. It should be
~150-160.

..Carsten