[wdmmg-dev] paster solr load departments on testing

Wed Apr 6 06:48:11 UTC 2011

Hi Carsten,

I've disabled some of the older solr cores (not used any longer) so
that we might not run out of resources as quickly and I've checked out
a new version of the schema to run in the testing.openspending.org
core. This means we have to rebuild the entire index for UAT (and
eventually also for production), so I've started a screen session on
us6 ("solr") to do that. Departments should be done quite soon :-) If
we see resource errors on eu4 again, we should investigate them more
closely (and perhaps raise the open files limit).

- Friedrich

On Tue, Apr 5, 2011 at 10:16 AM, Carsten Senger <senger at rehfisch.de> wrote:
> is there a secret how to load data into solr on testing?
>
> the solr powered search works. I can also connect to the solr admin page
> for the url listed in the paster ini file. For various subpages in the solr
> admin interface I get different http 500 error, like "Too many open files"
> (for [SCHEMA] or [CONFIG], but sometimes it works.). Maybe the solr server
> has to few resources?
>
> If I try to load a dataset into solr, I get this:
>
> (pyenv)okfn at us6:~/var/srvc/uat.openspending.org$ paster --plugin=wdmmg solr
> load departments
> Writing 500 records...
> Traceback (most recent call last):
>  File "/home/okfn/var/srvc/uat.openspending.org/pyenv/bin/paster", line 9,
> in <module>
>   load_entry_point('PasteScript==1.7.3', 'console_scripts', 'paster')()
>  File
> "/home/okfn/var/srvc/uat.openspending.org/pyenv/lib/python2.6/site-packages/paste/script/command.py",
> line 84, in run
>   invoke(command, command_name, options, args[1:])
>  File
> "/home/okfn/var/srvc/uat.openspending.org/pyenv/lib/python2.6/site-packages/paste/script/command.py",
> line 123, in invoke
>   exit_code = runner.run(args)
>  File
> "/home/okfn/var/srvc/uat.openspending.org/pyenv/lib/python2.6/site-packages/paste/script/command.py",
> line 218, in run
>   result = self.command()
>  File
> "/home/okfn/var/srvc/uat.openspending.org/pyenv/src/wdmmg/wdmmg/lib/cli.py",
> line 165, in command
>   wdmmg.lib.solrhelp.build_index(self.args[1])
>  File
> "/home/okfn/var/srvc/uat.openspending.org/pyenv/src/wdmmg/wdmmg/lib/solrhelp.py",
> line 116, in build_index
>   solr.add_many(buf)
>  File
> "/home/okfn/var/srvc/uat.openspending.org/pyenv/lib/python2.6/site-packages/solr/core.py",
> line 698, in add_many
>   return Solr.add_many(self, docs, commit=_commit)
>  File
> "/home/okfn/var/srvc/uat.openspending.org/pyenv/lib/python2.6/site-packages/solr/core.py",
> line 326, in wrapper
>   return self._update(content, query)
>  File
> "/home/okfn/var/srvc/uat.openspending.org/pyenv/lib/python2.6/site-packages/solr/core.py",
> line 550, in _update
>   rsp = self._post(selector, request, self.xmlheaders)
>  File
> "/home/okfn/var/srvc/uat.openspending.org/pyenv/lib/python2.6/site-packages/solr/core.py",
> line 639, in _post
>   return check_response_status(self.conn.getresponse())
>  File
> "/home/okfn/var/srvc/uat.openspending.org/pyenv/lib/python2.6/site-packages/solr/core.py",
> line 1096, in check_response_status
>   raise ex
> solr.core.SolrException: HTTP code=500, reason=Internal Server Error
>
>
> _______________________________________________
> wdmmg-dev mailing list
> wdmmg-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/wdmmg-dev
>