[wdmmg-dev] paster solr load departments on testing

[Friedrich Lindenberg]

Hi Carsten,

I've disabled some of the older solr cores (not used any longer) so
that we might not run out of resources as quickly and I've checked out
a new version of the schema to run in the testing.openspending.org
core. This means we have to rebuild the entire index for UAT (and
eventually also for production), so I've started a screen session on
us6 ("solr") to do that. Departments should be done quite soon :-) If
we see resource errors on eu4 again, we should investigate them more
closely (and perhaps raise the open files limit).

- Friedrich

On Tue, Apr 5, 2011 at 10:16 AM, Carsten Senger <senger at rehfisch.de> wrote:
> is there a secret how to load data into solr on testing?
>
> the solr powered search works. I can also connect to the solr admin page
> for the url listed in the paster ini file. For various subpages in the solr
> admin interface I get different http 500 error, like "Too many open files"
> (for [SCHEMA] or [CONFIG], but sometimes it works.). Maybe the solr server
> has to few resources?
>
> If I try to load a dataset into solr, I get this:
>
> (pyenv)okfn at us6:~/var/srvc/uat.openspending.org$ paster --plugin=wdmmg solr
> load departments
> Writing 500 records...
> Traceback (most recent call last):
>  File "/home/okfn/var/srvc/uat.openspending.org/pyenv/bin/paster", line 9,
> in <module>
>   load_entry_point('PasteScript==1.7.3', 'console_scripts', 'paster')()
>  File
> "/home/okfn/var/srvc/uat.openspending.org/pyenv/lib/python2.6/site-packages/paste/script/command.py",
> line 84, in run
>   invoke(command, command_name, options, args[1:])
>  File
> "/home/okfn/var/srvc/uat.openspending.org/pyenv/lib/python2.6/site-packages/paste/script/command.py",
> line 123, in invoke
>   exit_code = runner.run(args)
>  File
> "/home/okfn/var/srvc/uat.openspending.org/pyenv/lib/python2.6/site-packages/paste/script/command.py",
> line 218, in run
>   result = self.command()
>  File
> "/home/okfn/var/srvc/uat.openspending.org/pyenv/src/wdmmg/wdmmg/lib/cli.py",
> line 165, in command
>   wdmmg.lib.solrhelp.build_index(self.args[1])
>  File
> "/home/okfn/var/srvc/uat.openspending.org/pyenv/src/wdmmg/wdmmg/lib/solrhelp.py",
> line 116, in build_index
>   solr.add_many(buf)
>  File
> "/home/okfn/var/srvc/uat.openspending.org/pyenv/lib/python2.6/site-packages/solr/core.py",
> line 698, in add_many
>   return Solr.add_many(self, docs, commit=_commit)
>  File
> "/home/okfn/var/srvc/uat.openspending.org/pyenv/lib/python2.6/site-packages/solr/core.py",
> line 326, in wrapper
>   return self._update(content, query)
>  File
> "/home/okfn/var/srvc/uat.openspending.org/pyenv/lib/python2.6/site-packages/solr/core.py",
> line 550, in _update
>   rsp = self._post(selector, request, self.xmlheaders)
>  File
> "/home/okfn/var/srvc/uat.openspending.org/pyenv/lib/python2.6/site-packages/solr/core.py",
> line 639, in _post
>   return check_response_status(self.conn.getresponse())
>  File
> "/home/okfn/var/srvc/uat.openspending.org/pyenv/lib/python2.6/site-packages/solr/core.py",
> line 1096, in check_response_status
>   raise ex
> solr.core.SolrException: HTTP code=500, reason=Internal Server Error
>
>
> _______________________________________________
> wdmmg-dev mailing list
> wdmmg-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/wdmmg-dev
>