[wdmmg-dev] paster solr load departments on testing

Wed Apr 6 07:21:54 UTC 2011

--On Mittwoch, April 06, 2011 08:48:11 +0200 Friedrich Lindenberg 
<friedrich.lindenberg at okfn.org> wrote:

> Hi Carsten,
>
> I've disabled some of the older solr cores (not used any longer) so
> that we might not run out of resources as quickly and I've checked out
> a new version of the schema to run in the testing.openspending.org
> core. This means we have to rebuild the entire index for UAT (and
> eventually also for production), so I've started a screen session on
> us6 ("solr") to do that. Departments should be done quite soon :-) If
> we see resource errors on eu4 again, we should investigate them more
> closely (and perhaps raise the open files limit).

Thanks a lot. I can add a munin plugin that logs to munin.okfn.org
and set up an alert in munin.

..Carsten

> - Friedrich
>
> On Tue, Apr 5, 2011 at 10:16 AM, Carsten Senger <senger at rehfisch.de>
> wrote:
>> is there a secret how to load data into solr on testing?
>>
>> the solr powered search works. I can also connect to the solr admin page
>> for the url listed in the paster ini file. For various subpages in the
>> solr admin interface I get different http 500 error, like "Too many open
>> files" (for [SCHEMA] or [CONFIG], but sometimes it works.). Maybe the
>> solr server has to few resources?
>>
>> If I try to load a dataset into solr, I get this:
>>
>> (pyenv)okfn at us6:~/var/srvc/uat.openspending.org$ paster --plugin=wdmmg
>> solr load departments
>> Writing 500 records...
>> Traceback (most recent call last):
>>  File "/home/okfn/var/srvc/uat.openspending.org/pyenv/bin/paster", line
>> 9, in <module>
>>   load_entry_point('PasteScript==1.7.3', 'console_scripts', 'paster')()
>>  File
>> "/home/okfn/var/srvc/uat.openspending.org/pyenv/lib/python2.6/site-packa
>> ges/paste/script/command.py", line 84, in run
>>   invoke(command, command_name, options, args[1:])
>>  File
>> "/home/okfn/var/srvc/uat.openspending.org/pyenv/lib/python2.6/site-packa
>> ges/paste/script/command.py", line 123, in invoke
>>   exit_code = runner.run(args)
>>  File
>> "/home/okfn/var/srvc/uat.openspending.org/pyenv/lib/python2.6/site-packa
>> ges/paste/script/command.py", line 218, in run
>>   result = self.command()
>>  File
>> "/home/okfn/var/srvc/uat.openspending.org/pyenv/src/wdmmg/wdmmg/lib/cli.
>> py", line 165, in command
>>   wdmmg.lib.solrhelp.build_index(self.args[1])
>>  File
>> "/home/okfn/var/srvc/uat.openspending.org/pyenv/src/wdmmg/wdmmg/lib/solr
>> help.py", line 116, in build_index
>>   solr.add_many(buf)
>>  File
>> "/home/okfn/var/srvc/uat.openspending.org/pyenv/lib/python2.6/site-packa
>> ges/solr/core.py", line 698, in add_many
>>   return Solr.add_many(self, docs, commit=_commit)
>>  File
>> "/home/okfn/var/srvc/uat.openspending.org/pyenv/lib/python2.6/site-packa
>> ges/solr/core.py", line 326, in wrapper
>>   return self._update(content, query)
>>  File
>> "/home/okfn/var/srvc/uat.openspending.org/pyenv/lib/python2.6/site-packa
>> ges/solr/core.py", line 550, in _update
>>   rsp = self._post(selector, request, self.xmlheaders)
>>  File
>> "/home/okfn/var/srvc/uat.openspending.org/pyenv/lib/python2.6/site-packa
>> ges/solr/core.py", line 639, in _post
>>   return check_response_status(self.conn.getresponse())
>>  File
>> "/home/okfn/var/srvc/uat.openspending.org/pyenv/lib/python2.6/site-packa
>> ges/solr/core.py", line 1096, in check_response_status
>>   raise ex
>> solr.core.SolrException: HTTP code=500, reason=Internal Server Error
>>
>>
>> _______________________________________________
>> wdmmg-dev mailing list
>> wdmmg-dev at lists.okfn.org
>> http://lists.okfn.org/mailman/listinfo/wdmmg-dev
>>