[wdmmg-dev] paster solr load departments on testing

[Carsten Senger]

is there a secret how to load data into solr on testing?

the solr powered search works. I can also connect to the solr admin page
for the url listed in the paster ini file. For various subpages in the solr
admin interface I get different http 500 error, like "Too many open files"
(for [SCHEMA] or [CONFIG], but sometimes it works.). Maybe the solr server
has to few resources?

If I try to load a dataset into solr, I get this:

(pyenv)okfn at us6:~/var/srvc/uat.openspending.org$ paster --plugin=wdmmg solr 
load departments
Writing 500 records...
Traceback (most recent call last):
  File "/home/okfn/var/srvc/uat.openspending.org/pyenv/bin/paster", line 9, 
in <module>
    load_entry_point('PasteScript==1.7.3', 'console_scripts', 'paster')()
  File 
"/home/okfn/var/srvc/uat.openspending.org/pyenv/lib/python2.6/site-packages/paste/script/command.py", 
line 84, in run
    invoke(command, command_name, options, args[1:])
  File 
"/home/okfn/var/srvc/uat.openspending.org/pyenv/lib/python2.6/site-packages/paste/script/command.py", 
line 123, in invoke
    exit_code = runner.run(args)
  File 
"/home/okfn/var/srvc/uat.openspending.org/pyenv/lib/python2.6/site-packages/paste/script/command.py", 
line 218, in run
    result = self.command()
  File 
"/home/okfn/var/srvc/uat.openspending.org/pyenv/src/wdmmg/wdmmg/lib/cli.py", 
line 165, in command
    wdmmg.lib.solrhelp.build_index(self.args[1])
  File 
"/home/okfn/var/srvc/uat.openspending.org/pyenv/src/wdmmg/wdmmg/lib/solrhelp.py", 
line 116, in build_index
    solr.add_many(buf)
  File 
"/home/okfn/var/srvc/uat.openspending.org/pyenv/lib/python2.6/site-packages/solr/core.py", 
line 698, in add_many
    return Solr.add_many(self, docs, commit=_commit)
  File 
"/home/okfn/var/srvc/uat.openspending.org/pyenv/lib/python2.6/site-packages/solr/core.py", 
line 326, in wrapper
    return self._update(content, query)
  File 
"/home/okfn/var/srvc/uat.openspending.org/pyenv/lib/python2.6/site-packages/solr/core.py", 
line 550, in _update
    rsp = self._post(selector, request, self.xmlheaders)
  File 
"/home/okfn/var/srvc/uat.openspending.org/pyenv/lib/python2.6/site-packages/solr/core.py", 
line 639, in _post
    return check_response_status(self.conn.getresponse())
  File 
"/home/okfn/var/srvc/uat.openspending.org/pyenv/lib/python2.6/site-packages/solr/core.py", 
line 1096, in check_response_status
    raise ex
solr.core.SolrException: HTTP code=500, reason=Internal Server Error