[ckan-dev] problems configuring SOLR 4.5.0 with CKAN 2.0
Elena Camossi
elena.camossi at ext.jrc.ec.europa.eu
Tue Nov 5 16:24:45 UTC 2013
Hi everyone,
we're trying to configure CKAN 2.0 with SOLR 4.5.0, finally we managed to
have CKAN running, with database initialized and user admin created without
any error, but when we rebuild indexes and try inserting the test data, we
get again errors from SOLR. I post below the output of the two commands. Did
anyone manage to have the same configuration running?
Does anyone have any idea about why we got an HTML error sent to the console
when rebuilding the index, and what is the resource that is not available
(see below)?
Thanks a lot for any suggestion!
Kind regards,
-Elena
----------------------------------------------------------------------------
----------------------------------------------------------------------------
---------------------------------
REBUILDING THE INDEXES
paster --plugin=ckan search-index rebuild -c /etc/ckan/production.ini
Traceback (most recent call last):
File "/usr/local/ckan/pyenv/bin/paster", line 9, in <module>
load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')()
File
"/usr/local/ckan/pyenv/lib/python2.6/site-packages/paste/script/command.py",
line 104, in run
invoke(command, command_name, options, args[1:])
File
"/usr/local/ckan/pyenv/lib/python2.6/site-packages/paste/script/command.py",
line 143, in invoke
exit_code = runner.run(args)
File
"/usr/local/ckan/pyenv/lib/python2.6/site-packages/paste/script/command.py",
line 238, in run
result = self.command()
File "/usr/local/ckan/pyenv/src/ckan/ckan/lib/cli.py", line 370, in
command
self.rebuild()
File "/usr/local/ckan/pyenv/src/ckan/ckan/lib/cli.py", line 392, in
rebuild
defer_commit=(not self.options.commit_each))
File "/usr/local/ckan/pyenv/src/ckan/ckan/lib/search/__init__.py",
line 183, in rebuild
package_index.clear()
File "/usr/local/ckan/pyenv/src/ckan/ckan/lib/search/index.py", line 87,
in clear
clear_index()
File "/usr/local/ckan/pyenv/src/ckan/ckan/lib/search/index.py", line 58,
in clear_index
raise SearchIndexError(err)
ckan.lib.search.common.SearchIndexError: SOLR 'http://127.0.0.1:8080/solr'
exception: HTTP code=404, Reason=Not Found, body=<html><head><title>Apache
Tomcat/7.0.42 - Error
report</title><style><!--H1
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;fo
nt-size:22px;}
H2
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;fo
nt-size:16px;}
H3
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;fo
nt-size:14px;}
BODY
{font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;}
P
{font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:
12px;}A
{color : black;}A.name {color : black;}HR {color : #525D76;}--></style>
</head><body><h1>HTTP Status 404 - /solr/update</h1><HR size="1"
noshade="noshade"><p><b>type</b> Status report</p><p><b>message</b>
<u>/solr/update</u></p><p><b>description</b> <u>The requested resource is
not available.</u></p><HR size="1" noshade="noshade"><h3>Apache
Tomcat/7.0.42</h3></body></html>
CREATING THE TEST DATA
(pyenv)[ckan at drdsi-data ckan]$ paster create-test-data -c
/etc/ckan/production.ini Running setup_app() from ckan.websetup Creating
basic test data Traceback (most recent call last):
File "/usr/local/ckan/pyenv/bin/paster", line 9, in <module>
load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')()
File
"/usr/local/ckan/pyenv/lib/python2.6/site-packages/paste/script/command.py",
line 104, in run
invoke(command, command_name, options, args[1:])
File
"/usr/local/ckan/pyenv/lib/python2.6/site-packages/paste/script/command.py",
line 143, in invoke
exit_code = runner.run(args)
File
"/usr/local/ckan/pyenv/lib/python2.6/site-packages/paste/script/command.py",
line 238, in run
result = self.command()
File "/usr/local/ckan/pyenv/src/ckan/ckan/lib/cli.py", line 1297, in
command
CreateTestData.create_basic_test_data()
File "/usr/local/ckan/pyenv/src/ckan/ckan/lib/create_test_data.py",
line 25, in create_basic_test_data
cls.create()
File "/usr/local/ckan/pyenv/src/ckan/ckan/lib/create_test_data.py",
line 472, in create
model.repo.commit_and_remove()
File
"/usr/local/ckan/pyenv/lib/python2.6/site-packages/vdm/sqlalchemy/tools.py",
line 112, in commit_and_remove
self.commit()
File
"/usr/local/ckan/pyenv/lib/python2.6/site-packages/vdm/sqlalchemy/tools.py",
line 102, in commit
self.session.commit()
File
"/usr/local/ckan/pyenv/lib/python2.6/site-packages/sqlalchemy/orm/scoping.py
",
line 114, in do
return getattr(self.registry(), name)(*args, **kwargs)
File
"/usr/local/ckan/pyenv/lib/python2.6/site-packages/sqlalchemy/orm/session.py
",
line 656, in commit
self.transaction.commit()
File
"/usr/local/ckan/pyenv/lib/python2.6/site-packages/sqlalchemy/orm/session.py
",
line 314, in commit
self._prepare_impl()
File
"/usr/local/ckan/pyenv/lib/python2.6/site-packages/sqlalchemy/orm/session.py
",
line 290, in _prepare_impl
self.session.dispatch.before_commit(self.session)
File
"/usr/local/ckan/pyenv/lib/python2.6/site-packages/sqlalchemy/event.py",
line 291, in __call__
fn(*args, **kw)
File "/usr/local/ckan/pyenv/src/ckan/ckan/model/extension.py", line 103,
in before_commit
methodcaller('before_commit', session)
File "/usr/local/ckan/pyenv/src/ckan/ckan/model/extension.py", line 38,
in notify_observers
func(observer)
File "/usr/local/ckan/pyenv/src/ckan/ckan/model/modification.py",
line 37, in before_commit
self.notify(obj, domain_object.DomainObjectOperation.new)
File "/usr/local/ckan/pyenv/src/ckan/ckan/model/modification.py",
line 68, in notify
observer.notify(entity, operation)
File "/usr/local/ckan/pyenv/src/ckan/ckan/lib/search/__init__.py",
line 128, in notify
operation
File "/usr/local/ckan/pyenv/src/ckan/ckan/lib/search/__init__.py",
line 101, in dispatch_by_operation
index.insert_dict(entity)
File "/usr/local/ckan/pyenv/src/ckan/ckan/lib/search/index.py", line 75,
in insert_dict
return self.update_dict(data)
File "/usr/local/ckan/pyenv/src/ckan/ckan/lib/search/index.py", line 100,
in update_dict
self.index_package(pkg_dict, defer_commit)
File "/usr/local/ckan/pyenv/src/ckan/ckan/lib/search/index.py", line 259,
in index_package
raise SearchIndexError(msg)
ckan.lib.search.common.SearchIndexError: Solr returned an error: 404 Not
Found - <html><head><title>Apache Tomcat/7.0.42 - Error
report</title><style><!--H1
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;fo
nt-size:22px;}
H2
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;fo
nt-size:16px;}
H3
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;fo
nt-size:14px;}
BODY
{font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;}
P
{font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:
12px;}A
{color : black;}A.name {color : black;}HR {color : #525D76;}--></style>
</head><body><h1>HTTP Status 404 - /solr/update</h1><HR size="1"
noshade="noshade"><p><b>type</b> Status report</p><p><b>message</b>
<u>/solr/update</u></p><p><b>description</b> <u>The requested resource is
not available.</u></p><HR size="1" noshade="noshade"><h3>Apache
Tomcat/7.0.42</h3></body></html>
> -----Original Message-----
> From: ckan-dev-bounces at lists.okfn.org [mailto:ckan-dev-
> bounces at lists.okfn.org] On Behalf Of Elena Camossi
> Sent: giovedì 31 ottobre 2013 17:02
> To: ckan-dev at lists.okfn.org
> Subject: [ckan-dev] problems configuring SOLR 4.5.0 with CKAN 2.0
>
>
> Hi everyone,
>
> I'm trying to install CKAN 2.0 on Linux RedHAT using last stable version
of Solr,
> 4.5.0, with Tomcat 7.
> After having solved some issue with solr installation (it has changed
quite a
> bit with respect to the steps in CKAN documentation, that were for an
older
> version of Solr apparently) I have my Solr instance running, but it's
still not
> working correctly with CKAN.
>
> I'm at the final configuration steps, but when I create the database
tables
> with command
>
> paster db init -c /etc/ckan/production.ini
>
> I get the following error
>
> 2013-10-31 15:17:07,344 ERROR [ckan.lib.search.common] HTTP code=404,
> reason=Not Found Traceback (most recent call last):
> File "/usr/local/ckan/pyenv/src/ckan/ckan/lib/search/common.py", line
51,
> in is_available
> conn.query("*:*", rows=1)
> File "/usr/local/ckan/pyenv/lib/python2.6/site-packages/solr/core.py",
> line 703, in query
> return self.select(*args, **params)
> File "/usr/local/ckan/pyenv/lib/python2.6/site-packages/solr/core.py",
> line 798, in __call__
> xml = self.raw(**params)
> File "/usr/local/ckan/pyenv/lib/python2.6/site-packages/solr/core.py",
> line 823, in raw
> rsp = conn._post(self.selector, request, conn.form_headers)
> File "/usr/local/ckan/pyenv/lib/python2.6/site-packages/solr/core.py",
> line 639, in _post
> return check_response_status(self.conn.getresponse())
> File "/usr/local/ckan/pyenv/lib/python2.6/site-packages/solr/core.py",
> line 1097, in check_response_status
> raise ex
> SolrException: HTTP code=404, reason=Not Found
> 2013-10-31 15:17:07,345 WARNI [ckan.lib.search] Problems were found while
> connecting to the SOLR server
> 2013-10-31 15:17:07,516 INFO [ckan.model] CKAN database version remains
> as:
> 68
> 2013-10-31 15:17:07,564 INFO [ckan.model] Database initialised
Initialising
> DB: SUCCESS
>
> when I add the user admin
> (with command paster --plugin=ckan sysadmin add admin -c
> /etc/ckan/production.ini, because I found out that paster --plugin=ckan
user
> add admin -c /etc/ckan/production.ini doesn't
> work)
>
> I got a similar error
>
> paster --plugin=ckan sysadmin add admin -c /etc/ckan/production.ini
> 2013-10-31 15:24:31,262 ERROR [ckan.lib.search.common] HTTP code=404,
> reason=Not Found Traceback (most recent call last):
> File "/usr/local/ckan/pyenv/src/ckan/ckan/lib/search/common.py", line
51,
> in is_available
> conn.query("*:*", rows=1)
> File "/usr/local/ckan/pyenv/lib/python2.6/site-packages/solr/core.py",
> line 703, in query
> return self.select(*args, **params)
> File "/usr/local/ckan/pyenv/lib/python2.6/site-packages/solr/core.py",
> line 798, in __call__
> xml = self.raw(**params)
> File "/usr/local/ckan/pyenv/lib/python2.6/site-packages/solr/core.py",
> line 823, in raw
> rsp = conn._post(self.selector, request, conn.form_headers)
> File "/usr/local/ckan/pyenv/lib/python2.6/site-packages/solr/core.py",
> line 639, in _post
> return check_response_status(self.conn.getresponse())
> File "/usr/local/ckan/pyenv/lib/python2.6/site-packages/solr/core.py",
> line 1097, in check_response_status
> raise ex
> SolrException: HTTP code=404, reason=Not Found
> 2013-10-31 15:24:31,263 WARNI [ckan.lib.search] Problems were found while
> connecting to the SOLR server
>
> Then user admin is added
>
> User "admin" not found
> Create new user: admin? [y/n]y
> Password:
> Confirm password:
> Creating admin user
> Added admin as sysadmin
>
> and when I check the user list with command paster --plugin=ckan user list
-c
> /etc/ckan/production.ini i see the user has been added but I have still an
> error
>
> Traceback (most recent call last):
> File "/usr/local/ckan/pyenv/src/ckan/ckan/lib/search/common.py", line
51,
> in is_available
> conn.query("*:*", rows=1)
> File "/usr/local/ckan/pyenv/lib/python2.6/site-packages/solr/core.py",
> line 703, in query
> return self.select(*args, **params)
> File "/usr/local/ckan/pyenv/lib/python2.6/site-packages/solr/core.py",
> line 798, in __call__
> xml = self.raw(**params)
> File "/usr/local/ckan/pyenv/lib/python2.6/site-packages/solr/core.py",
> line 823, in raw
> rsp = conn._post(self.selector, request, conn.form_headers)
> File "/usr/local/ckan/pyenv/lib/python2.6/site-packages/solr/core.py",
> line 639, in _post
> return check_response_status(self.conn.getresponse())
> File "/usr/local/ckan/pyenv/lib/python2.6/site-packages/solr/core.py",
> line 1097, in check_response_status
> raise ex
> SolrException: HTTP code=404, reason=Not Found
> 2013-10-31 15:39:13,128 WARNI [ckan.lib.search] Problems were found while
> connecting to the SOLR server
> Users:
> count = 4
> name=
> name=admin
> name=logged_in
> name=visitor
>
> I went through the next steps of CKAN configuration, but of course it
doesn't
> work, and the problem is most likely related to Solr configuration.
> I post below the steps I followed for configuring Solr, hoping that
someone
> can give me some hint on when i went wrong.
>
> regards,
> -Elena
>
>
>
> -------------------------------------- Installing and configuring SOLR
>
----------------------------------------------------------------------------
> -----
>
>
> 1) Download and extract Apache SOLR (if you get an error, check if the
> name of the .tgz file is solr-xxx.tgz or apache-solr-xxx.tgz) curl
> http://archive.apache.org/dist/lucene/solr/4.5.0/solr-4.5.0.tgz | tar xzf
-
>
> 2) Create directories to hold SOLR cores (just one, in our case, called
> ckan):
>
> mkdir -p /usr/share/solr/ckan /var/lib/solr/data/ckan /etc/solr/ckan
>
> . /usr/share/solr/ckan is solr core home, where directory conf is
> . /var/lib/solr/data/ckan is data directory
> . /etc/solr/ckan is the actual folder for core configuration files
>
> 3) Cd to the directory where solr-4.5.0.tgz was downloaded and
> unpacked
> and Copy the Apache SOLR war to the desired location (you can eventually
> change the name of the file to solr.war)
>
> cp solr-4.5.0/dist/solr-4.5.0.war /usr/share/solr cp -r solr-
> 4.5.0/example/solr/collection1/conf /etc/solr/ckan
>
>
> 4) Create directory in solr home for libs and copy all .jar there
>
> mkdir /usr/share/solr/lib
> cp solr-4.5.0/dist/* /usr/share/solr/lib cp solr-4.5.0/contrib/analysis-
> extras/lib/* /usr/share/solr/lib cp solr-4.5.0/contrib/clustering/lib/*
> /usr/share/solr/lib cp solr-4.5.0/contrib/dataimporthandler/lib/*
> /usr/share/solr/lib cp solr-4.5.0/contrib/extraction/lib/*
/usr/share/solr/lib
> cp solr-4.5.0/contrib/langid/lib/* /usr/share/solr/lib cp solr-
> 4.5.0/contrib/uima/lib/* /usr/share/solr/lib cp solr-
> 4.5.0/contrib/velocity/lib/* /usr/share/solr/lib
>
> To avoid copying all jar, consider to edit file
> /etc/solr/ckan/conf/solrconfig.xml to remove all unnecessary sections
(what
> is not required by CKAN?)
>
> 5) Remove or comment in /etc/solr/ckan/conf/solrconfig.xml < dataDir>
> section, which is no longer necessary
>
> 6) Create a symbolic link between the configurations in /etc and /usr.
>
> ln -s /etc/solr/ckan/conf /usr/share/solr/ckan/conf
>
> 7) Remove the provided schema from the configured core and link the
> schema files in the CKAN source.
>
> rm -f /etc/solr/ckan/conf/schema.xml
> ln -s /usr/local/ckan/pyenv/src/ckan/ckan/config/solr/schema-2.0.xml
> /etc/solr/ckan/conf/schema.xml
>
> 8) For SOLR multicore installation, for each core create in
> /opt/tomcat7/conf/Catalina/localhost/ a new xml file specifying the
context
> (the solr home) for each core, each with the following content
>
> <Context docBase="/usr/share/solr/solr-4.5.0.war" crossContext="true">
> <Environment name="solr/home" type="java.lang.String"
> value="[complete path to solr core home]" override="true" /> </Context>
>
> For example, create file /opt/tomcat7/conf/Catalina/localhost/solr.xml:
>
> <Context docBase="/usr/share/solr/solr-4.5.0.war" crossContext="true">
> <Environment name="solr/home" type="java.lang.String"
> value="/usr/share/solr" override="true" /> </Context>
>
> 8) Create a new file, called /usr/share/solr/solr.xml, specifying the
> configuration for each core. For core ckan is:
> <solr persistent="true" sharedLib="lib">
> <cores adminPath="/admin/cores" defaultCoreName="ckan">
> <core name ="ckan-schema-2.0" instanceDir="ckan"> <property
> name="dataDir" value="/var/lib/solr/data/ckan" /></core>
> </cores>
> </solr>
>
>
> Enabling Tomcat
>
> 1) Copy the solr jars library from /usr/share/solr/conf/lib/ into your
> container's main lib directory (Tomcat). These jars will set up SLF4J and
log4j,
> Solr's new logging features.
>
> cd <solr download directory>
> cp -f solr-4.5.0/dist/* /opt/tomcat7/lib/
>
> 2) Set Permissions to enable tomcat user to access solr configuration
> chown -R tomcat:tomcat /usr/share/solr /var/lib/solr
>
> 3) Set up tomcat to start at startup, or restart the service is it's
> running
>
> chkconfig tomcat on
> service tomcat start
> or
> service tomcat restart
>
> 4) Check if Tomcat and Solr are running correctly:
> go to http://localhost:8080/ and check if tomcat is running.
> Then, go to http://localhost:8080/manager/html and connect as
> admin.
>
> Among the applications you should see solr, and going to
> http://localhost:80804/solr you should access Solr admin page.
> The path depends on the name of the file in
> /opt/tomcat7/conf/Catalina/localhost. It is [.]/solr if the name of the
file is
> /opt/tomcat7/conf/Catalina/localhost/solr.xml.
>
>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
More information about the ckan-dev
mailing list