[ckan-dev] Error running solr in a container

Tue May 31 14:57:32 UTC 2016

Thanks Pedro - adding in the "ckan" to the `solr_url` key was the key issue. I wonder why this is not documented for CKAN?

>>> Pedro G Silva <pedro.gracio at deimos.com.pt> 30/05/16 5:03 PM >>>
 `solr_url` key is set correctly ( http://my.ip:8983/solr  ).
Using solr 5 you must set the url to 
solr_url = http://127.0.0.1:8983/solr/ckan

the reason is on the version on the docs creates the core automatically and there is no need to specify a core if there is only one. on v5, this does not happen even if you have only one core. refer to the solr documentation on this. a vanilla installation of ckan with solr5 with this setting on dev.ini, prod.ini or whatever works out of the box.

On Mon, May 30, 2016 at 3:11 PM, Derek Hohls <dhohls at csir.co.za> wrote:
Hi Pedro

1. The developer of the solr docker suggest mounting the .jar file in at runtime, so I will do that.
2. The "/solr/select/" that I refer to is the one generated inside the Python code; http://my.ip:8983/solr/ckan/select called from a browser certainly works... but that is not what the CKAN code creates (see my original stack trace)
3. I am using production.ini (not dev.ini or development.ini - neither file exists), and the `solr_url` key is set correctly ( http://my.ip:8983/solr  ).

I will add a temporary patch to the code to hard-code in "ckan" and see if I can progress  any further.

Thanks
Derek

>>> Pedro G Silva <pedro.gracio at deimos.com.pt> 30/05/16 1:02 PM >>>
Hi DerekThe JTS.jar is to enable the spatial support in solr. If you don't need it, skip it. Or maybe try to mkdir -p and cp the jar before installing solr, that way you don't have to deal with restarting the service
About the 404 error,  /solr/select/  should be  /solr/ckan/select/
check solr_url in dev.ini

On Mon, May 30, 2016 at 10:39 AM, Derek Hohls <dhohls at csir.co.za> wrote:
Hi again

After some further work, it seems as though the ckan "core" is now correctly setup for solr; the admin page at http://my.ip:8983/solr/#/ckan  displays without any obvious errors (although, to be honest, I do not  know enough to tell if there is something missing that should be there).

I have tried to implement all Pedro's changes for my solr 5 instance (adding the new .jar file is a bit tricky, as one cannot restart the solr service from inside the container...)

I am still hitting the same error though: the call being made to /solr/select/ by CKAN causes a 404 error; which can be duplicated in the browser:
HTTP ERROR 404 Problem accessing /solr/select/. Reason: 
    Not FoundPowered by Jetty://

There is nothing obvious in the solr.log file - running a tail while doing a page refresh does not trigger any new log entry,

I am not quite sure how to proceed from here?  Any other ideas or suggestions would be welcome!

Thanks
Derek

>>> "Derek Hohls" <dhohls at csir.co.za> 27/05/16 5:16 PM >>>
Just for the record, the log is under:

/opt/solr/server/logs/solr.log

I am working on automating the install by extracting the necessary files from a " run once" instance of a solr container, then overwriting the default schema.xml with the one supplied by CKAN, then remounting all these files back into the final instance of solr  (as per instructions in https://github.com/docker-solr/docker-solr/blob/master/5.3/scripts/docker-entrypoint.sh) to auto-create the ckan core files.

The solr container is working; but I need to get the CKAN container to communicate with it.

Thanks for the help so far.

Derek

>>> Pedro G Silva <pedro.gracio at deimos.com.pt> 27/05/16 11:45 AM >>>
logs are in /var/solr/logsif you want to automate the install you can just curl or wget that webpage to create the core

On Thu, May 26, 2016 at 2:09 PM, Derek Hohls <dhohls at csir.co.za> wrote:
Hi Pedro

Thanks for all the help here! - I did suspect it might be a missing core issue.  I am still on a learning curve with these technologies and am not sure what is / is not a requirement or pre-requisite for each.

I obviously missed the part in the CKAN install guide where I had to manually create the core; but I will look that up as I need to find a way to fully automate it at install time (i.e. without access to a browser as you suggest).

Have you any quick pointers about where the solr log is?  One of my frustrations about Java-based installs is that developers seem to get very creative about where everything goes (even between installs of exactly the same package).  It appears that the Docker install has placed everything under /opt/solr but I see your commands reference /var/solr ...

I will check, in the meantime, whether these other files you referenced, are correct on my side.

Thanks
Derek

>>> Pedro G Silva <pedro.gracio at deimos.com.pt> 25/05/16 4:55 PM >>>
Im using solr 5. Derek try 2 things:-see if you have a proxy set somewhere
-check the solr logs for more detailed info
-check that in that solr version that you have you created the core, on version 5 the link to solr is solr_url:8983/solr/core_name , and you have to manually create this core, since solr 5 doesn't automatically create cores. also you should check that all needed files were copied. I'll leave a snippet of my commands. Hope it helps
1
2
3
4
5
6
7
8
9
10
11
wget https://archive.apache.org/dist/lucene/solr/5.1.0/solr-5.1.0.tgz
tar xzf solr-5.1.0.tgz solr-5.1.0/bin/install_solr_service.sh --strip-components=2
sudo bash ./install_solr_service.sh solr-5.1.0.tgz
sudo -u solr mkdir -p /var/solr/data/configsets/ckan/conf
sudo ln -s /usr/lib/ckan/default/src/ckan/ckan/config/solr/schema.xml /var/solr/data/configsets/ckan/conf/schema.xml
sudo -u solr cp /opt/solr/server/solr/configsets/basic_configs/conf/solrconfig.xml /var/solr/data/configsets/ckan/conf/.
sudo -u solr cp /opt/solr/server/solr/configsets/basic_configs/conf/synonyms.txt /var/solr/data/configsets/ckan/conf/.
sudo -u solr cp /opt/solr/server/solr/configsets/basic_configs/conf/protwords.txt /var/solr/data/configsets/ckan/conf/.
wget http://search.maven.org/remotecontent?filepath=com/vividsolutions/jts/1.13/jts-1.13.jar -O jts-1.13.jar
sudo -u solr cp jts-1.13.jar /opt/solr/server/solr-webapp/webapp/WEB-INF/lib/.
sudo service solr restart

Go to the following link to create the SOLR Core

http://localhost:8983/solr/admin/cores?action=CREATE&name=ckan&configSet=ckan

On Wed, May 25, 2016 at 3:18 PM, David Read <david.read at hackneyworkshop.com> wrote:
Derek,

 Where did you get the idea ckan only supports solr v3.6? Because
 wherever it is, it needs fixing.

 I think most people run SOLR v4 and several are on v5.
 e.g. https://github.com/ckan/ckan/issues/2318

 Dave

 On 25 May 2016 at 10:55, Derek Hohls <dhohls at csir.co.za> wrote:
 > Hi
 >
 > Hope the list can help with this problem...
 >
 > I have setup and run a full CKAN install without problems in a VM.
 >
 > Now I am trying to run solr in a separate container.  I have done this,
 > using the Docker image from https://hub.docker.com/r/idguk/docker-solr/ (the
 > official solr Docker could not be used as it does not support the very old
 > version - 3.6 -  which is used by CKAN). The Docker is linked, via the "-v"
 > option, to the `schema.xml` file sourced from
 > `python-ckan_2.5-trusty_amd64.deb`.
 >
 > However, calling:
 >
 >     ckan db init
 >
 > triggers off an error; specifically this:
 >
 > Traceback (most recent call last):
 >   File "/usr/lib/ckan/default/src/ckan/ckan/lib/search/common.py", line 51,
 > in is_available
 >     conn.query("*:*", rows=1)
 >   File
 > "/usr/lib/ckan/default/local/lib/python2.7/site-packages/solr/core.py", line
 > 703, in query
 >     return self.select(*args, **params)
 >   File
 > "/usr/lib/ckan/default/local/lib/python2.7/site-packages/solr/core.py", line
 > 798, in __call__
 >     xml = self.raw(**params)
 >   File
 > "/usr/lib/ckan/default/local/lib/python2.7/site-packages/solr/core.py", line
 > 823, in raw
 >     rsp = conn._post(self.selector, request, conn.form_headers)
 >   File
 > "/usr/lib/ckan/default/local/lib/python2.7/site-packages/solr/core.py", line
 > 639, in _post
 >     return check_response_status(self.conn.getresponse())
 >   File
 > "/usr/lib/ckan/default/local/lib/python2.7/site-packages/solr/core.py", line
 > 1097, in check_response_status
 >     raise ex
 >
 > On checking the `solr/core.py` file, it seems as if its the call to
 > `solr/select/` that triggers the issue.
 >
 > Access via a browser to this URL (the container version of solr) shows a
 > Java error:
 >
 > java.lang.NullPointerException
 >     at org.apache.solr.servlet.SolrServlet.doGet(SolrServlet.java:92)
 >     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
 >     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
 >     at
 > org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
 >     at
 > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
 >     at
 > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:306)
 >     at
 > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
 >     at
 > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
 >     at
 > org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
 >     at
 > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
 >     at
 > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
 >     at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
 >     at
 > org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
 >     at
 > org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
 >     at
 > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
 >     at org.mortbay.jetty.Server.handle(Server.java:326)
 >     at
 > org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
 >     at
 > org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
 >     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
 >     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
 >     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
 >     at
 > org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
 >     at
 > org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
 >
 > But in the VM-version of solr, the same call generates an XML document:
 >
 > <?xml version="1.0" encoding="UTF-8"?>
 > <response>
 >     <lst name="responseHeader">
 >     <int name="status">0</int>
 >     <int name="QTime">1</int>
 >     <lst name="params"/></lst>
 >     <result name="response" numFound="0" start="0"/>
 > </response>
 >
 > I'd appreciate if anyone could highlight what I have missed in the
 > configuration that could cause this error.
 >
 > Thanks
 > Derek
 >
 >
 > --
 > This message is subject to the CSIR's copyright terms and conditions, e-mail
 > legal notice, and implemented Open Document Format (ODF) standard.
 > The full disclaimer details can be found at
 > http://www.csir.co.za/disclaimer.html.
 >
 >
 > This message has been scanned for viruses and dangerous content by
 > MailScanner,
 > and is believed to be clean.
 >
 >
 > Please consider the environment before printing this email.
 >
 >

> _______________________________________________
 > ckan-dev mailing list
 > ckan-dev at lists.okfn.org
 > https://lists.okfn.org/mailman/listinfo/ckan-dev
 > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
 >
 _______________________________________________
 ckan-dev mailing list
 ckan-dev at lists.okfn.org
 https://lists.okfn.org/mailman/listinfo/ckan-dev
 Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev

-- 

Pedro Gracio Silva 
Aerospace Systems / Production Center

Av. D. Joao II, Lt 1.17.01, 10 ? 1998-023 Lisboa, Portugal 
Ph: +351 21 893 3020 ? Fax: +351 21 896 9099

--  
This message is subject to the CSIR's copyright terms and conditions, e-mail legal notice, and implemented Open Document Format (ODF) standard. 
The full disclaimer details can be found at http://www.csir.co.za/disclaimer.html.  
This message has been scanned for viruses and dangerous content by MailScanner,  
and is believed to be clean. 