[ckan-dev] Error running solr in a container

Wed Jun 1 06:50:49 UTC 2016

In an earlier message, David Read said:

  Where did you get the idea ckan only supports solr v3.6? 
  Because wherever it is, it needs fixing.

  I think most people run SOLR v4 and several are on v5.
  e.g. https://github.com/ckan/ckan/issues/2318

The reason I was using 5.x is because that is what the official solr Docker site supports (as well 6).

If the docs could clarify this, it would be great!

(As an aside, my experience with other open source projects suggests that if your project does not keep with changes and developments in your core dependencies - especially major releases - your project will start to experience problems at some point.)

>>> Adrià Mercader <adria.mercader at okfn.org> 31/05/16 7:41 PM >>>
Yes, we need to clarify this Sole core issue in the docs because it has caused quite a lot of confusion 
 On 31 May 2016 17:05, "Pedro G Silva" <pedro.gracio at deimos.com.pt> wrote:
I replied to derek (mistakly) private in other email:Hi DerekThe JTS.jar is to enable the spatial support in solr. If you don't need it, skip it. Or maybe try to mkdir -p and cp the jar before installing solr, that way you don't have to deal with restarting the service
About the 404 error,  /solr/select/  should be  /solr/ckan/select/
check solr_url in dev.ini

Derek it might not be documented in ckan documentation since ckan still oficially uses solr 3 or 4 (?) and you are using solr 5.

On Tue, May 31, 2016 at 3:59 PM, Derek Hohls <dhohls at csir.co.za> wrote:
Thanks Adrià

Yes it does - Pedro sent me the same clue "off list" and it is has worked (up to the point of the next error).

>>> Adrià Mercader <adria.mercader at okfn.org> 31/05/16 11:28 AM >>>
Hi Derek,
If I understood correctly, you created a Solr core named "ckan". In that case your sorl_url setting in the CKAN ini file should be:

solr_url = http://{solr_host}/solr/ckan

Replace {solr_host} with the appropriate value.

Does this help?

Adrià

On 30 May 2016 at 10:39, Derek Hohls <dhohls at csir.co.za> wrote:
Hi again

After some further work, it seems as though the ckan "core" is now correctly setup for solr; the admin page at http://my.ip:8983/solr/#/ckan  displays without any obvious errors (although, to be honest, I do not  know enough to tell if there is something missing that should be there).

I have tried to implement all Pedro's changes for my solr 5 instance (adding the new .jar file is a bit tricky, as one cannot restart the solr service from inside the container...)

I am still hitting the same error though: the call being made to /solr/select/ by CKAN causes a 404 error; which can be duplicated in the browser:
HTTP ERROR 404 Problem accessing /solr/select/. Reason: 
    Not FoundPowered by Jetty://

There is nothing obvious in the solr.log file - running a tail while doing a page refresh does not trigger any new log entry,

I am not quite sure how to proceed from here?  Any other ideas or suggestions would be welcome!

Thanks
Derek

>>> "Derek Hohls" <dhohls at csir.co.za> 27/05/16 5:16 PM >>>
Just for the record, the log is under:

/opt/solr/server/logs/solr.log

I am working on automating the install by extracting the necessary files from a " run once" instance of a solr container, then overwriting the default schema.xml with the one supplied by CKAN, then remounting all these files back into the final instance of solr  (as per instructions in https://github.com/docker-solr/docker-solr/blob/master/5.3/scripts/docker-entrypoint.sh) to auto-create the ckan core files.

The solr container is working; but I need to get the CKAN container to communicate with it.

Thanks for the help so far.

Derek

>>> Pedro G Silva <pedro.gracio at deimos.com.pt> 27/05/16 11:45 AM >>>
logs are in /var/solr/logsif you want to automate the install you can just curl or wget that webpage to create the core

On Thu, May 26, 2016 at 2:09 PM, Derek Hohls <dhohls at csir.co.za> wrote:
Hi Pedro

Thanks for all the help here! - I did suspect it might be a missing core issue.  I am still on a learning curve with these technologies and am not sure what is / is not a requirement or pre-requisite for each.

I obviously missed the part in the CKAN install guide where I had to manually create the core; but I will look that up as I need to find a way to fully automate it at install time (i.e. without access to a browser as you suggest).

Have you any quick pointers about where the solr log is?  One of my frustrations about Java-based installs is that developers seem to get very creative about where everything goes (even between installs of exactly the same package).  It appears that the Docker install has placed everything under /opt/solr but I see your commands reference /var/solr ...

I will check, in the meantime, whether these other files you referenced, are correct on my side.

Thanks
Derek

>>> Pedro G Silva <pedro.gracio at deimos.com.pt> 25/05/16 4:55 PM >>>
Im using solr 5. Derek try 2 things:-see if you have a proxy set somewhere
-check the solr logs for more detailed info
-check that in that solr version that you have you created the core, on version 5 the link to solr is solr_url:8983/solr/core_name , and you have to manually create this core, since solr 5 doesn't automatically create cores. also you should check that all needed files were copied. I'll leave a snippet of my commands. Hope it helps
1
2
3
4
5
6
7
8
9
10
11
wget https://archive.apache.org/dist/lucene/solr/5.1.0/solr-5.1.0.tgz
tar xzf solr-5.1.0.tgz solr-5.1.0/bin/install_solr_service.sh --strip-components=2
sudo bash ./install_solr_service.sh solr-5.1.0.tgz
sudo -u solr mkdir -p /var/solr/data/configsets/ckan/conf
sudo ln -s /usr/lib/ckan/default/src/ckan/ckan/config/solr/schema.xml /var/solr/data/configsets/ckan/conf/schema.xml
sudo -u solr cp /opt/solr/server/solr/configsets/basic_configs/conf/solrconfig.xml /var/solr/data/configsets/ckan/conf/.
sudo -u solr cp /opt/solr/server/solr/configsets/basic_configs/conf/synonyms.txt /var/solr/data/configsets/ckan/conf/.
sudo -u solr cp /opt/solr/server/solr/configsets/basic_configs/conf/protwords.txt /var/solr/data/configsets/ckan/conf/.
wget http://search.maven.org/remotecontent?filepath=com/vividsolutions/jts/1.13/jts-1.13.jar -O jts-1.13.jar
sudo -u solr cp jts-1.13.jar /opt/solr/server/solr-webapp/webapp/WEB-INF/lib/.
sudo service solr restart

Go to the following link to create the SOLR Core

http://localhost:8983/solr/admin/cores?action=CREATE&name=ckan&configSet=ckan

On Wed, May 25, 2016 at 3:18 PM, David Read <david.read at hackneyworkshop.com> wrote:
Derek,

 Where did you get the idea ckan only supports solr v3.6? Because
 wherever it is, it needs fixing.

 I think most people run SOLR v4 and several are on v5.
 e.g. https://github.com/ckan/ckan/issues/2318

 Dave

 On 25 May 2016 at 10:55, Derek Hohls <dhohls at csir.co.za> wrote:
 > Hi
 >
 > Hope the list can help with this problem...
 >
 > I have setup and run a full CKAN install without problems in a VM.
 >
 > Now I am trying to run solr in a separate container.  I have done this,
 > using the Docker image from https://hub.docker.com/r/idguk/docker-solr/ (the
 > official solr Docker could not be used as it does not support the very old
 > version - 3.6 -  which is used by CKAN). The Docker is linked, via the "-v"
 > option, to the `schema.xml` file sourced from
 > `python-ckan_2.5-trusty_amd64.deb`.
 >
 > However, calling:
 >
 >     ckan db init
 >
 > triggers off an error; specifically this:
 >
 > Traceback (most recent call last):
 >   File "/usr/lib/ckan/default/src/ckan/ckan/lib/search/common.py", line 51,
 > in is_available
 >     conn.query("*:*", rows=1)
 >   File
 > "/usr/lib/ckan/default/local/lib/python2.7/site-packages/solr/core.py", line
 > 703, in query
 >     return self.select(*args, **params)
 >   File
 > "/usr/lib/ckan/default/local/lib/python2.7/site-packages/solr/core.py", line
 > 798, in __call__
 >     xml = self.raw(**params)
 >   File
 > "/usr/lib/ckan/default/local/lib/python2.7/site-packages/solr/core.py", line
 > 823, in raw
 >     rsp = conn._post(self.selector, request, conn.form_headers)
 >   File
 > "/usr/lib/ckan/default/local/lib/python2.7/site-packages/solr/core.py", line
 > 639, in _post
 >     return check_response_status(self.conn.getresponse())
 >   File
 > "/usr/lib/ckan/default/local/lib/python2.7/site-packages/solr/core.py", line
 > 1097, in check_response_status
 >     raise ex
 >
 > On checking the `solr/core.py` file, it seems as if its the call to
 > `solr/select/` that triggers the issue.
 >
 > Access via a browser to this URL (the container version of solr) shows a
 > Java error:
 >
 > java.lang.NullPointerException
 >     at org.apache.solr.servlet.SolrServlet.doGet(SolrServlet.java:92)
 >     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
 >     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
 >     at
 > org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
 >     at
 > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
 >     at
 > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:306)
 >     at
 > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
 >     at
 > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
 >     at
 > org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
 >     at
 > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
 >     at
 > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
 >     at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
 >     at
 > org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
 >     at
 > org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
 >     at
 > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
 >     at org.mortbay.jetty.Server.handle(Server.java:326)
 >     at
 > org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
 >     at
 > org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
 >     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
 >     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
 >     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
 >     at
 > org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
 >     at
 > org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
 >
 > But in the VM-version of solr, the same call generates an XML document:
 >
 > <?xml version="1.0" encoding="UTF-8"?>
 > <response>
 >     <lst name="responseHeader">
 >     <int name="status">0</int>
 >     <int name="QTime">1</int>
 >     <lst name="params"/></lst>
 >     <result name="response" numFound="0" start="0"/>
 > </response>
 >
 > I'd appreciate if anyone could highlight what I have missed in the
 > configuration that could cause this error.
 >
 > Thanks
 > Derek
 >
 >
 > --
 > This message is subject to the CSIR's copyright terms and conditions, e-mail
 > legal notice, and implemented Open Document Format (ODF) standard.
 > The full disclaimer details can be found at
 > http://www.csir.co.za/disclaimer.html.
 >
 >
 > This message has been scanned for viruses and dangerous content by
 > MailScanner,
 > and is believed to be clean.
 >
 >
 > Please consider the environment before printing this email.
 >
 >

> _______________________________________________
 > ckan-dev mailing list
 > ckan-dev at lists.okfn.org
 > https://lists.okfn.org/mailman/listinfo/ckan-dev
 > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
 >
 _______________________________________________
 ckan-dev mailing list
 ckan-dev at lists.okfn.org
 https://lists.okfn.org/mailman/listinfo/ckan-dev
 Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev

-- 

Pedro Gracio Silva 
Aerospace Systems / Production Center

Av. D. Joao II, Lt 1.17.01, 10 ? 1998-023 Lisboa, Portugal 
Ph: +351 21 893 3020 ? Fax: +351 21 896 9099

--  
This message is subject to the CSIR's copyright terms and conditions, e-mail legal notice, and implemented Open Document Format (ODF) standard. 
The full disclaimer details can be found at http://www.csir.co.za/disclaimer.html.  
This message has been scanned for viruses and dangerous content by MailScanner,  
and is believed to be clean. 