[ckan-dev] Error running solr in a container

Adrià Mercader adria.mercader at okfn.org
Tue May 31 17:40:20 UTC 2016 

Yes, we need to clarify this Sole core issue in the docs because it has
caused quite a lot of confusion
On 31 May 2016 17:05, "Pedro G Silva" <pedro.gracio at deimos.com.pt> wrote:

> I replied to derek (mistakly) private in other email:
> Hi Derek
> The JTS.jar is to enable the spatial support in solr. If you don't need
> it, skip it. Or maybe try to mkdir -p and cp the jar before installing
> solr, that way you don't have to deal with restarting the service
> About the 404 error,  /solr/select/  should be  /solr/ckan/select/
> check solr_url in dev.ini
>
> Derek it might not be documented in ckan documentation since ckan still
> oficially uses solr 3 or 4 (?) and you are using solr 5.
>
> On Tue, May 31, 2016 at 3:59 PM, Derek Hohls <dhohls at csir.co.za> wrote:
>
>> Thanks Adrià
>>
>> Yes it does - Pedro sent me the same clue "off list" and it is has worked
>> (up to the point of the next error).
>>
>>  >>> Adrià Mercader <adria.mercader at okfn.org> 31/05/16 11:28 AM >>>
>> Hi Derek,
>>
>> If I understood correctly, you created a Solr core named "ckan". In that
>> case your sorl_url setting in the CKAN ini file should be:
>>
>> solr_url = http://{solr_host}/solr/ckan
>>
>> Replace {solr_host} with the appropriate value.
>>
>> Does this help?
>>
>> Adrià
>>
>>
>>
>> On 30 May 2016 at 10:39, Derek Hohls <dhohls at csir.co.za> wrote:
>>
>>> Hi again
>>>
>>> After some further work, it seems as though the ckan "core" is now
>>> correctly setup for solr; the admin page at
>>> http://my.ip:8983/solr/#/ckan displays without any obvious errors
>>> (although, to be honest, I do not know enough to tell if there is something
>>> missing that should be there).
>>>
>>> I have tried to implement all Pedro's changes for my solr 5 instance
>>> (adding the new .jar file is a bit tricky, as one cannot restart the solr
>>> service from inside the container...)
>>>
>>> I am still hitting the same error though: the call being made to
>>> /solr/select/ by CKAN causes a 404 error; which can be duplicated in the
>>> browser:
>>> HTTP ERROR 404
>>>
>>> Problem accessing /solr/select/. Reason:
>>>
>>>     Not Found
>>>
>>> ------------------------------
>>> *Powered by Jetty://*
>>>
>>> There is nothing obvious in the solr.log file - running a tail while
>>> doing a page refresh does not trigger any new log entry,
>>>
>>> I am not quite sure how to proceed from here?  Any other ideas or
>>> suggestions would be welcome!
>>>
>>> Thanks
>>> Derek
>>>
>>>
>>> >>> "Derek Hohls" <dhohls at csir.co.za> 27/05/16 5:16 PM >>>
>>> Just for the record, the log is under:
>>>
>>> /opt/solr/server/logs/solr.log
>>>
>>> I am working on automating the install by extracting the necessary files
>>> from a " run once" instance of a solr container, then overwriting the
>>> default schema.xml with the one supplied by CKAN, then remounting all these
>>> files back into the final instance of solr  (as per instructions in
>>> https://github.com/docker-solr/docker-solr/blob/master/5.3/scripts/docker-entrypoint.sh)
>>> to auto-create the ckan core files.
>>>
>>> The solr container is working; but I need to get the CKAN container to
>>> communicate with it.
>>>
>>> Thanks for the help so far.
>>>
>>> Derek
>>>
>>>
>>> >>> Pedro G Silva <pedro.gracio at deimos.com.pt> 27/05/16 11:45 AM >>>
>>> logs are in /var/solr/logs
>>> if you want to automate the install you can just curl or wget that
>>> webpage to create the core
>>>
>>> On Thu, May 26, 2016 at 2:09 PM, Derek Hohls <dhohls at csir.co.za> wrote:
>>>
>>>> Hi Pedro
>>>>
>>>> Thanks for all the help here! - I did suspect it might be a missing
>>>> core issue.  I am still on a learning curve with these technologies and am
>>>> not sure what is / is not a requirement or pre-requisite for each.
>>>>
>>>> I obviously missed the part in the CKAN install guide where I had to
>>>> manually create the core; but I will look that up as I need to find a way
>>>> to fully automate it at install time (i.e. without access to a browser as
>>>> you suggest).
>>>>
>>>> Have you any quick pointers about where the solr log is?  One of my
>>>> frustrations about Java-based installs is that developers seem to get very
>>>> creative about where everything goes (even between installs of exactly the
>>>> same package).  It appears that the Docker install has placed
>>>> everything under /opt/solr but I see your commands reference /var/solr ...
>>>>
>>>> I will check, in the meantime, whether these other files you
>>>> referenced, are correct on my side.
>>>>
>>>> Thanks
>>>> Derek
>>>>
>>>>
>>>> >>> Pedro G Silva <pedro.gracio at deimos.com.pt> 25/05/16 4:55 PM >>>
>>>> Im using solr 5. Derek try 2 things:
>>>> -see if you have a proxy set somewhere
>>>> -check the solr logs for more detailed info
>>>> -check that in that solr version that you have you created the core, on
>>>> version 5 the link to solr is solr_url:8983/solr/core_name , and you have
>>>> to manually create this core, since solr 5 doesn't automatically create
>>>> cores. also you should check that all needed files were copied. I'll leave
>>>> a snippet of my commands. Hope it helps
>>>> 1
>>>> 2
>>>> 3
>>>> 4
>>>> 5
>>>> 6
>>>> 7
>>>> 8
>>>> 9
>>>> 10
>>>> 11
>>>> wget https://archive.apache.org/dist/lucene/solr/5.1.0/solr-5.1.0.tgz
>>>> tar xzf solr-5.1.0.tgz solr-5.1.0/bin/install_solr_service.sh
>>>> --strip-components=2
>>>> sudo bash ./install_solr_service.sh solr-5.1.0.tgz
>>>> sudo -u solr mkdir -p /var/solr/data/configsets/ckan/conf
>>>> sudo ln -s /usr/lib/ckan/default/src/ckan/ckan/config/solr/schema.xml
>>>> /var/solr/data/configsets/ckan/conf/schema.xml
>>>> sudo -u solr cp
>>>> /opt/solr/server/solr/configsets/basic_configs/conf/solrconfig.xml
>>>> /var/solr/data/configsets/ckan/conf/.
>>>> sudo -u solr cp
>>>> /opt/solr/server/solr/configsets/basic_configs/conf/synonyms.txt
>>>> /var/solr/data/configsets/ckan/conf/.
>>>> sudo -u solr cp
>>>> /opt/solr/server/solr/configsets/basic_configs/conf/protwords.txt
>>>> /var/solr/data/configsets/ckan/conf/.
>>>> wget http://search.maven.org/remotecontent?filepath=com
>>>> /vividsolutions/jts/1.13/jts-1.13.jar -O jts-1.13.jar
>>>> sudo -u solr cp jts-1.13.jar
>>>> /opt/solr/server/solr-webapp/webapp/WEB-INF/lib/.
>>>> sudo service solr restart
>>>>
>>>> Go to the following link to create the SOLR Core
>>>>
>>>>
>>>> http://localhost:8983/solr/admin/cores?action=CREATE&name=ckan&configSet=ckan
>>>>
>>>> On Wed, May 25, 2016 at 3:18 PM, David Read <
>>>> david.read at hackneyworkshop.com> wrote:
>>>>
>>>>> Derek,
>>>>>
>>>>> Where did you get the idea ckan only supports solr v3.6? Because
>>>>> wherever it is, it needs fixing.
>>>>>
>>>>> I think most people run SOLR v4 and several are on v5.
>>>>> e.g. https://github.com/ckan/ckan/issues/2318
>>>>>
>>>>> Dave
>>>>>
>>>>> On 25 May 2016 at 10:55, Derek Hohls <dhohls at csir.co.za> wrote:
>>>>> > Hi
>>>>> >
>>>>> > Hope the list can help with this problem...
>>>>> >
>>>>> > I have setup and run a full CKAN install without problems in a VM.
>>>>> >
>>>>> > Now I am trying to run solr in a separate container.  I have done
>>>>> this,
>>>>> > using the Docker image from
>>>>> https://hub.docker.com/r/idguk/docker-solr/ (the
>>>>> > official solr Docker could not be used as it does not support the
>>>>> very old
>>>>> > version - 3.6 -  which is used by CKAN). The Docker is linked, via
>>>>> the "-v"
>>>>> > option, to the `schema.xml` file sourced from
>>>>> > `python-ckan_2.5-trusty_amd64.deb`.
>>>>> >
>>>>> > However, calling:
>>>>> >
>>>>> >     ckan db init
>>>>> >
>>>>> > triggers off an error; specifically this:
>>>>> >
>>>>> > Traceback (most recent call last):
>>>>> >   File "/usr/lib/ckan/default/src/ckan/ckan/lib/search/common.py",
>>>>> line 51,
>>>>> > in is_available
>>>>> >     conn.query("*:*", rows=1)
>>>>> >   File
>>>>> >
>>>>> "/usr/lib/ckan/default/local/lib/python2.7/site-packages/solr/core.py", line
>>>>> > 703, in query
>>>>> >     return self.select(*args, **params)
>>>>> >   File
>>>>> >
>>>>> "/usr/lib/ckan/default/local/lib/python2.7/site-packages/solr/core.py", line
>>>>> > 798, in __call__
>>>>> >     xml = self.raw(**params)
>>>>> >   File
>>>>> >
>>>>> "/usr/lib/ckan/default/local/lib/python2.7/site-packages/solr/core.py", line
>>>>> > 823, in raw
>>>>> >     rsp = conn._post(self.selector, request, conn.form_headers)
>>>>> >   File
>>>>> >
>>>>> "/usr/lib/ckan/default/local/lib/python2.7/site-packages/solr/core.py", line
>>>>> > 639, in _post
>>>>> >     return check_response_status(self.conn.getresponse())
>>>>> >   File
>>>>> >
>>>>> "/usr/lib/ckan/default/local/lib/python2.7/site-packages/solr/core.py", line
>>>>> > 1097, in check_response_status
>>>>> >     raise ex
>>>>> >
>>>>> > On checking the `solr/core.py` file, it seems as if its the call to
>>>>> > `solr/select/` that triggers the issue.
>>>>> >
>>>>> > Access via a browser to this URL (the container version of solr)
>>>>> shows a
>>>>> > Java error:
>>>>> >
>>>>> > java.lang.NullPointerException
>>>>> >     at org.apache.solr.servlet.SolrServlet.doGet(SolrServlet.java:92)
>>>>> >     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>>>> >     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>> >     at
>>>>> >
>>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>>> >     at
>>>>> >
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>>>> >     at
>>>>> >
>>>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:306)
>>>>> >     at
>>>>> >
>>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>> >     at
>>>>> >
>>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>> >     at
>>>>> >
>>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>>> >     at
>>>>> >
>>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>>> >     at
>>>>> >
>>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>> >     at
>>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>>> >     at
>>>>> >
>>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>>> >     at
>>>>> >
>>>>> org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
>>>>> >     at
>>>>> >
>>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>> >     at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>> >     at
>>>>> >
>>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>> >     at
>>>>> >
>>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>> >     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>> >     at
>>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>> >     at
>>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>> >     at
>>>>> >
>>>>> org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
>>>>> >     at
>>>>> >
>>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>> >
>>>>> > But in the VM-version of solr, the same call generates an XML
>>>>> document:
>>>>> >
>>>>> > <?xml version="1.0" encoding="UTF-8"?>
>>>>> > <response>
>>>>> >     <lst name="responseHeader">
>>>>> >     <int name="status">0</int>
>>>>> >     <int name="QTime">1</int>
>>>>> >     <lst name="params"/></lst>
>>>>> >     <result name="response" numFound="0" start="0"/>
>>>>> > </response>
>>>>> >
>>>>> > I'd appreciate if anyone could highlight what I have missed in the
>>>>> > configuration that could cause this error.
>>>>> >
>>>>> > Thanks
>>>>> > Derek
>>>>> >
>>>>> >
>>>>> > --
>>>>> > This message is subject to the CSIR's copyright terms and
>>>>> conditions, e-mail
>>>>> > legal notice, and implemented Open Document Format (ODF) standard.
>>>>> > The full disclaimer details can be found at
>>>>> > http://www.csir.co.za/disclaimer.html.
>>>>> >
>>>>> >
>>>>> > This message has been scanned for viruses and dangerous content by
>>>>> > MailScanner,
>>>>> > and is believed to be clean.
>>>>> >
>>>>> >
>>>>> > Please consider the environment before printing this email.
>>>>> >
>>>>> >
>>>>> > _______________________________________________
>>>>> > ckan-dev mailing list
>>>>> > ckan-dev at lists.okfn.org
>>>>> > https://lists.okfn.org/mailman/listinfo/ckan-dev
>>>>> > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>>> >
>>>>> _______________________________________________
>>>>> ckan-dev mailing list
>>>>> ckan-dev at lists.okfn.org
>>>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>>
>>>> [image: DEIMOS Engenharia, SA] <http://www.deimos.pt>
>>>> *Pedro Gracio Silva*
>>>> Aerospace Systems / Production Center
>>>>
>>>> *Av. D. Joao II, Lt 1.17.01, 10 ? 1998-023 Lisboa, Portugal*
>>>> Ph: +351 21 893 3020 ? Fax: +351 21 896 9099
>>>>
>>>>
>>>>
>>>> --
>>>> This message is subject to the CSIR's copyright terms and conditions,
>>>> e-mail legal notice, and implemented Open Document Format (ODF) standard.
>>>> The full disclaimer details can be found at
>>>> http://www.csir.co.za/disclaimer.html.
>>>>
>>>>
>>>> This message has been scanned for viruses and dangerous content by
>>>> *MailScanner* <http://www.mailscanner.info/>,
>>>> and is believed to be clean.
>>>>
>>>>
>>>> Please consider the environment before printing this email.
>>>>
>>>> --
>>>> This message is subject to the CSIR's copyright terms and conditions,
>>>> e-mail legal notice, and implemented Open Document Format (ODF) standard.
>>>> The full disclaimer details can be found at
>>>> http://www.csir.co.za/disclaimer.html.
>>>>
>>>>
>>>> This message has been scanned for viruses and dangerous content by
>>>> *MailScanner* <http://www.mailscanner.info/>,
>>>> and is believed to be clean.
>>>>
>>>>
>>>> Please consider the environment before printing this email.
>>>>
>>>>
>>>
>>>
>>> --
>>>
>>>
>>> [image: DEIMOS Engenharia, SA] <http://www.deimos.pt>
>>> *Pedro Gracio Silva*
>>> Aerospace Systems / Production Center
>>>
>>> *Av. D. Joao II, Lt 1.17.01, 10 ? 1998-023 Lisboa, Portugal*
>>> Ph: +351 21 893 3020 ? Fax: +351 21 896 9099
>>>
>>>
>>>
>>> --
>>> This message is subject to the CSIR's copyright terms and conditions,
>>> e-mail legal notice, and implemented Open Document Format (ODF) standard.
>>> The full disclaimer details can be found at
>>> http://www.csir.co.za/disclaimer.html.
>>>
>>>
>>> This message has been scanned for viruses and dangerous content by
>>> *MailScanner* <http://www.mailscanner.info/>,
>>> and is believed to be clean.
>>>
>>>
>>> Please consider the environment before printing this email.
>>>
>>> --
>>> This message is subject to the CSIR's copyright terms and conditions,
>>> e-mail legal notice, and implemented Open Document Format (ODF) standard.
>>> The full disclaimer details can be found at
>>> http://www.csir.co.za/disclaimer.html.
>>>
>>>
>>> This message has been scanned for viruses and dangerous content by
>>> *MailScanner* <http://www.mailscanner.info/>,
>>> and is believed to be clean.
>>>
>>>
>>> Please consider the environment before printing this email.
>>>
>>> --
>>> This message is subject to the CSIR's copyright terms and conditions,
>>> e-mail legal notice, and implemented Open Document Format (ODF) standard.
>>> The full disclaimer details can be found at
>>> http://www.csir.co.za/disclaimer.html.
>>>
>>>
>>> This message has been scanned for viruses and dangerous content by
>>> *MailScanner* <http://www.mailscanner.info/>,
>>> and is believed to be clean.
>>>
>>>
>>> Please consider the environment before printing this email.
>>>
>>>
>>> _______________________________________________
>>> ckan-dev mailing list
>>> ckan-dev at lists.okfn.org
>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>
>>>
>>
>> --
>> This message is subject to the CSIR's copyright terms and conditions,
>> e-mail legal notice, and implemented Open Document Format (ODF) standard.
>> The full disclaimer details can be found at
>> http://www.csir.co.za/disclaimer.html.
>>
>>
>> This message has been scanned for viruses and dangerous content by
>> *MailScanner* <http://www.mailscanner.info/>,
>> and is believed to be clean.
>>
>>
>> Please consider the environment before printing this email.
>>
>> --
>> This message is subject to the CSIR's copyright terms and conditions,
>> e-mail legal notice, and implemented Open Document Format (ODF) standard.
>> The full disclaimer details can be found at
>> http://www.csir.co.za/disclaimer.html.
>>
>>
>> This message has been scanned for viruses and dangerous content by
>> *MailScanner* <http://www.mailscanner.info/>,
>> and is believed to be clean.
>>
>>
>> Please consider the environment before printing this email.
>>
>>
>
>
> --
>
>
> [image: DEIMOS Engenharia, SA] <http://www.deimos.pt>
> *Pedro Gracio Silva*
> Aerospace Systems / Production Center
>
> *Av. D. Joao II, Lt 1.17.01, 10 • 1998-023 Lisboa, Portugal*
> Ph: +351 21 893 3020 • Fax: +351 21 896 9099
>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20160531/b93fa17b/attachment-0003.html>

More information about the ckan-dev mailing list