[CKAN-support] [Open Knowledge Foundation] Update: Solr with Oracle JDK

Sat Apr 5 07:01:50 UTC 2014

##- Please type your reply above this line -##

[Open Knowledge Foundation] Update: Solr with Oracle JDK

You are registered as a CC on this support request (278). Reply to this email to add a comment to the request.

----------------------------------------------

George Sattler, Apr 05 08:01

Hi Nigel

I’ve successfully made a switch on our demo site ckan.xvt.com.au from using the solr-jetty package to Solr 4 with Jetty 9.

So far, so good. I’ve been hitting it again with scripted requests and found memory to be stable. The HTML dashboard that comes with Solr 4 doesn’t show any increase in JVM memory usage at all.

If anyone is interested in the method I used…

1. Remove solr-jetty package

2. Remove openjdk-6-jdk

3. Install openjdk-7-jdk

4. I followed this script to install Jetty 9 and Solr 4 https://gist.github.com/vumaasha/6756423 . I actually ran the commands by hand and replaced the globals at the start of the script with updated URLs.

5. This is my /etc/default/jetty configuration:

NO_START=0
JAVA_HOME=/usr/bin/java
JETTY_HOME=/opt/jetty
JETTY_HOST=0.0.0.0
JETTY_ARGS=jetty.port=8983
JETTY_USER=jetty
JAVA_OPTIONS="-Dsolr.solr.home=/opt/solr/cores/production -Xms16M -Xmx64M -XX:Ma
#JAVA_OPTIONS='"-Dsolr.solr.home='$SOLR_HOME'/cores/production '$JAVA_OPTIONS'"'

6. Replaced /opt/solr/cores/production/collection1/conf/schema.xml with CKAN schema

7. Restart Jetty

Cheers

George

----------------------------------------------

Nigel Babu, Apr 04 04:51

Hi George,

We think it could be connected to security updates as well because it's hitting sites where we haven't done anything new for a while. If you have better performance on Solr 4 with Tomcat, we may end up going that route as well. I'm going to send an email to the dev list to see if more people have noticed than us.

Thanks for taking a deeper look into it!

Nigel.

----------------------------------------------

George Sattler, Apr 04 04:46

Hi Nigel

Our Java guys here have tried to find what the problem is but it seems to not be something that’s very obvious. It just seems to be that heap space gets used up until Solr fails. Then a restart of Jetty is required.

It’s strange, but the problem just suddenly occurred. I’m not sure if this is the cause but about a week (approx.) before the problem was noticed, we had run a heap of security updates on the machine for SSQ. If you guys did the same thing before the problem started occurring then it might be the cause.

I’ve done some testing with an internet-facing, vanilla CKAN installation (source install). I just ran a script with wget to hit the front page 5 times per second. Within a few hours, the process ‘jscv’ had used up 50% memory. After I stopped the script, the memory used remained at 50%. Looks like some unreleased memory of some kind.

I’m going to experiment with Solr 4 with Jetty 9 (or Solr with Tomcat) this weekend and I’ll let you know if I come up with a good combination.

I know that you used tomcat for data.nsw.gov.au and it’s been working just fine.

Cheers

George

----------------------------------------------

Nigel Babu, Apr 03 17:06

Hi George,

Like you've noticed, we're running into the exact problem on multiple Solr machines and we're experimenting with fixes and monitoring. OracleJDK is not contraindicated for CKAN for sure and we'd love to hear your results on it about this particular problem.

In the past, I've deployed CKAN for a client on Red Hat with Oracle JDK and it worked just fine.

----------------------------------------------

George Sattler, Apr 03 09:30

Hi ckan support

Just further to my email below, i noticed that demo.ckan.org<http://demo.ckan.org> seems to be returning an error right now (8:28am UTC) - this looks like it might possibly be a solr OutOfMemory error - the site seems to be presenting the same symptoms!
Cheers
George

Sent from my iPad

Begin forwarded message:

From: George Sattler <George.sattler at xvt.com.au<mailto:George.sattler at xvt.com.au>>
Date: 27 March 2014 8:42:13 pm AEDT
To: "support at ckan.org<mailto:support at ckan.org>" <support at ckan.org<mailto:support at ckan.org>>
Subject: Solr with Oracle JDK

Hi ckan support
Another quick question (i hope)...

I'm having some Solr issues - running out of heap space memory for data.qld.gov.au<http://data.qld.gov.au> and publications.qld.gov.au<http://publications.qld.gov.au>. Increasing heap space memory has been tried and the cause of the problem appears to be quite illusive.

I'm thinking of switching to OracleJDK instead of OpenJDK to see if it makes any difference.

So, is there any reason why _not_ to use OracleJDK? Just asking in case there's some known issues with doing this.

Thanks
George

Sent from my iPad

________________________________
This email and any files transmitted with it is confidential and intended solely for the use of the addressee. The unauthorised use, dissemination, forwarding, printing or copying of this communication is strictly prohibited. If you have received this communication in error please notify us immediately by reply email and destroy this communication. Any views and opinions presented in this email are solely those of the author and do not necessarily represent the views of XVT Solutions. The recipient should check this email and any attachments for viruses. XVT Solutions accepts no liability for the content of this email, and any damage caused by any viruses that could potentially be transmitted through this email.

----------------------------------------------

George Sattler, Mar 27 09:43

Hi ckan support
Another quick question (i hope)...

I'm having some Solr issues - running out of heap space memory for data.qld.gov.au and publications.qld.gov.au. Increasing heap space memory has been tried and the cause of the problem appears to be quite illusive.

I'm thinking of switching to OracleJDK instead of OpenJDK to see if it makes any difference.

So, is there any reason why _not_ to use OracleJDK? Just asking in case there's some known issues with doing this.

Thanks
George

Sent from my iPad

This email and any files transmitted with it is confidential and intended solely for the use of the addressee. The unauthorised use, dissemination, forwarding, printing or copying of this communication is strictly prohibited. If you have received this communication in error please notify us immediately by reply email and destroy this communication. Any views and opinions presented in this email are solely those of the author and do not necessarily represent the views of XVT Solutions. The recipient should check this email and any attachments for viruses. XVT Solutions accepts no liability for the content of this email, and any damage caused by any viruses that could potentially be transmitted through this email.

--------------------------------
This email is a service from Open Knowledge Foundation.

[GS28-9Z14]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.okfn.org/mailman/private/ckan-support/attachments/20140405/983d8577/attachment-0003.html>