[ckan-dev] Solr out of memory

Sun Dec 6 09:51:49 UTC 2015

Hi all,

Recently we moved from an m3.large aws instance to a 4GB linode, having
discovered that we didn't require nearly as much power as we thought.
However, since then, our Solr core has been causing some major issues. To
begin with, it runs out of memory very quickly (with a JavaOutOfMemory
exception). Other times, I simply get a 500 from solr, with no other info.
I have only about 1200 datasets - surely I don't need more than 4GB of RAM
to search this?

Could anyone point me in the direction of the best way to debug these
issues? I find myself restarting jetty every ten minutes in order to get my
search back, which is a little unsustainable ;)

The problem seems to have manifested itself after google decided to crawl
all our search pages yesterday, 1 per second. I've brought this rate down
to 0.3 a second, which has helped a little. Prior to this, we had
reasonably stable search for about a month.

I notice that there are quite a large number of indexed fields in
schema.xml - are these all necessary? Same goes for stored fields. (I'm
using a slightly modified version of the data.gov.uk schema).

Really though, I'm just shooting in the dark - I don't know if it's my
schema, or if it's anything else, and so some info on how to debug this
would be great.

Cheers,
Carl
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20151206/93faa419/attachment-0002.html>