[ckan-dev] Solr out of memory

[Carl Lange]

OK, so it looks like the issue has been mitigated for now. Thanks Ross and
Marianne for your help!

First off, I set up nginx to cache everything it could get its little hands
on, and also gzip everything. This helped with the crawlers, and with
general site responsiveness (in fact, it's a miracle we'd gotten by without
it before now).

I set the solr max heap limit (in /etc/default/jetty, $JAVA_OPTIONS, -Xmx
to 1536M), and set apache's prefork conf to some lower values (to prevent
solr and apache both getting too big for their boots). This bought some
time, but I still had to restart jetty every 45 minutes or so, as it would
run out of memory, try to GC, and basically stop the world forever.

Next, I did a lot of diagnostics with new relic and the like, which were
immensely helpful, especially to see the impact of any changes I'd made.

Also, I set up monit to restart jetty if it stops (just in case, and so I
could get some sleep :) ).

Then, I set a few defaults in solrconfig.xml to be lower than the usual,
particularly switching on *autoCommit* and *autoSoftCommit*.

Finally, *disabling storage of a few fields in schema.xml*
(particularly *validated_data_dict
*and *data_dict*), *disabling index of a few fields *that we don't ever use
for search, and reindexing has basically fixed everything. The size of my
solr data dir (/usr/share/solr/solr-4.3.1/example/solr/collection1/data)
has gone down a huge amount (from 120MB to 18MB), and solr's ram usage more
than halved. Since this, I have not seen any issues at all.

I have not seen any detrimental effects from this - anywhere in particular
I should look?

All in all, the site is nearly 10x faster at responding than this time last
week, and can now handle more than three people getting the dcat catalogue
at once :)

Cheers,
Carl

On 6 December 2015 at 13:21, Exversion <marianne at exversion.com> wrote:

> There's a neat suggestion in Solr documentation that might help you debug:
>
> If you want to see the heap when OOM occurs set
> "-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/path/to/the/dump"
>
> Reference: https://wiki.apache.org/solr/SolrPerformanceFactors
>
>
> Sent from my iPhone
>
> On Dec 6, 2015, at 7:59 AM, Carl Lange <carl at derilinx.com> wrote:
>
> I hadn't set -Xmx previously, so I've set it to 2048M. However, now solr
> simply dies when it hits that limit, which seems like odd behaviour…
>
> On 6 December 2015 at 12:12, Exversion <marianne at exversion.com> wrote:
>
>> First thing I would check is how your heap size compares to your actual
>> memory size. The JVM is greedy. If you set your heap size to 5GB on a 4GB
>> machine it will routinely run itself into the ground
>>
>> Sent from my iPhone
>>
>> On Dec 6, 2015, at 4:51 AM, Carl Lange <carl at derilinx.com> wrote:
>>
>> Hi all,
>>
>> Recently we moved from an m3.large aws instance to a 4GB linode, having
>> discovered that we didn't require nearly as much power as we thought.
>> However, since then, our Solr core has been causing some major issues. To
>> begin with, it runs out of memory very quickly (with a JavaOutOfMemory
>> exception). Other times, I simply get a 500 from solr, with no other info.
>> I have only about 1200 datasets - surely I don't need more than 4GB of RAM
>> to search this?
>>
>> Could anyone point me in the direction of the best way to debug these
>> issues? I find myself restarting jetty every ten minutes in order to get my
>> search back, which is a little unsustainable ;)
>>
>> The problem seems to have manifested itself after google decided to crawl
>> all our search pages yesterday, 1 per second. I've brought this rate down
>> to 0.3 a second, which has helped a little. Prior to this, we had
>> reasonably stable search for about a month.
>>
>> I notice that there are quite a large number of indexed fields in
>> schema.xml - are these all necessary? Same goes for stored fields. (I'm
>> using a slightly modified version of the data.gov.uk schema).
>>
>> Really though, I'm just shooting in the dark - I don't know if it's my
>> schema, or if it's anything else, and so some info on how to debug this
>> would be great.
>>
>> Cheers,
>> Carl
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>
>>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20151207/bae0e65e/attachment-0003.html>