[ckan-dev] Solr out of memory

[Carl Lange]

Correction: disabling storage of *data_dict* will break everything. So
don't do that.

On 7 December 2015 at 10:02, Carl Lange <carl at derilinx.com> wrote:

> OK, so it looks like the issue has been mitigated for now. Thanks Ross and
> Marianne for your help!
>
> First off, I set up nginx to cache everything it could get its little
> hands on, and also gzip everything. This helped with the crawlers, and with
> general site responsiveness (in fact, it's a miracle we'd gotten by without
> it before now).
>
> I set the solr max heap limit (in /etc/default/jetty, $JAVA_OPTIONS, -Xmx
> to 1536M), and set apache's prefork conf to some lower values (to prevent
> solr and apache both getting too big for their boots). This bought some
> time, but I still had to restart jetty every 45 minutes or so, as it would
> run out of memory, try to GC, and basically stop the world forever.
>
> Next, I did a lot of diagnostics with new relic and the like, which were
> immensely helpful, especially to see the impact of any changes I'd made.
>
> Also, I set up monit to restart jetty if it stops (just in case, and so I
> could get some sleep :) ).
>
> Then, I set a few defaults in solrconfig.xml to be lower than the usual,
> particularly switching on *autoCommit* and *autoSoftCommit*.
>
> Finally, *disabling storage of a few fields in schema.xml* (particularly *validated_data_dict
> *and *data_dict*), *disabling index of a few fields *that we don't ever
> use for search, and reindexing has basically fixed everything. The size of
> my solr data dir (/usr/share/solr/solr-4.3.1/example/solr/collection1/data)
> has gone down a huge amount (from 120MB to 18MB), and solr's ram usage more
> than halved. Since this, I have not seen any issues at all.
>
> I have not seen any detrimental effects from this - anywhere in particular
> I should look?
>
> All in all, the site is nearly 10x faster at responding than this time
> last week, and can now handle more than three people getting the dcat
> catalogue at once :)
>
> Cheers,
> Carl
>
> On 6 December 2015 at 13:21, Exversion <marianne at exversion.com> wrote:
>
>> There's a neat suggestion in Solr documentation that might help you
>> debug:
>>
>> If you want to see the heap when OOM occurs set
>> "-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/path/to/the/dump"
>>
>> Reference: https://wiki.apache.org/solr/SolrPerformanceFactors
>>
>>
>> Sent from my iPhone
>>
>> On Dec 6, 2015, at 7:59 AM, Carl Lange <carl at derilinx.com> wrote:
>>
>> I hadn't set -Xmx previously, so I've set it to 2048M. However, now solr
>> simply dies when it hits that limit, which seems like odd behaviour…
>>
>> On 6 December 2015 at 12:12, Exversion <marianne at exversion.com> wrote:
>>
>>> First thing I would check is how your heap size compares to your actual
>>> memory size. The JVM is greedy. If you set your heap size to 5GB on a 4GB
>>> machine it will routinely run itself into the ground
>>>
>>> Sent from my iPhone
>>>
>>> On Dec 6, 2015, at 4:51 AM, Carl Lange <carl at derilinx.com> wrote:
>>>
>>> Hi all,
>>>
>>> Recently we moved from an m3.large aws instance to a 4GB linode, having
>>> discovered that we didn't require nearly as much power as we thought.
>>> However, since then, our Solr core has been causing some major issues. To
>>> begin with, it runs out of memory very quickly (with a JavaOutOfMemory
>>> exception). Other times, I simply get a 500 from solr, with no other info.
>>> I have only about 1200 datasets - surely I don't need more than 4GB of RAM
>>> to search this?
>>>
>>> Could anyone point me in the direction of the best way to debug these
>>> issues? I find myself restarting jetty every ten minutes in order to get my
>>> search back, which is a little unsustainable ;)
>>>
>>> The problem seems to have manifested itself after google decided to
>>> crawl all our search pages yesterday, 1 per second. I've brought this rate
>>> down to 0.3 a second, which has helped a little. Prior to this, we had
>>> reasonably stable search for about a month.
>>>
>>> I notice that there are quite a large number of indexed fields in
>>> schema.xml - are these all necessary? Same goes for stored fields. (I'm
>>> using a slightly modified version of the data.gov.uk schema).
>>>
>>> Really though, I'm just shooting in the dark - I don't know if it's my
>>> schema, or if it's anything else, and so some info on how to debug this
>>> would be great.
>>>
>>> Cheers,
>>> Carl
>>>
>>> _______________________________________________
>>> ckan-dev mailing list
>>> ckan-dev at lists.okfn.org
>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>
>>>
>>> _______________________________________________
>>> ckan-dev mailing list
>>> ckan-dev at lists.okfn.org
>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>
>>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20151207/eff5c7f3/attachment-0003.html>