[knowledgeforge-user] maintenance work

Mon Feb 1 23:19:48 UTC 2010

Rufus Pollock <rufus.pollock at okfn.org> wrote:
> 
> 2009/5/3 Mark Longair <mark-knowledgeforge at mythic-beasts.com>:
>> Rufus Pollock <rufus.pollock at okfn.org> wrote:
>>>
>>> 2009/4/20 Mark Longair <mark-knowledgeforge at mythic-beasts.com>:
> [snip]
>>>> Unfortunately, I still find that the connection is lost during
>>>> "git svn" updates, sometimes after it has been running for for
>>>> half an hour, sometimes up to an hour.  What are Apache's
>>>> timeout and keepalive timeout settings set to?
>>>
>>> KeepAlive On
>>> Timeout 300
>>> MaxKeepAliveRequests 500
>>> KeepAliveTimeout 5
>>
>> Hm.  I suppose that KeepAliveTimeout could be a bit higher, but
>> it seems to me that it's a bit unlikely to be the problem given
>> the symptoms I'm seeing - I've noticed that when I get a
>> connection reset it becomes impossible to connect to port 80 of
>> project.knowledgeforge.net for a couple of minutes afterwards,
>> either from JANET or from my home ADSL connection.
>>
>> So that you have a time point to find in the logs when this
>> definitely happened, I set "git svn clone --stdlayout
>> http://knowledgeforge.net/ukparse/svn/" going earlier from the
>> client 129.215.197.50 and got the connection reset by peer error [1]
>> at:
>>
>>  Sun May  3 12:57:40 BST 2009
>>
>> I could connect to port 80 again at 13:01, but there were a
>> couple of minutes before that wwhere every connection attempt
>> from each network I tried failed.
>
> Monit reboots apache when system overloads and this is what seems to
> be happening here.
>
> My "guess" (from experience) is that because this is "graceful" it can
> be slow since it waits for existing connections to die/time-out and
> this may explain the long period
>
>> I hope this is of some help in tracking down the problem.
>
> Will see what we can do :)
>
> [snip]
>
> Rufus

I've found a workaround for this now, so I thought it would be
worth sending an update to the list with more details of the
problem.  The quick version is "make sure you have
'http-compression = yes' set in ~/.subversion/servers".  For more
details, read on...

It seems that when I was doing very long subversion operations
the memory consumption of Apache grew until monit was provoked
into restarting it, as if there was a memory leak of some kind.
It seems most likely that this is the same problem as described
in this long-standing bug report against mod_dav_svn:

  http://subversion.tigris.org/issues/show_bug.cgi?id=3084

We very briefly tried one of the earlier suggested workarounds
(SVNAllowBulkUpdates Off) but it caused svn operations to be
unbearably slow, so that was unacceptable.

In November last year, someone updated the bug report suggesting
that it was the same as this one:

  http://svn.haxx.se/dev/archive-2009-08/0274.shtml

The "massive" memory leak described there can be reproduced when
the server is set up to use mod_deflate but the client doesn't
indicate that it can cope with compression.  It does indeed seem
that this was the problem for me - setting "http-compression =
yes" in my client means that operations that fetch thousands of
revisions can now run to completion correctly.

This means that finally I have a "git svn" import of the whole
of ukparse, which is a joy :)  (The repository with complete
history is 9GB while the working tree is 23GB.)

I hope that's of some use.

regards,
mark

P.S.  Short of actually fixing the mod_dav_svn bug, the other
possible workaround we discussed was to patch kforge to offer
subversion over the svn protocol as well as the HTTP-based
transport.  Obviously that's much more work, though.