[knowledgeforge-user] maintenance work

Rufus Pollock rufus.pollock at okfn.org
Wed Feb 3 20:36:09 UTC 2010

Great stuff Mark!

Just so I'm clear: have you deployed this patch on knowledgeforge.net
(I assume you have but I wasn't sure from reading your email).

Lastly, I agree that this is definitely easier than the svnserve route
(even though I know that isn't *that* hard John!)

Francis: I prefer mercurial too but this seems to be a emacs vs. vim
like debate :) I guess that now the git conversion is done converting
to mercurial isn't that hard.



On 1 February 2010 23:19, Mark Longair
<mark-knowledgeforge at mythic-beasts.com> wrote:
> Rufus Pollock <rufus.pollock at okfn.org> wrote:
>> 2009/5/3 Mark Longair <mark-knowledgeforge at mythic-beasts.com>:
>>> Rufus Pollock <rufus.pollock at okfn.org> wrote:
>>>> 2009/4/20 Mark Longair <mark-knowledgeforge at mythic-beasts.com>:
>> [snip]
>>>>> Unfortunately, I still find that the connection is lost during
>>>>> "git svn" updates, sometimes after it has been running for for
>>>>> half an hour, sometimes up to an hour.  What are Apache's
>>>>> timeout and keepalive timeout settings set to?
>>>> KeepAlive On
>>>> Timeout 300
>>>> MaxKeepAliveRequests 500
>>>> KeepAliveTimeout 5
>>> Hm.  I suppose that KeepAliveTimeout could be a bit higher, but
>>> it seems to me that it's a bit unlikely to be the problem given
>>> the symptoms I'm seeing - I've noticed that when I get a
>>> connection reset it becomes impossible to connect to port 80 of
>>> project.knowledgeforge.net for a couple of minutes afterwards,
>>> either from JANET or from my home ADSL connection.
>>> So that you have a time point to find in the logs when this
>>> definitely happened, I set "git svn clone --stdlayout
>>> http://knowledgeforge.net/ukparse/svn/" going earlier from the
>>> client and got the connection reset by peer error [1]
>>> at:
>>>  Sun May  3 12:57:40 BST 2009
>>> I could connect to port 80 again at 13:01, but there were a
>>> couple of minutes before that wwhere every connection attempt
>>> from each network I tried failed.
>> Monit reboots apache when system overloads and this is what seems to
>> be happening here.
>> My "guess" (from experience) is that because this is "graceful" it can
>> be slow since it waits for existing connections to die/time-out and
>> this may explain the long period
>>> I hope this is of some help in tracking down the problem.
>> Will see what we can do :)
>> [snip]
>> Rufus
> I've found a workaround for this now, so I thought it would be
> worth sending an update to the list with more details of the
> problem.  The quick version is "make sure you have
> 'http-compression = yes' set in ~/.subversion/servers".  For more
> details, read on...
> It seems that when I was doing very long subversion operations
> the memory consumption of Apache grew until monit was provoked
> into restarting it, as if there was a memory leak of some kind.
> It seems most likely that this is the same problem as described
> in this long-standing bug report against mod_dav_svn:
>  http://subversion.tigris.org/issues/show_bug.cgi?id=3084
> We very briefly tried one of the earlier suggested workarounds
> (SVNAllowBulkUpdates Off) but it caused svn operations to be
> unbearably slow, so that was unacceptable.
> In November last year, someone updated the bug report suggesting
> that it was the same as this one:
>  http://svn.haxx.se/dev/archive-2009-08/0274.shtml
> The "massive" memory leak described there can be reproduced when
> the server is set up to use mod_deflate but the client doesn't
> indicate that it can cope with compression.  It does indeed seem
> that this was the problem for me - setting "http-compression =
> yes" in my client means that operations that fetch thousands of
> revisions can now run to completion correctly.
> This means that finally I have a "git svn" import of the whole
> of ukparse, which is a joy :)  (The repository with complete
> history is 9GB while the working tree is 23GB.)
> I hope that's of some use.
> regards,
> mark
> P.S.  Short of actually fixing the mod_dav_svn bug, the other
> possible workaround we discussed was to patch kforge to offer
> subversion over the svn protocol as well as the HTTP-based
> transport.  Obviously that's much more work, though.

Open Knowledge Foundation
Promoting Open Knowledge in a Digital Age
http://www.okfn.org/ - http://blog.okfn.org/

More information about the knowledgeforge-user mailing list