[knowledgeforge-user] maintenance work

[Rufus Pollock]

Great stuff Mark!

Just so I'm clear: have you deployed this patch on knowledgeforge.net
(I assume you have but I wasn't sure from reading your email).

Lastly, I agree that this is definitely easier than the svnserve route
(even though I know that isn't *that* hard John!)

Francis: I prefer mercurial too but this seems to be a emacs vs. vim
like debate :) I guess that now the git conversion is done converting
to mercurial isn't that hard.

Regards,

Rufus

On 1 February 2010 23:19, Mark Longair
<mark-knowledgeforge at mythic-beasts.com> wrote:
> Rufus Pollock <rufus.pollock at okfn.org> wrote:
>>
>> 2009/5/3 Mark Longair <mark-knowledgeforge at mythic-beasts.com>:
>>> Rufus Pollock <rufus.pollock at okfn.org> wrote:
>>>>
>>>> 2009/4/20 Mark Longair <mark-knowledgeforge at mythic-beasts.com>:
>> [snip]
>>>>> Unfortunately, I still find that the connection is lost during
>>>>> "git svn" updates, sometimes after it has been running for for
>>>>> half an hour, sometimes up to an hour.  What are Apache's
>>>>> timeout and keepalive timeout settings set to?
>>>>
>>>> KeepAlive On
>>>> Timeout 300
>>>> MaxKeepAliveRequests 500
>>>> KeepAliveTimeout 5
>>>
>>> Hm.  I suppose that KeepAliveTimeout could be a bit higher, but
>>> it seems to me that it's a bit unlikely to be the problem given
>>> the symptoms I'm seeing - I've noticed that when I get a
>>> connection reset it becomes impossible to connect to port 80 of
>>> project.knowledgeforge.net for a couple of minutes afterwards,
>>> either from JANET or from my home ADSL connection.
>>>
>>> So that you have a time point to find in the logs when this
>>> definitely happened, I set "git svn clone --stdlayout
>>> http://knowledgeforge.net/ukparse/svn/" going earlier from the
>>> client 129.215.197.50 and got the connection reset by peer error [1]
>>> at:
>>>
>>>  Sun May  3 12:57:40 BST 2009
>>>
>>> I could connect to port 80 again at 13:01, but there were a
>>> couple of minutes before that wwhere every connection attempt
>>> from each network I tried failed.
>>
>> Monit reboots apache when system overloads and this is what seems to
>> be happening here.
>>
>> My "guess" (from experience) is that because this is "graceful" it can
>> be slow since it waits for existing connections to die/time-out and
>> this may explain the long period
>>
>>> I hope this is of some help in tracking down the problem.
>>
>> Will see what we can do :)
>>
>> [snip]
>>
>> Rufus
>
> I've found a workaround for this now, so I thought it would be
> worth sending an update to the list with more details of the
> problem.  The quick version is "make sure you have
> 'http-compression = yes' set in ~/.subversion/servers".  For more
> details, read on...
>
> It seems that when I was doing very long subversion operations
> the memory consumption of Apache grew until monit was provoked
> into restarting it, as if there was a memory leak of some kind.
> It seems most likely that this is the same problem as described
> in this long-standing bug report against mod_dav_svn:
>
>  http://subversion.tigris.org/issues/show_bug.cgi?id=3084
>
> We very briefly tried one of the earlier suggested workarounds
> (SVNAllowBulkUpdates Off) but it caused svn operations to be
> unbearably slow, so that was unacceptable.
>
> In November last year, someone updated the bug report suggesting
> that it was the same as this one:
>
>  http://svn.haxx.se/dev/archive-2009-08/0274.shtml
>
> The "massive" memory leak described there can be reproduced when
> the server is set up to use mod_deflate but the client doesn't
> indicate that it can cope with compression.  It does indeed seem
> that this was the problem for me - setting "http-compression =
> yes" in my client means that operations that fetch thousands of
> revisions can now run to completion correctly.
>
> This means that finally I have a "git svn" import of the whole
> of ukparse, which is a joy :)  (The repository with complete
> history is 9GB while the working tree is 23GB.)
>
> I hope that's of some use.
>
> regards,
> mark
>
> P.S.  Short of actually fixing the mod_dav_svn bug, the other
> possible workaround we discussed was to patch kforge to offer
> subversion over the svn protocol as well as the HTTP-based
> transport.  Obviously that's much more work, though.
>



-- 
Open Knowledge Foundation
Promoting Open Knowledge in a Digital Age
http://www.okfn.org/ - http://blog.okfn.org/