[okfn-coord] Our major server disappeared!

[Becky Hogge]

2009/10/29 Rufus Pollock <rufus.pollock at okfn.org>:
> Thanks for asking Becky.
>
> Main effort will have to come from the more techhy among us. From the
> board I would like general suggestion about what to do about LT
> (layeredtech -- our hoster).
>
> There is one major update here: at 7am our time (after I went to bed)
> LT support came back with something a bit more reasonable in response
> to my pleas than the previous "your server is gone and we can't help":
>
> <quote>
> There is a 150/hr admin fee to locate and place your server back
> online. The server has already been placed on a pallet and shrink
> wrapped for shipping purposes. We will need to break down the pallet
> and locate your specific server. There are about 70 servers on a
> pallet and it may be at the bottom of the stack.
>
> Please confirm you want to proceed with this and we will place your
> server back online for 24 hours. We need a response before the pallet
> is shipped off to be disposed of.
> </quote>

If we decide to go down this route and pay for the server back, I'm
happy to make a £150 donation to OKFN to cover 1hr of their time.
Although I don't think these playboys deserve any of my money, I'd
like to make the donation in the spirit of helping the OKF community.

If we decide to name and shame, I'm happy to post link to naming and
shaming on my blog and Twitter.

I don't have a good track record of picking good ISPs, so I'm not sure
I can volunteer to find a replacement.

Cheers

Becky





>
> What should we do about this? (I've already asked for clarification on
> how many hours they think this would take.)
>
> The main thing we are currently missing is mailing list memberships.
> We need to weigh the benefit of getting this info (and a really
> complete backup generally) against the cost. Depending on the time
> involved I guess this might be worth it.
>
> NB: I should add that the charges will almost certainly come out of
> the 1 year in advance hosting fee I paid them in August (as of
> yesterday they had cancelled our contract and were about to refund our
> fee and I assume they will dock any maintenance work out of this).
>
> I also wonder what we should do once we get the data. I'm seriously
> annoyed with LT. I understand that they did send emails over last
> month but I really feel for something this serious they would need
> positive confirmation that stuff was OK (particularly since they knew
> we were still using the server as of the 2nd of Oct when I sent a
> reboot request!). Any thoughts here are appreciated.
>
> Regards,
>
> Rufus
>
> 2009/10/29 Becky Hogge <becky.hogge at gmail.com>:
>> Rufus
>>
>> What can we do to help?
>>
>> Becky
>>
>> 2009/10/29 Rufus Pollock <rufus.pollock at okfn.org>:
>>> At approximately 6pm UK time yesterday one of our two main servers
>>> which hosts all our mailing lists, our email, our blog and
>>> knowledgeforge.net was removed permanently and (it appears)
>>> irrevocably from service by our hosting provider layeredtech.com.
>>> Details behind this are given below, but suffice it to stay this was
>>> extraordinarily unfortunate since several key parts of infrastructure
>>> went immediately offline together with all of their associated data.
>>>
>>> We are fortunate in having reasonable backups of most of our material,
>>> however these backups had never actually been tested out. (work has
>>> been done to improve the backup process over the last few months but
>>> this effort has not been completed). However, even with the backups in
>>> hand we had just lost 1 of our 2 major servers.
>>>
>>> Faced with this 'crisis', over the last 8h, myself together with
>>> Jonathan Gray and John Bywater have managed to:
>>>
>>>  * Reinstall blog.okfn.org successfully from backup
>>>  * Migrate our email server to eu1.okfn.org
>>>  * Migrate the bulk of knowledgeforge.net data (22GB) to a new
>>> temporary location but restart of the service will not happen until
>>> tomorrow and there is likely to be some data loss (though thankfully
>>> fairly minor)
>>>
>>> The one major issue that has been discovered is with our mailing list
>>> backups. While we do have fairly full backups of the list archives (as
>>> of last night) it has become apparent that some key information, most
>>> importantly **list memberships**, have not been (adequately) backed up
>>> (by me!).
>>>
>>> Many of our smaller lists can be reconstructed by hand -- including
>>> this one (which is the first one I have brought back online). However
>>> for our larger lists full reconstruction may not be possible.
>>>
>>> I still have a small (but fading) hope that we will get access to the
>>> original machine. But assuming we do not we shall have to deal with
>>> some permanent data loss.
>>>
>>> We will also have to talk with our community and partners (such as
>>> oneclickorgs) about the incident, and deal as rapidly as possible with
>>> bringing existing services back online. I would welcome assistance
>>> from the board in doing this and hope we can put in place the
>>> processes we never encounter a crisis like this one again.
>>>
>>> Regards,
>>>
>>> Rufus
>>>
>>> ### Background to Server Disconnection
>>>
>>> At approximately the same time I discovered an email sent at 3am
>>> yesterday morning (24h ago) from our host informing us that the
>>> machine was going to be decomissioned and "recycled" with the next 24h
>>> and our account cancelled (unfortunately I have been away since
>>> Tuesday am at workshops and seminars in London and Oxford).
>>>
>>> I immediately contacted our server provider and in so doing discovered
>>> several messages on their support system from the last month stating
>>> that they were planning to migrate machines. Unfortunately not one of
>>> these messages had arrived via email to us (except for the last one!),
>>> due most unfortunately to them being allocated at spam by gmail.
>>>
>>> I immediately requested postponement of the removal of the machine
>>> and/or reconnection pending us getting key data off the machine but
>>> the support response was just that it was too late and there was
>>> nothing they were going to do. I have been most disappointed by the
>>> attitude and helpfulness of the support staff at LT.
>>>
>>> _______________________________________________
>>> okfn-coord mailing list
>>> okfn-coord at lists.okfn.org
>>> http://lists.okfn.org/mailman/listinfo/okfn-coord
>>>
>>
>
>
>
> --
> Join our "100 supporters pledge": http://www.pledgebank.com/support-okfn
> Promoting Open Knowledge in a Digital Age - http://www.okfn.org/
>