[okfn-help] Backup work

[James Casbon]

Apologies for the delay in getting back to you on this, Rufus.

On 25 May 2010 15:21, Rufus Pollock <rufus.pollock at okfn.org> wrote:
> Dear James and Martin,
>
> I'm still not quite sure about the state of your guys efforts on the
> backup front. As I missed you guys last night I had a stab on my own
> at taking things forward.
>
> 1. I updated this document to reflect the current situation (as I
> deciphered it):
>
> <http://knowledgeforge.net/okfn/tasks/wiki/BackupPlan>
>
> 2. I updated fabfile.py backup_report and started on backup_setup method.
>
> At this point 'd be really grateful :) for some info on what machines
> you've (starting) setup backup on already (and current status). I
> would also like thoughts on:

I don't have this info (martin was doing it), which is why I started
writing the report on the backup status of the machine so that I could
get this info automatically.  I left this comment on the ticket:
"""
Nothing on eu3, eu4, eu1.

us0, eu0, eu2 has the backup hooks in /etc.
"""
but the fact that cron was not working meant just checking for the
hooks was not enough.  I'm still not sure what happened to the cron
diagnostics, last I heard Martin was checking the process accounting
logs (I can't find them).

>  * s3 snapshots of ebs backups

I think that the main problem we might face is a long downtime in a
particular availability zone that makes both backup and host
unavailable.  However, at this point amazon would be loosing a lot of
cash so I think they should bring it up quickly enough.

>  * recovery from backup plans (we clearly have to automate (and test)
> recovery off backups for this to be really useful)

Yes indeed, we need to do this.  It is also worth noting the backups
on eu0 still look like they are happening (according to the mtime)[1]
- so we may want to look at the cost of this in terms of bandwidth.

[1] okfn at eu0:~$ ls /mnt/backup/* -l
/mnt/backup/eu0:
total 16
drwxr-xr-x 3 root root 4096 Jun  7 07:08 daily.0
drwxr-xr-x 3 root root 4096 Jun  6 07:06 daily.1
drwxr-xr-x 3 root root 4096 Jun  5 07:07 daily.2
drwxr-xr-x 3 root root 4096 Jun  4 07:07 daily.3

/mnt/backup/eu1:
total 16
drwxr-xr-x 3 root root 4096 Jun  7 07:09 daily.0
drwxr-xr-x 3 root root 4096 Jun  6 07:06 daily.1
drwxr-xr-x 3 root root 4096 Jun  5 07:08 daily.2
drwxr-xr-x 3 root root 4096 Jun  4 07:08 daily.3
ls: cannot open directory /mnt/backup/lost+found: Permission denied

/mnt/backup/us1:
total 4
drwxr-xr-x 3 root root 4096 Jun  7 07:18 daily.0

/mnt/backup/us2:
total 16
drwxr-xr-x 3 root root 4096 Jun  7 07:24 daily.0
drwxr-xr-x 3 root root 4096 Jun  6 07:21 daily.1
drwxr-xr-x 3 root root 4096 Jun  5 07:22 daily.2
drwxr-xr-x 3 root root 4096 Jun  4 07:24 daily.3