[okfn-help] backup, monitoring, etc

[James Casbon]

For monitoring, I want to get munin back running -
http://knowledgeforge.net/okfn/tasks/ticket/133
Munin is running but munin.okfn,org is not resolving.  Who has the DNS details?

Also, I think it is worth discussing what is happening for backup.

We now have a few scripts running on eu0 that can back things up
easily enough. Look at /etc/cron.daily/daily_backup_snapshot_eu1.  You
can adjust the hostname, target directory and begin backing up another
host.
All these backups go into /mnt/backup/$HOSTNAME/snapshot.name.  There
are three days of backups kept (I think this is important to prevent
an automated backup blowing away your backup copy in case you delete
your original).  /mnt/backup is on a separate physical disk on eu0.

This is a slight departure from the existing backups which backup in
specific directories into other specific directories.  The drawback of
this approach is that your backup scripts need to know how to exclude
backups which means storing all the information about where the
backups are in multiple scripts.

With this in mind, the current kforge mirror from us1 to eu0 can be
converted to use the above script easy enough, and the result would be
that this would live under /mnt/backup/us1 rather that /home/kforge.
Then when we want to backup eu1 we can do the whole of /home without
picking up kforge again.  This also needs to proper database dumps to
be done before the mirror.

Backing up eu0 is high priority now, and should be done to us1.

One thing we can look at after these system level backups are sorted
is doing application backups onto EBS.  This provides snapshotable,
diff based backups that can be shared.  So we can offer people the
ability to clone kforge, ckan from these.

One final thing is that the svn admin things need some documentation.
Currently the us1:/home/okfn/svn-okfn repo complains about bad keys so
I cannot commit. But I need to know so more about the whole setup
here.  My quick guess at how it should be done is that you maintain a
patch queue on top of the debian vanilla config, with branches per
host.   However, a policy based approach is probably better here.

James