[okfn-help] backup, monitoring, etc

Wed Nov 4 23:56:10 GMT 2009

2009/11/4 James Casbon <casbon at gmail.com>:
> For monitoring, I want to get munin back running -
> http://knowledgeforge.net/okfn/tasks/ticket/133
> Munin is running but munin.okfn,org is not resolving.  Who has the DNS details?

DNS for all our stuff is run off http://www.everydns.net/ -- see
<http://knowledgeforge.net/okfn/tasks/wiki/SystemOrganization#DNS>

> Also, I think it is worth discussing what is happening for backup.
>
> We now have a few scripts running on eu0 that can back things up
> easily enough. Look at /etc/cron.daily/daily_backup_snapshot_eu1.  You
> can adjust the hostname, target directory and begin backing up another
> host.
> All these backups go into /mnt/backup/$HOSTNAME/snapshot.name.  There
> are three days of backups kept (I think this is important to prevent
> an automated backup blowing away your backup copy in case you delete
> your original).  /mnt/backup is on a separate physical disk on eu0.

Sounds good.

> This is a slight departure from the existing backups which backup in
> specific directories into other specific directories.  The drawback of
> this approach is that your backup scripts need to know how to exclude
> backups which means storing all the information about where the
> backups are in multiple scripts.

I think generic rsync is maybe better though we need to think about db
-- suggest we dump all dbs to a directory called backupdb which we do
rsync ...

> With this in mind, the current kforge mirror from us1 to eu0 can be
> converted to use the above script easy enough, and the result would be
> that this would live under /mnt/backup/us1 rather that /home/kforge.
> Then when we want to backup eu1 we can do the whole of /home without
> picking up kforge again.  This also needs to proper database dumps to
> be done before the mirror.

See previous comment.

> Backing up eu0 is high priority now, and should be done to us1.

I was just working on this earlier though it is not finished. Script
in svn-okfn/trunk/bin/sync/mirror-okfn.py does /home/okfn but as you
suggest above we might as well generalize (though we have to be
careful -- we don't want to sync kforge off eu0 as that is just
syncing from us0 ...)

This currently sycns to /home/okfn/mirror/{hostname}/...

I've currently also scripted, run once, but not cron-tabbed mailman
syncs off eu1 onto us0:/home/okfn/mirror/{host}/mailman

> One thing we can look at after these system level backups are sorted
> is doing application backups onto EBS.  This provides snapshotable,
> diff based backups that can be shared.  So we can offer people the
> ability to clone kforge, ckan from these.

OK nice.

> One final thing is that the svn admin things need some documentation.
> Currently the us1:/home/okfn/svn-okfn repo complains about bad keys so
> I cannot commit. But I need to know so more about the whole setup
> here.  My quick guess at how it should be done is that you maintain a
> patch queue on top of the debian vanilla config, with branches per
> host.   However, a policy based approach is probably better here.

I think storing /etc/ stuff and storing generals scripts is probably
to be separated. Also some of /etc/ is consistent across machines will
some is not. Views on best practice here v. welcome.

Rufus