[ckan-dev] Backup strategy for CKAN

Wed Apr 9 03:15:00 UTC 2014

Hi Stéphane,

we're deploying our CKAN using Docker linux containers. In our docker image
build process we copy out storage folders and the database from the (non
persistent) container into persistent directories within a BTRFS
snapshotting file system.
That simplifies a few things for us:
- All read-only files (software, config, dependencies) are located within
the Docker image, which also contains all installed extensions,
- All read/write files, the installed / set up / populated database, plus
uploaded attachments are located within the persistent folder, making
migration a "build the image and copy the persist folder" job,
- the snapshotting file system allows us to roll back the CKAN instance to
a sane state, should bad things happen, instead of having to
migrate/install.

Out backup strategy is:
- manual backup of dev/test servers: synchronise persistent folder to a
save place using rsync.
- automated backup of prod servers: configure snapshotting intervals of
BTRFS appropriately, and schedule a cron job to rsync persistent folder for
good measure.
We'll be updating our Docker image in the next weeks, but feel free to have
a squiz at the code (rsync backup settings) at
https://bitbucket.org/dpaw/dpaw_docker/src/tip/ckan/

Cheers,
Florian Mayer
Department of Parks & Wildlife, Western Australia

On Wed, Apr 9, 2014 at 8:43 AM, Hayden Waring <hayden at opengovgear.com>wrote:

> Hey Stéphane,
>
> I recently did a CKAN migration. I simply dumped the database, copied the
> storage folders, and took notes of which extensions I had installed.
> I loaded the database onto a second instance of CKAN, swapped in my
> storage folders, and installed my extensions from git.
> With a basic script I was able to reboot the exact machine in under ten
> minutes. Depending on how much content you have I'm sure that would be able
> to get comparable results restoring a failed machine.
>
> DB synchronization is an option that we have been working with but it is
> slightly more expensive.
>
> I would recommend it if you are in a situation where downtime is not an
> option or you are dealing with large amounts of data that cannot be moved
> and loaded quickly.
>
> Hayden Waring
> OpenGovGear
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20140409/b86ff437/attachment-0003.html>