[ckan-dev] Backup strategy for CKAN

[Florian May]

Hi Stéphane,

we're deploying our CKAN using Docker linux containers. In our docker image
build process we copy out storage folders and the database from the (non
persistent) container into persistent directories within a BTRFS
snapshotting file system.
That simplifies a few things for us:
- All read-only files (software, config, dependencies) are located within
the Docker image, which also contains all installed extensions,
- All read/write files, the installed / set up / populated database, plus
uploaded attachments are located within the persistent folder, making
migration a "build the image and copy the persist folder" job,
- the snapshotting file system allows us to roll back the CKAN instance to
a sane state, should bad things happen, instead of having to
migrate/install.

Out backup strategy is:
- manual backup of dev/test servers: synchronise persistent folder to a
save place using rsync.
- automated backup of prod servers: configure snapshotting intervals of
BTRFS appropriately, and schedule a cron job to rsync persistent folder for
good measure.
We'll be updating our Docker image in the next weeks, but feel free to have
a squiz at the code (rsync backup settings) at
https://bitbucket.org/dpaw/dpaw_docker/src/tip/ckan/

Cheers,
Florian Mayer
Department of Parks & Wildlife, Western Australia



On Wed, Apr 9, 2014 at 8:43 AM, Hayden Waring <hayden at opengovgear.com>wrote:

> Hey Stéphane,
>
> I recently did a CKAN migration. I simply dumped the database, copied the
> storage folders, and took notes of which extensions I had installed.
> I loaded the database onto a second instance of CKAN, swapped in my
> storage folders, and installed my extensions from git.
> With a basic script I was able to reboot the exact machine in under ten
> minutes. Depending on how much content you have I'm sure that would be able
> to get comparable results restoring a failed machine.
>
> DB synchronization is an option that we have been working with but it is
> slightly more expensive.
>
> I would recommend it if you are in a situation where downtime is not an
> option or you are dealing with large amounts of data that cannot be moved
> and loaded quickly.
>
> Hayden Waring
> OpenGovGear
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20140409/b86ff437/attachment-0003.html>