[ckan-dev] Backup strategy for CKAN

Thu Apr 10 09:17:31 UTC 2014

Hi Vítor,

our public-facing instance runs branded as "demo" at
http://data-demo.dpaw.wa.gov.au/ and will in the near future move to
data.dpaw.wa.gov.au. Disclaimer: we're collecting anonymous usage stats via
google analytics.

So far we haven't had any troubles with Docker and CKAN.

The only caveat I can see lies in the nature of CKAN requiring the site url
(which will change between different containers running off the same image)
in its config file, which is baked into the image during the Docker build.
This required us to create a two-step build process, where the first step
builds the image and copies out the config file (ensuing manual changes to
site_url), and the second step builds the image, now copying the manually
modifled config file back into the image. So we have to build the image
with updated site urls for each url we wish to run the container on, which
- thanks to Docker re-using its cache - only takes seconds. We're working
on a way of simplifying that step.

In case anything goes pear-shaped inside the running image, we can always
attach to the image and inspect the log files, also the running container
streams its logs to stdout.

Is there a way of soft-coding the site url, and for good measure the
datapusher url?

Cheer,s
Florian

On Thu, Apr 10, 2014 at 2:12 AM, Vitor Baptista <vitor at vitorbaptista.com>wrote:

> Hi Florian,
>
> That's really cool. Is your CKAN instance already online? Where is it? Any
> caveats on using it with Docker?
>
> Cheers,
>
>
> 2014-04-09 0:15 GMT-03:00 Florian May <florian.wendelin.mayer at gmail.com>:
>
> Hi Stéphane,
>>
>> we're deploying our CKAN using Docker linux containers. In our docker
>> image build process we copy out storage folders and the database from the
>> (non persistent) container into persistent directories within a BTRFS
>> snapshotting file system.
>> That simplifies a few things for us:
>> - All read-only files (software, config, dependencies) are located within
>> the Docker image, which also contains all installed extensions,
>> - All read/write files, the installed / set up / populated database, plus
>> uploaded attachments are located within the persistent folder, making
>> migration a "build the image and copy the persist folder" job,
>> - the snapshotting file system allows us to roll back the CKAN instance
>> to a sane state, should bad things happen, instead of having to
>> migrate/install.
>>
>> Out backup strategy is:
>> - manual backup of dev/test servers: synchronise persistent folder to a
>> save place using rsync.
>> - automated backup of prod servers: configure snapshotting intervals of
>> BTRFS appropriately, and schedule a cron job to rsync persistent folder for
>> good measure.
>> We'll be updating our Docker image in the next weeks, but feel free to
>> have a squiz at the code (rsync backup settings) at
>> https://bitbucket.org/dpaw/dpaw_docker/src/tip/ckan/
>>
>> Cheers,
>> Florian Mayer
>> Department of Parks & Wildlife, Western Australia
>>
>>
>>
>> On Wed, Apr 9, 2014 at 8:43 AM, Hayden Waring <hayden at opengovgear.com>wrote:
>>
>>> Hey Stéphane,
>>>
>>> I recently did a CKAN migration. I simply dumped the database, copied
>>> the storage folders, and took notes of which extensions I had installed.
>>> I loaded the database onto a second instance of CKAN, swapped in my
>>> storage folders, and installed my extensions from git.
>>> With a basic script I was able to reboot the exact machine in under ten
>>> minutes. Depending on how much content you have I'm sure that would be able
>>> to get comparable results restoring a failed machine.
>>>
>>> DB synchronization is an option that we have been working with but it is
>>> slightly more expensive.
>>>
>>> I would recommend it if you are in a situation where downtime is not an
>>> option or you are dealing with large amounts of data that cannot be moved
>>> and loaded quickly.
>>>
>>> Hayden Waring
>>> OpenGovGear
>>>
>>> _______________________________________________
>>> ckan-dev mailing list
>>> ckan-dev at lists.okfn.org
>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>
>>>
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>
>>
>
>
> --
>
> Vítor Baptista
>
> Developer  |  http://vitorbaptista.com | LinkedIn<http://www.linkedin.com/in/vitorbaptista>|
> @vitorbaptista <http://twitter.com/vitorbaptista>
>
> The Open Knowledge Foundation <http://okfn.org>
>
> *Empowering through Open Knowledge*
>
> http://okfn.org/  |  @okfn <http://twitter.com/okfn>  |  OKF on Facebook<https://www.facebook.com/OKFNetwork> |
> Blog <http://blog.okfn.org/>  |  Newsletter<http://okfn.org/about/newsletter/>
>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20140410/519c799e/attachment-0003.html>