[ckan-dev] Backup strategy for CKAN

Thu Apr 10 16:02:31 UTC 2014

Hi Florian,

Thanks for the info! Yeah, that's a problem that's somewhat easy (and
useful) to be solved: being able to set the config variables through
environment variables. This will bring us closer to following the 12 factor
app principles (specifically http://12factor.net/config), and will make it
easier to run not only on Docker, but also on Heroku and other PaaS. This
is what Pyramid does for its own config variables. It accepts a .ini as we
have in CKAN, but the values can be overwritten by using environment
variables (
http://docs.pylonsproject.org/projects/pyramid/en/latest/narr/environment.html
)

@rossjones is doing a pull request somewhat related in
https://github.com/ckan/ckan/pull/1600. But that's just to be able to set
up which .ini file CKAN will use through an environment variable, not the
actual contents.

I would be happy to see someone fixing this issue in CKAN.

Cheers,

2014-04-10 6:17 GMT-03:00 Florian May <florian.wendelin.mayer at gmail.com>:

> Hi Vítor,
>
> our public-facing instance runs branded as "demo" at
> http://data-demo.dpaw.wa.gov.au/ and will in the near future move to
> data.dpaw.wa.gov.au. Disclaimer: we're collecting anonymous usage stats
> via google analytics.
>
> So far we haven't had any troubles with Docker and CKAN.
>
> The only caveat I can see lies in the nature of CKAN requiring the site
> url (which will change between different containers running off the same
> image) in its config file, which is baked into the image during the Docker
> build. This required us to create a two-step build process, where the first
> step builds the image and copies out the config file (ensuing manual
> changes to site_url), and the second step builds the image, now copying the
> manually modifled config file back into the image. So we have to build the
> image with updated site urls for each url we wish to run the container on,
> which - thanks to Docker re-using its cache - only takes seconds. We're
> working on a way of simplifying that step.
>
> In case anything goes pear-shaped inside the running image, we can always
> attach to the image and inspect the log files, also the running container
> streams its logs to stdout.
>
> Is there a way of soft-coding the site url, and for good measure the
> datapusher url?
>
> Cheer,s
> Florian
>
>
> On Thu, Apr 10, 2014 at 2:12 AM, Vitor Baptista <vitor at vitorbaptista.com>wrote:
>
>> Hi Florian,
>>
>> That's really cool. Is your CKAN instance already online? Where is it?
>> Any caveats on using it with Docker?
>>
>> Cheers,
>>
>>
>> 2014-04-09 0:15 GMT-03:00 Florian May <florian.wendelin.mayer at gmail.com>:
>>
>> Hi Stéphane,
>>>
>>> we're deploying our CKAN using Docker linux containers. In our docker
>>> image build process we copy out storage folders and the database from the
>>> (non persistent) container into persistent directories within a BTRFS
>>> snapshotting file system.
>>> That simplifies a few things for us:
>>> - All read-only files (software, config, dependencies) are located
>>> within the Docker image, which also contains all installed extensions,
>>> - All read/write files, the installed / set up / populated database,
>>> plus uploaded attachments are located within the persistent folder, making
>>> migration a "build the image and copy the persist folder" job,
>>> - the snapshotting file system allows us to roll back the CKAN instance
>>> to a sane state, should bad things happen, instead of having to
>>> migrate/install.
>>>
>>> Out backup strategy is:
>>> - manual backup of dev/test servers: synchronise persistent folder to a
>>> save place using rsync.
>>> - automated backup of prod servers: configure snapshotting intervals of
>>> BTRFS appropriately, and schedule a cron job to rsync persistent folder for
>>> good measure.
>>> We'll be updating our Docker image in the next weeks, but feel free to
>>> have a squiz at the code (rsync backup settings) at
>>> https://bitbucket.org/dpaw/dpaw_docker/src/tip/ckan/
>>>
>>> Cheers,
>>> Florian Mayer
>>> Department of Parks & Wildlife, Western Australia
>>>
>>>
>>>
>>> On Wed, Apr 9, 2014 at 8:43 AM, Hayden Waring <hayden at opengovgear.com>wrote:
>>>
>>>> Hey Stéphane,
>>>>
>>>> I recently did a CKAN migration. I simply dumped the database, copied
>>>> the storage folders, and took notes of which extensions I had installed.
>>>> I loaded the database onto a second instance of CKAN, swapped in my
>>>> storage folders, and installed my extensions from git.
>>>> With a basic script I was able to reboot the exact machine in under ten
>>>> minutes. Depending on how much content you have I'm sure that would be able
>>>> to get comparable results restoring a failed machine.
>>>>
>>>> DB synchronization is an option that we have been working with but it
>>>> is slightly more expensive.
>>>>
>>>> I would recommend it if you are in a situation where downtime is not an
>>>> option or you are dealing with large amounts of data that cannot be moved
>>>> and loaded quickly.
>>>>
>>>> Hayden Waring
>>>> OpenGovGear
>>>>
>>>> _______________________________________________
>>>> ckan-dev mailing list
>>>> ckan-dev at lists.okfn.org
>>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>>
>>>>
>>>
>>> _______________________________________________
>>> ckan-dev mailing list
>>> ckan-dev at lists.okfn.org
>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>
>>>
>>
>>
>> --
>>
>> Vítor Baptista
>>
>> Developer  |  http://vitorbaptista.com | LinkedIn<http://www.linkedin.com/in/vitorbaptista>|
>> @vitorbaptista <http://twitter.com/vitorbaptista>
>>
>> The Open Knowledge Foundation <http://okfn.org>
>>
>> *Empowering through Open Knowledge*
>>
>> http://okfn.org/  |  @okfn <http://twitter.com/okfn>  |  OKF on Facebook<https://www.facebook.com/OKFNetwork> |
>> Blog <http://blog.okfn.org/>  |  Newsletter<http://okfn.org/about/newsletter/>
>>
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>
>>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>
>

-- 

Vítor Baptista

Developer  |  http://vitorbaptista.com |
LinkedIn<http://www.linkedin.com/in/vitorbaptista>|
@vitorbaptista <http://twitter.com/vitorbaptista>

The Open Knowledge Foundation <http://okfn.org>

*Empowering through Open Knowledge*

http://okfn.org/  |  @okfn <http://twitter.com/okfn>  |  OKF on
Facebook<https://www.facebook.com/OKFNetwork> |
Blog <http://blog.okfn.org/>  |  Newsletter<http://okfn.org/about/newsletter/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20140410/8675e11d/attachment-0003.html>