[ckan-dev] CKAN Multi-site

[Florian May]

Hi Ross, all,

not sure whether this is helpful, but we're working on a multi-site CKAN
(master branch because #1779) on AWS (Ubuntu 14.04 LTS / t2.medium / 100GB
hd) for an Australian state government agency. We're running three CKANs
(three separate .ini configs running as apache wsgi apps, three plus three
ckan / datastore databases, three separate filestore locations) sharing one
SolR core, and having three supervisor'd celery/redis task queues. The AWS
VM is orchestrated using saltstack and proxied both on AWS's and our side
using nginx.

Our three production instances are one internal-facing for our sensitive
and unpublished/cooling-off data sets, one external-facing (to be
officially sanctioned later), and one internal-facing sandbox for users to
learn and developers to test their scripts without regrets. We felt it more
practicable to use the same install for all three instances, rather than
installing three separate CKAN instances.

Diverging from standard installation procedure, we installed all valuables
(CKAN, postgres tablespace, file store) underneath one folder which we can
snapshot as-is using btrfs.
For disaster recovery, Amazon keeps a 30-day nightly snapshot, plus we
create btrfs file system snapshots, plus we cron-job a psql db dump of ckan
and datastore and rsync the filestore to a safe place.

If we can separate out sensitive config settings and data we could provide
the clean AWS snapshot if that's any help. Realistically that should
include the 2.3 release.

If our deployment architecture makes any sense to you all, in order to
upscale to multiple agencies as per US Open Data Institute's suggestion
we'd simply spin up one instance of our  AWS image per agency. The only
effort would be to separate out sensitive settings for db passwords, maybe
a default CKAN sysadmin account, nginx reverse proxy settings and salt
server orchestration settings. Getting all sensitive settings from one
global config would be great, but it would require CKAN to read settings
from environment variables (cf. site_url and docker images).

Let us know what you think, any feedback would be greatly appreciated!

Cheers,
Florian

Dept Parks & Wildlife, WA

On Thu, Dec 11, 2014 at 4:01 AM, Ross Jones <ross at servercode.co.uk> wrote:

> Hi all,
>
> The US Open Data Institute node have recently written a blog post at
> https://usodi.org/2014/12/08/ckan-multisite/ about their desire to see
> CKAN extended to better support lots of CKAN instances (at once). Apologies
> if you’ve seen it and read it, you can probably stop reading this now.
>
> The post begins,  "We want to make it much easier for governments to
> create and host open data repositories” which seems to be a very admirable
> goal  :) The proposal written by OKFN (yes, I’m still calling it OKFN even
> though I think they dropped the N) is at
> https://github.com/opendata/CKAN-Multisite/blob/master/Proposal%20Draft.md and
> I’ve broken down the individual points in the proposal into separate issues
> at https://github.com/opendata/CKAN-Multisite/issues
>
> This is really a call for help, I’ve started adding a few comments, but I
> am pretty sure the ODI guys would love more feedback on the proposal, so if
> you had time to read the proposal, and maybe record any feedback on the
> issues there, that would be great!
>
> There’s a lot of valuable CKAN knowledge and experience outside of the
> people on the tech team calls*, and that input would be very welcome I’m
> sure.
>
>
> Ross.
>
> * If you want to be on the calls, just shout, you may^H^H^H will however
> be assigned tickets.
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20141211/83b9efb7/attachment-0003.html>