[ckan-dev] CKAN Enterprise Stack on AWS

[Shane Davis]

Hi All,

I'm starting to immerse myself more deeply into the CKAN dev world.

Having read https://lists.okfn.org/pipermail/ckan-dev/2016-July/010130.html
by Luiz Felipe I thought it worthwhile to share where Link is at with
Enterprise CKAN builds given we want to open source our work once we are
happy with it's stability.

What we've been working on for quite some time is a robust, scalable and
easily maintained HA CKAN stack. Our technology of choice has been AWS, and
more specifically, AWS CloudFormation and OpsWorks.

Where we are at the moment is a fairly stable build that has the following
features:
- Gluster File System for HA shared file store across two service nodes
- SolrCloud with Apache Zookeeper across two service nodes
- CKAN 2.5.2 across load balanced EC2-Instances
- AWS Postgres RDS with Multi-AZ and Postgis extensions installed and
enabled.

The CloudFormation templates we have allow highly configurable DEV,UAT and
Production VPCs to be provisioned.
Standalone OpsWorks stacks with the features described above can then be
provisioned into any one of the VPCs.

OpsWorks provides the following key benefits:
Layers are used to define service roles. Hence our build has seperate
layers for all the key components such as GFS, Solr, Zookeeper etc.
An instance can be assigned to one or more layers. Thus you might have two
service nodes that provide GFS, Solr and Redis. Later as your environment's
requirements expand, you can create instances that take those roles on
separately.
CKAN extensions can be deployed as apps within the stack. These apps (CKAN
Extensions) can then be deployed to individual instances/servers, or to a
whole layer.

The existing ruby based Chef recipes should provide pretty good insights to
anyone interested in adapting them to other environments or cloud
platforms. They are as close to bare basics as possible which should lend
them towards adaptation easily.

Longer term our ambition is to have a straight out CloudFormation template
for those not interested in running an OpsWorks stack. We're also fairly
confident that the build can be turned into Docker containers so rather
than using OpsWorks, or indeed AWS, it would be possible to deploy a fully
scalable HA stack using Docker on one or more systems.

As things keep progressing I'll keep the list updated. We plan to provide
the repo with a blog post and possibly video taped presentation of some
sort.

Cheers.
Shane.


------------------------------------------------------------------
*SHANE DAVIS*  |  *SERVICE DELIVERY MANAGER*
Link Digital
shane.davis at linkdigital.com.au
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20160721/30d072ad/attachment-0002.html>