[ckan-dev] Docker (Was ckanext-spatial deploy to production environment (datacats))

Fri Jun 3 20:27:34 UTC 2016

Adrià/Ian,

Thanks! We will take a look at /contrib/docker to contribute to moving
forward. Two questions re the scope of supporting docker within the
community repo:

   1. We're working to make our Docker work production ready, taking an
   "Infrastructure as Code" approach to deploy via docker in a
   stateful/persistent cloud environment (but looking for feedback on our
   current thinking on the tool(s) to do so):
      - Saltstack - would be used to setup a clustering of docker
      containers on a host that can be remotely provisioned. Salt (and Salt
      Cloud) is a provisioning/configuration management tool like
      ansible/chef/puppet but IMHO takes a better approach to dynamically
      managing the stack (using a master/agent vs something like ansible that
      provisions a system but falls short of managing that system
      once provisioned). Salt is written in pure python, uses yaml for
templates,
      and jinja2 for filtering/logic to declare the system state in excels at
      managing docker containers allowing you to scale just by changing the
      yaml/jinja codebase. Example demo managing docker at scale
      <https://www.youtube.com/watch?v=EVq6hn25lYM>. Here are the docs for
      managing Docker Containers
      <https://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.dockerng.html>
and
      beta support for docker-compose
      <https://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.dockercompose.html>.
      While those can be used to config any machine Salt Cloud can be used to
      provision the actual machines... then provision and deploy docker state
      onto them - https://docs.saltstack.com/en/latest/topics/cloud/. As
      everything needed to potentially deploy CKAN would be stored as
code - its
      feasible to also support this within the /contrib of the repo?
      - Kubernetes - While we're set on supporting saltstack, another
      option for offering the community a means to deploy docker CKAN is
      Kubernetes. http://kubernetes.io/. While saltstack is more
      declarative, kubernetes takes a "wwll manage that container all for you"
      approach to docker container - but its not so easy to setup and scale a
      kubernetes cluster (actually it uses saltstack to setup a kubernetes
      cluster
      <https://github.com/kubernetes/kubernetes/tree/master/cluster/saltbase>).
      It too uses yaml for its deployment/scheduling plans so this is something
      that could also be considered in the code base (caveated with the need to
      setup kubernetes one self)
   2. If the docker within the CKAN/CKAN repowork namely to be used for
   dev/test - should we aim to enable "everything"? Like ckan-spatial, pycsw,
   plugins so that we can use it to test ongoing compatibility as well as
   showcase the full set of features/services CKAN can support?

On Fri, Jun 3, 2016 at 7:18 AM, Adrià Mercader <adria.mercader at okfn.org>
wrote:

> Hi John,
>
> On 2 June 2016 at 15:28, John Jediny - XAAB <john.jediny at gsa.gov> wrote:
>
>> It seems there have been a few attempts now at creating a docker based
>> CKAN...
>>
>>    1. Datacat being the best example - but I've heard the core
>>    contributors have more or less moved on/away, can anyone confirm status?
>>    While we assessed the use of Datacats, one of the issues we foresaw was
>>    that it was using python wrappers for the management of docker on a single
>>    host (which is completely valid for most use-cases). But we needed to be
>>    able to scale out across multiple hosts to pool those resources to truly
>>    get a benefit in using docker. This however comes with its own issues as
>>    managing shared volumes, queues, ips, port mapping, cron jobs, etc, etc are
>>    "Platform" problems that Docker alone doesn't solve (hence the python
>>    management of Datacats)...
>>
>> Deployment of multiple CKAN instances to a cloud it's a use case that
> probably isn't relevant to most users of CKAN. I know that Viderum and Link
> Digital have worked on their own cloud platforms for CKAN so perhaps they
> can share more details about their own experiences. These are of course
> CKAN vendors deploying multiple sites, so again not the most common use
> case.
>
>
>
>>
>>    1. I know the docker repo on the CKAN github org has been deprecated,
>>    but there was recently a minimal version added within the main repo -
>>    https://github.com/ckan/ckan/tree/master/contrib/docker. Can anyone
>>    comment on  this (is it only intended for dev/test)? If there are those
>>    interested we at Data.gov would like to contribute/collaborate on our
>>    docker CKAN work with the community. Any ideas/suggests how best to
>>    collaborate on this moving forward would be appreciated:
>>    - https://hub.docker.com/u/datagov/
>>       - https://github.com/GSA/catalog-app
>>
>>
> Ian will know more about this but my understanding is that the Docker and
> Docker cloud files on the main repo are being updated and that they should
> form the basis for ongoing Docker support in CKAN. It would be awesome if
> you could contribute any feedback or patches you have for these!
>
> See
>
> https://github.com/ckan/ckan/pull/3029
> https://github.com/ckan/ckan/issues/3075
>
> Best,
>
> Adrià
>
>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>
>

-- 
Chief Data Engineer
202-341-0191
@Data.gov
@Office of Citizen Science and Innovative Technologies/18F
<http://www.gsa.gov/portal/category/25729>
General Services Administration

Work in the Open... ideate, innovate, iterate...
@github <https://github.com/JJediny> | @projectopendata
<https://github.com/project-open-data>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20160603/01a37375/attachment-0003.html>