[ckan-dev] Q&A for datacats installs?

Wed Jan 13 21:29:12 UTC 2016

The quick answer from me is yes :)

At Link Digital manage CKAN for various jurisdictions and it makes some
sense to isolate the infrastructure in this case. Within those
jurisdictions we are doing more 2nd generation work with the agencies now
to establish data pipelines for open publishing. This often includes the
establishment of an agency CKAN as a clearing house or review check point
prior to release. It also serves as an internal data catalogue for those
agencies and can be extended to support their specific governance and
workflow requirements.

We've found that docker can be handy for small platforms but the benefits
become issues when those environments need to scale, so isolating processes
better to allow for easier scaling would be great.

Along similar thinking I've been working on finding clients and use cases
that would help us fund development of an approach that would abstract data
storage into what I'd call 'resource containers'. This would be along the
same lines as resource views and allow platform managers to configure
options for resource storage. An example might be to configure cloud
storage on either S3 or Azure and allow custodians to select which they
prefer, with the option of specifying their own account for the storage and
therefore cover the raw costs involved.

Resource containers of this type could be suitable for configuration of
third party technology and deal with use cases that involve big data,
unstructured data, automatically cleansed data, etc.

Perhaps a good example to highlight the diversity of options this would
open up is to consider an integration with GitHub.com and GitHub Enterprise
as resource containers for code. There is strong relationship between data
and code when you consider civic technology and where Government seeks to
stimulate public-private partnerships that leverage open data it makes
sense to use CKAN as the source of truth for the Governance of code repos
alongside relevant code. The same use cases apply where there is a change
of Government and portfolios are rearranged. Custodianship of data and open
source projects can then be easily shifted. Again, the same use cases apply
for discover-ability and the potential for standardization of civic
applications on top of standardized data improves.

But back to the point on docker... I'm sure our solution architect would be
interested in joining a call to discuss approaches and share our
experiences. We'd be happy to collaborate with others and help move things
forward in a common direction.

Cheers,
Steven

*STEVEN DE COSTA *|
*EXECUTIVE DIRECTOR*www.linkdigital.com.au

On 14 January 2016 at 04:31, John Jediny - XAAB <john.jediny at gsa.gov> wrote:

> Derek... I'm used to replying to you on the Geonode <http://geonode>
> mailing list (small world)...
>
> I recently joined this list as well as the data.gov team, and we've been
> looking for a similar solution to datacats but our need requires us to
> deploy CKAN using as little infrastructure as possible to scaffold out our
> future deployments which will be docker based. Our plan is to use
> http://bosh.io (an InfrastructureaaS agnostic
> deployment/orchestration/health manager) to managed our future deployments
> of docker/docker-swarm to handle the management of multiple CKANs as well
> as load balancing...
>
> Currently testing a docker based (alpha) repo that would then be deployed
> onto the docker-swarm <https://github.com/GSA/catalog-app> (but this is
> currently a rough cut at packaging CKAN with Docker - there is still ALOT
> of work to be done to refactor the application to isolate
> services/processes per container and more inline with http://12factor.net...
> our current plan has use looking at our current production and test
> environments to identify all our custom processes that tend to hide
> themselves when any application that is running on a single virtual machine
> along side the various services (like Solr/Postgres/Reddis/ApacheORNginx...
>
> Our hope is to move fast but we'd also like to do so in the open and
> hopefully within the existing CKAN community. Are there others out there
> that would be interested in supporting a community repo for a docker-based
> CKAN? I know there has been some work in the past on this (and I'm sure
> there's more ongoing) - so this too could be an opportunity to consolidate?
>
>
> On Mon, Jan 11, 2016 at 9:01 AM, Derek Hohls <dhohls at csir.co.za> wrote:
>
>> Hi Ian
>>
>> Thanks for the info.  I see you refer to the datacats git site for
>> issues; but I also see some issues that are outstanding.
>>
>> The specific issue I am interested in is the inclusion of CSW/pycsw into
>> the datacats environment because that will be our primary source type for
>> geospatial metadata.  I see there is one issue logged and one outstanding
>> pull request (https://github.com/datacats/datacats/issues/301 and
>> https://github.com/datacats/datacats/pull/303 ). Rather than reinvent
>> any wheels; is there likely to be any movement on this in the near future
>> (otherwise the only alternative seems to be to go back to a fully manual
>> CKAN install process without all the benefits that Docker offers)?
>>
>> Derek
>>
>> >>> Ian Ward <ian at excess.org> 01/08/16 4:38 PM >>>
>> Hi Derek,
>>
>> We don't have a mailing list, but feel free to use the issues on
>> github for problems you run into.
>>
>> Installing most extensions in datacats just requires dropping the
>> extension in your source directory and running "datacats install".
>> Anything else you might need to do to get set up can usually be done
>> with the "datacats paster" (run a paster command from the extension in
>> the current directory) or "datacats shell" (open a shell inside the
>> environment)
>>
>> Ian
>>
>> On Fri, Jan 8, 2016 at 9:13 AM, Derek Hohls <dhohls at csir.co.za> wrote:
>> > Hi
>> >
>> > I'd like to know if this is the correct mailing list to post questions
>> > around the usage of the datacats (Docker-based CKAN) software? I am
>> > struggling with adapting the usual approach to CKAN extension
>> installations
>> > to the one used by datacats (e.g. adapting the
>> > http://docs.ckan.org/projects/ckanext-spatial/en/latest/install.html ).
>> >
>> > If this is not the correct forum, please point me in the right
>> direction.
>> >
>> > Thanks
>> > Derek
>> >
>> > --
>> > This message is subject to the CSIR's copyright terms and conditions,
>> e-mail
>> > legal notice, and implemented Open Document Format (ODF) standard.
>> > The full disclaimer details can be found at
>> > http://www.csir.co.za/disclaimer.html.
>> >
>> >
>> > This message has been scanned for viruses and dangerous content by
>> > MailScanner,
>> > and is believed to be clean.
>> >
>> >
>> > Please consider the environment before printing this email.
>> >
>> >
>> > _______________________________________________
>> > ckan-dev mailing list
>> > ckan-dev at lists.okfn.org
>> > https://lists.okfn.org/mailman/listinfo/ckan-dev
>> > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>> >
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>
>> --
>> This message is subject to the CSIR's copyright terms and conditions,
>> e-mail legal notice, and implemented Open Document Format (ODF) standard.
>> The full disclaimer details can be found at
>> http://www.csir.co.za/disclaimer.html.
>>
>> This message has been scanned for viruses and dangerous content by
>> MailScanner,
>> and is believed to be clean.
>>
>> Please consider the environment before printing this email.
>>
>>
>> --
>> This message is subject to the CSIR's copyright terms and conditions,
>> e-mail legal notice, and implemented Open Document Format (ODF) standard.
>> The full disclaimer details can be found at
>> http://www.csir.co.za/disclaimer.html.
>>
>>
>> This message has been scanned for viruses and dangerous content by
>> *MailScanner* <http://www.mailscanner.info/>,
>> and is believed to be clean.
>>
>>
>> Please consider the environment before printing this email.
>>
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>
>>
>
>
> --
> Chief Data Engineer
> @Data.gov
> OCSIT | 18F
> General Services Administration
> 202-341-0191
>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20160114/600c9eed/attachment-0003.html>