[ckan-dev] Q&A for datacats installs?

Wed Jan 13 21:32:30 UTC 2016

correction:  it makes sense to use CKAN as the source of truth for the
Governance of code repos alongside relevant *data*

*STEVEN DE COSTA *|
*EXECUTIVE DIRECTOR*www.linkdigital.com.au

On 14 January 2016 at 08:29, Steven De Costa <
steven.decosta at linkdigital.com.au> wrote:

> The quick answer from me is yes :)
>
> At Link Digital manage CKAN for various jurisdictions and it makes some
> sense to isolate the infrastructure in this case. Within those
> jurisdictions we are doing more 2nd generation work with the agencies now
> to establish data pipelines for open publishing. This often includes the
> establishment of an agency CKAN as a clearing house or review check point
> prior to release. It also serves as an internal data catalogue for those
> agencies and can be extended to support their specific governance and
> workflow requirements.
>
> We've found that docker can be handy for small platforms but the benefits
> become issues when those environments need to scale, so isolating processes
> better to allow for easier scaling would be great.
>
> Along similar thinking I've been working on finding clients and use cases
> that would help us fund development of an approach that would abstract data
> storage into what I'd call 'resource containers'. This would be along the
> same lines as resource views and allow platform managers to configure
> options for resource storage. An example might be to configure cloud
> storage on either S3 or Azure and allow custodians to select which they
> prefer, with the option of specifying their own account for the storage and
> therefore cover the raw costs involved.
>
> Resource containers of this type could be suitable for configuration of
> third party technology and deal with use cases that involve big data,
> unstructured data, automatically cleansed data, etc.
>
> Perhaps a good example to highlight the diversity of options this would
> open up is to consider an integration with GitHub.com and GitHub Enterprise
> as resource containers for code. There is strong relationship between data
> and code when you consider civic technology and where Government seeks to
> stimulate public-private partnerships that leverage open data it makes
> sense to use CKAN as the source of truth for the Governance of code repos
> alongside relevant code. The same use cases apply where there is a change
> of Government and portfolios are rearranged. Custodianship of data and open
> source projects can then be easily shifted. Again, the same use cases apply
> for discover-ability and the potential for standardization of civic
> applications on top of standardized data improves.
>
> But back to the point on docker... I'm sure our solution architect would
> be interested in joining a call to discuss approaches and share our
> experiences. We'd be happy to collaborate with others and help move things
> forward in a common direction.
>
> Cheers,
> Steven
>
> *STEVEN DE COSTA *|
> *EXECUTIVE DIRECTOR*www.linkdigital.com.au
>
>
>
> On 14 January 2016 at 04:31, John Jediny - XAAB <john.jediny at gsa.gov>
> wrote:
>
>> Derek... I'm used to replying to you on the Geonode <http://geonode>
>> mailing list (small world)...
>>
>> I recently joined this list as well as the data.gov team, and we've been
>> looking for a similar solution to datacats but our need requires us to
>> deploy CKAN using as little infrastructure as possible to scaffold out our
>> future deployments which will be docker based. Our plan is to use
>> http://bosh.io (an InfrastructureaaS agnostic
>> deployment/orchestration/health manager) to managed our future deployments
>> of docker/docker-swarm to handle the management of multiple CKANs as well
>> as load balancing...
>>
>> Currently testing a docker based (alpha) repo that would then be
>> deployed onto the docker-swarm <https://github.com/GSA/catalog-app> (but
>> this is currently a rough cut at packaging CKAN with Docker - there is
>> still ALOT of work to be done to refactor the application to isolate
>> services/processes per container and more inline with
>> http://12factor.net... our current plan has use looking at our current
>> production and test environments to identify all our custom processes that
>> tend to hide themselves when any application that is running on a single
>> virtual machine along side the various services (like
>> Solr/Postgres/Reddis/ApacheORNginx...
>>
>> Our hope is to move fast but we'd also like to do so in the open and
>> hopefully within the existing CKAN community. Are there others out there
>> that would be interested in supporting a community repo for a docker-based
>> CKAN? I know there has been some work in the past on this (and I'm sure
>> there's more ongoing) - so this too could be an opportunity to consolidate?
>>
>>
>> On Mon, Jan 11, 2016 at 9:01 AM, Derek Hohls <dhohls at csir.co.za> wrote:
>>
>>> Hi Ian
>>>
>>> Thanks for the info.  I see you refer to the datacats git site for
>>> issues; but I also see some issues that are outstanding.
>>>
>>> The specific issue I am interested in is the inclusion of CSW/pycsw into
>>> the datacats environment because that will be our primary source type for
>>> geospatial metadata.  I see there is one issue logged and one outstanding
>>> pull request (https://github.com/datacats/datacats/issues/301 and
>>> https://github.com/datacats/datacats/pull/303 ). Rather than reinvent
>>> any wheels; is there likely to be any movement on this in the near future
>>> (otherwise the only alternative seems to be to go back to a fully manual
>>> CKAN install process without all the benefits that Docker offers)?
>>>
>>> Derek
>>>
>>> >>> Ian Ward <ian at excess.org> 01/08/16 4:38 PM >>>
>>> Hi Derek,
>>>
>>> We don't have a mailing list, but feel free to use the issues on
>>> github for problems you run into.
>>>
>>> Installing most extensions in datacats just requires dropping the
>>> extension in your source directory and running "datacats install".
>>> Anything else you might need to do to get set up can usually be done
>>> with the "datacats paster" (run a paster command from the extension in
>>> the current directory) or "datacats shell" (open a shell inside the
>>> environment)
>>>
>>> Ian
>>>
>>> On Fri, Jan 8, 2016 at 9:13 AM, Derek Hohls <dhohls at csir.co.za> wrote:
>>> > Hi
>>> >
>>> > I'd like to know if this is the correct mailing list to post questions
>>> > around the usage of the datacats (Docker-based CKAN) software? I am
>>> > struggling with adapting the usual approach to CKAN extension
>>> installations
>>> > to the one used by datacats (e.g. adapting the
>>> > http://docs.ckan.org/projects/ckanext-spatial/en/latest/install.html
>>> ).
>>> >
>>> > If this is not the correct forum, please point me in the right
>>> direction.
>>> >
>>> > Thanks
>>> > Derek
>>> >
>>> > --
>>> > This message is subject to the CSIR's copyright terms and conditions,
>>> e-mail
>>> > legal notice, and implemented Open Document Format (ODF) standard.
>>> > The full disclaimer details can be found at
>>> > http://www.csir.co.za/disclaimer.html.
>>> >
>>> >
>>> > This message has been scanned for viruses and dangerous content by
>>> > MailScanner,
>>> > and is believed to be clean.
>>> >
>>> >
>>> > Please consider the environment before printing this email.
>>> >
>>> >
>>> > _______________________________________________
>>> > ckan-dev mailing list
>>> > ckan-dev at lists.okfn.org
>>> > https://lists.okfn.org/mailman/listinfo/ckan-dev
>>> > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>> >
>>> _______________________________________________
>>> ckan-dev mailing list
>>> ckan-dev at lists.okfn.org
>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>
>>> --
>>> This message is subject to the CSIR's copyright terms and conditions,
>>> e-mail legal notice, and implemented Open Document Format (ODF) standard.
>>> The full disclaimer details can be found at
>>> http://www.csir.co.za/disclaimer.html.
>>>
>>> This message has been scanned for viruses and dangerous content by
>>> MailScanner,
>>> and is believed to be clean.
>>>
>>> Please consider the environment before printing this email.
>>>
>>>
>>> --
>>> This message is subject to the CSIR's copyright terms and conditions,
>>> e-mail legal notice, and implemented Open Document Format (ODF) standard.
>>> The full disclaimer details can be found at
>>> http://www.csir.co.za/disclaimer.html.
>>>
>>>
>>> This message has been scanned for viruses and dangerous content by
>>> *MailScanner* <http://www.mailscanner.info/>,
>>> and is believed to be clean.
>>>
>>>
>>> Please consider the environment before printing this email.
>>>
>>>
>>> _______________________________________________
>>> ckan-dev mailing list
>>> ckan-dev at lists.okfn.org
>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>
>>>
>>
>>
>> --
>> Chief Data Engineer
>> @Data.gov
>> OCSIT | 18F
>> General Services Administration
>> 202-341-0191
>>
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20160114/cdfbc644/attachment-0003.html>