[okfn-help] The Future of Our Tech Infrastructure: RFC

[Rufus Pollock]

Useful associated diagram:
<http://knowledgeforge.net/okfn/tasks/wiki/ServiceArchitecture>
(Source version of this text at [1])

The Future
==========

Having developed, deployed and maintained apps and services at the
OKFN for the last 5 1/2 years we are not at a point of relative
maturity and I think it is time to take stock of where we are and look
to the future.

We currently support more than 10 production or semi-production
application services (apps or app services from hereon) as well as
lots of bleeding edge development. The majority of these apps are OKFN
projects but we also do some service provision to others.

It appears likely that both internal and external demand is only
likely to grow. Furthermore, there is some important re-architecting
we can do that both ease our ability to scale and significantly
improve the app development process by providing key base services
(see below) in a standardized way.

Our aim: solid, scalable infrastructure on which to develop the open
knowledge application "stack"

Some Observations on the World at Present
=========================================

  * We are in a world in which hardware/bw are **cheap** and rented
  * Larger machines are cheaper per compute unit than smaller machines.
  * Clear economies of scope across projects and services
  * Many application services require a common a set of 'base'
services/infrastructure ranging from file storage to the full-text
search.  While each of these can be provided as part of the individual
app (i.e. by integrating search software into the app) it is
increasingly attractive to provide these at the **service** level.
  * In general, we (and others) are moving towards a more
"service-oriented" architecture in which different services are
integrated together

Service-oriented architecture
=============================

**Aim:** Reap benefits of economies of scope and scale, simplify our
internal architecture, and follow the UNIX philosophy (each tool
(service) does one thing and one thing well).

Key base services provided:

  * Bare-metal machines
  * Relational databases: mysql and postgresql
  * RDF: 4store
  * Keyvalue: redis + mongo (couchdb?)
  * Filestorage (?): more than bare-metal machines
  * Messaging and job queue: celery + redis + rabbitmq
  * Indexing: solr + xapian (?)
  * Mailing lists: lists.okfn.org
  * MX: mail exchange but no mail accounts

For each of these we deliver:

  * Provisioning and availability
  * Backup
  * Redeployment from backup

General architecture for a application service detailing its use of
key base services is shown in diagram here:

<http://knowledgeforge.net/okfn/tasks/wiki/ServiceArchitecture>

Appendix: The Unix Philosophy
=============================

See unix-philosophy.txt

[1]: <http://knowledgeforge.net/okfn/sysadmin/file/b9c53a5e1765/doc/future-of-infrastructure.txt>