[openspending-dev] OpenSpending's future architecture

[Tryggvi Björgvinsson]

Hi all,

I wanted to give all of you update on list about where we're at with
OpenSpending. As some of you might know, others might have deduced and
some maybe not known about, we've been going through a trial period
since the beginning of the year, to try to understand better what
architecture we'd go for with OpenSpending in the future.

There are two approaches we've wanted to choose between:

* Continue with the monolithic code base we have but port it from the
obsolete pylons to flask (something Friedrich has done to a large extent
-- thanks Friedrich!)
* Go for the micro-services based approach proposed in the OpenSpending
Enhancement Proposal #1:
https://github.com/openspending/osep/blob/gh-pages/01-approach-and-architecture-of-openspending.md

Ultimately this boils down to preference. With micro-services we
increase deployment complexity at the cost of more flexibility. It is
hard for me to make an unbiased decision as I was involved in the
micro-services approach proposal (OSEP#1) so it should not come as a
surprise that I am going to propose that we *accept* OSEP#1.

This does not mean the flask port gets thrown out the window and never
looked at again. It means that it won't be *the* service. There are
definitely things in it we can use so we might strip out a few things
(This is one of the reasons I settled for flask as the preferred web
framework for OpenSpending services - as discussed at the February dev
meeting)

I came to this conclusion by looking at both users and developers but I
do admit (like I said before) that this might be a biased view.

# Users

Users of OpenSpending or potential users want to be able to do different
things. Some would like to be able to compare different datasets, some
would just like to provide datasets, some want to visualise datasets on
their site without having to rely on an external service, some want to
be able to find source data.

People want to do different things and they come to OpenSpending for
different reasons. Currently OpenSpending is kind of designed to be an
analysis service. You model a dataset, based on a source file, you load
it, you extract the data you want to make pretty visualisations. Nobody
other than you is likely to use your dataset.

We could of course modify the existing code base to better serve the
users' needs and provide different contact points to the process but I
think the micro-service approach, which inevitably has to be built
around conformed or standardised data passing between the services,
allows for better flexibility and gives people a better chance to hook
into the process based on their needs (key thing here being standardised
or conformed datasets).

# Developers

Now with developers I am mostly thinking about the OpenSpending
"services" developers. I see some users as being developers (those who
want to re-use some part or all of the OpenSpending services to fulfill
their needs) but I'm not focussing on those. However I interpret the
OpenSpending "services" developers more broadly than I do for the
current OpenSpending site.

There are a lot of very interesting projects/services that we could
easily re-use, build around or incorporate without having to merge them
into our code base, or others could re-use our services:

* Just today Paul Walsh announced a tabular validator project on the
okfn-labs mailing list (disclaimer: I'm involved in that project).
* Friedrich Lindenberg has a really great idea to have a central civic
app login system (something he calls CitizenID) and there are other such
services (more general) like lastuser by hasgeek.
* Nathan Hilbert has been using I believe some parts of OpenSpending to
create a data management platform in a project he's working on.

These can be a part of OpenSpending but can also stand as services on
their own and developers can work on them without having to understand
or worry about the whole OpenSpending project (like how to do a data
warehouse or how to provide search).

So I believe that with this, we can be more flexible in what we use and
do to create a central repository for conformed budget data and provide
different services around it, based on users' needs. Just like the
single statement summary says:

> We want to centralize data but decentralize "presentation" ("views")

It's not something we do tomorrow. It will take us a long time to get
there but this is what I believe will make OpenSpending more valuable to
users (and developers).

It's been a really difficult decision to make but I wanted to share this
with you in advance and give you a chance to comment before I
"officially" accept OSEP#1 this Saturday.

/Tryggvi