[openspending-dev] OpenSpending's future architecture

Fri Feb 20 10:10:04 UTC 2015

Hi Tryggvi,

I guess this decision is OKFN's to make. So I will respect it.

Allow me to say, however, that my experience in trying to contribute to
OpenSpending over the last two months has been very frustrating.

I know that I am not a great team player. When I have an approach that I
think might work, I enjoy going ahead and seeing where it leads. I'm a
cowboy coder.

While trying to make the recent changes, however, I've very much tried to
engage you and others in an open discussion - on #833 and by sending lots
of email updates to the list, discussing ideas with Stiivi on IRC/ML etc.
The response on these from you and Rufus has been extremely uniform: "We
have a big plan. We must do the big plan. This is not the big plan."

At the same time, the individual contributions I have suggested - Flask
port, a better ETL pipeline, more flexible data modelling, richer metadata
- have not been given consideration or feedback. Their substance has mostly
been disregarded, because they deviate from the "big plan".

It's not that any of them actually contradict the micro-services model. By
and large, they are orthogonal to that question. I just don't rate the
intrinsic value of micro-services as highly as you do. I think OpenSpending
faces more important challenges: it's impossible to find datasets on the
site, hard to model the data, there has been zero new value on the
analytics/datavis end for years etc.

An exclusive focus on OSEP #1 would make some sense if OKFN had received
major funding for OpenSpending and was able to pull off a major refactor
within a short time-frame. But as far as I can tell this is not the case.
OSEP #1 has been around for two years, and zero progress has been made on
it. If there is some actual development here, it would be good for OKFN to
be more open about it.

In the Jan 8 developer meeting we made an agreement: we'd prototype both
approaches until Open Data Day and then make a decision. To the best of my
knowledge, there isn't any prototype of OSEP #1. I understand that we all
have other work to do. But not doing any of the proposed prototyping and
then (without any discussion) declaring victory on theoretical merits just
seems very bad style.

At the same time, you still won't engage in any way with the changes I have
proposed and the working code base which I have delivered. This leaves me
incredibly frustrated, and, sadly, leaves me unsure of how I can, if at
all, contribute to OpenSpending.

I'm not sure what the way forward is for me. My fellowship commitments with
Code for Africa might be best served if I set up a localised fork for the
African countries in which we are active. I will try to find an option that
is more constructive to the OS community.

In any case, I wish you the best of luck with the micro-services. Hopefully
I'm wrong, and OpenSpending really needs more HTTP, not more UX, and
hopefully you have the resource to actually make it happen.

Regards,

- Friedrich

On Thu, Feb 19, 2015 at 11:53 PM, Tryggvi Björgvinsson <
tryggvi.bjorgvinsson at okfn.org> wrote:

> Hi all,
>
> I wanted to give all of you update on list about where we're at with
> OpenSpending. As some of you might know, others might have deduced and
> some maybe not known about, we've been going through a trial period
> since the beginning of the year, to try to understand better what
> architecture we'd go for with OpenSpending in the future.
>
> There are two approaches we've wanted to choose between:
>
> * Continue with the monolithic code base we have but port it from the
> obsolete pylons to flask (something Friedrich has done to a large extent
> -- thanks Friedrich!)
> * Go for the micro-services based approach proposed in the OpenSpending
> Enhancement Proposal #1:
>
> https://github.com/openspending/osep/blob/gh-pages/01-approach-and-architecture-of-openspending.md
>
> Ultimately this boils down to preference. With micro-services we
> increase deployment complexity at the cost of more flexibility. It is
> hard for me to make an unbiased decision as I was involved in the
> micro-services approach proposal (OSEP#1) so it should not come as a
> surprise that I am going to propose that we *accept* OSEP#1.
>
> This does not mean the flask port gets thrown out the window and never
> looked at again. It means that it won't be *the* service. There are
> definitely things in it we can use so we might strip out a few things
> (This is one of the reasons I settled for flask as the preferred web
> framework for OpenSpending services - as discussed at the February dev
> meeting)
>
> I came to this conclusion by looking at both users and developers but I
> do admit (like I said before) that this might be a biased view.
>
> # Users
>
> Users of OpenSpending or potential users want to be able to do different
> things. Some would like to be able to compare different datasets, some
> would just like to provide datasets, some want to visualise datasets on
> their site without having to rely on an external service, some want to
> be able to find source data.
>
> People want to do different things and they come to OpenSpending for
> different reasons. Currently OpenSpending is kind of designed to be an
> analysis service. You model a dataset, based on a source file, you load
> it, you extract the data you want to make pretty visualisations. Nobody
> other than you is likely to use your dataset.
>
> We could of course modify the existing code base to better serve the
> users' needs and provide different contact points to the process but I
> think the micro-service approach, which inevitably has to be built
> around conformed or standardised data passing between the services,
> allows for better flexibility and gives people a better chance to hook
> into the process based on their needs (key thing here being standardised
> or conformed datasets).
>
> # Developers
>
> Now with developers I am mostly thinking about the OpenSpending
> "services" developers. I see some users as being developers (those who
> want to re-use some part or all of the OpenSpending services to fulfill
> their needs) but I'm not focussing on those. However I interpret the
> OpenSpending "services" developers more broadly than I do for the
> current OpenSpending site.
>
> There are a lot of very interesting projects/services that we could
> easily re-use, build around or incorporate without having to merge them
> into our code base, or others could re-use our services:
>
> * Just today Paul Walsh announced a tabular validator project on the
> okfn-labs mailing list (disclaimer: I'm involved in that project).
> * Friedrich Lindenberg has a really great idea to have a central civic
> app login system (something he calls CitizenID) and there are other such
> services (more general) like lastuser by hasgeek.
> * Nathan Hilbert has been using I believe some parts of OpenSpending to
> create a data management platform in a project he's working on.
>
> These can be a part of OpenSpending but can also stand as services on
> their own and developers can work on them without having to understand
> or worry about the whole OpenSpending project (like how to do a data
> warehouse or how to provide search).
>
> So I believe that with this, we can be more flexible in what we use and
> do to create a central repository for conformed budget data and provide
> different services around it, based on users' needs. Just like the
> single statement summary says:
>
> > We want to centralize data but decentralize "presentation" ("views")
>
> It's not something we do tomorrow. It will take us a long time to get
> there but this is what I believe will make OpenSpending more valuable to
> users (and developers).
>
> It's been a really difficult decision to make but I wanted to share this
> with you in advance and give you a chance to comment before I
> "officially" accept OSEP#1 this Saturday.
>
> /Tryggvi
>
> _______________________________________________
> openspending-dev mailing list
> openspending-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/openspending-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/openspending-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/openspending-dev/attachments/20150220/5d443507/attachment-0002.html>