[openspending-dev] Micro-services: OpenSpending's future architecture

Sun Dec 28 00:39:30 UTC 2014

Hey Friedrich,

Thank you very much for a great email. I'm very happy to get this
discussion off the ground even if the holidays kept me from replying
appropriately.

I'll reply in-line as usual.

On mán 22.des 2014 17:16, Friedrich Lindenberg wrote:
> Hey Tryggvi, 
>
> thanks for writing up these thoughts, I think this is an incredibly
> valuable discussion for us to have around OpenSpending. In many ways,
> I agree with you: I also believe that OpenSpending would be a better
> piece of software if it was more modular.
>
> That change would help us to define great APIs, make the code base
> clearer and perhaps it would also lead to more contributions -
> although I can't help to consider a hypothesis, rather than a natural
> conclusion.
>
> I think the main reason OS hasn't gathered massive numbers of
> contributors is that the intersection of people who a) know about BI
> to some extent, b) care about public finance from a civic point of
> view, c) care about open source and d) aren't into starting their own
> thing is just very, very small.

I don't agree. The fact that since September last year the number of
contributors to OpenSpending increased 83% shows that there is a lot of
interest in the project (and I'm only looking at the python web app).
That said, I've noticed over the last 1.5 year that many help out a bit,
and then they drop off. So we get interested people on board but for
some reason they don't stick.

Based on my observations (not thorough research) I suspect that the
reason is that it's a very steep learning curve and too much going on so
even if you help out it still is just a bit too much to handle. The
micro-service architecture will result in a better software
architecture, but it also creates smaller bits you can work on and get
comfortable with before you take in the bigger picture. You can start
focussing on things you know before you start learning about other things.

But you are right in that this is a hypothesis or a conjecture. I don't
know if I'm right or not. So let's do this because it leads to better
software, not because it may or may not attract more developers, i.e.
let's leave the theoretical open source discussion out of this :-)

> What you are describing is a very appealing vision - a notion of small
> pieces loosely joined. It represents the best of FOSS design.
>
> Unfortunately, I'm not sure that FOSS design is what's going to help
> OpenSpending have real-world impact. I believe your choice of primary
> target audience (developers and data wranglers) is determined by OKF's
> financial constraints and not by looking at the kinds of problems
> which OpenSpending could help to solve.

That's not correct. Please don't mix Open Knowledge into this. This is a
really weird thing to say when I'm proposing something that makes it
even easier for others to join the project and own different new parts
of it (or run things on their own). I'm proposing something that already
includes everything we're doing now and allows for more stuff to happen.

But you are correct in that this tailors to a specific target audience
which I have implicitly chosen (based on research done as part of the
budget data package development). We can take that discussion here
because it's very important to have when discussing this new architecture.

So here's the target audience of OpenSpending as I see it. OpenSpending
is never going to be an end-user (citizen) facing solution. Budget and
budget information is consumed via information intermediaries, entities
that provide context (possibly after analysis) to budget data and by
doing so help citizens understand it. Citizens (the ultimate target
group of budget transparency) don't really care that much about the
contents of the budgets per se, only after it has been provided to them
with some context+analysis.

These information intermediaries are various organisations, researchers,
journalists, hackers, etc. Groups that either have access to or can get
access to developers and data wranglers. OpenSpending is a tool to help
this group, the information intermediaries do the analysis, get a hold
of the budgets and do "their jobs" and as such needs to make their work
easier.

There's another target group which is "the budget nerds". People who
care about budget transparency and want to load budgets into
OpenSpending to make it possible for information intermediaries to
analyse and deliver to citizens. These budget nerds can be government
officials or members of the information intermediaries, but at them
moment I think this is mostly made up of just people who care and want
to do good.

So our number one goal, imo, is to tailor to people who provide the
budget data and the people who turn that budget data into something
understandable to the public. As in, really easy to upload stuff and a
plethora of options to analyse it, preferably in a way that can be
replicated across datasets (which can save costs for information
intermediaries).

<snip: unrelated discussion about decentralization which is a totally
different thing than micro-services that expose more contact surface of
a still centralized system>

> But I just can't see very much evidence that it actually applies to
> OpenSpending. The people who provide analytical services in this field
> - let's name SpendNetwork and OpenGov.com - don't actually need to
> access our large repository of data (or our APIs). Their customers are
> cities, and these cities bring their own data (and APIs are easy to code).

Their customers (who they care about) are not the same as our indirect
user (the citizen) so our approach is different. We're opening up the
data to provide transparency and understanding for citizens (of course,
that might be the goal of most cities who use SpendNetwork or
OpenGov.com or the analytical services themselves, and if that's the
case, they could also use OpenSpending). We don't restrict ourselves to
any group, we just focus, imo, on helping information intermediaries to
provide context and thus understanding to citizens.

> This makes OpenSpending unlike OpenStreetMap, and it makes developers
> an unrealistic and unwilling target audience for the project. I think
> the budgetary constraints on OpenSpending have lead to a shift in
> thinking. The discussion you're now having is not what problems need
> to be solved, but: which ones are cheap to solve. Putting the code for
> a bunch of APIs on GitHub and storing lots of CSVs on S3 is incredibly
> cheap, I'm just not sure whose problem it solves.

Let's make it super clear that the budgetary constraints on OpenSpending
you are alluding to are that OpenSpending is a community project and
Open Knowledge pays for its running costs which we are grateful for but
that's perhaps not how we as a project want it to be run but that's a
different discussion, so again, don't mix this into the discussion.
We're not talking about this from a financial perspective.

I did add the CSV storage into the micro-services as something new (as
an addition, not a replacement) not because I just want a solution that
can store CSV files cheaply because something-something-which
you-believe-but-I-don't-understand. I added it there because we don't
have it at the moment. We rely on LINKS ON THE INTERNET if something
happens to our database which is _not_ a resilient solution. Sure we
could just hook a downloader into our existing loading mechanism but
that would just be the same as putting the new underpants over the old
ones you urinated in. In the end you'll just get burned as badly by the
pee :-)

Sorry Friedrich, I don't intend this to be mean if it sounds that way. I
just don't like it when I'm accused of not being honest (by having a
secret agenda to what I'm doing) when I'm clearly suggesting an addition
which would make the project more reliable and resilient (well I just
don't like it when I'm accused of not being honest period because I
always try to be, but in this case it should be clear that I have the
project's interests at heart like always and not something else).

> OpenSpending could be a strong open source service, if it did two
> things: a) actually start thinking even more about who it's end-users
> are and start to provide them value, and b) convince a set of funders
> to financially support the site until something fundamentally better
> is available.
>
> OpenSpending, if it is addressed (directly and through it's
> satellites) at citizens, journalists and policy analysts, is a public
> service. It needs to find a funding mode that reflects this: grant
> funding, perhaps even public funding.
>
> OpenSpending, if it is addressed at a group of "other developers" who
> magically need it's services and data yet don't face the same kind of
> constraints OKF has and instead provide great public services, is a
> fiction.

I think we need to align ourselves in what problem we're solving for the
target audience because it seems you might have the same target group in
mind as I do (but please do share as I asked earlier because we need to
be sure).

Are we (the project) doing the in-context analysis in your
"non-fictional" suggestion? I don't think we can. I do think we can
provide context providers with the tools and data they need to do it
(and I do think the big problem they face is that they don't have that
access which is what we solve). That might be fiction, so please
elaborate a little bit about what you're thinking.

> So, in summary: yes, let's make OS a modular application, because it's
> the right thing to do. But let's not adopt the idea that a modular set
> of tools is a replacement for a user-facing web service in 2014. Let's
> find a model for OS to have an impact that doesn't involve the open
> source narrative prop of "other developers" who don't have our problems.

Great!

Note that there still is a user-facing web site, it's just not the most
important thing of OpenSpending as a project. In my opinion, the web
site is the gateway into the project, not the project per se.

> I apologise for the length of my response.

Mine too, but I think it's really great to have this discussion and
please other chime in with your views.

By the way. Great things you're doing with the loadkit. I'll have a look
at it.

/Tryggvi