[openspending-dev] Modularisation

Sun Mar 10 10:45:42 UTC 2013

Þann lau 9.mar 2013 22:30, skrifaði Friedrich Lindenberg:
> Well, I'm not entirely sure we're facing the "really big software" issue so
> much: we're talking about something around 7000 lines of Python (including
> tests). I've worked on individual Java classes that had half that line
> count. Out of this, roughly 500 lines are actually the cubes code - most of
> the effort goes into user guidance and data loading.

All the more reason to start thinking about proper software design now.
It becomes a bigger headache to work with design debt as the lines of
code increase.

> Sorry, but if you really want to start supporting deployments, you're in
> deep, deep trouble. That would actually require releases and release
> management. People can do it, but do you really want to make it your
> responsibility to fix it? This will drain your time to work on stuff
> immediately while not fixing any issue other than bad communication.

No I don't want to support deployments but I don't want to be in the way
for those who want to deploy on their own. They might actually
contribute back to the project (which I'll always see as an indicator of
a good open source project).

> Side-note: most German budgets are single-entry bk and last time I checked
> we had about 50 of these loaded. There is very little that I can think of
> that can be added to OpenSpending core to make support for these better.

Good to know :-) This is something we need to let Bernhard and Hannes
know about.

>> In their case they're going to pay
>> for some development and could divert that dev attention to the
>> "spending analytical machine (spam?)" and leave things like what does
>> openspending.content_root in the .ini file do to the people who work on
>> offenerhaushalt.at
> Wut? Not sure I follow this?

It's not that important. The way OpenSpending is constructed now is a
big pile of moderately related components. If you want to improve a core
part of it (like e.g. the cube) you'll have to grok unrelated pieces
(well I would) which from my perspective doesn't make a whole lot of
sense. This was just an example of what I find weird, why does spending
analysis software have a configuration for "content root"? (I know the
reason -- this is just an example)

> The biggest benefit I see with having a modularised system is that it
>> makes it easier for us to swap out the modules. What if the python cubes
>> project is good enough for us? A modularised system would allow us to
>> drop our homebuilt solution (which is probably better than cubes for our
>> needs at the moment) and replace it with cubes. We would become
>> contributors to cubes but active maintenance would be somewhere else
>> (raising some other issues which are I think easier to deal with than
>> maintaining things on our own). This would actually help us focus more
>> on our mission and real problems (although it's always easy to hack
>> together a working piece of software, maintaining it is difficult -
>> especially when it's not the core focus of the project).
>>
> YAGNI until proven otherwise. This is complete speculation. Cubes has cool
> stuff in it - no question - but the actual effort for using it would be in
> re-architecting most of the rest of OS to use it: it doesn't have data
> loading/table generation, input validation or some component for data
> modelling in the browser.

Of course this is complete speculation. This is not even in the plans
and I'm not proposing it, but you prove my point. If swapping out one
module for another requires refactoring and re-designing the rest of the
software there is something illogical in the design that needs to be
fixed. That's exactly why I feel modularisation is good.

> Re-architecting your entire codebase based on the feeling that you may or
> may not want to do something for one of any number of reasons at some point
> later down the road is just a really bad idea.

Why? Why have a software designed for only now instead of designing it
for future? I don't see why that's a bad idea to make maintenance and
future development easier?

> Can you give a sketch of what modules you envision?

So here's the rough idea I have for how to break the software into
external modules (different projects). This idea revolves around
changing as little as possible (try to immitate OpenSpending's current
structure - just shifting things around a bit):

Four core projects:

1. Spending cube
Spending analysis as RESTful service (albeit using json). Interface
would be via input and output API nothing else. Similar to how the API
work is now but we need an input API (with OAuth). I see it as hosted
standalone on api.openspending.org or data.openspending.org or something
like that. This is the core of our services and it shouldn't change rapidly.

2. Visualisation library
The home of openspendingjs and could be a merge between the gallery and
apps.openspending.org (I don't want to go into that discussion because I
know you, Friedrich, don't like that idea and I haven't formed an
opinion). This is where we'd focus most of our attention since this is
the extension and utilisation of the spending cube, it's what our users
see and can reuse. Again, I see this a standalone service that relies on
our

3. Bob the builder (the modeller)
I'm putting this as a special project because as you said it is a
generalised product and I think it might help other projects (such as a
plugin to CKAN with direct upload to the spending cube). This wouldn't
be a standalone service but something dev can use on their satellite
sites (e.g. to upload stuff from there to our spending cube).

4. Official website
OpenSpending.org as a "satellite site" that uses the 3 other projects
with focus on describing the project, blog posts, how to use it, how to
contribute, etc.

There are other projects we need to think about as well such as the
assembly kit for satellite sites and some decisions we'd have to make
like where do we put Solr (it would most likely be a part of the
spending cube). Then we'd also have to think about the internal
modularisation of each of these projects (like splitting out the
datastore in the spending cube, what template engine to use for the
official website, etc.).

Is something like this really that bad?

> What I really wanted to convince you of is that the environment you're
> working in has a special dynamic with regards to infrastructure work. I'm
> trying to say that I'm concerned about OS getting distracted and tech-happy
> instead of serving its users.

Do I understand you correctly. You're saying OKF doesn't really want to
focus on infrastructure but we still want to be the central store for
spending data? How is that supposed to work? I think we can serve our
users (and ourselves) better with a good technical infrastructure.
Oohh... let me know say it... with "clean foundations" ;-)

But I think you're right in that we should try to avoid being
tech-happy, we're here to help the world and software is just how we
accomplish it. We're not the other way around, happy software hackers
where helping people is a by-product :-)

> So we've gone through three partial rewrites of the platform over the last
> few years, but in terms of user-facing outputs all it does is bubbles,
> treemaps and tables (this is early 2009: http://bund.offenerhaushalt.de/,
> we couldn't even do the bar charts out of the box now). Vitor is our hope
> here (yay, linegraphs!).
>
> Getting other people to actually do cool new vis types like maps, project
> trackers, tell a story, run a statistical analysis on some data or to have
> fun with the data we have in some other way is a much more interesting and
> realistic than to hope that we will get someone to add roll-ups to our
> backend for no particular reason.

Ahh... more fancy stuff to put in our gallery that others can see and
use on their satellite sites. Yes, I agree. It's urgent and we also need
to show possible contributors how they can help us (and themselves) by
contributing cool visualisations.

/Tryggvi