[openspending-dev] Modularisation

Sun Mar 10 12:11:34 UTC 2013

Hey,

On Sun, Mar 10, 2013 at 11:45 AM, Tryggvi Björgvinsson <
tryggvi.bjorgvinsson at okfn.org> wrote:

> Þann lau 9.mar 2013 22:30, skrifaði Friedrich Lindenberg:
> > Well, I'm not entirely sure we're facing the "really big software" issue
> so
> > much: we're talking about something around 7000 lines of Python
> (including
> > tests). I've worked on individual Java classes that had half that line
> > count. Out of this, roughly 500 lines are actually the cubes code - most
> of
> > the effort goes into user guidance and data loading.
>
> All the more reason to start thinking about proper software design now.
> It becomes a bigger headache to work with design debt as the lines of
> code increase.
>

I'm not at all against thinking about software design (but would contest
the notion that its just "starting" ;), just against designing things for
the sake of design and searching beauty in complexity.

> > YAGNI until proven otherwise. This is complete speculation. Cubes has
> cool
> > stuff in it - no question - but the actual effort for using it would be
> in
> > re-architecting most of the rest of OS to use it: it doesn't have data
> > loading/table generation, input validation or some component for data
> > modelling in the browser.
>
> Of course this is complete speculation. This is not even in the plans
> and I'm not proposing it, but you prove my point. If swapping out one
> module for another requires refactoring and re-designing the rest of the
> software there is something illogical in the design that needs to be
> fixed. That's exactly why I feel modularisation is good.
>

What is the inherent value of "swapability"? My point is that OS right now
has a number of integrated components (a data modelling interface that
generates models in a specific format, a data validation service that
evaluates these models, a query service that builds aggregations based on
this model). Swapping out one of these would necessitate adapting the other
parts.

It's a bit like complaining that Wordpress is horrible because it can't
render content from a Drupal database (and that we all need to define a
common concept of content so that all CMS can access the same database -
Java people actually did this in proposals like JSR 168 and 170, not with
very much success. It's infrastructure for its own sake.)

> > Re-architecting your entire codebase based on the feeling that you may or
> > may not want to do something for one of any number of reasons at some
> point
> > later down the road is just a really bad idea.
>
> Why? Why have a software designed for only now instead of designing it
> for future? I don't see why that's a bad idea to make maintenance and
> future development easier?
>

I guess its a line between avoiding technical debt and pre-empting
requirements, what I believe is the cardinal sin of software development
particularly within OKFN: "Always implement things when you actually need
them, never when you just foresee that you need them."

> Can you give a sketch of what modules you envision?
>
> So here's the rough idea I have for how to break the software into
> external modules (different projects). This idea revolves around
> changing as little as possible (try to immitate OpenSpending's current
> structure - just shifting things around a bit):
>
> Four core projects:
>
> 1. Spending cube
> Spending analysis as RESTful service (albeit using json). Interface
> would be via input and output API nothing else. Similar to how the API
> work is now but we need an input API (with OAuth). I see it as hosted
> standalone on api.openspending.org or data.openspending.org or something
> like that. This is the core of our services and it shouldn't change
> rapidly.
>
> 2. Visualisation library
> The home of openspendingjs and could be a merge between the gallery and
> apps.openspending.org (I don't want to go into that discussion because I
> know you, Friedrich, don't like that idea and I haven't formed an
> opinion). This is where we'd focus most of our attention since this is
> the extension and utilisation of the spending cube, it's what our users
> see and can reuse. Again, I see this a standalone service that relies on
> our
>
> 3. Bob the builder (the modeller)
> I'm putting this as a special project because as you said it is a
> generalised product and I think it might help other projects (such as a
> plugin to CKAN with direct upload to the spending cube). This wouldn't
> be a standalone service but something dev can use on their satellite
> sites (e.g. to upload stuff from there to our spending cube).
>

Let me be clear: I didn't say this is generic, it's just the thing that
doesn't exist yet. I personally believe that's because in order to make it
good to use for non BI-folks, you have to make it specific (model your data
... for what?). If you want to simplify data loading for OpenSpending, add
more guidance and specificity - don't rip it out and dump all CKAN use
cases onto it just so it becomes completely impossible to keep simple.

btw. Bob (query builder) != Model editor

> 4. Official website
> OpenSpending.org as a "satellite site" that uses the 3 other projects
> with focus on describing the project, blog posts, how to use it, how to
> contribute, etc.
>

Ok, so the major change relative to now that you're proposing is that the
API and front-facing site become separate components. I'm not saying thats
a bad idea. But, you now have three fairly complex integration issues: what
belongs where (model editor, data validation output), shared authorisation
(OAuth means users will be bothered with this?) and SEO (does OS data show
up in Google if it gets rendered client-side? or is the front-end
implemented as a new layer on the server side?). An external developer who
doesn't like user interfaces can now set up the backend on its own, but
your team now has to manage two web applications.

I'm really not sure which problem you've solved.

There are other projects we need to think about as well such as the
> assembly kit for satellite sites and some decisions we'd have to make
> like where do we put Solr (it would most likely be a part of the
> spending cube). Then we'd also have to think about the internal
> modularisation of each of these projects (like splitting out the
> datastore in the spending cube, what template engine to use for the
> official website, etc.).
>
> Is something like this really that bad?
>

Nah, most of it just doesn't generate value to users. This is not
short-sightedness, its just keeping the eye on the ball.

> What I really wanted to convince you of is that the environment you're
> > working in has a special dynamic with regards to infrastructure work. I'm
> > trying to say that I'm concerned about OS getting distracted and
> tech-happy
> > instead of serving its users.
>
> Do I understand you correctly. You're saying OKF doesn't really want to
> focus on infrastructure but we still want to be the central store for
> spending data? How is that supposed to work? I think we can serve our
> users (and ourselves) better with a good technical infrastructure.
> Oohh... let me know say it... with "clean foundations" ;-)
>

What I'm saying is that OKFN likes nothing better than creating
infrastructure, just that people usually don't end up using it because most
of it is too far from the reality of people who actually do stuff. We're
building great foundations for fantasy castles.

> Ahh... more fancy stuff to put in our gallery that others can see and
> use on their satellite sites. Yes, I agree. It's urgent and we also need
> to show possible contributors how they can help us (and themselves) by
> contributing cool visualisations.
>

Yeah, fancy stuff, but really: useful stuff. Let's not get stuck
implementing a service-oriented architecture for bubbles.

Cheers,

 - Friedrich
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/openspending-dev/attachments/20130310/440d2d19/attachment.html>