[openspending-dev] Modularisation

Sun Mar 10 20:42:17 UTC 2013

Hey Martin!

On Sun, Mar 10, 2013 at 8:47 PM, Martin Keegan <martin.keegan at okfn.org>wrote:

> On Sun, Mar 10, 2013 at 12:11 PM, Friedrich Lindenberg
> <friedrich.lindenberg at okfn.org> wrote:
> >> Of course this is complete speculation. This is not even in the plans
> >> and I'm not proposing it, but you prove my point. If swapping out one
> >> module for another requires refactoring and re-designing the rest of the
> >> software there is something illogical in the design that needs to be
> >> fixed. That's exactly why I feel modularisation is good.
> >
> > What is the inherent value of "swapability"? My point is that OS right
> now
>
> The value of specialisation, and the interchangeability of parts, has
> been understood since the writing of Adam Smith in the 18th century.
>

I think we need to build on this link! Let's have a panel session with
Margaret Thatcher and Wolfgang Schäuble to discuss the applicability of the
CAP theorem to the European single market!

> has a number of integrated components (a data modelling interface that
> > generates models in a specific format, a data validation service that
> > evaluates these models, a query service that builds aggregations based on
> > this model). Swapping out one of these would necessitate adapting the
> other
> > parts.
>
> This is somewhat begging the question, though, isn't it?
>
> A modular system is one in which you don't have to change one
> component in order to change another. It's common ground that
> OpenSpending does not have this property.
>

You mean openspending/openspending, right? Not the other components like
openspending/osvalidate, openspending/dotorg and
openspending/openspendingjs or any of the dozen independent satellite
sites? Then agreed.

The observation that "swapping out one of [the components] would
> necessitate adapting the other parts" is a statement that OpenSpending
> is not modular, rather than an argument that it should not be made
> more modular.
>

Actually, the point I was trying to make was that we're talking about
making OpenSpending's domain model exchangeable. As things to out-source
go, an application's domain model is one of the more exotic candidates. (In
fact, I'm keen to collect some examples - I think some of OSGi does this,
but I'm not sure.)

> It's not clear to me that the components you refer to in the paragraph
> above should be disaggregated from each other. However, whether it
> makes sense to keep them together does not determine questions such as
> whether the OLAP backend should be modular.
>

I'm not sure what that means, but I get the impression that other than
"Adam Smith said so" you didn't really make a concrete point in favor of
extracting the aggregation mechanism from OpenSpending? Will Adam
contribute functionality to the site?

> It's a bit like complaining that Wordpress is horrible because it can't
> > render content from a Drupal database (and that we all need to define a
> > common concept of content so that all CMS can access the same database -
> > Java people actually did this in proposals like JSR 168 and 170, not with
> > very much success. It's infrastructure for its own sake.)
>
> This is an analogy, not a deductive argument.
>

Correct. But its a nice analogy.

> Let me be clear: I didn't say this is generic, it's just the thing that
> > doesn't exist yet. I personally believe that's because in order to make
> it
> > good to use for non BI-folks, you have to make it specific (model your
> data
>
> What's the basis for this belief? (I'm not saying it's untrue, just
> wondering where it comes from)
>

It's based on discussions we had during the handover on how we could make
the data loader more accessible. It would be great to make it into more of
a questionnaire, where the user is lead through a series of questions (Does
this budget have a functional classification? -> Autogenerate a dimension;
Does this transactional dataset have a geographic identifier? -> Make a
dimension). This would require users to answer questions from their area of
expertise (budgets) rather than having them deal with a fairly abstract
interface that uses business intelligence language). It would also mean we
start collecting higher-level semantics of the datasets coming in.

> ... for what?). If you want to simplify data loading for OpenSpending, add
> > more guidance and specificity - don't rip it out and dump all CKAN use
> cases
> > onto it just so it becomes completely impossible to keep simple.
>
> This seems like a straw man - is anyone proposing to rip out and dump
> all CKAN use cases onto OS? (Someone may be, of course, I just wasn't
> aware; you'll recall I was bit proponent of the "optional" compromise
> between the "mandatory" and "prohibited" approaches to the use of CKAN
> for loading data into OS).
>

Well yes. Gavin and Rufus would like to have a model editor in CKAN,
Tryggvi proposed: "3. Bob the builder (the modeller)
I'm putting this as a special project because as you said it is a generalised
product and I think it might help other projects (such as a plugin to CKAN
with direct upload to the spending cube)." - which raises the question:
what about all the data in CKAN that isn't spending?

> Nah, most of it just doesn't generate value to users. This is not
> > short-sightedness, its just keeping the eye on the ball.
>
> Did changing from SQL to Mongo and back to SQL again generate value for
> users?
>

Changing to Mongo was done to achieve multi-tenancy, the basic value
proposition of OpenSpending. Going back to SQL was done to increase
availability of the service. Error isn't that useful.

Sometimes you need to do things which are totally irrelevant to users.
> Focus on user needs should be a high priority, not a rule which
> excludes all other activity.
>

All I'm saying is: are we doing this because we actually have some concrete
architectural issue that gets solved or because it turns on our inner geek?

Cheers,

 - Friedrich
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/openspending-dev/attachments/20130310/387135e8/attachment.html>