[wdmmg-dev] Proposal: Architectural Changes - Part I. (long)

Fri Jul 8 07:48:23 UTC 2011

On Thu, Jul 07, 2011 at 05:36:24PM -0700, Stefan Urbanek wrote:
> Hi,

Gday Stefan,

thanks for putting in the time and effort to write this up!

> * legacy application package wdmmg and wdmmg-ext will be split into more logical and functionally coupled
>   packges:
> 
>   * OpenSpending Web - web front-end application, with interface for pluggable visualisations
>   * OpenSpending ETL - extraction, transformation and loading package
>   * OpenSpending OLAP - online analytical processing package

Certainly, splitting ETL and the web frontend from each other is a goer, and we should see what the shape of the 
remaining pieces is after that.

As discussed in Berlin, we're going to split out the part of the web app which writes to the database (basically, the 
webapp on sandbox which initiates import and reports errors during import).

> * main OpenSpending packages should focus on "core business" - that is providing analytical insight into
>   spending data, either through web based interface, search engine or API. 
> * analytical dataset is read-only for web application

Ergo, we need a writable db for annotations, flags, etc, and unique ids for entries.

> * most of master data sources (classifications, lists, enumeration) should be available as CKAN packages as
>   well, mainly list of entities, classfications ? *Reasons*: same as reasons for dataset source being stored
>   in CKAN; introduces better reusability of classifications at source level ? potential data providers can be
>   pointed to open and public existing classifications to make their data comply with OpenSpending
>   requirements.

I do worry how we're going to deal with exchange rate datasets: we'll want a list of (currency, currency, rate, date) 
tuples, but only for the dates we care about. Does that count as a real dataset?

> Note: search engine might or might not be part of OLAP module, recommended is indexing out-of OLAP module
> with references to OLAP objects (multidimensional aggregates, detailed facts)

Yes - what are your (and other people's) views on Solr and how it fits in?

Mk