[openspending-dev] OpenSpending - Thoughts on Approach and Architecture

Sat May 4 09:22:09 UTC 2013

On 3 May 2013 18:18, Tryggvi Björgvinsson <tryggvi.bjorgvinsson at okfn.org> wrote:
> On fös 3.maí 2013 16:59, Rufus Pollock wrote:
>
>> The point of data.openspending.org would be, in fact, to become *the*
>> primary datastore for OpenSpending data.
>>
>> OpenSpending.org would be a frontend and would be the home page for
>> the project but it as a datastore or catalog it would be based off
>> data.openspending.org.
>>
>> Think again of OpenStreetMap. When you go to openstreetmap.org you are
>> seeing the frontend map for the project. But in fact the heart of OSM
>> is the database and the tooling built around it not openstreetmap.org
>> itself.
>
> I see (and agree). This is in-line with my ideas. However I don't see that
> shining through in your diagram. In your diagram you have the "Data Store"
> and then you have the "OS Primary Database" (they are joined via an arrow
> labeled "Map" but it's hard to understand what you mean).

OS Primary Database refers to the relational database that currently
powers OS.org. The "Map" is because in loading from a DataStore (with
CSVs in, say, "simple data format") we need to apply an "mapping" to
put in the OLAP structure (measures, dimensions, from, to, time etc)

> Then in Part IIa "to OpenSpending.org" you talk about map creation and "do
> import" which for me is what we already do. Do you mean that it would just
> be a frontend for data.openspending.org? That's how I envision things as
> well. I see openspending.org function more like a satellite site, connecting
> data.openspending.org (db write) and perhaps api.openspending.org (db read).

I think the key point here was that the primary "DataStore" would
*not* be the current relational DB with its OLAP structure. Instead it
could be e.g. flat files with some structured metadata. This data
could then go several places:

- Into OLAP db and then displayed on OS.org
- Into OLAP db and then via API to elsewhere
- Pulled down by others into whatever analytical tool they want ...
(see the why section of this issue
https://github.com/openspending/thingstodo/issues/4)
- Pull out info for my own viz that does not run off the API

Rufus