[openspending-dev] Experiment: flat-file aggregator "API"

Wed Apr 29 13:22:54 UTC 2015

Hey all,

I just wanted to send out a quick note to people on this list about a
little hack I did to experiment with flat-file aggregates. The idea is to
move OffenerHaushalt off the OpenSpending API, as the platform is currently
unmaintained.

There's a prototype for this idea that I hacked up under
https://github.com/mapthemoney/cubepress - it would basically load a given
CSV or Excel file into an (in-memory) database and then permute through all
possible combinations of the API endpoint which might be accessed by a
given set of visualisations, storing the result to name-coded JSON files.

This is based on a schema file, which includes an ultra-lightweight version
of an OpenSpending mapping and model:

https://github.com/mapthemoney/cubepress/blob/master/test_data/awards.yaml

This YAML model would be complimentary to the visualisation specs that I've
been using to manage the OffenerHaushalt datasets until now:

https://github.com/okfde/offenerhaushalt.de/blob/master/sites

The success of this approach is probably going to be very varied: some
datasets would get by with only a few thousands of permutations, but the
larger ones (like the German federal budget) will explode and yield
millions or billions of permutations. This will probably fail with issues
as basic as file system inodes.

So, in all, I'm not sure it's a good idea, a proper API still seems like
the way to go (so I've also spent some time making a light-weight version
of that, but more on that later).

The tool isn't using cubes at the moment, mostly because I was on a train
while writing it and couldn't download any dependencies. Which is somewhat
nice, since it doesn't really have dependencies beyond messytables and
sqlalchemy.

Would love to hear what other operators of OpenSpending satellite sites
think :)

- Friedrich
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/openspending-dev/attachments/20150429/6419ff7d/attachment.html>