[openspending-dev] Experiment: flat-file aggregator "API"

Wed Apr 29 14:18:59 UTC 2015

On 29 April 2015 at 14:22, Friedrich Lindenberg <friedrich at pudo.org> wrote:

> Hey all,
>
> I just wanted to send out a quick note to people on this list about a
> little hack I did to experiment with flat-file aggregates. The idea is to
> move OffenerHaushalt off the OpenSpending API, as the platform is currently
> unmaintained.
>

Re unmaintained. That isn't what has been said. The service and site is
maintained and is monitored by sysadmin. As set out in the Roadmap we are
not prioritising fixes to the loading system as we want to move to the new,
agreed, setup.

In any case, delighted you are looking at flat-file stuff and could this be
contributed as part of the Roadmap for OpenSpending "v2" platform.

> There's a prototype for this idea that I hacked up under
> https://github.com/mapthemoney/cubepress - it would basically load a
> given CSV or Excel file into an (in-memory) database and then permute
> through all possible combinations of the API endpoint which might be
> accessed by a given set of visualisations, storing the result to name-coded
> JSON files.
>

Sounds interesting. Again, would it be worth seeing if this can align or be
part of the aggregation system for OS v2. Early stage outline at:

http://labs.openspending.org/osep/osep-07.html

> This is based on a schema file, which includes an ultra-lightweight
> version of an OpenSpending mapping and model:
>
> https://github.com/mapthemoney/cubepress/blob/master/test_data/awards.yaml
>

I assume you've seen work on http://labs.openspending.org/osep/osep-04.html
including the recent pull request.

This YAML model would be complimentary to the visualisation specs that I've
> been using to manage the OffenerHaushalt datasets until now:
>
> https://github.com/okfde/offenerhaushalt.de/blob/master/sites
>
> The success of this approach is probably going to be very varied: some
> datasets would get by with only a few thousands of permutations, but the
> larger ones (like the German federal budget) will explode and yield
> millions or billions of permutations. This will probably fail with issues
> as basic as file system inodes.
>

Interesting to see how this goes then.

> So, in all, I'm not sure it's a good idea, a proper API still seems like
> the way to go (so I've also spent some time making a light-weight version
> of that, but more on that later).
>
> The tool isn't using cubes at the moment, mostly because I was on a train
> while writing it and couldn't download any dependencies. Which is somewhat
> nice, since it doesn't really have dependencies beyond messytables and
> sqlalchemy.
>
> Would love to hear what other operators of OpenSpending satellite sites
> think :)
>

One blunt question: are you basically intent on forking here are you doing
this as part of a way to continue to contribute to OpenSpending "v2"?

Regards,

Rufus

>
> - Friedrich
>
>
> _______________________________________________
> openspending-dev mailing list
> openspending-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/openspending-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/openspending-dev
>
>

-- 

*Rufus PollockFounder and President | skype: rufuspollock | @rufuspollock
<https://twitter.com/rufuspollock>Open Knowledge <http://okfn.org/> - see
how data can change the world**http://okfn.org/ <http://okfn.org/> | @okfn
<http://twitter.com/OKFN> | Open Knowledge on Facebook
<https://www.facebook.com/OKFNetwork> |  Blog <http://blog.okfn.org/>*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/openspending-dev/attachments/20150429/08db5c51/attachment-0002.html>