[openspending-dev] Porting OpenSpending to Flask

Mon Dec 22 12:06:46 UTC 2014

Hey Friedrich,

This is really great and something I have wanted to do for a long time
(move away from pylons)! Thank you!

I've been hatching the micro-services plan (which Rufus hinted at), in
my head for a long time and after our last developer meeting, Rufus
Pollock, Paul Walsh and I sat down to combine and align our thoughts and
ideas so that we could share them with the list for a better and more
thorough discussion (my initial thought was we could do it at the next
developer meeting).

So, your email comes at a great time. We can just kickstart the
discussion now to align all of us and see how what Rufus, Paul and I
have been thinking about resonates with the rest of the list. To keep
things clean and because this is important, I'll describe what we're
thinking and our approach in a separate thread but I just wanted to say
that it's probably not time well spent to begin with the flask port
before we've agreed on the overall approach and even started moving some
of the pieces to micro-services because this is going to change the
system quite a lot.

More comments in-line.

On sun 21.des 2014 22:42, Friedrich Lindenberg wrote:
> I agree that it's be cool to split up OS into micro-services, and I'd
> say that only gets easier once we have everything divided into blueprints.

Yes. I agree. The idea with the micro-services is to "unixify"
OpenSpending: Make independent things (where each one does only a few
things) work well together and to do them well and then allow people to
tap into OpenSpending at different end-points. If you want the raw data
you can access that, but you can also get analysis results for standard
aggregation queries you can, etc.

> As for OSEP2, I want to focus on the bits that generate end-user
> value. My interpretation is that raw data storage would be useful, but
> less so than simple file upload. They may end up being the same thing,
> but with different attitudes :)

I agree with you there as well. The raw data storage is still very
important. Currently OpenSpending does not save the raw data but
downloads it, imports it and then forgets about it. This means we cannot
rebuild things if we notice a bug in the import or half the earth gets
attacked by aliens and we lose all our backups (because if half the
world gets destroyed, there will still be OpenSpending fanatics who want
to continue to use our awesome software).

I think the interface should still be a simple file upload even if in
the backend we're storing a copy of the data, but I think we need to
focus on standardized input (more about that later) for the future
because it's not really helpful from a global user base point of view to
have non-standard data in OpenSpending.

> If I had to guess what the most pressing challenge is for the
> platform, I would go with domain-specific metadata. OS has apparently
> got 2000 datasets now (massive jump, what happened?) - but it's near
> impossible to find out which areas are covered, which datasets are
> current and how one would update those that aren't.

Yes, I really liked your idea of browsing the data from a metadata
perspective instead of titles. I actually think that would be the way to
go but I still have to digest and have a think about it some more. Would
you be willing to elaborate a little (e.g. with mockups) what you're
thinking?

> More than that, the OS home page does a horrible job linking out to
> the cool satellite sites like Spending.jp, WDMMG, budzeti.ba
> <http://budzeti.ba>, CameroonBudget or OffenerHaushalt. These
> shouldn't just be mentioned in random blog post, but featured in the
> main system when people browse for budget data.

Oh yes. I agree with you there as well. It's a real shame we don't have
some sort of a "call home" feature and a registry of who's using
OpenSpending datasets (meaning a non-editorial approach).

> Another random comment: reading the OS codebase today, I have to say
> that I haven't learned to love the BDP. At the moment, it's turning
> into a parallel, non-UI loading mechanism, when really the BDP should
> link into the process much more smoothly. One mechanism --
> model/mapping or BDP should be the "truth", and the other one should
> map onto it. I'm really not sure what the best approach is.

So more on the standard input thing. What you're talking about is
exactly what I want. The BDP importer in the code base basically just
converts the budget data package into an OpenSpending model (dimensions
and attributes) at the moment. It's far from how I would like
OpenSpending to treat the BDP, but it's still a more user-friendly way
of loading it than what https://github.com/openspending/biab does (so
it's only focussing on API at the moment).

What we are proposing in our new approach (the micro-services thing) is
that BDP will be "the truth" and others will have to map onto it
(perhaps in their own external services). We might in the transitioning
period have an unstructured importer as an "official" micro-service but
hopefully we can design it in such a way that the end-result will still
be a budget data package.

I know this means we might actually lose information that's stored in
OpenSpending (since some budget data has more information than what the
BDP stores) but I don't think we're really after ALL the data, just the
most important one, and if we do need some more data, we can just
propose that as an addition to the BDP spec or have a separate
micro-service that stores and links to the budget data packages to
provide more context ("official" or maintained by other).

In any case, I believe settling on BDP as "the truth" is important if
the datasets in OpenSpending are supposed to be useful for more people
than only those who are building a visualisation on top of their own
dataset.

I welcome all comment but I'm now going to move over to starting the
other thread (which is an email that will probably take me a while to
write because I want to get it right with such a big suggestion/proposal).

/Tryggvi

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/openspending-dev/attachments/20141222/5f656468/attachment-0002.html>