[openspending-dev] Roadmaps and release schedules

Sat Mar 9 20:46:51 UTC 2013

Hey guys,

fantastic discussion that's starting here, thanks Tryggvi for kicking it
off. It may be hard to believe, but I fully agree with the developing
consensus on most of the issues (Roadmap good, release cycles bad, Jinija2
nice to have, API breakage: let's begin to properly version the whole thing
and move off non-namespaced JSON). The thing I'd speak against is
modularisation.

This has a somewhat non-technical reason: OKFN has infrastructure disease.
The basic assumption is that, given open knowledge and the right
infrastructure, people will go and do amazing things. That's probably true,
but our resulting plan is naive: we'll just lobby for the data and build
the infrastructure. The truth is that OKFN neither has any database
experts, nor does it know very much about data processes (people who have
actually read a Kimball book, raise your hands now). So we generally suck
at making infrastructure, but we love trying (it's kind of sad to watch,
myself included).

The point of the OpenSpending project is to set a counter-point: let's take
a fairly random problem that could be solved with open data - people don't
understand how their government spends money - and try to fix that.

Unless OKFN has decided a major investment into OS since I've left, you
will have fairly scarce resources. The price you can ask for making a
budget vis for government turns out to be close to the amount of effort it
requires to interact with a public institution or NGO. This thing is now
easy enough to use so people don't really depend on us for the basic
service. There isn't really an easy cash cow.

So, you can either take the existing resources and play infrastructure, or
you can stick with the actual problem: explaining government spending to
people. Making OS into modules is a nice technical thing to do. Maybe it'll
even get us a piece of generalised software (it's actually the model
editor, I think, not the cube itself). But I would argue that modularising
OS has a fairly low value with regards to achieving that goal. (There's
going to be a "clean foundations" in the response here, but is shit really
broken? Do you really think that it'll be easier to get contributors if its
ten packages, rather than two? Why not solve the "satellite contributors"
problem first - that's where actual value is added most easily.)

Figuring out what the next step for OS is in terms of fulfilling its
mission is a) much harder than building an OLAP cube and b) an OKFNy
problem. If you want to introduce better development practices, thats where
I'd start: measurement. OS has some KPI, but they're shit. Traffic is
horrible. Let's make like a startup, think about the customers (are these
the people paying you or the people using the site?). What if the next step
in the solution turns out to not actually require an OLAP cube, just
sending some nice email to people?

Anyway, just my $.02,

 - Friedrich

On Sat, Mar 9, 2013 at 12:54 PM, Tryggvi Björgvinsson <
tryggvi.bjorgvinsson at okfn.org> wrote:

> Þann fös 8.mar 2013 16:46, skrifaði Vitor Baptista:
> > I feel the trend in the webdev world is to deploy as often as possible,
> > getting to extremes like GitHub (
> > https://github.com/blog/1241-deploying-at-github) doing dozens of
> deploys
> > per day. In 2012, they did 175 deploys in a single day!
>
> Hi,
>
> Alright, seems like we shouldn't put up a rigid release schedule (which
> is just fine, it was only a suggestion to get the conversation started).
> I can live with a pure rolling release (deploy when ready). I think it
> can be done bu we must take special care (to quote Spiderman -- not
> Voltaire: With great power there must also come -- great
> responsibility!). These considerations come down to the emails I
> referred to in my last email. So I'll lay them out (and start each
> thread individually when I've given it some thought based on a pure
> rolling release - you can start them as well if you feel like it). I'll
> enumerate them as I go along... hold on tight, this is going to be long.
>
> The way OpenSpending works is that we have a central service
> (openspending.org) and then a bunch of satellite sites (which we may or
> may not know about) that rely on that service to work correctly. It
> should be our absolute priority (imo) not to break those sites, these
> are the people who want to use OpenSpending (governments do not have the
> same patience with broken software as hackers do).
>
> If we deploy via a pure rolling release we need to take good care of
> those sites and never ever break things unless we give them a heads up
> and some time to get ready (and a migration time frame). This is
> something the 6 month release cycle solves by giving maintainers of
> those sites two dates (every 6 months) when they can see if there site
> still works).
>
> As far as I know we don't have a continuous integration server but we
> should (we could do it on the staging server I think but iirc there are
> some monitoring services we're using which we might be able to use). We
> need it if we're continuously deploying (even if we're a small community
> - I'd like us to be able to grow without breaking our infrastructure).
> We need to do a lot of testing, especially regression testing. So this
> brings me to one of the emails:
>
> 1. Test framework / Continuous integration
>
> We have to build a rock solid test framework/suite for OpenSpending.
> Every new feature should come with a test that cannot be removed unless
> the community has had time to prepare / respond. All tests should pass
> before we can deploy.
>
> We will all have to agree on a workflow (and we have to document it),
> which is one of the emails:
>
> 2. Work flow of OpenSpending
>
> Vitor actually already started discussing this and has some ideas:
>
> > The next steps, now that we have a staging site
> > (thanks, Nigel!), could be to work on our deployment process. Configure a
> > deployment tool (maybe Fabric?), configure Travis (or Jenkins), and set
> up
> > a way to deploy, automatically, every commit to our staging site, if the
> > tests pass.
>
> There are other concerns like how do we deal with pull requests, how do
> things get merged? Should we make it a requirement that all merges have
> to be vouched by one/two other devs before they can be committed? We
> need to check if new features come with tests. How do we know everybody
> agrees on where OpenSpending is heading? We could have monthly dev
> meetings, with a community built agenda, on IRC which are logged.
>
> I'm still keen on having a roadmap (an idea of where we are going) and
> just as Nigel said we can have a roadmap even if we have a pure rolling
> release, we can update it whenever (e.g. on the monthly meetings if we
> have those). At least it gives us a good reason to celebrate when we
> reach our goals -- I really like celebrations ;-)
>
> Our roadmaps only need to map some bigger feature which take a lot more
> effort to implement. For example we have to decide what we want to do
> with Pylons. It's not in active development since the core pylon group
> started focusing on Pyramid. We need to move away from Pylons, maybe go
> to Pyramid (logical step from Pylons), maybe go to Flask (which has been
> suggested). I'm a fan of asynchronous frameworks (Twisted/Tornado) but
> that would be a massive change so I don't think we should go there).
> Whatever we agree on this is a rather big change which we all need to be
> aware of and we need to prepare our code. These can be big and small
> changes.
>
> Nigel and Martin have already done some work on a move from Genshi to
> Jinja2 as our templating engine. This doesn't have to _the_ templating
> engine but jinja2 is quite widely supported (and easier to maintain imo
> than Genshi). I've also heard some ideas to move to pystache (the python
> port of mustache) to avoid logic in the template. I think the template
> discussion can help us figure out how we deal with a medium sized change
> that everybody need to agree on, giving us another of the emails:
>
> 3. Template engine
>
> It's a smaller discussion and I know this can lead to bikeshedding (if
> what we're figuring out is how we can agree on moderate effort changes).
> Another, possibly slightly larger discussion would be:
>
> 4. Web framework
>
> What web framework should we use instead of Pylons? We might have to
> discuss this before (or parallel to) the template engine discussion.
> These are discussions we need to have very soon and they can be use
> cases for how we build up a roadmap (or if we build up a roadmap).
>
> Another even larger discussion would be about modularisation. This is
> probably one of the most important discussions we need to have (so it's
> a bad idea to have put it last in such a long email - but here's to
> hoping that people at least skim through the enumerated parts):
>
> 5. Modularisation
>
> There might actually be different paces to different parts of
> OpenSpending and I think the code base is too large to be useful (for
> maintenance, for contributions and just software engineering principles
> in general). It is doing too much at the same time. We have our
> analytical machine that powers the satellite sites and then we have some
> visualisation (demos - not a core part but a requirement). If I'm an
> OLAP specialist and I want to hack on that part (which I think is the
> core part of the OpenSpending platform) I still have to install the
> complete package with Solr, the template engine and whatnot. I think
> that this might drive contributors away, it makes maintenance more
> difficult, it might result in forks, and it makes testing more difficult.
>
> This is a huge change which requires quite a big effort (and therefore
> justifies to be on a roadmap since we have to take baby steps in that
> direction). It can involve breaking OpenSpending into different projects
> (e.g. make OpenSpending.org a satellite site on top of the spending
> cube) and it can involve smaller changes like breaking big python
> modules into more specific ones (with a clearer focus). An example of
> this is Martin and Michael's work on the data store. They've created an
> experimental branch where they split the data store out from the model
> (cube/dataset separation).
>
> This discussion would basically be about the future of the code base and
> imo requires a roadmap and a lot of planning.
>
> Alright, some things off my chest. I hope you made it this far. Feel
> free to attack my opinions or add yours. I can change my mind based on
> good arguments and I never take things personally so you don't really
> have to be polite but that would still be good netiquette on a public
> mailing list :-)
>
> /Tryggvi
>
>
>
> _______________________________________________
> openspending-dev mailing list
> openspending-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/openspending-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/openspending-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/openspending-dev/attachments/20130309/3a8a1335/attachment.html>