[openspending-dev] SpenDB progress update

Thu Jan 15 15:07:01 UTC 2015

Hey all,

I wanted to give a brief update on the technical work that I am doing on
the "spendb" branch of OpenSpending:

On Fri, Jan 9, 2015 at 11:21 AM, Friedrich Lindenberg <friedrich at pudo.org>
wrote:
>
> SpenDB, the version I intend to work on, will retain a single core
> application which handles data loading and the web site. The following are
> my priorities for development:
>
> * Replace the current data loader with a more lightweight approach based
> on loadkit.
>

Done. The previous loading system is replaced with an S3-based loader now.
This also inverts part of the loading flow, so data will be imported into
the database before the model is finished (meaning the model is mutable
after loading data, making it much easier to do this confusing part of work
iteratively).

On the minus side, I've had to move the BDP code out of the core package
for now: the BDP source data field name alignment needs to become its own
ETL stage, and I haven't gotten around to that yet. The model converter
also needs to be converted into a more stand-alone part (possibly as part
of a "fiscalmodel" package, see below).

> * Remove Solr as a core dependency, provide per-dataset search via
> Postgres.
>

It hasn't really bothered me too much yet, so it's still there.

> * Move logging to flat files on S3 (logs make up 40% of the production DB
> right now).
>

This is done, but a bit ugly since S3 doesn't seem to do streaming write
(so you log locally and then upload at the very end).

> * Migrate to BootStrap3, make the site responsive and nice
>

It's running on bootstrap 3 now, but could really use a face lift. Does OKF
have an in-house designer? Is there any way we could set up a form of
barter or so to get some of their time?

> * Rebuild the dataset importer frontend in AngularJS and make a nicer
> model editor.
>

In progress. I've migrated OS to run off Angular (locale switcher, cookie
thingie) and you can create datasets via Angular. Next up, editing the
metadata and team. Then the model editor. Still looking for good examples
on this.

To do this, I've started mocking up bits of an API v3. You can see an
overview of the endpoints which are needed to replace the old dataset
editor here:

https://github.com/openspending/spendb/blob/master/doc/api_v3.rst

> * If time permits, vis builder in Angular.
>

Not started yet.

In the process of doing this, I've also made a few other changes:

* Moved osvalidate into openspending core. I want to separate out more of
the model part into a separate package later ("fiscalmodel"?), but for now
it's easier to work like this instead of maintaining correlating branches.

* Dropped the source and log_record domain objects (both live on S3 now).

* Disabled tests on the dataset manager view for the moment, it's broken
and will be migrated into the API v3 calls.

* I've removed BDP loading support from the API v2.

https://twitter.com/ananelson/status/445705093199826944

Cheers & any comments welcome,

- Friedrich
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/openspending-dev/attachments/20150115/9fa93058/attachment.html>