[openspending-dev] Table widget for Search and Aggregates

Mon Mar 5 16:09:50 UTC 2012

Hey Nick & al,

thanks for this write-up, very happy about the perspective of getting tables kicked off properly. As mentioned before, I though that I may be able to contribute with a v2 search API. Again, the goal here is:

* Drive the next version of the embeddable DGU widget (presumably with data tables in the front end).  
* Unify the representations of entities that we hand out (the search v1 will give out flattened entries which are different from the ones in the DB).
* Eventually kill the browser in the front- and backend by replacing it with JS widgets.  

To get this started, I'd propose creating the following basic API:

/api/2/search?q=solrQuery&limit=100&page=1&filter=other:selection&dataset=cra&callback=json

Questions about this:  

* page or offset? limit or size?
* filter or fq (solr's name)
* dataset is just a filter, do we want it? I like it for being simple.

This would return a JSON object like:

query: { … GET params state… },
num_entries: NN,
entries: [
{ … full repr .. },
]

This would just serve to get things started, we'd still need (stats) facets in the medium term.

- Friedrich   

On Monday, March 5, 2012 at 3:53 PM, Nick Stenning wrote:

> Hi Anna and others,
>  
> Thanks for the contributions. For the time being, I've experimented enough with jQuery DataTables [http://datatables.net/] to establish that it's going to be fine for our usage in OpenSpending.
>  
> Slickgrid does seem nice (and enormous), but the requirement for "infinite scroll" over millions of entries seems a bit artificial (what's wrong with pagination?) so I'm not too worried about this. DataTables has many nice features including multisorting, "lambda columns" (as Friedrich has coined them), and is easy to tie into a serverside search API. It's only major downside as far as I can see is an aesthetic one: the original author wrote it using a variant of Hungarian notation. But I've trained my nausea response and realised that it's actually a mature, well-tested piece of software.
>  
> I'll update the list when we get this properly integrated into OS.
>  
> Best,
> N
>  
> On 1 Mar 2012, at 19:07, Anna Powell-Smith <annapowellsmith at gmail.com (mailto:annapowellsmith at gmail.com)> wrote:
>  
> > I don't know all your requirements, but SlickGrid is worth another look if you want to be able to present and filter hundreds of thousands of rows, because as far as I know it's the only plugin that can currently do that: http://stackoverflow.com/questions/2402953/javascript-data-grid-for-millions-of-rows  
> >  
> > On the downside, it's feature-heavy, as you say, has its own idea of what a data model should be, and the documentation is minimal.  
> >  
> > On 1 March 2012 16:26, Friedrich Lindenberg <friedrich.lindenberg at okfn.org (mailto:friedrich.lindenberg at okfn.org)> wrote:
> > > Just a quick update on this: it seems like Nick is reasonably happy with the way datatables works and can be intercepted to hook in other backends, so we'll do a spike on getting this to run on top of the aggregator API in a clean way (i.e. unlike the current datatables plugin). Having gone through some of the feature ideas with Nick, I think this is going to work OK and will be the quickest way for us to provide a stable widget.  
> > >  
> > > I'll also try to do a strawman draft of the search v2 API soon so that we can build a newer browser for this, too.   
> > >  
> > > - Friedrich  
> > >  
> > > On Wednesday, February 29, 2012 at 10:20 AM, Rufus Pollock wrote:
> > >  
> > > > On 20 February 2012 21:09, Friedrich Lindenberg
> > > > <friedrich.lindenberg at okfn.org (mailto:friedrich.lindenberg at okfn.org)> wrote:
> > > > > Hi all,
> > > > >  
> > > >  
> > > >  
> > > > Sorry to not get to this earlier :-)
> > > >  
> > > > > I'm still undecided on which way to go with the new data tables in
> > > > > OpenSpending. To recap the scope, they are first going to be used for
> > > > > an embeddable search widget and then later also for displaying
> > > > > aggregates (i.e. replace datatables). There are a couple of options
> > > > > for this:
> > > > >  
> > > > > 1) Use recline. Its nice because it already does a lot of what we need
> > > > > and since its developed "in-house" we can influence its further
> > > > > development to do more custom bits. But, to be honest, I'm also
> > > > > feature-creeped out by it: going through the list of things that it is
> > > > > supposed to do now or in the future, it's clear that we will have to
> > > > > hack it very seriously to keep it down to match its task. (And, to be
> > > > >  
> > > >  
> > > >  
> > > > Is that really a true statement. Right now you can just grab model.js
> > > > plus view-grid.js [1] and you are go.
> > > >  
> > > > [1]: https://github.com/okfn/recline/blob/master/src/view-grid.js  
> > > >  
> > > > > a bit brutal, that it will not be able to do all these things well
> > > > > with the resources we have and that it therefore will remain an
> > > > > experiment.) This is already becoming an issue in debates about it
> > > > > including multiple views or a generalized query format.
> > > > >  
> > > >  
> > > >  
> > > > I think it is our currently our best bet and more advanced than any
> > > > other option. It is also pretty easy to add to.
> > > >  
> > > > > 2) Use slickgrid, jqgrid or datatables. To be honest, I'm not too
> > > > > excited about any of them - some (datatables) are trying to keep it
> > > > > simple and then only accept APIs in a very specific format, while
> > > > > others (slickgrid) are incredibly complex and try to boot what feels
> > > > > like a few copies of excel in your browser. I haven't seen a nice
> > > > > trade-off yet. On the other hand, you're likely to get mature code and
> > > > > lots of configuration options.
> > > > >  
> > > >  
> > > >  
> > > > We've used slickgrid before and various other options. We specifically
> > > > thought about using Slickgrid for the grid component in Recline and
> > > > decided against, the primary reason being that Backbone already
> > > > provided you with a lot of the proper 'data-oriented' structure that
> > > > Slickgrid provides in a cleaner way.
> > > >  
> > > > > 3) Roll our own, probably a simplified/feature-limited fork of recline
> > > > > or the old spend-browser. Of course it's tempting but also probably a
> > > > > huge waste of resources. The argument here is that we don't want that
> > > > >  
> > > >  
> > > >  
> > > > :-) -- it probably is though I very much understand of having complete control.
> > > >  
> > > > > much from it, but that those things are probably going to be fairly  
> > > > > specific (around query state with facets and embeddability).
> > > > >  
> > > >  
> > > >  
> > > > The issue always is that the dev curves for rolling one's own versus
> > > > using other stuff always are a mirror of each other:
> > > >  
> > > > a) Roll-your-own: easy at the start, and then progressively more costly
> > > > b) Re-use something else: more costly at the start but then relatively cheaper.
> > > >  
> > > > Where these intersect and b) becomes better than a) determines what  
> > > > you should do. Given the innate tendency of coders to underestimate
> > > > the cost of a) and to prefer rolling their own my general rule is that
> > > > you should have *always* made one good faith attempt to build on an
> > > > existing library or solution before you roll your own.
> > > >  
> > > > > I honestly don't know where I stand on this, and I'd appreciate any
> > > > > input. @Rufus: I know you feel strongly about recline, but please put
> > > > > yourself in your OpenSpending shoes for this :)
> > > > >  
> > > >  
> > > >  
> > > > Understood. I wrote the original table browser. Following my
> > > > injunction to reuse I used ajax-solr and the jquery tablesorter. I
> > > > think it worked OK though AJAX solr was probably overkill for what we
> > > > ended up doing.
> > > >  
> > > > The core of my view on this is that you probably do want a proper
> > > > internal structure and you probably want to use Backbone because it
> > > > gives you good structure while remaining lightweight. Once you are
> > > > doing that you are going to at least re-invent the model part of
> > > > Recline. Copying and pasting that is no big deal so that's definitely
> > > > an option (remember its the conceptual components that take most of
> > > > the time ;-) ). Following my point above I'd still definitely
> > > > recommend trying one good faith effort to re-use Recline -- even if
> > > > you decide against you'll have avoided the trap of underestimating a)
> > > > and you'll have a better sense of what you are doing.
> > > >  
> > > > Rufus  
> > >  
> > >  
> > > _______________________________________________
> > > openspending-dev mailing list
> > > openspending-dev at lists.okfn.org (mailto:openspending-dev at lists.okfn.org)
> > > http://lists.okfn.org/mailman/listinfo/openspending-dev
> > >  
> >  
> >  
> >  
> > --  
> > 07812 357037
> > _______________________________________________
> > openspending-dev mailing list
> > openspending-dev at lists.okfn.org (mailto:openspending-dev at lists.okfn.org)
> > http://lists.okfn.org/mailman/listinfo/openspending-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/openspending-dev/attachments/20120305/3cd8c21b/attachment.html>