[OpenSpending] Help needed

[Tryggvi Björgvinsson]

On mán 3.jún 2013 14:58, Pierre Chrzanowski wrote:
> Hi Tryggvi, just to clarify the process, do you go through the entire
> dataset before validating compatibliity?
> I thought only the first rows were checked and then the entire dataset
> during the upload.

'Test a sample' goes through only the first row. 'Load' goes through the
entire dataset. The problem we're dealing with now is that both of these
are handled by the same "workers" which are all currently working on
'loading' bigger datasets into the database (this blocks those who are
just waiting for a sample or analysis).

The fix I'm thinking about would attack this on two fronts:

1. Split 'loading' away from analysis and samples so that they don't
block others from working with testing their datasets.
2. Limit each dataset to only one 'load' at a time. This might force
uploaders to visit openspending.org more often "just to press a button"
but that will hopefully distribute the loading processes more equally
among all users.

I'm interested in hearing if somebody has anything against the latter
approach before I implement it. This will make openspending.org a just
slightly more cumbersome to use but it will distribute our resources
more equally between all of us :-)

-- 

Tryggvi Björgvinsson

Technical Lead, OpenSpending

The Open Knowledge Foundation <http://okfn.org>

/Empowering through Open Knowledge/

http://okfn.org/ | @okfn <http://twitter.com/OKFN> | OKF on Facebook
<https://facebook.com/OKFNetwork> | Blog <http://blog.okfn.org/> |
Newsletter <http://okfn.org/about/newsletter>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/openspending/attachments/20130603/e89cf65a/attachment.html>