[openspending-dev] OpenSpending Uploader

Sun Dec 22 15:21:37 UTC 2013

Hey guys,

I was playing with the OS uploader a bit on Friday, and the thing just
struck me as very odd. I have two main criticisms that I quickly wanted to
submit here. I'm aware of the relevant GitHub issues, but IMO this deserves
some proper discussion.

1) The primary goal of the uploader is to allow normal users to submit
source data to OpenSpending when they're trying to load a dataset. I fail
to see how the UX of this uploader can be considered superior to an
integrated uploader (also known as: a button and a one-field form). It
seems like UX has taken a very clear back seat to architecture here.

At the same time, I would say that its architectural merits are debatable.
The uploader doesn't seem like SOA to me, it's just plucking out a
user-facing feature into a separate code base in a hipper programming
language, producing lots of unnecessary integration requirements. In other
words, it's technical debt.

2) The secondary design goal of the uploader is to solidify the handling of
source data in OpenSpending. It would seem to me that a sensible way of
ensuring control of source data would be to move from only knowing a
source's URL, to forcing the source data to be stored as S3 keys, managed
by OS. That would also mean that data submitted to OS via a URL would get
downloaded before actually being loaded.

Of course, OS core can now be retrofitted to upload URL-based sources via
the uploader, but that essentially means replacing a well-proven S3 API
with the uploader's API, creating so many new problems (authz, sync,
dataset association, ...) it's almost fun to think about. To me, this also
indicates that there is no re-usability to be had with the uploader, unless
you actually want to make it generic enough for it to be indistinguishable
from the S3 API.

I would really ask you guys to reconsider your design decision here. I've
shown this to people not involved in OS and they though I was kidding.

This is not meant to re-ignite the debate over whether OS should be more
modular, let's keep it to this use case. In this case, you're taking it out
on a) the OS users and b) the quality of the data that gets collected.
These things, I would argue, are more important than a form of modularity
that has no real benefits.

</rant>, Merry Christmas!

- Friedrich
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.okfn.org/pipermail/openspending-dev/attachments/20131222/455de895/attachment.html>