[open-government] Sunlight Foundation ID's $1.3T in "broken" federal spending data

Wed Sep 8 20:44:15 UTC 2010

I think that perhaps I invited some trouble by adopting an overly diplomatic
position in that sentence.  Personally, I think the advocacy community
should have worked on this problem more intensely once it became common
knowledge among some that problems existed.

However, I do think that the data is not as accessible as you're implying.
 FAADS and FAADS PLUS are distinct reporting systems; FAADS' availability
doesn't actually speak to the data on USASpending.gov (there are different
reporting requirements and timelines, for one thing).  FAADS PLUS records
can now be retrieved from USASpending.gov, but this capability only came
online with the site relaunch earlier this year (and in fact we received the
data prior to this, by dealing directly with the vendor who runs the site
and shipping hard drives to Maryland). Prior to that relaunch an API
existed, but was designed with aggregate queries in mind.  It wasn't
suitable for the kind of granular data quality analysis that we conducted.

The CFDA is now available as a CSV data dump, but prior to this year was
released only as a PDF.  Even with the improved format, the obligation
figures still are expressed as prose, and required some technical work to
extract as quantities.

FPDS-NG wasn't a part of our analysis, so I won't speak to it, but similar
concerns apply.

The point being: although some of this information was arguably available,
there were substantial barriers to obtaining it.  I still wish that these
data quality problems had been pursued a bit more doggedly in the past --
particularly by organizations like GAO and OMB that clearly had the data on
hand.

Tom

2010/9/8 Pranesh Prakash <pranesh at cis-india.org>

> Dear Steven and Gabriela,
>
>
> On Wednesday 08 September 2010 11:04 PM, Steven Clift wrote:
>
>> There's also a larger point about this analysis: it's not news -- not
>> really (especially to all of you).  Yet, instead of fixing those
>> problems, those responsible for USASpending.gov decided to build new
>> websites on top of the same lousy data, collecting superficial
>> plaudits from advocates who were excited about the apparent potential
>> of the site, but didn't have sufficiently direct access to the data to
>> see its flaws.
>>
>
> I don't quite understand this paragraph.
>
> 1. As per the information provided on the Clearspending website, the FAADS
> dataset as well as the FAADS-PLUS dataset are openly available for free
> download.
>
> Quoting from the background page on Clearspending[1] :
>
>> All FAADS reports are freely available to interested parties as sequential
>> text files that can be downloaded directly from the FAADS website.
>>
> and
>
>> And the new information contained in FAADS is not available to the public
>> until several months after the close of each fiscal quarter, while the new
>> information contained in FAADS-PLUS is made available to the public on a
>> daily basis.
>>
>
> 2. From what I can infer from that same page, CFDA and FPDS-NG are also
> available on their own websites.
>
> 3. As the Clearspending website notes, the government itself has noted the
> inaccuracies:
>
>> The GAO specifically notes that “OMB has not implemented a process to
>> identify nonreporting agencies as originally planned and instead has relied
>> on agencies’ voluntary compliance with OMB guidance to ensure complete and
>> accurate reporting.
>>
>
> Given all this, who "didn't have sufficiently direct access to the data"?
>  I would argue that this is a great case for showcasing the benefits of open
> data.  While I don't feel we disagree, the negative tone of that last bit of
> the paragraph quoted above seems to indicate that we don't quite agree
> either.  Could you please explain your viewpoint further?
>
> As an aside: There is many a similar story in India about inaccuracies in
> data being revealed due to right to information requests -- sometimes the
> same datum collected from different departments ends up being different, and
> at other times it is quite obvious that the data are inaccurate because they
> go against reason, such as the data relating to startling declines in murder
> rates in Mumbai in stark contrast to its burgeoning population.  These
> wouldn't have come to light -- at least not as easily as they have now --
> without the Right to Information Act.  And I believe such inconsistencies
> would be easier to detect if the data were available online.
>
> Regards,
> Pranesh
>
>  [1]: http://sunlightfoundation.com/clearspending/background/
>
>
>
> --
> Pranesh Prakash
> Programme Manager
> Centre for Internet and Society
> W: http://cis-india.org | T: +91 80 40926283
>
>
> _______________________________________________
> open-government mailing list
> open-government at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/open-government
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/open-government/attachments/20100908/1e576bd4/attachment-0001.html>