[euopendata] A critical look at the Open Data Index methodology

[David Cabo]

 Hi,

 When the Web Foundation's Open Data Index was published last September a number of Open Data activists here in Spain thought that the good score given to Spain was unjustified, as everyone aware of the situation on the ground knows there's no consistent national policy on the matter, and that the critical data sets (health, education, crime, spending…) are published in non-reusable formats, or not at all. We've written an open letter [1] - reproduced below - where we elaborate our criticism of the Index methodology in general, and Spain's score in particular.

 Regards,

/david

PS: The letter is available online at: http://www.access-info.org/es/open-government-data/302-spain-is-a-world-leader-in-open-data-says-who

----

Spain is a world leader in Open Data. Says who?

In September 2012 the Web Foundation published the first edition of its Open Data Index, “a specific set of 14 indicators directly targeted at measuring open data worldwide”. Many open data and transparency activists in Spain were surprised to find Spain in the leading pack, since Spain still doesn't have an access to information law and there is no coherent national Open Data policy or practice. The only actively maintained Open Data initiatives are those started by a few local and regional governments, with no coordination or support from the national level. More importantly, key datasets about health, education, public procurement or official agendas are still being withheld by the administration,with no plan to release them.  

When asked for an explanation and rationale for these results, the Web Foundation responded that these results are "based on perception". In particular, in the perception of one person in Spain, who asked not to be identified, and about whom we know nothing.  

Also, we've been told the questions measure “availability”, not “openness”. If so, the name of the Open Data Index is seriously misleading. When the Spanish government was asked in March 2012 by member of parliament Alberto Garzon to release the national budget in machine-readable format, the official government response said "transparency is about the extent of the information provided, not about formal aspects of presentation" [1]. In spite of this, and of the fact budget execution (spending) data has little detail, procurement data is fragmented across many sources - most often in non-reusable formats -, and of the fact citizens have no access whatsoever to actual invoices, Spain gets a score of 8/10 on spending data [2].

The question about crime data [3] is also particularly interesting. The Spanish government promised in its Open Government Partnership action plan to release the data in April 2012 [4], but once published it fell short compared with the detailed information available in other countries: only a summary of provincial level crime figures is made available once per quarter, in PDF [5]. According to the unidentified expert who contributed to the Open Data Index, Spain deserves a score of 10/10 for availability of crime data.  

Reviewing all the index scores falls beyond the scope of this open letter, but similar arguments could be made for the health or education datasets. Because of this, we call on the Web Foundation to:

 * Review the Open Data Index score for Spain.

 * Identify the expert who carried out the analysis for Spain.

 * Revise its methodology to:
   - Move away from perception-based scoring of one or two experts to results which are subject to fact-checking and empirical verification;
   - Base the results on a full open data standards which includes that the data is made available in machine-readable, open formats, is comprehensive, raw data and is regularly updated in a timely manner;
   - Provide links to all data sets and other sources used in the Index so that others can review and assess the scores.  

Signed:

Victoria Anderica, Access Info Europe
David Cabo, Civio Foundation
Javier de la Cueva, Lawyer
Helen Darbishire, Access Info Europe
Jacobo Elosua, Civio Foundation
José Luis Marin, EuroAlert

[1]: “En todo caso, el concepto de transparencia se refiere a la suficiencia de información suministrada y no al mero aspecto formal de presentación de datos.” http://www.agarzon.net/?p=1758
[2]: Question Q23b - “To what extent are there Government data on the Web in the following areas: detailed data on budgeted and actual spending on different departments”
[3]: Question Q23j - “To what extent are there Government data on the Web in the following areas: data and statistics on crime”
[4]: http://www.access-info.org/es/coalicion-pro-acceso/237-spain-in-ogp   
[5]: http://www.interior.gob.es/file/58/58149/58149.pdf


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/euopendata/attachments/20121113/c3f49300/attachment.html>