[euopendata] A critical look at the Open Data Index methodology

Wed Nov 14 08:31:51 UTC 2012

Dear All 

This is an important issue. We would be really happy to have comments from
other countries and we will post them on our website. 

In parallel there is another debate going on about how we measure “openness”
which you can read here: 

http://www.access-info.org/en/rti-rating/303-measuring-transparency-debate 

And in the meantime we are working on a set of standards for data sets to be
published which will be included in the Open Government Standards initiative
www.opengovstandards.org. Recommendations are welcome on which datasets and
the level of detail needed. 

best, Helen 

_________________________

Helen Darbishire, Executive Director

Access Info Europe,  <http://www.access-info.org/> www.access-info.org 

mobile tel: + 34 667 685 319

Skype: helen_darbishire

Twitter: @Access_Info, @helen_access

From: euopendata-bounces at lists.okfn.org
[mailto:euopendata-bounces at lists.okfn.org] On Behalf Of Paola Di Maio
Sent: Tuesday, November 13, 2012 8:59 PM
To: David Cabo
Cc: EU Open Data Working Group
Subject: Re: [euopendata] A critical look at the Open Data Index methodology

Important!

Thank you for noting, and for sharing.

This should be a good opportunity to refine the methodology.

Is it open? Can it be massaged into something more precise?

One way to build a sample of 'real data'  (even a small onem maybe testing
other countries too, surely if Spain is swayed ,others could be too) and see
what figures comes up

when analyzed, then challenging the outcome from there, and see how the
methodology can be fixed. Take the exercise as 'evaluation' and feed it back
into the work.

I take a quick look here 

http://thewebindex.org/documents/2012-webindex-dataset#Q22

and have some doubts about the indicators and the datasources, possibly not
sufficiently representative or other possible bias

PDM

On Tue, Nov 13, 2012 at 7:01 PM, David Cabo <david.cabo at gmail.com> wrote:

 Hi,

 When the Web Foundation's Open Data Index was published last September a
number of Open Data activists here in Spain thought that the good score
given to Spain was unjustified, as everyone aware of the situation on the
ground knows there's no consistent national policy on the matter, and that
the critical data sets (health, education, crime, spending
) are published
in non-reusable formats, or not at all. We've written an open letter [1] -
reproduced below - where we elaborate our criticism of the Index methodology
in general, and Spain's score in particular.

 Regards,

/david

PS: The letter is available online at:
http://www.access-info.org/es/open-government-data/302-spain-is-a-world-lead
er-in-open-data-says-who

----

Spain is a world leader in Open Data. Says who?

In September 2012 the Web Foundation published the first edition of its Open
Data Index, “a specific set of 14 indicators directly targeted at measuring
open data worldwide”. Many open data and transparency activists in Spain
were surprised to find Spain in the leading pack, since Spain still doesn't
have an access to information law and there is no coherent national Open
Data policy or practice. The only actively maintained Open Data initiatives
are those started by a few local and regional governments, with no
coordination or support from the national level. More importantly, key
datasets about health, education, public procurement or official agendas are
still being withheld by the administration,with no plan to release them. 

When asked for an explanation and rationale for these results, the Web
Foundation responded that these results are "based on perception". In
particular, in the perception of one person in Spain, who asked not to be
identified, and about whom we know nothing. 

Also, we've been told the questions measure “availability”, not “openness”.
If so, the name of the Open Data Index is seriously misleading. When the
Spanish government was asked in March 2012 by member of parliament Alberto
Garzon to release the national budget in machine-readable format, the
official government response said "transparency is about the extent of the
information provided, not about formal aspects of presentation" [1]. In
spite of this, and of the fact budget execution (spending) data has little
detail, procurement data is fragmented across many sources - most often in
non-reusable formats -, and of the fact citizens have no access whatsoever
to actual invoices, Spain gets a score of 8/10 on spending data [2].

The question about crime data [3] is also particularly interesting. The
Spanish government promised in its Open Government Partnership action plan
to release the data in April 2012 [4], but once published it fell short
compared with the detailed information available in other countries: only a
summary of provincial level crime figures is made available once per
quarter, in PDF [5]. According to the unidentified expert who contributed to
the Open Data Index, Spain deserves a score of 10/10 for availability of
crime data. 

Reviewing all the index scores falls beyond the scope of this open letter,
but similar arguments could be made for the health or education datasets.
Because of this, we call on the Web Foundation to:

 * Review the Open Data Index score for Spain.

 * Identify the expert who carried out the analysis for Spain.

 * Revise its methodology to:

   - Move away from perception-based scoring of one or two experts to
results which are subject to fact-checking and empirical verification;

   - Base the results on a full open data standards which includes that the
data is made available in machine-readable, open formats, is comprehensive,
raw data and is regularly updated in a timely manner;

   - Provide links to all data sets and other sources used in the Index so
that others can review and assess the scores. 

Signed:

Victoria Anderica, Access Info Europe

David Cabo, Civio Foundation

Javier de la Cueva, Lawyer

Helen Darbishire, Access Info Europe

Jacobo Elosua, Civio Foundation

José Luis Marin, EuroAlert

[1]: “En todo caso, el concepto de transparencia se refiere a la suficiencia
de información suministrada y no al mero aspecto formal de presentación de
datos.” http://www.agarzon.net/?p=1758

[2]: Question Q23b - “To what extent are there Government data on the Web in
the following areas: detailed data on budgeted and actual spending on
different departments”

[3]: Question Q23j - “To what extent are there Government data on the Web in
the following areas: data and statistics on crime”

[4]: http://www.access-info.org/es/coalicion-pro-acceso/237-spain-in-ogp  

[5]: http://www.interior.gob.es/file/58/58149/58149.pdf

_______________________________________________
euopendata mailing list
euopendata at lists.okfn.org
http://lists.okfn.org/mailman/listinfo/euopendata
Unsubscribe: http://lists.okfn.org/mailman/options/euopendata

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/euopendata/attachments/20121114/c4fe42dd/attachment-0001.html>