[euopendata] A critical look at the Open Data Index methodology

Thu Feb 21 08:30:34 UTC 2013

Hi Paola,

WF issued a public response in December:
http://www.webfoundation.org/2012/12/user-feedback-on-the-open-data-index/

We have collected and analyzed all the feedback so far and are working on an improved methodology for the 2013 Web Index (and Open Data Index)

Cheers,
Josema.

El 20/02/2013, a las 14:17, Paola Di Maio escribió:
> David, Josema and all
> just following up on an earlier conversation
> (cant believe it was in November!!)
> any updates on this methodology, and how can we make this index a bit more
> 'open science' ?  I am forwarding to the public lists the correspondence trail
> Look forward to your reply
> PDM
> 
> ---------- Forwarded message ----------
> From: Paola Di Maio <paola.dimaio at gmail.com>
> Date: Wed, Nov 14, 2012 at 5:06 PM
> Subject: Re: A critical look at the Open Data Index methodology
> To: David Cabo <david.cabo at gmail.com>
> Cc: "Jose M. Alonso" <josema at webfoundation.org>
> 
> 
> Thank you David
> 
> Interested  in the issue from more than one angle-
> among others, this could be a  good case of #openscience 
> 
> re point 2)
> if that's the case, then it is obvious where the flaw of the methodology is, and it can be easily fixed, by a sufficient (representative) number of scorer and reviewers and then perform a meta-analysis of all the reviews
> 
> as with all cognitive science, people opinions vary depend on what knowledge they hold. assume someone is really ignorant of some fact/issue, then their opinion is likely to be falst/biased
> 
> we do not need to know the identity of the scorer/reviewr, but yes it would be nice to see what data they have analysed .
> would another scorer/reviewer with a different profile come up with a different outcome?
> 
> how was the scorer/reviewers selected? based on what criteria?
> were they paid? were they volunteers?
> 
> these are all issues which may impact the outcome of the study
> 
> Happy to have this portion of the conversation posted to list, I agree the conversation should be in the public domain
> 
> PDM
> 
> 
> On Wed, Nov 14, 2012 at 11:26 AM, David Cabo <david.cabo at gmail.com> wrote:
>  Hi Paola,
> 
>  I'll let Josema reply to your questions fully, but regarding point 2), that's the response I got when I asked the Web Foundation about the scores through team-webindex at webfoundation.org. Unfortunately because that's an email address, not a public forum, I can't link to it, but I'll quote:
> 
> > This is a perception-based index, i.e. the questions are answered by experts in each country and then peer-reviewed. Different people have different opinions. The full methodology is available as part of the Web Index report [1]. We cannot always disclose the name of scorers and reviewers. We offered all of them to be acknowledged but some preferred not to be known or have his name published, so we need to keep this information confidential. This is the case of the person who scored Spain. It's not the same for the reviewer so I could contact that person separately and ask him if he wouldn't mind to be put in contact with you.
> [1] http://thewebindex.org/2012/10/2012-Web-Index-Key-Findings.pdf
>  
>  I think it's important we continue this conversation in public, so others can follow. I just found out, btw, that there are similar concerns in Mexico: https://twitter.com/oso/status/268451120521826304
> 
>  Regards,
> 
> /david
> 
> On Wednesday, November 14, 2012 at 9:51 AM, Paola Di Maio wrote:
>> Dear Josema
>> 
>> I receive the statement below and very interested to learn more
>> about the issue raised (working on Open Data indexing myself!)
>> 
>> Can you please share information about 
>> 1. what data was analysed (for each country)
>> 2. why does the post say the result is based on perception of only one person
>> (as this does not seem clear in the methodology)
>> 3. Is there a proper methodology paper/document that I can use how were the results obtained?
>> 
>> Thank you
>> Best
>> 
>> PDM
>> 
>> 
>> ---------- Forwarded message ----------
>> From: David Cabo <david.cabo at gmail.com>
>> Date: Tue, Nov 13, 2012 at 7:01 PM
>> Subject: [euopendata] A critical look at the Open Data Index methodology
>> To: EU Open Data Working Group <euopendata at lists.okfn.org>
>> 
>> 
>>  Hi,
>> 
>>  When the Web Foundation's Open Data Index was published last September a number of Open Data activists here in Spain thought that the good score given to Spain was unjustified, as everyone aware of the situation on the ground knows there's no consistent national policy on the matter, and that the critical data sets (health, education, crime, spending…) are published in non-reusable formats, or not at all. We've written an open letter [1] - reproduced below - where we elaborate our criticism of the Index methodology in general, and Spain's score in particular.
>> 
>>  Regards,
>> 
>> /david
>> 
>> PS: The letter is available online at: http://www.access-info.org/es/open-government-data/302-spain-is-a-world-leader-in-open-data-says-who
>> 
>> ----
>> 
>> Spain is a world leader in Open Data. Says who?
>> 
>> In September 2012 the Web Foundation published the first edition of its Open Data Index, “a specific set of 14 indicators directly targeted at measuring open data worldwide”. Many open data and transparency activists in Spain were surprised to find Spain in the leading pack, since Spain still doesn't have an access to information law and there is no coherent national Open Data policy or practice. The only actively maintained Open Data initiatives are those started by a few local and regional governments, with no coordination or support from the national level. More importantly, key datasets about health, education, public procurement or official agendas are still being withheld by the administration,with no plan to release them. 
>> 
>> When asked for an explanation and rationale for these results, the Web Foundation responded that these results are "based on perception". In particular, in the perception of one person in Spain, who asked not to be identified, and about whom we know nothing. 
>> 
>> Also, we've been told the questions measure “availability”, not “openness”. If so, the name of the Open Data Index is seriously misleading. When the Spanish government was asked in March 2012 by member of parliament Alberto Garzon to release the national budget in machine-readable format, the official government response said "transparency is about the extent of the information provided, not about formal aspects of presentation" [1]. In spite of this, and of the fact budget execution (spending) data has little detail, procurement data is fragmented across many sources - most often in non-reusable formats -, and of the fact citizens have no access whatsoever to actual invoices, Spain gets a score of 8/10 on spending data [2].
>> 
>> The question about crime data [3] is also particularly interesting. The Spanish government promised in its Open Government Partnership action plan to release the data in April 2012 [4], but once published it fell short compared with the detailed information available in other countries: only a summary of provincial level crime figures is made available once per quarter, in PDF [5]. According to the unidentified expert who contributed to the Open Data Index, Spain deserves a score of 10/10 for availability of crime data. 
>> 
>> Reviewing all the index scores falls beyond the scope of this open letter, but similar arguments could be made for the health or education datasets. Because of this, we call on the Web Foundation to:
>> 
>>  * Review the Open Data Index score for Spain.
>> 
>>  * Identify the expert who carried out the analysis for Spain.
>> 
>>  * Revise its methodology to:
>>    - Move away from perception-based scoring of one or two experts to results which are subject to fact-checking and empirical verification;
>>    - Base the results on a full open data standards which includes that the data is made available in machine-readable, open formats, is comprehensive, raw data and is regularly updated in a timely manner;
>>    - Provide links to all data sets and other sources used in the Index so that others can review and assess the scores. 
>> 
>> Signed:
>> 
>> Victoria Anderica, Access Info Europe
>> David Cabo, Civio Foundation
>> Javier de la Cueva, Lawyer
>> Helen Darbishire, Access Info Europe
>> Jacobo Elosua, Civio Foundation
>> José Luis Marin, EuroAlert
>> 
>> [1]: “En todo caso, el concepto de transparencia se refiere a la suficiencia de información suministrada y no al mero aspecto formal de presentación de datos.” http://www.agarzon.net/?p=1758
>> [2]: Question Q23b - “To what extent are there Government data on the Web in the following areas: detailed data on budgeted and actual spending on different departments”
>> [3]: Question Q23j - “To what extent are there Government data on the Web in the following areas: data and statistics on crime”
>> [4]: http://www.access-info.org/es/coalicion-pro-acceso/237-spain-in-ogp  
>> [5]: http://www.interior.gob.es/file/58/58149/58149.pdf
>> 
>> 
>> _______________________________________________
>> euopendata mailing list
>> euopendata at lists.okfn.org
>> http://lists.okfn.org/mailman/listinfo/euopendata
>> Unsubscribe: http://lists.okfn.org/mailman/options/euopendata
>> 
>> 
> 
> 
> 
> _______________________________________________
> euopendata mailing list
> euopendata at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/euopendata
> Unsubscribe: http://lists.okfn.org/mailman/options/euopendata

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/euopendata/attachments/20130221/a02296b9/attachment-0001.html>