[Open-data-census] Serious inconsistencies in the application of the methodology

Mon Nov 3 11:20:20 UTC 2014

Hi list, I am forwarding a message from Simon Chignard who is concerned
about the lack of quality and consistency in the current submissions.

I think his feedbacks should be carefully taken into account for the
reviewing process.

Best
Pierre

Ps : text below is a Google translate from email wrote in French to okf
france members list

---
Hello all,

I spotted this weekend which seems to me to be serious inconsistencies in
the application of the methodology of the Open Data Index since 2014. I
alert you that the question of the reliability of the tool.

1 / An example: the assessment of open Zipcodes / Postcodes.

Consider the postal code file for Spain, Sweden, Canada and France.

In these four countries, the situation is the same: a more or less public
operator (Correos, Postnummer, Canada Post and La Poste) sells, on demand,
the postal code file.

Yet, these are the scores on the same file:

Zipcode / Canada: 55%
http://global.census.okfn.org/entry/ca/postcodes

Zipcode / Spain: 45%
http://global.census.okfn.org/entry/es/postcodes

Zipcode / France: 10%
http://global.census.okfn.org/entry/fr/postcodes

Zipcode / Sweden: 55%
http://global.census.okfn.org/entry/se/postcodes

2 / What is at issue

The question posed here is that of chaining or independence criteria.

In France we (collectively) have considered that the criteria chained. This
means that if the data is not available then we put red all other criteria.
However, in all other countries I could see they took each criterion
separately. They consider that given legally sold and closed may still be
available online, be current, be downloaded in bulk, etc ...

I took the example of Zipcodes but there is the same problem for other
evaluations, for example here:
http://global.census.okfn.org/entry/si/companies

3 / An assessment that differs between countries

When we look in detail on the evaluation, we also see that the application
of the criteria is more or less strict.

An example: Zipcode / Slovania: 55%
http://global.census.okfn.org/entry/si/postcodes - the commentary states:
Data is available from Post of Slovenia, purpose is hidden in HTML format,
not available in bulk and Additional skills are needed to extract it.
Geodetska uprava (Slovenian equivalent of UK Ordnance Survey) resells bulk
data with GIS Additional information.

Just scrap the data then it deserves a score of 55%?

One for the road: Finland / Spending: 90%
http://global.census.okfn.org/entry/fi/spending - Certain assets data are
available on Finnish data portal Avoindata.fi. More information from Netra
Will Be ouvert in the future.

There was clearly a problem for the application of the methodology
described, for evaluating a current and non-availability "in the future."

3 / A reviewer who is also the editor for a country

I looked in detail ratings for the Isle of Man, who gets such good scores
for Government Spending file (100%).
That evaluation and comment: http://global.census.okfn.org/entry/im/spending

The proposed link is this one: http://financereports.gov.im - it in no way
corresponds to the criteria of the methodology.

The problem seems even more serious for this country - and unlike the
response Mor was Peter - it is one and the same person who proposed the
evaluation and validated once.

4 / Why is that a problem?

It was therefore clearly major inconsistencies in how to apply the criteria
for each country. But if the goal is to produce a ranking of countries -
not to assess individually), it is a problem. And even a serious problem to
the extent that 10 places to play close to 10%!

The only solution, to me it seems, is that the OKF can ensure that the
assessment is consistent for all countries .. if it is the credibility of
the ranking is questioned.

Simon

PS: also the issue had already been raised in 2012 for the classification
of W3C https://lists.okfn.org/pipermail/euopendata/2013-February/001153.html
- so I do not feel that the only problem is discovered now.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/open-data-census/attachments/20141103/99ca3879/attachment.html>