[Open-data-census] Serious inconsistencies in the application of the methodology

[Mor Rubinstein]

Hi guys,

Good input which can help to improve the Index. However, notice the
following points:

1. You gave examples of data that has been reviewed with data that has not
been reviewed yet. In addition, notice that the Canadians gave an example
of an Open data website which gives the information for free as a
government initiative and not as part of the policies that were mentioned.
I would suggest that we would wait until all data sets have been reviewed
before comparing them.

2. Slovenia data is from 2013 and wasn't updated this year, so does the
Finnish entry. The current display of the census survey show all entries
from all years, but this does not mean that they will be published in the
2014 one. It is confusing, I know, but we are not going to show irrelevant
submission. Regarding scoring systems to the Index, it can be found in the
methodology section : http://census.okfn.org/global/methodology/

Just to be clear again - the ranking that you see now in the census site
*do not* represent this year's index.

3. Yes, you are right, we have the same editor for the Isle of Man, which
is a small place with not a lot of activists. We are trying to solve this
and to  have different reviewers who did not contribute. In some place is
easy than other (depend on language skills for example). We do not pay any
of the people who contribute to the Index, which makes it particularly
hard. In order to solve this inconsistency, we would have a task force
which will examine entries by topic and not by country (and if any of you
out there want to help, email me).

4. This is an important discussion and we should keep having it, but we
need to look at the same points of reference. In any case, keep sending
feedback to us!

Best,
Mor

On Mon, Nov 3, 2014 at 11:20 AM, Pierre Chrzanowski <
pierre.chrzanowski at gmail.com> wrote:

> Hi list, I am forwarding a message from Simon Chignard who is concerned
> about the lack of quality and consistency in the current submissions.
>
> I think his feedbacks should be carefully taken into account for the
> reviewing process.
>
> Best
> Pierre
>
> Ps : text below is a Google translate from email wrote in French to okf
> france members list
>
> ---
> Hello all,
>
> I spotted this weekend which seems to me to be serious inconsistencies in
> the application of the methodology of the Open Data Index since 2014. I
> alert you that the question of the reliability of the tool.
>
> 1 / An example: the assessment of open Zipcodes / Postcodes.
>
> Consider the postal code file for Spain, Sweden, Canada and France.
>
> In these four countries, the situation is the same: a more or less public
> operator (Correos, Postnummer, Canada Post and La Poste) sells, on demand,
> the postal code file.
>
> Yet, these are the scores on the same file:
>
> Zipcode / Canada: 55%
> http://global.census.okfn.org/entry/ca/postcodes
>
> Zipcode / Spain: 45%
> http://global.census.okfn.org/entry/es/postcodes
>
> Zipcode / France: 10%
> http://global.census.okfn.org/entry/fr/postcodes
>
> Zipcode / Sweden: 55%
> http://global.census.okfn.org/entry/se/postcodes
>
>
> 2 / What is at issue
>
> The question posed here is that of chaining or independence criteria.
>
> In France we (collectively) have considered that the criteria chained.
> This means that if the data is not available then we put red all other
> criteria. However, in all other countries I could see they took each
> criterion separately. They consider that given legally sold and closed may
> still be available online, be current, be downloaded in bulk, etc ...
>
> I took the example of Zipcodes but there is the same problem for other
> evaluations, for example here:
> http://global.census.okfn.org/entry/si/companies
>
> 3 / An assessment that differs between countries
>
> When we look in detail on the evaluation, we also see that the application
> of the criteria is more or less strict.
>
> An example: Zipcode / Slovania: 55%
> http://global.census.okfn.org/entry/si/postcodes - the commentary states:
> Data is available from Post of Slovenia, purpose is hidden in HTML format,
> not available in bulk and Additional skills are needed to extract it.
> Geodetska uprava (Slovenian equivalent of UK Ordnance Survey) resells bulk
> data with GIS Additional information.
>
> Just scrap the data then it deserves a score of 55%?
>
> One for the road: Finland / Spending: 90%
> http://global.census.okfn.org/entry/fi/spending - Certain assets data are
> available on Finnish data portal Avoindata.fi. More information from Netra
> Will Be ouvert in the future.
>
> There was clearly a problem for the application of the methodology
> described, for evaluating a current and non-availability "in the future."
>
> 3 / A reviewer who is also the editor for a country
>
> I looked in detail ratings for the Isle of Man, who gets such good scores
> for Government Spending file (100%).
> That evaluation and comment:
> http://global.census.okfn.org/entry/im/spending
>
> The proposed link is this one: http://financereports.gov.im - it in no
> way corresponds to the criteria of the methodology.
>
> The problem seems even more serious for this country - and unlike the
> response Mor was Peter - it is one and the same person who proposed the
> evaluation and validated once.
>
> 4 / Why is that a problem?
>
> It was therefore clearly major inconsistencies in how to apply the
> criteria for each country. But if the goal is to produce a ranking of
> countries - not to assess individually), it is a problem. And even a
> serious problem to the extent that 10 places to play close to 10%!
>
> The only solution, to me it seems, is that the OKF can ensure that the
> assessment is consistent for all countries .. if it is the credibility of
> the ranking is questioned.
>
> Simon
>
> PS: also the issue had already been raised in 2012 for the classification
> of W3C
> https://lists.okfn.org/pipermail/euopendata/2013-February/001153.html -
> so I do not feel that the only problem is discovered now.
>
>
> _______________________________________________
> open-data-census mailing list
> open-data-census at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/open-data-census
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/open-data-census/attachments/20141103/2e09a09f/attachment.html>