[Open-data-census] Serious inconsistencies in the application of the methodology

Wed Nov 5 14:59:54 UTC 2014

Sorry Mor, I don't understand why this message appears now on the mailing list. 
In the meantime you did provide an answer to my "chained / independent" question. Thank you !

Regards,

Simon

> Le 3 nov. 2014 à 14:06, "Simon Chignard - data.gouv.fr" <simon at data.gouv.fr> a écrit :
> 
> Thank you Mor for your response.
> 
> My response below:
>> 
>> 1. You gave examples of data that has been reviewed with data that has not been reviewed yet. In addition, notice that the Canadians gave an example of an Open data website which gives the information for free as a government initiative and not as part of the policies that were mentioned.
> 
> I don’t see this information for Canada, neither here http://global.census.okfn.org/entry/ca/postcodes or here: http://global.census.okfn.org/submission/c231c2ed-659b-453f-b321-3b46c219d942 ?
> 
>> I would suggest that we would wait until all data sets have been reviewed before comparing them.
>> 
>> 2. Slovenia data is from 2013 and wasn't updated this year, so does the Finnish entry. The current display of the census survey show all entries from all years, but this does not mean that they will be published in the 2014 one. It is confusing, I know, but we are not going to show irrelevant submission. Regarding scoring systems to the Index, it can be found in the methodology section : http://census.okfn.org/global/methodology/
> 
> The methodology is unclear about the independence (or chaining) of every criteria. This was the main point of my alert. Sorry but I don’t think the OKF provides a clear answer to that.
> 
>> Just to be clear again - the ranking that you see now in the census site *do not* represent this year's index.
>> 
>> 3. Yes, you are right, we have the same editor for the Isle of Man, which is a small place with not a lot of activists. We are trying to solve this and to  have different reviewers who did not contribute. In some place is easy than other (depend on language skills for example). We do not pay any of the people who contribute to the Index, which makes it particularly hard. In order to solve this inconsistency, we would have a task force which will examine entries by topic and not by country (and if any of you out there want to help, email me).
>> 
>> 4. This is an important discussion and we should keep having it, but we need to look at the same points of reference. In any case, keep sending feedback to us!
> 
> I agree with you, we shouldn’t compare 2013 and 2014 results. But we definitely need to have a shared understanding of the criteria (and their applications).
> 
> Best,
> 
> Simon
> 
>> Best,
>> Mor
>> 
>>> On Mon, Nov 3, 2014 at 11:20 AM, Pierre Chrzanowski <pierre.chrzanowski at gmail.com> wrote:
>>> Hi list, I am forwarding a message from Simon Chignard who is concerned about the lack of quality and consistency in the current submissions.
>>> 
>>> I think his feedbacks should be carefully taken into account for the reviewing process.
>>> 
>>> Best 
>>> Pierre
>>> 
>>> Ps : text below is a Google translate from email wrote in French to okf france members list
>>> 
>>> ---
>>> Hello all, 
>>> 
>>> I spotted this weekend which seems to me to be serious inconsistencies in the application of the methodology of the Open Data Index since 2014. I alert you that the question of the reliability of the tool. 
>>> 
>>> 1 / An example: the assessment of open Zipcodes / Postcodes. 
>>> 
>>> Consider the postal code file for Spain, Sweden, Canada and France. 
>>> 
>>> In these four countries, the situation is the same: a more or less public operator (Correos, Postnummer, Canada Post and La Poste) sells, on demand, the postal code file. 
>>> 
>>> Yet, these are the scores on the same file: 
>>> 
>>> Zipcode / Canada: 55% 
>>> http://global.census.okfn.org/entry/ca/postcodes 
>>> 
>>> Zipcode / Spain: 45% 
>>> http://global.census.okfn.org/entry/es/postcodes 
>>> 
>>> Zipcode / France: 10% 
>>> http://global.census.okfn.org/entry/fr/postcodes 
>>> 
>>> Zipcode / Sweden: 55% 
>>> http://global.census.okfn.org/entry/se/postcodes 
>>> 
>>> 
>>> 2 / What is at issue 
>>> 
>>> The question posed here is that of chaining or independence criteria. 
>>> 
>>> In France we (collectively) have considered that the criteria chained. This means that if the data is not available then we put red all other criteria. However, in all other countries I could see they took each criterion separately. They consider that given legally sold and closed may still be available online, be current, be downloaded in bulk, etc ... 
>>> 
>>> I took the example of Zipcodes but there is the same problem for other evaluations, for example here: http://global.census.okfn.org/entry/si/companies 
>>>  
>>> 3 / An assessment that differs between countries 
>>> 
>>> When we look in detail on the evaluation, we also see that the application of the criteria is more or less strict. 
>>> 
>>> An example: Zipcode / Slovania: 55% 
>>> http://global.census.okfn.org/entry/si/postcodes - the commentary states: Data is available from Post of Slovenia, purpose is hidden in HTML format, not available in bulk and Additional skills are needed to extract it. Geodetska uprava (Slovenian equivalent of UK Ordnance Survey) resells bulk data with GIS Additional information. 
>>> 
>>> Just scrap the data then it deserves a score of 55%? 
>>> 
>>> One for the road: Finland / Spending: 90% 
>>> http://global.census.okfn.org/entry/fi/spending - Certain assets data are available on Finnish data portal Avoindata.fi. More information from Netra Will Be ouvert in the future. 
>>> 
>>> There was clearly a problem for the application of the methodology described, for evaluating a current and non-availability "in the future." 
>>> 
>>> 3 / A reviewer who is also the editor for a country 
>>> 
>>> I looked in detail ratings for the Isle of Man, who gets such good scores for Government Spending file (100%). 
>>> That evaluation and comment: http://global.census.okfn.org/entry/im/spending 
>>> 
>>> The proposed link is this one: http://financereports.gov.im - it in no way corresponds to the criteria of the methodology. 
>>> 
>>> The problem seems even more serious for this country - and unlike the response Mor was Peter - it is one and the same person who proposed the evaluation and validated once. 
>>> 
>>> 4 / Why is that a problem? 
>>> 
>>> It was therefore clearly major inconsistencies in how to apply the criteria for each country. But if the goal is to produce a ranking of countries - not to assess individually), it is a problem. And even a serious problem to the extent that 10 places to play close to 10%! 
>>> 
>>> The only solution, to me it seems, is that the OKF can ensure that the assessment is consistent for all countries .. if it is the credibility of the ranking is questioned. 
>>> 
>>> Simon 
>>> 
>>> PS: also the issue had already been raised in 2012 for the classification of W3C https://lists.okfn.org/pipermail/euopendata/2013-February/001153.html - so I do not feel that the only problem is discovered now.
>>> 
>>> 
>>> _______________________________________________
>>> open-data-census mailing list
>>> open-data-census at lists.okfn.org
>>> https://lists.okfn.org/mailman/listinfo/open-data-census
> 
> _______________________________________________
> open-data-census mailing list
> open-data-census at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/open-data-census
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/open-data-census/attachments/20141105/068c5d0b/attachment.html>