[Open-data-census] Open Data Census_proposals for improvement from Russia

Mon Oct 28 21:50:08 GMT 2013

Thank you Tatyana, for this very elaborate set of thoughts. As Ton
mentions, this is highly useful.
This week we are fully focused on the Open Government Partnership
activities, where the Index will also be presented - and thereafter we'll
have more bandwidth to start the conversion on future improvements - based
on this kind of feedback from the community.

Feedback on these ideas from other list members is also very welcome!

-Christian

-- 

Christian Villum

Community Manager, Open Government Data + Local Groups Network
skype: christianvillum  |  @villum <http://www.twitter.com/villum>
The Open Knowledge Foundation <http://okfn.org/>
*Empowering through Open Knowledge
*http://okfn.org/  |  @okfn <http://twitter.com/OKFN>  |  OKF on
Facebook<https://www.facebook.com/OKFNetwork> |
Blog <http://blog.okfn.org/>  |  Newsletter<http://okfn.org/about/newsletter>

On Mon, Oct 28, 2013 at 11:14 AM, Ton Zijlstra <ton.zijlstra at gmail.com>wrote:

> Hi Tatyana,
>
> useful feedback! Thank you.
>
> best,
>
> Ton
>
> ---------------------------------------------------------------------------
> Interdependent Thoughts
> Ton Zijlstra
>
> ton at tonzijlstra.eu
> +31-6-34489360
>
> http://zylstra.org/blog
>
> ---------------------------------------------------------------------------
>
>
> On Mon, Oct 28, 2013 at 9:52 AM, Tatyana Tolsteneva <
> tatyana.tolsteneva at svobodainfo.org> wrote:
>
>> Dear colleagues,
>>
>> Since September of this year, Ivan Pavlov, Chair of the Freedom of
>> Information Foundation Board;  and I have taken part as country editors
>> in the Open Data Census. We have done this with great pleasure and interest.
>> We are very grateful that this unique opportunity exists and that we
>> have a chance to observe the operations from the inside of the process.
>>
>> In Russia, the Open Data Census is valued so high that the country's
>> Census rating it is used as an indicator of efficiency for Russia's
>> national open data development strategy implementation.
>>
>> This year, we have paid close attention to the OKFN's initiative and have
>> noticed that both the evaluation process and the Census interface are now
>> even better than when we started working as country editors. Clearly, The
>> Open Knowledge Foundation is striving to ensure that the Census is
>> recognized as an impartial measure of transparency. We greatly respect the
>> open manner of work you have chosen in order to improve the initiative, and
>> your willingness to entertain proposals on the study improvement and
>> readiness to discuss them in a constructive way.
>>
>> It is in this spirit of the OKFN's desire for improvement and
>> impartiality that we wish to make some additional recommendations that may
>> not yet have been taken into consideration in developing the Census
>> process. We understand that in order to run a successful crowd-sourcing
>> project, the information sought should be kept simple and the interface
>> quick and easy to use. Unfortunately, we also see that this approach has
>> its weaknesses as well, namely, the simplification of an evaluation process
>> may negatively impact the fairness of a study.
>>
>> Our Foundation has been carrying out comparative studies and ratings
>> based on informational openness of government entities in Russia and
>> countries of the former USSR for the last nine years. It is our great hope
>> that our specific experience in this area could be of use for development
>> of the Open Data Census.
>>
>> There are a few specific areas where we see the potential to improve the
>> impartiality of the system:
>>
>>    - *Publish Methodology: *We believe that the methodology of your
>>    studies of open data development level in various countries should be
>>    formalized and published. While the FAQ section covers some aspects of the
>>    research in the FAQ is not enough to answer specific questions should they
>>    arise. In our opinion, the methodology should also include a clear
>>    description of contents for each information category included in each
>>    dataset for evaluation, and a comprehensive description of each dataset
>>    evaluation criterion (Data availability Questions). The international
>>    nature of the study necessitates this: in different countries, datasets
>>    evaluated within Open Data Census are formed in different ways, and their
>>    contents are differently interpreted at the national level.
>>
>>
>>    - *Benefit of the Doubt Policy: *Evaluators can also meet
>>    difficulties in evaluating data that can impact the results of the study.
>>    In these conditions, when evaluating the situation in the Open Data for
>>    Russia, we based on *in dubio pro reo* principle: in other words, the
>>    research subject receives the benefit of the doubt related to
>>    evaluating datasets. This would ensure that a weakness of the study would
>>    not result in a negative bias towards the subject (in this case, country).
>>    We implement this same principle when carrying out research and evaluation
>>    for our Foundation.
>>
>>
>>    - *Sub-Categories for Data Sets:* One potential area for methodology
>>    improvement could be the possibility for separate evaluation of each
>>    sub-category of information included in a specific dataset. For example,
>>    when evaluating transport schedules, one should realize clearly what
>>    sub-categories of information can/must be present in such schedules. Since
>>    schedule contents (quality of datasets) can be different in different
>>    countries, each sub-category included in this dataset should be evaluated
>>    separately for the study to be objective.
>>
>> From our point of view, current options for answering the Open Data
>> Census questions* – *"yes", "no", and "unclear" – often do not allow to
>> expand the study potential in full volume. For instance, if part of a
>> dataset under evaluation is provided for free and part for fee, we cannot
>> reflect this correctly within the three-point evaluation scale; therefore,
>> when results summarized, such a situation will not be reflected reliably.
>> We believe this can be solved if there will be a clear list of conditions
>> a dataset should meet to be evaluated as "yes", "no", or "unclear" for each
>> sub-category included in it.
>>
>> ·       *Set out a clear description of study stages from the outset:*This will help Country Editors and participants in the crowd sourcing
>> process to set expectations and resources related to the time and outcomes
>> of the study stages and the entire process. It also lends clarity and
>> additional transparency to the process.
>>
>> ·       *“Terms of Use” vs. “Openly Licensed”:* As you may be aware, our
>> organisation is comprised mainly of lawyers, sociologists and IT
>> specialists. Our lawyer training causes us to pay particular attention to
>> wording, including that for the rules governing the study. ****
>>
>> ·       One particular instance which caught our attention are the
>> evaluation criterion for “openly licensed” as stated in the FAQ:
>>
>> *"It needs to state the** terms of use or license that allow anyone to
>> freely use, reuse or redistribute the data (subject at most to attribution
>> or sharealike requirements). It is vital that a license is available (if
>> there's no license, the data is not openly licensed).*
>>
>> As you can see, one phrase allows "terms of use" be a condition for
>> positive evaluation as a license , but another phrase states that a license
>> is mandatory. Such wording is contradictory, and we consider this
>> contradiction to be a rather important point.
>>
>> Open license is not only a technical, but also a legal issue.  Different
>> countries can solve this legal issue in quite different ways, depending on
>> the peculiarities of their national legal regimes regulating informational
>> relations. Such peculiarities should not impact the country's positions in
>> the Census rating. We believe that if a lawyer familiar well with both
>> international and national specificities of open data legal regulation
>> takes part in the methodology upgrade, this will help to improve the
>> methodology significantly.
>>
>> Data use conditions similar to those contained in an open data license
>> can be also guaranteed by national legislation, regarding not only specific
>> datasets but entire information categories. If fixed in such a way, such
>> conditions/terms of use can cover specific datasets *by default* without
>> a direct reference to any license for any case of dataset publication;
>> moreover, the terms of use may even be not mentioned together with the
>> dataset. In the same time, such default terms of use can fully meet the
>> sense and spirits of your study.
>>
>> We think it is unjust if the formal absence of a license for use of state
>> information deprives a country of 30 points for each of 10 datasets
>> evaluated, even if national legislation contains direct provision(s)
>> meaning that terms of use of the state information meet all conditions of
>> the "open data license". This is what happened in the case with Russia, and
>> possibly other countries.
>>
>> We would like to humbly propose a few additional points that could be
>> useful in improving procedural aspects of the study.
>>
>> 1.     The mechanism of Country Editors' work would become more
>> transparent if the system stored and displayed all versions of scores for
>> any dataset and any country approved by a Country Editor at any time,
>> together with the name of the Country Editor that approved them. ****
>>
>> 2.     It would become easier to process each dataset if the system
>> allowed the user to post comments for any of nine questions for each
>> dataset, also providing possibility to post more than one comment, and
>> displaying comment author's name.****
>>
>> 3.     The study could be essentially enriched and extended if there
>> were a possibility for constructive discussion of each country's evaluation
>> scores. The current version of the Census does not take into account lack
>> of consensus between different Country Editors. At the same time, different
>> experts can have different points of view regarding evaluation of the same
>> dataset. The current version shows only the latest entered scores, not
>> allowing to display existence of alternate views on evaluation of current
>> data.****
>>
>> 4.     There is one more fact of practical significance: when printing
>> data for a single country, color coding of evaluation scores is not
>> displayed so one is to add data on current scores manually.
>>
>> A number of the  improvements recently introduced by the Census, for
>> instance, indication of scoring points for each dataset and public display
>> of Country Editors' names at the country census page, has made the Census
>> more usable, and the evaluation procedure for each country more
>> transparent. At the same time, we notice some more definite methodology
>> changes in the current annual cycle of the Census; in particular, the
>> result scoring system has changed. Improvement of the system is surely very
>> important, but we believe it would be optimal to modify methodology *
>> before* the beginning of a new evaluation cycle for all participants of
>> the Census to learn and to understand the game rules ahead of the game.
>>
>> We understand that if implemented, some of our advice could create
>> difficulties for using crowd-sourcing approach in your study since the
>> evaluation procedure will become more complicated and resource-taking for
>> anyone involved in it. We believe that, to solve this problem, a single
>> study cycle could cover one or two datasets instead of ten.
>>
>> We will be very glad to have a possibility to take part in discussing
>> those and/or any other proposals on the Census development with other
>> interested parties. Any questions from your side are welcome.****
>>
>> **Terribly sorry for such a long letter and thanks a lot for attention
>> to it. **
>>   Yours sincerely,
>>
>> Tatyana Tolsteneva
>> Development manager
>> tatyana.tolsteneva at svobodainfo.org
>>
>> Freedom of Information Foundation
>> (formerly known as Institute for Information Freedom Development)
>> P.O. Box 527, St.-Petersburg, 192007, Russia
>> Phone:  +7 812 766-03-66
>> Fax: +7 812 766-52-61
>> Email: info at svobodainfo.org <info at svobodainfo.org>
>> www.svobodainfo.org
>>
>>
>> _______________________________________________
>> Open-data-census mailing list
>> Open-data-census at lists.okfn.org
>> http://lists.okfn.org/mailman/listinfo/open-data-census
>>
>>
>
> _______________________________________________
> Open-data-census mailing list
> Open-data-census at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/open-data-census
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/open-data-census/attachments/20131028/381ae7a1/attachment-0001.htm>