[Open-data-census] Open Data Census_proposals for improvement from Russia

Mon Oct 28 11:14:45 GMT 2013

Hi Tatyana,

useful feedback! Thank you.

best,

Ton

---------------------------------------------------------------------------
Interdependent Thoughts
Ton Zijlstra

ton at tonzijlstra.eu
+31-6-34489360

http://zylstra.org/blog

---------------------------------------------------------------------------

On Mon, Oct 28, 2013 at 9:52 AM, Tatyana Tolsteneva <
tatyana.tolsteneva at svobodainfo.org> wrote:

> Dear colleagues,
>
> Since September of this year, Ivan Pavlov, Chair of the Freedom of
> Information Foundation Board;  and I have taken part as country editors
> in the Open Data Census. We have done this with great pleasure and interest.
> We are very grateful that this unique opportunity exists and that we have
> a chance to observe the operations from the inside of the process.
>
> In Russia, the Open Data Census is valued so high that the country's
> Census rating it is used as an indicator of efficiency for Russia's
> national open data development strategy implementation.
>
> This year, we have paid close attention to the OKFN's initiative and have
> noticed that both the evaluation process and the Census interface are now
> even better than when we started working as country editors. Clearly, The
> Open Knowledge Foundation is striving to ensure that the Census is
> recognized as an impartial measure of transparency. We greatly respect the
> open manner of work you have chosen in order to improve the initiative, and
> your willingness to entertain proposals on the study improvement and
> readiness to discuss them in a constructive way.
>
> It is in this spirit of the OKFN's desire for improvement and
> impartiality that we wish to make some additional recommendations that may
> not yet have been taken into consideration in developing the Census
> process. We understand that in order to run a successful crowd-sourcing
> project, the information sought should be kept simple and the interface
> quick and easy to use. Unfortunately, we also see that this approach has
> its weaknesses as well, namely, the simplification of an evaluation process
> may negatively impact the fairness of a study.
>
> Our Foundation has been carrying out comparative studies and ratings based
> on informational openness of government entities in Russia and countries of
> the former USSR for the last nine years. It is our great hope that our
> specific experience in this area could be of use for development of the
> Open Data Census.
>
> There are a few specific areas where we see the potential to improve the
> impartiality of the system:
>
>    - *Publish Methodology: *We believe that the methodology of your
>    studies of open data development level in various countries should be
>    formalized and published. While the FAQ section covers some aspects of the
>    research in the FAQ is not enough to answer specific questions should they
>    arise. In our opinion, the methodology should also include a clear
>    description of contents for each information category included in each
>    dataset for evaluation, and a comprehensive description of each dataset
>    evaluation criterion (Data availability Questions). The international
>    nature of the study necessitates this: in different countries, datasets
>    evaluated within Open Data Census are formed in different ways, and their
>    contents are differently interpreted at the national level.
>
>
>    - *Benefit of the Doubt Policy: *Evaluators can also meet difficulties
>    in evaluating data that can impact the results of the study. In these
>    conditions, when evaluating the situation in the Open Data for Russia, we
>    based on *in dubio pro reo* principle: in other words, the researchsubject receives the benefit of the doubt related to evaluating datasets.
>    This would ensure that a weakness of the study would not result in a
>    negative bias towards the subject (in this case, country). We implement
>    this same principle when carrying out research and evaluation for our
>    Foundation.
>
>
>    - *Sub-Categories for Data Sets:* One potential area for methodology
>    improvement could be the possibility for separate evaluation of each
>    sub-category of information included in a specific dataset. For example,
>    when evaluating transport schedules, one should realize clearly what
>    sub-categories of information can/must be present in such schedules. Since
>    schedule contents (quality of datasets) can be different in different
>    countries, each sub-category included in this dataset should be evaluated
>    separately for the study to be objective.
>
> From our point of view, current options for answering the Open Data Census
> questions* – *"yes", "no", and "unclear" – often do not allow to expand
> the study potential in full volume. For instance, if part of a dataset
> under evaluation is provided for free and part for fee, we cannot reflect
> this correctly within the three-point evaluation scale; therefore, when
> results summarized, such a situation will not be reflected reliably.  We
> believe this can be solved if there will be a clear list of conditions a
> dataset should meet to be evaluated as "yes", "no", or "unclear" for each
> sub-category included in it.
>
> ·       *Set out a clear description of study stages from the outset:*This will help Country Editors and participants in the crowd sourcing
> process to set expectations and resources related to the time and outcomes
> of the study stages and the entire process. It also lends clarity and
> additional transparency to the process.
>
> ·       *“Terms of Use” vs. “Openly Licensed”:* As you may be aware, our
> organisation is comprised mainly of lawyers, sociologists and IT
> specialists. Our lawyer training causes us to pay particular attention to
> wording, including that for the rules governing the study. ****
>
> ·       One particular instance which caught our attention are the
> evaluation criterion for “openly licensed” as stated in the FAQ:
>
> *"It needs to state the** terms of use or license that allow anyone to
> freely use, reuse or redistribute the data (subject at most to attribution
> or sharealike requirements). It is vital that a license is available (if
> there's no license, the data is not openly licensed).*
>
> As you can see, one phrase allows "terms of use" be a condition for
> positive evaluation as a license , but another phrase states that a license
> is mandatory. Such wording is contradictory, and we consider this
> contradiction to be a rather important point.
>
> Open license is not only a technical, but also a legal issue.  Different
> countries can solve this legal issue in quite different ways, depending on
> the peculiarities of their national legal regimes regulating informational
> relations. Such peculiarities should not impact the country's positions in
> the Census rating. We believe that if a lawyer familiar well with both
> international and national specificities of open data legal regulation
> takes part in the methodology upgrade, this will help to improve the
> methodology significantly.
>
> Data use conditions similar to those contained in an open data license can
> be also guaranteed by national legislation, regarding not only specific
> datasets but entire information categories. If fixed in such a way, such
> conditions/terms of use can cover specific datasets *by default* without
> a direct reference to any license for any case of dataset publication;
> moreover, the terms of use may even be not mentioned together with the
> dataset. In the same time, such default terms of use can fully meet the
> sense and spirits of your study.
>
> We think it is unjust if the formal absence of a license for use of state
> information deprives a country of 30 points for each of 10 datasets
> evaluated, even if national legislation contains direct provision(s)
> meaning that terms of use of the state information meet all conditions of
> the "open data license". This is what happened in the case with Russia, and
> possibly other countries.
>
> We would like to humbly propose a few additional points that could be
> useful in improving procedural aspects of the study.
>
> 1.     The mechanism of Country Editors' work would become more
> transparent if the system stored and displayed all versions of scores for
> any dataset and any country approved by a Country Editor at any time,
> together with the name of the Country Editor that approved them. ****
>
> 2.     It would become easier to process each dataset if the system
> allowed the user to post comments for any of nine questions for each
> dataset, also providing possibility to post more than one comment, and
> displaying comment author's name.****
>
> 3.     The study could be essentially enriched and extended if there were
> a possibility for constructive discussion of each country's evaluation
> scores. The current version of the Census does not take into account lack
> of consensus between different Country Editors. At the same time, different
> experts can have different points of view regarding evaluation of the same
> dataset. The current version shows only the latest entered scores, not
> allowing to display existence of alternate views on evaluation of current
> data.****
>
> 4.     There is one more fact of practical significance: when printing
> data for a single country, color coding of evaluation scores is not
> displayed so one is to add data on current scores manually.
>
> A number of the  improvements recently introduced by the Census, for
> instance, indication of scoring points for each dataset and public display
> of Country Editors' names at the country census page, has made the Census
> more usable, and the evaluation procedure for each country more
> transparent. At the same time, we notice some more definite methodology
> changes in the current annual cycle of the Census; in particular, the
> result scoring system has changed. Improvement of the system is surely very
> important, but we believe it would be optimal to modify methodology *
> before* the beginning of a new evaluation cycle for all participants of
> the Census to learn and to understand the game rules ahead of the game.
>
> We understand that if implemented, some of our advice could create
> difficulties for using crowd-sourcing approach in your study since the
> evaluation procedure will become more complicated and resource-taking for
> anyone involved in it. We believe that, to solve this problem, a single
> study cycle could cover one or two datasets instead of ten.
>
> We will be very glad to have a possibility to take part in discussing
> those and/or any other proposals on the Census development with other
> interested parties. Any questions from your side are welcome.****
>
> **Terribly sorry for such a long letter and thanks a lot for attention to
> it. **
>   Yours sincerely,
>
> Tatyana Tolsteneva
> Development manager
> tatyana.tolsteneva at svobodainfo.org
>
> Freedom of Information Foundation
> (formerly known as Institute for Information Freedom Development)
> P.O. Box 527, St.-Petersburg, 192007, Russia
> Phone:  +7 812 766-03-66
> Fax: +7 812 766-52-61
> Email: info at svobodainfo.org <info at svobodainfo.org>
> www.svobodainfo.org
>
>
> _______________________________________________
> Open-data-census mailing list
> Open-data-census at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/open-data-census
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/open-data-census/attachments/20131028/2c3aae48/attachment-0001.htm>