[Open-data-census] Scoring System

[Graeme Jones]

Just some thoughts so far.  I think a scoring system is certainly important
but perhaps the census is too blunt to always be productive in
political/government lobbying.

For example, peer group review of Isle of Man with Jersey, Guernsey are all
in the same sort of range and that undermines one of the main headlines to
push change - relative competition.  It is easy to dismiss lobbying if big
neighbouring countries have got a multiple of the points and much larger
resources such as UK or (theoretically at least) Ireland.

A government can almost cheat if they quickly outsource a narrow service
area to a supplier that has arms length open documents or open data but the
government policy on open documents and open data isn't progressed in other
service areas.

Not all service areas are equal expectation or equal effort.  A government
could open up something basic like election results but something more
advanced like monopoly geolocation postcodes and low level ariel
photography in digital mapping is more likely to indicate and push wider
and deeper open government policy.  So, should there be variable points
with each service area to reward relative boldness/effort or each extra
service area to reward momentum?

The scoring system struggles to convey content scale such as latest
election results or all election and by-election results back 50-100 years
or extra reports such as voting trend analysis.

It could be more progressive to itemise points "earned" in more detail on
each indicator so that action points can be recommended as fast wins such
as GTFS bus timetables, CSV/XLS of annual budgets or month end spending,
acceptance and encouragement of crowdsourced data cleansing.

Graeme Jones
Isle of Man

Date: Mon, 7 Oct 2013 18:53:32 +0400

> From: Tatyana Tolsteneva <tatyana.tolsteneva at svobodainfo.org>
> Subject: Re: [Open-data-census] [okfn-discuss] A question about the
>         Open    Data census scoring system
> To: Christian Villum <christian.villum at okfn.org>,       open-data-census
>         <open-data-census at lists.okfn.org>
> Message-ID: <CA93CA2F-4EBD-4ECE-87B6-9BC57AD90256 at svobodainfo.org>
> Content-Type: text/plain; charset="utf-8"
>
> Dear colleagues,
> let me please ask you about the Census's results calculation system.
> I have found the table with scoring system at the FAQ page
> http://2013.census.okfn.org/faq/ and have found some explanation:
> "The Open Data Census measures the state of openness of 10 data sets for
> each country. The overall score for a dataset is based on the response to
> specific questions with varying weightings - the weighting for each
> question is listed in the question table above. The overall country score
> is then calculated from the score on each dataset."
> But as far as i can see from the main Census page
> http://2013.census.okfn.org/country/  the "up to date" score is still
> excluded form the calculation system as it was explained here
> https://github.com/okfn/opendatacensus/issues/33?source=cc. On the main
> Census page there is no special green/yellow/red mark for "up to date"
> (actually we have 9 questions for each dataset, but only 8
>  green/yellow/red marks for answers)
> Would you be so kind to help me with any explanation of the Census's
>  results calculation?
> Thanks in advance!
>
> Yours sincerely,??
>
> Tatyana Tolsteneva
> Development manager
> tatyana.tolsteneva at svobodainfo.org
>
> Freedom of Information Foundation?
> (formerly known as Institute for Information Freedom Development)?
> P.O. Box 527, St.-Petersburg, 192007, Russia
> Phone:  +7 812 766-03-66 ?
> Fax: +7 812 766-52-61 ?
> Email: info at svobodainfo.org?
> www.svobodainfo.org
>
>
>
> 01.10.2013, ? 16:39, Pierre Chrzanowski ???????(?):
>
> > Hi All, I already got some feedbacks about the new census results and
> most of the time people are quite surprised with the total score and
> position of France. I think the fact that France (whose most of datasets
> are openly licensed) and China (whose datasets are not open) share almost
> the same score is one of the main reasons.
> >
> > I already expressed my concerns to Christian that the scoring system
> could be misleading, or even irrelevant.
> > And regarding the first feedbacks I got, I must say I am now quite sure
> it is misleading ;)
> >
> > Then, I would suggest to :
> >
> > 1/ remove the actual score which does not reflect openess of datasets in
> the countries
> >
> > and
> >
> > 2/ create a scoring system that really value openess (i.e score 0 if
> dataset not open)
> >
> > That being said, I am also wondering if it would not be better not have
> a score at all ...
> >
> > What do you think ?
> >
> > I found a discussion on the scoring calculation methodology and how it
> has evolved here
> https://github.com/okfn/opendatacensus/issues/33?source=cc
> >
> >
> >
> > On Fri, Aug 23, 2013 at 12:49 PM, Christian Villum <
> christian.villum at okfn.org> wrote:
> > Dear Tatyana,
> >
> > Thank you for offering to help, we will definitely be explaining the
> methodology on the site as part of the coming updates.
> > We'll also look into the EXMO tool, thanks for the tip.
> >
> > Let's stay in touch!
> >
> > -Christian
> >
> > --
> >
> > Christian Villum
> >
> > Community Manager, Open Government Data + Local Groups Network
> > skype: christianvillum  |  @villum
> >
> >
> > The Open Knowledge Foundation
> > Empowering through Open Knowledge
> > http://okfn.org/  |  @okfn  |  OKF on Facebook  |  Blog  |  Newsletter
> >
> > Have you registered for OKCon 2013?
> >
> >
> >
> > On Wed, Aug 21, 2013 at 11:49 AM, Tatyana Tolsteneva <
> tatyanatolsteneva at gmail.com> wrote:
> > Dear Rufus,
> >
> > thanks a lot for your respond.
> > I would like suggest to join to the idea of developing and publishing
> full methodology of the Census.
> > We in the Freedom of Information Foundation (Russia) have already done
> tis job for our monitoring (please see our methodology on our web-site in
> English http://www.svobodainfo.org/en/node/1014).
> > We would be happy to provide translation support and translate the
> Census methodology in Russian.
> >
> > Also we would be pleasant to offer you our ICT monitoring tool - EXMO
> system, specially developed for government web-sites monitoring. Please
> find the article about EXMO work here (in English)
> http://www.svobodainfo.org/en/node/2162
> > Just note, that unique quantity of EXMO is it's interaction part, which
> allows not only measure government openness level, but also improve it via
> internet communication with officials.
> >
> > Yours sincerely,?> ?>
> >
> > Tatyana Tolsteneva
> > Development manager
> > tatyana.tolsteneva at svobodainfo.org
> >
> > Freedom of Information Foundation?>
> > (formerly known as Institute for Information Freedom Development)?>
> > P.O. Box 527, St.-Petersburg, 192007, Russia
> > Phone:  +7 812 766-03-66 ?>
> > Fax: +7 812 766-52-61 ?>
> > Email: info at svobodainfo.org?>
> > www.svobodainfo.org
> >
> >
> > 01.08.2013, ? 14:06, Rufus Pollock ???????(?):
> >
> >> On 1 August 2013 10:24, Tatyana Tolsteneva <tatyanatolsteneva at gmail.com>
> wrote:
> >> Dear Rufus,
> >>
> >> thanks a lot for your explanation, i have already found it here
> https://github.com/okfn/opendatacensus/issues/33.
> >> Please take my apologies for "OKNF", it was just a misprint.
> >>
> >> No worries :-) (Just making sure not a confusion as opposed to typo!)
> >>
> >> Here in Russia the Census it considered to be most authoritative
> comparative study of open data around the world. And here we really feel
> some lack of official information about the Census methodology.
> >> It seems, this problem could be solved by official publication of the
> methodology on the Census website.
> >> Could it be possible? What do you think about the methodology
> publication?
> >>
> >> Yes this is a very good idea and we'll get moving on this right away.
> Any suggestions for specific things to include or language are very welcome!
> >>
> >> Rufus
> >>
> >> Yours sincerely,?>> ?>>
> >>
> >> Tatyana Tolsteneva
> >> Development manager
> >> tatyana.tolsteneva at svobodainfo.org
> >>
> >> Freedom of Information Foundation?>>
> >> (formerly known as Institute for Information Freedom Development)?>>
> >> P.O. Box 527, St.-Petersburg, 192007, Russia
> >> Phone:  +7 812 766-03-66 ?>>
> >> Fax: +7 812 766-52-61 ?>>
> >> Email: info at svobodainfo.org?>>
> >> www.svobodainfo.org
> >>
> >> 31.07.2013, ? 22:28, Rufus Pollock ???????(?):
> >>
> >>> [changing mailing list from okfn-discuss to open-data-census]
> >>>
> >>> On 23 July 2013 12:06, Tatyana Tolsteneva <tatyanatolsteneva at gmail.com>
> wrote:
> >>> Dear colleagues,
> >>> my name is Tatyana and i'm a development manager of the Freedom of
> Information Foundation - Russian FOI-advocasy NGO based in St. Petersburg.
> >>> I'm interested in the OKNF Open Data Census scoring system because i
> can't catch it's idea of it http://census.okfn.org/G8/
> >>>
> >>> OKFN (Open Knowledge Foundation (Network))
> >>>
> >>> At first i had a hypothesis, that each "yes" costs 1 poin and points
> just should to be summarized.
> >>>
> >>> That's almost the case but not quite: whether data is up to date does
> not count towards the total score.
> >>>
> >>> This idea has been perfect, if maximum for all questions would have
> been 7.
> >>> But we have 6 as maximum.
> >>> So my question is about "data existence" column
> >>> IN case all  "yes" should cost 1 points, it should be 7 as maximum.
> >>>
> >>> See above.
> >>>
> >>> Form the table http://census.okfn.org/G8/ i can see that some times
> "yes" per "data existence" is taken into the account.
> >>>
> >>> It should be always taken into account.
> >>>
> >>> Fo example,  Transport timetable is Russia, Government spending is
> France and in Italy.
> >>> But in the majority of cases YES per "data existence" is not included
> into the account. For example Zip-codes Russia
> >>>
> >>> It is included there - it's up to date that is not included.
> >>>
> >>> And speaking about Russian case, company register for Russia  has 2/6
> and Transport timetable has 2/6, but Company register is up to date and
> Transport timetable is not.
> >>>
> >>> See above.
> >>>
> >>> Would you be so kind to explain this scoring and situation with points
> for "data existence" and i woul be specially glad for explaining in the
> context of the Russian score.
> >>>
> >>> Hope the above clarifies things. Also thanks for asking as your
> questions will encourage us to put up further documentation on the website!
> >>>
> >>> Rufus
> >>>
> >>>
> >>
> >>
> >>
> >>
> >> --
> >> Rufus Pollock
> >> Founder and Executive Director  |  skype: rufuspollock  |  @rufuspollock
> >> The Open Knowledge Foundation
> >> Empowering through Open Knowledge
> >> http://okfn.org/  |  @okfn  |  OKF on Facebook  |  Blog  |  Newsletter
> >>
> >>
> >
> >
> > _______________________________________________
> > Open-data-census mailing list
> > Open-data-census at lists.okfn.org
> > http://lists.okfn.org/mailman/listinfo/open-data-census
> >
> >
> >
> > _______________________________________________
> > Open-data-census mailing list
> > Open-data-census at lists.okfn.org
> > http://lists.okfn.org/mailman/listinfo/open-data-census
> >
> >
> >
> >
> > --
> > Pierre Chrzanowski
> > Open Knowledge Foundation France
> >
> > Mail: pierre.chrzanowski at gmail.com
> > Mobile: +33 (0)7 855 71 292 | Skype: pierre.chrzanowski | Twitter:
> @piezanowski
>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://lists.okfn.org/pipermail/open-data-census/attachments/20131007/c77c0604/attachment.htm
> >
>
> ------------------------------
>
> _______________________________________________
> Open-data-census mailing list
> Open-data-census at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/open-data-census
>
>
> End of Open-data-census Digest, Vol 5, Issue 39
> ***********************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/open-data-census/attachments/20131008/6016eb6b/attachment-0001.htm>