[data-protocols] json-stat

Fri Aug 10 09:44:54 BST 2012

Hey Vincenzo and Xavier,

thanks for both of your responses, this is really interesting.

On Fri, Aug 10, 2012 at 10:24 AM, Vincenzo Patruno <patrunomeister at gmail.com
> wrote:

> In short:
>
>    - I think the key issue is to have the possibility to get and to use
>    "structured" datasets
>    - We should obtain this datasets using APIs (Restful, of course!). I
>    add that we should converge on "standard" APIs with standard query string
>    schema (Open Services)
>
> I agree, having a standard API for statistical (or other
multi-dimensional) data would be fantastic, and REST takes care of a lot of
the necessary environment. I guess the query language/schema is the major
open bit, cf. http://www.unqlspec.org/display/UnQL/Home - but I don't think
that ever took off.

>
>    - Using the standard query string schema we can have the possibility
>    to get only data or both data and metadata depending on the needs of data
>    consumers
>
> That is why I think its worthwhile to separate model and data - different
users may also want to pull in different parts of the data - especially if
your reference data is more complex and pass-by-reference, such as the
stuff stored at http://opencorporates.com/ ..

>
>    - To do that is important to model  datasets (data and metadata)
>    before inserting them into a DB. The big problem we have is about metadata.
>    Which metadata and code lists to use? Everyone can use his own metadata,
>    but I think we should supply a way to converge toward common "public"
>    metadata (if and when possible). Probably we should imagine a cloud
>    platform to manage metadata you can match during the dataset modeling step.
>
> Exactly! What we need is a common, open and easy-to-use reference data
storage service! There's a discussion about standardisation under the OGP
where I'm trying to pitch this idea - I think this mix of government and
other actors would be a nice home for such a platform. Compare:

https://docs.google.com/drawings/d/1UInjPigMHO9nQD8kFU-CYZrULyNOAZT4xQ-LhO9J1nw/edit

>
>    - Coming back to json-stat, I think It's a really good starting point.
>    :-)
>
> Agreed!

 - Friedrich

> On Thu, Aug 9, 2012 at 2:18 PM, Friedrich Lindenberg <
> friedrich.lindenberg at okfn.org> wrote:
>
>> Dear Vincenzo,
>>
>> this is really fascinating, thanks for the link! I've always found SDMX a
>> bit impenetrable (not only because its XML), and I fully agree that JSON is
>> a much easier way to represent this information. What I find interesting is
>> that you have decided to describe both the metadata and data model in JSON,
>> but to also package it with the actual data.
>>
>> As an alternative approach, Google's DSPL (
>> https://developers.google.com/public-data/) splits these out - using XML
>> for the metadata and normal CSV for the actual observations. Nick and I
>> have done a bit of additional work to transfer this into JSON, you can find
>> a simple converter here: https://github.com/nickstenning/dspljson ...
>>
>> I slightly prefer the split approach, as nothing can be quite as concise
>> for data as a CSV file (and it's now no longer an issue to load that via
>> JavaScript). What do you think?
>>
>> - Friedrich
>>
>> On Wed, Aug 8, 2012 at 9:56 AM, Vincenzo Patruno <
>> patrunomeister at gmail.com> wrote:
>>
>>> Hi all!
>>>
>>> I know,... it's summertime and most of you are probably on vacation.....
>>> Anyway, my name is Vincenzo Patruno and I'm IT specialist at ISTAT, the
>>> italian statistical office.
>>>
>>> As probably you already know, some national statistical offices and
>>> seven international organizations (like for instance IMF, OECD, Eurostat,
>>> ...) are supporting the SDMX initiative (http://sdmx.org/)
>>>
>>> In short, me and some colleagues around the world, we believe we should
>>>  need a different architectural model and a different protocol for data
>>> transmission. The reason is to permit developers to use data easily inside
>>> web apps and mobile apps.
>>>
>>> This is the sense of Json-Stat initiative (http://json-stat.org/)
>>> lanched "from the bottom" last january.
>>> Json-stat is a "json based" schema that permits to use easily datasets
>>> inside applications. Just to be clear, for dataset we intend something
>>> modeled as follow:
>>>
>>> Suppose we have the following table:
>>>
>>> Demographic Balance for the year 2010
>>> Italy
>>>                                                         males       females
>>>    total
>>> Resident population on 1st January 2928740331052925 60340328Live births
>>> 289185 272759561944Deaths 286094301394587488 Natural increase3091 -28635
>>> -25544
>>>
>>>
>>>
>>> A dataset "Demographic Balance" can be:
>>>
>>> geo        time         sex       demographic balance               value
>>>           italy       2010         m        Resident population on 1st
>>> jan 29287403 italy               2010 f           Resident population
>>> on 1st jan 31052925[....]
>>>
>>> or, better
>>>
>>> geo       time          sex      demographic balance                value
>>>           37         2010         m        001
>>>                          29287403 37                 2010 f           001
>>>                                                  31052925
>>>
>>>
>>> where "37" is the code for "italy" and 001 the code for "Resident
>>> population on 1st jan" etc
>>>
>>> This is what we call usually "multidimensional dataset". (But every
>>> dataset is "multidimensional" because has almost the "geo" and "time"
>>> dimension ....) What we need is probably to promote the culture for
>>> modeling dataset. What we should promote is modeling both data and
>>> structured metadata.
>>>
>>> To obtain something like this (open it on your web browser)
>>>
>>> http://apistat.istat.it/?q=getdatajson&dataset=DCIS_POPSTRBIL&dim=1,0,0,0&lang=1&tr=&te=
>>>
>>> that is a structured dataset that can be easily used inside applications
>>> for an "interactive" use of data.
>>>
>>> Here some example.
>>>
>>> http://www.vincenzopatruno.org/json-stat/
>>> Enjoy!
>>>
>>> VP
>>>
>>>
>>>
>>> --
>>> Vincenzo Patruno
>>> http://www.segnalazionit.org
>>> http://www.vincenzopatruno.org
>>>
>>> “ If you want a track team to win the high jump you find one person who
>>> can jump seven feet, not seven people who can jump one foot. ”.
>>>
>>> _______________________________________________
>>> data-protocols mailing list
>>> data-protocols at lists.okfn.org
>>> http://lists.okfn.org/mailman/listinfo/data-protocols
>>>
>>>
>>
>
>
> --
> Vincenzo Patruno
> http://www.segnalazionit.org
> http://www.vincenzopatruno.org
>
> “ If you want a track team to win the high jump you find one person who
> can jump seven feet, not seven people who can jump one foot. ”.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/data-protocols/attachments/20120810/0a17e65e/attachment-0001.htm>