[open-government] [CivicAccess-discuss] Open Data Standards

Tue Apr 28 14:54:14 UTC 2015

Hi Tracey,

I don’t know of any comprehensive list of data standards, mainly because it would be so large that it would no longer be helpful. As Stéphane mentioned, once you get into domain-specific standards, you end up with a lot of competing standards. Also, I’m using “standard” here in the colloquial, loose sense to mean any specification.

The best organized source for linked data vocabularies is http://lov.okfn.org/ <http://lov.okfn.org/> There are other linked data vocabulary search engines, but LOV is the best in my experience, and it is actively maintained.

For other lists of specs, you can look at:
- http://en.wikipedia.org/wiki/Metadata_standards <http://en.wikipedia.org/wiki/Metadata_standards>
- https://en.wikipedia.org/wiki/List_of_types_of_XML_schemas <https://en.wikipedia.org/wiki/List_of_types_of_XML_schemas>
- http://en.wikipedia.org/wiki/List_of_XML_markup_languages <http://en.wikipedia.org/wiki/List_of_XML_markup_languages>
- http://en.wikipedia.org/wiki/List_of_web_service_specifications <http://en.wikipedia.org/wiki/List_of_web_service_specifications>

So far on this list, it seems that the suggestions in the replies to your original message have been mostly about specs that people find interesting - but not necessarily specs that have much widespread adoption. With the exception of Humanitarian Data Exchange (HDX), those listed by Stéphane are from the community surrounding Code for America. The oldest of those is Yelp’s from 2012. Its partnership with Socrata may propel the spec to widespread use: http://www.socrata.com/newsroom-article/socrata-yelp-partner-open-restaurant-data-improve-global-public-health/ <http://www.socrata.com/newsroom-article/socrata-yelp-partner-open-restaurant-data-improve-global-public-health/>

The non-spec pages with more context (e.g. list of adopters and people involved) for OpenTrails and HouseFacts are:
- https://www.codeforamerica.org/specifications/trails/ <https://www.codeforamerica.org/specifications/trails/>
- http://www.codeforamerica.org/our-work/data-formats/housefacts/ <http://www.codeforamerica.org/our-work/data-formats/housefacts/>

The HDX spec was restarted after 2013; the earlier version can be seen at http://web.archive.org/web/20140221133907/http://hxl.humanitarianresponse.info/ <http://web.archive.org/web/20140221133907/http://hxl.humanitarianresponse.info/> The new version is still in development.

As for the various Data Packages, hopefully Rufus can tell us more about their adoption, and whether they are stable or are still under active development.

By the way, Popolo inventories the terms of specs that it researched as prior work (so that it may reuse the best of those specs) at http://www.popoloproject.com/appendices/terms.html <http://www.popoloproject.com/appendices/terms.html> As you can see, spec developers have come up with at least a dozen ways to express anything.

Your list of specifications/standards looks good to me! I think Budget Data Package (http://fiscal.dataprotocols.org/ <http://fiscal.dataprotocols.org/>) should be substituted for the linked OpenSpending spec. My understanding is that Budget Data Package was pursued instead of further development on that earlier spec.

Cheers,

James

> On Apr 28, 2015, at 9:05 AM, Tracey P. Lauriault <tlauriau at gmail.com> wrote:
> 
> Thank you Stephane and Rufus;
> 
> That was really helpful.
> 
> Rufus, please forgive my ignorance, but protocols, packages, standards and specifications?
> 
> I know what specs and standards are but not sure what you mean by protocols and packages?  Also, perhaps naive of me, but my trust in things increases when I see the community involved in things being produced and communities of adopters.  In this list you sent how would a novice like myself be able to judge when looking at the pages? Of course I have trust in the sender, you, but how do I trust the robustness of what you sent, who is community developing this, what is the context, who are the creators, users, implementers?  The site does not have much of that context info.
> 
> Cheers
> t
> 
> On Tue, Apr 28, 2015 at 1:48 PM, Stéphane Guidoin <stephane at opennorth.ca <mailto:stephane at opennorth.ca>> wrote:
> Hey Tracey,
> 
> We (Open north) are continuing our work around open data standards and some new content should become available before IODC.
> 
> Before we get there, if I may add a few comments:
> - In our current work, we are mainly focusing on the open data building blocks: standards and best practices that make open data usable, things like DCAT but also things like character encoding. In my view this is really some sort of "meta" open standards because they make open data available/possible (and unfortunatly many of these basic block are not that well-known yet)
> 
> - Then there are what we can call "generic" standards. Those are standards that can be integrated in any data. Things like IANA code, ISO 639, ISO 8601 are usually not that useful alone but are pretty helpful when you have to put a date (for example) in another dataset.
> 
> - After you have domain specific open data standards which are full-fledge self-sufficient (although they can borrow from others) standards that allow to represent a very specific information (e.g transit schedules.) The difficulty with those is that they tend to proliferate. Also it's not always obvious how a standards is (for) open data.
> 
> In any case, to your list I would add:
> - Open referral (aka open211) : http://openreferral.org/ <http://openreferral.org/>
> - Humanitarian data exchange: http://docs.hdx.rwlabs.org/standards/ <http://docs.hdx.rwlabs.org/standards/>
> - Health score for food inspection : http://www.yelp.ca/healthscores <http://www.yelp.ca/healthscores>
> - House facts for home/salubrity inspections: https://sites.google.com/site/housefactsdatastandard/home/specification <https://sites.google.com/site/housefactsdatastandard/home/specification>
> - Open trails for ... trails: https://www.codeforamerica.org/specifications/trails/spec.html <https://www.codeforamerica.org/specifications/trails/spec.html>
> 
> On Tue, Apr 28, 2015 at 8:14 AM, Tracey P. Lauriault <tlauriau at gmail.com <mailto:tlauriau at gmail.com>> wrote:
> Thanks Rufus!
> 
> Any thoughts on the list I sent? Anything missing?  It is for a technical framework doc under discussion at the moment
> 
> I am finding a disconnect in open data communities re-standards that support exchange and interoperability beyond the data format or linked data.  For example, in that list I sent, everything after ISO 8601 is what i added.
> 
> Any help would be greatly appreciated.  This document goes for discussion and potential signoff tmr!
> 
> Cheers
> t
> 
> On Tue, Apr 28, 2015 at 12:53 PM, Rufus Pollock <rufus.pollock at okfn.org <mailto:rufus.pollock at okfn.org>> wrote:
> Hi Tracey,
> 
> That's a great question. I note that the Data Protocols <http://dataprotocols.org/> has a nascent "reference" section and there are plans to extend it:
> 
> http://dataprotocols.org/reference/ <http://dataprotocols.org/reference/>
> 
> New additions are very welcome!
> 
> Regards,
> 
> Rufus
> 
> On 28 April 2015 at 10:59, Tracey P. Lauriault <tlauriau at gmail.com <mailto:tlauriau at gmail.com>> wrote:
> Hi Gang;
> 
> Is there a place which lists open data standards?
> 
> I found the Open North Inventory,  https://docs.google.com/file/d/0BxxUHhraTIHhcnNhUU9JRnc1YlU/edit <https://docs.google.com/file/d/0BxxUHhraTIHhcnNhUU9JRnc1YlU/edit>
> 
> For example if I wanted to find a page of the most important data, metadata, and transfer standards in order to make a recommendation, where would I go fishing?
> 
> Here is a working list to date:
> Short Title
> Title
> Domain
> Standardisation Body
> URL
> DCAT
> Data Catalog Vocabulary
> Metadata
> W3C
> http://www.w3.org/TR/vocab-dcat/ <http://www.w3.org/TR/vocab-dcat/>
> INSPIRE
> Infrastructure for Spatial Information in the European Community
> Spatial / Environmental
> EC
> http://inspire.ec.europa.eu/ <http://inspire.ec.europa.eu/>
> DCMI
> Dublin Core Metadata Initiative
> Metadata
> Dublin Core
> http://dublincore.org/documents/dcmi-terms/ <http://dublincore.org/documents/dcmi-terms/>
> SDMX
> Statistical Data and Metadata eXchange
> Statistical
> SDMX
> http://sdmx.org/ <http://sdmx.org/>
> DataCube
> Data Cube Vocabulary
> Statistical
> W3C
> http://www.w3.org/TR/vocab-data-cube/ <http://www.w3.org/TR/vocab-data-cube/>
> XBRL
> eXtensible Business Reporting Language
> Business
> XBRL
> http://www.xbrl.org/ <http://www.xbrl.org/>
> IANA
> IANA Media Types
> Media/File Types
> Internet Assigned Numbers Authority
> http://www.iana.org/assignments/media-types/media-types.xhtml <http://www.iana.org/assignments/media-types/media-types.xhtml>
> ISO 639
> Language codes
> Language
> ISO
> http://www.iso.org/iso/home/standards/language_codes.htm <http://www.iso.org/iso/home/standards/language_codes.htm>
> ISO 8601
> Date and time format
> Date/Time
> ISO
> http://www.iso.org/iso/home/standards/iso8601.htm <http://www.iso.org/iso/home/standards/iso8601.htm>
> WMS
> Map Service Interface Standard
> HTTP interface for requesting geo-registered map images from one or more distributed geospatial databases.
> OGC
> http://www.opengeospatial.org/standards/wms <http://www.opengeospatial.org/standards/wms>
> WFS
> Web Feature Service
> allows a client to retrieve and update geospatial data encoded in GML from multiple WFS
> OGC
> http://www.opengeospatial.org/standards/wfs <http://www.opengeospatial.org/standards/wfs>
> Gazetteer Service
> 
> Application Profile of the Web Feature Service Best Practice
> OGC
> https://portal.opengeospatial.org/files/?artifact_id=46964 <https://portal.opengeospatial.org/files/?artifact_id=46964>
> Open511
> 
> standard for road event information
> Open511.org
> http://open511.org/ <http://open511.org/>
> IATI
> International Aid Transparency Initiative
> framework for publishing information on development cooperation activities in a timely, comprehensive and forward-looking manner.
> IATI
> http://iatistandard.org/ <http://iatistandard.org/>
> OpenSpending
> Public spending data standard
> International standard for transaction-level spending data.
> Open Spending
> http://community.openspending.org/research/standard/technical/ <http://community.openspending.org/research/standard/technical/>
> GTFS-Realtime
> General Transit Feed Specification
> Feed specification that allows public transportation agencies to provide realtime updates about their fleet
> Google
> https://developers.google.com/transit/gtfs-realtime/ <https://developers.google.com/transit/gtfs-realtime/>
> GTFS
> General Transit Feed Specification
> common format for public transportation schedules and associated geographic information
> Google
> https://developers.google.com/transit/gtfs/ <https://developers.google.com/transit/gtfs/>
> Open Contracting
> 
> formalize how contracting data and documents can be published in an accessible, structured and repeatable way
> Open Contracting
> http://ocds.open-contracting.org/standard/r/1__0__RC/en/standard/intro/ <http://ocds.open-contracting.org/standard/r/1__0__RC/en/standard/intro/>
> 
> Popolo
> International open government data specifications
> specifications relating to the legislative branch
> Open North
> http://www.popoloproject.com/ <http://www.popoloproject.com/>
> thanks
> t
> --
> Tracey P. Lauriault
> http://www.maynoothuniversity.ie/progcity/contributors/tracey-p-lauriault/ <http://www.maynoothuniversity.ie/progcity/contributors/tracey-p-lauriault/>
> https://gcrc.carleton.ca/confluence/display/GCRCWEB/Lauriault <https://gcrc.carleton.ca/confluence/display/GCRCWEB/Lauriault>
> http://datalibre.ca/ <http://datalibre.ca/>
> @TraceyLauriault
> Skype: Tracey.P.Lauriault
> 
> _______________________________________________
> open-government mailing list
> open-government at lists.okfn.org <mailto:open-government at lists.okfn.org>
> https://lists.okfn.org/mailman/listinfo/open-government <https://lists.okfn.org/mailman/listinfo/open-government>
> Unsubscribe: https://lists.okfn.org/mailman/options/open-government <https://lists.okfn.org/mailman/options/open-government>
> 
> 
> 
> 
> --
> Rufus Pollock
> Founder and President  |  skype: rufuspollock  |  @rufuspollock <https://twitter.com/rufuspollock>
> Open Knowledge <http://okfn.org/> - see how data can change the world
> http://okfn.org/ <http://okfn.org/>  |  @okfn <http://twitter.com/OKFN>  |  Open Knowledge on Facebook <https://www.facebook.com/OKFNetwork>  |  Blog <http://blog.okfn.org/>
> 
> 
> --
> Tracey P. Lauriault
> http://www.maynoothuniversity.ie/progcity/contributors/tracey-p-lauriault/ <http://www.maynoothuniversity.ie/progcity/contributors/tracey-p-lauriault/>
> https://gcrc.carleton.ca/confluence/display/GCRCWEB/Lauriault <https://gcrc.carleton.ca/confluence/display/GCRCWEB/Lauriault>
> http://datalibre.ca/ <http://datalibre.ca/>
> @TraceyLauriault
> Skype: Tracey.P.Lauriault
> 
> _______________________________________________
> open-government mailing list
> open-government at lists.okfn.org <mailto:open-government at lists.okfn.org>
> https://lists.okfn.org/mailman/listinfo/open-government <https://lists.okfn.org/mailman/listinfo/open-government>
> Unsubscribe: https://lists.okfn.org/mailman/options/open-government <https://lists.okfn.org/mailman/options/open-government>
> 
> 
> 
> 
> --
> Stéphane Guidoin
> Director of product and service development
> Open North
> 514-862-0084 <tel:514-862-0084>
> http://opennorth.ca <http://opennorth.ca/>
> Twitter: @opennorth / @hoedic
> 
> 
> Inscrivez-vous <http://www.eventbrite.ca/e/canadian-open-data-summit-2015-sommet-canadien-des-donnees-ouvertes-2015-tickets-15458440612> au Sommet Canadien des Données Ouvertes!
> 
>  <http://www.opendatasummit.ca/>
> Register now <http://www.eventbrite.ca/e/canadian-open-data-summit-2015-sommet-canadien-des-donnees-ouvertes-2015-tickets-15458440612> for the Canadian Open Data Summit!
> 
> 
> 
> 
> --
> Tracey P. Lauriault
> http://www.maynoothuniversity.ie/progcity/contributors/tracey-p-lauriault/ <http://www.maynoothuniversity.ie/progcity/contributors/tracey-p-lauriault/>
> https://gcrc.carleton.ca/confluence/display/GCRCWEB/Lauriault <https://gcrc.carleton.ca/confluence/display/GCRCWEB/Lauriault>
> http://datalibre.ca/ <http://datalibre.ca/>
> @TraceyLauriault
> Skype: Tracey.P.Lauriault
> _______________________________________________
> CivicAccess-discuss mailing list
> CivicAccess-discuss at civicaccess.ca <mailto:CivicAccess-discuss at civicaccess.ca>
> http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss <http://lists.pwd.ca/mailman/listinfo/civicaccess-discuss>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/open-government/attachments/20150428/12752193/attachment-0003.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.okfn.org/pipermail/open-government/attachments/20150428/12752193/attachment-0003.sig>