[data-protocols] Extending the JSON Table Schema for scientific applications

[Tom Aldcroft]

Hi -

I am working on a standard for text data tables for use in science
applications, in particular in the context of the Python astropy package (
http://astropy.org).  This includes support for reading and writing ASCII
tables in various formats that are common in astronomy (
http://astropy.readthedocs.org/en/latest/io/ascii/index.html).

The draft proposal I submitted baselines an approach that is very similar
to the Tabular Data Package standard in data-protocols.  After discussion
we are very interested in adopting the JSON Table Schema from Data
Protocols.  See:

 https://github.com/taldcroft/astropy-APEs/blob/ape6/APE6.rst
 https://github.com/astropy/astropy-APEs/pull/7

The question I have is to what extent your organization would be interested
in extending the JSON Table Schema standard to include more optional
elements that would be common in science applications.  As a rough outline,
we would like to see:

At the top level:

- "schema-name": optional, name (or URL) of detailed schema which allows
for interpretation and validation of the table and header content
- "keywords": optional, list of keyword structures which includes {"name"
(req'd), "value" (req'd), "unit" (optional), "description" (optional)
- "comments": optional, list of general string comments
- "history": optional, list of records indicating processing history
(providing provenance of the data in the file).

In the standard "fields" specification:

 - "unit": optional, specifies physical unit of data (e.g m/s)
 - "dtype": optional, detailed data type which indicates a specific binary
representation (e.g. float32, int8, uint8, float64).  This can be important
in numerical applications and is required to round-trip a data structure to
file and back with minimal information loss.

At this point I'm mostly interested in general discussion of whether it's
worth opening a pull request to extend the JSON Table Schema in the
direction I've outlined, with details TBD.

Thanks,
Tom Aldcroft
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.okfn.org/pipermail/data-protocols/attachments/20140415/0ed6127f/attachment.html>