[datacatalogs] DCIP - Bulk catalog access option: access to all datasets in a single single file

Thu Nov 22 13:45:41 UTC 2012

Hi All,

I'd like to discuss a substantive addition to DCIP in for the of a
Bulk catalog access option:

<https://github.com/dataprotocols/data-catalog-spec/issues/7>

I've inlinded some of the detail below for ease of discussion.

Regards,

Rufus

## Bulk catalog access option: access to all datasets in a single single file

This proposes a substantive change to the DCIP spec. Key features

* Provision of all datasets in a single file
* Format would be a simple list of each dataset with each dataset
serialized as in DCIP
  * with default of JSON but options for n3 etc
* Location likely specified by a meta field in head as per API location

This option could be provided both in addition to and as substitute
for the full API option.

Benefits:

* Catalog operators:
  * simpler and easier to do than a full API. Very easy to get started.
* Consumers:
  * All datasets in one go - no need to walk through the API

Possible problems:

* Catalog operators:
  * For larger catalogs the file is very large. Inefficient both for
creation, storage, and transmission.
* Consumers:
 * File could be large if catalog is large.
 * to get whole file even if only one dataset has changed