[ckan-discuss] question on datastore, filestore and data/metadata storage in CKAN

Tue Nov 12 22:22:07 UTC 2013

Hi Elena,

Overall structure:
- DB (postgres) is mainly used to store metadata (dataset, resources) and
the overall structure of CKAN (organizations, groups, users, history of
modifications, etc.)

- The "default" behaviour of CKAN is to point the resource where it is.
(thus, a catalog)

- Most of the implementations will use the "Filestore" to store locally
some data. When you create a dataset, if the filestore is enabled, you can
upload a file. You can also push some files via the CKAN API.

- Harvesters use the default behaviour: collect the metadata and leave the
data wherever it is. As someone said recently, the archiver extension could
be use to download them but I did not try it yet.

- If you activate the datastore, the DB (postgres) can be used to store
atomized data (usually it will be in a different DB that the main
database... could even be on a different server.

- SOLR leaves it own life.

Hope it answers your questions.

On Tue, Nov 12, 2013 at 11:14 AM, Elena Camossi <
elena.camossi at ext.jrc.ec.europa.eu> wrote:

> Hi everyone,
>
> I have a basic question that will probably sound silly, but I'm getting
> more
> and more confused on how CKAN organizes physically datasets...
>
> Question is: What is exactly the data/metadata storage model CKAN uses?
>
> To be more clear, are data from datasets always stored with metadata, or
> just metadata are locally stored in the CKAN instance, and the dataset can
> remain stored somewherelse? (I'm thinking of the case of a harvested
> dataset, not to a dataset which is inserted from scratch).
> What is it actually stored in the backend postgres database? Just metadata,
> and data go eventually to the file system or remain remote? SOLR, in this
> architecture, index both data and metadata?
>
> Finally, what is the exact function of the filestore? Is it used to store
> locally the data? Or just the metadata?
> Does the datastore duplicate data/metadata already included in the
> filestore?
>
> Thanks a lot for putting some light on this...
>
> Cheers,
> -Elena
>
>
>
>
>
>
> _______________________________________________
> ckan-discuss mailing list
> ckan-discuss at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-discuss
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-discuss
>

-- 
Stéphane Guidoin
Director, Transportation
Open North
514-862-0084
http://opennorth.ca
Twitter: @opennorth / @hoedic
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-discuss/attachments/20131112/fcbd196f/attachment.htm>