[ckan-dev] Relationship between resource and datastore

Fri Jan 4 14:34:44 UTC 2013

Hi Lasse,

On 2 Jan 2013, at 12:07, Lasse Vestergaard <ibbernik at gmail.com> wrote:

> Hi again.
> 
>> This was also how I thought it, but how do I *only* use the datastore? I
>>> might be misinterpreting you, but aren't you saying that I can have a
>>> datastore as a resource? This way I don't need to upload a CSV file, and
>> I
>>> can just create a datastore with a specifik structure? Or do you just
>> mean
>>> that I shouldn't use the CSV file I uploaded?
>> 
>> To be honest, there is no clean way at the moment. You will need to create
>> a new resource with a temporary url and then create the datastore table
>> manually through the API. Then change the url to point to the datastore (or
>> a url that points to the converter that returns CSV). In this case you
>> won't need the original CSV file but you will have to do a lot of things
>> manually.
>> Alternatively, you can add the original resource as a CSV, let the
>> datastorer import it and then change the url of the resource in the
>> datastore. The second option will have the advantage, that the data will
>> already be in the datastore and is probably the preferred option.
>> 
> 
> I have been looking into this, and there are some things I don't
> understand. I get the first, of the above solutions, but the latter seems
> impossible to do. The problem already arises when I create a new CSV
> resource. It seems that the datastorer won't import the CSV file because
> there is no existing datastore. This makes perfect sense, as I perceive it,
> because a datastore is always "owned" by a resource, according to the
> documentation (you need a resource id to create a datastore). This means
> that it doesn't make sense that the datastorer tries to update a datastore
> with data from the CSV file - there is not yet created any datastore
> because the resource has not been created until now.

The datastorer can create the table in the datastore [1]. What I wanted to say with the second option is that you create a resource just as you would do it normally. You also set the resource URL to the original CSV file and the type to CSV. Wait until the datastorer has created the table in the datastore and imported the whole content. Then change the url of the CKAN resource to the datastore endpoint (i.e. http://ckan.org/api/action/datastore_search?resource_id=0c7eb0c3-e655-437e-bed5-91e4a725aecb) or the url to a converter that converts the datastore to CSV. It worked for me. If you don't see the preview, wait until this pull request has been accepted [2].

> 
> Furthermore I can't change the url of the resource in the datastore. There
> is only registered an id of the resource in the datastore (and that is the
> name of the table, as I perceive it). When a resource has been created,
> then I can't change it - I cannot change a link resource to an uploaded
> file, so that I can use the same resource id.

Don't change the datastore URL, change the resource URL/ link in CKAN.

> 
> Thanks in advance.
> 
> Lasse
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev

[1] https://github.com/okfn/ckanext-datastorer/blob/master/ckanext/datastorer/tasks.py#L152
[2] https://github.com/okfn/ckan/pull/260