[ckan-dev] [Services-coord] Datastore and Filestore consolidation

Fri May 24 10:45:11 UTC 2013

On 24 May 2013 10:50, Dominik Moritz <dominik.moritz at okfn.org> wrote:
> On 24 May 2013 11:42, Adrià Mercader <adria.mercader at okfn.org> wrote:

>>> We'll disable data previews for data that is not in the datastore, because the dataproxy is too unreliable. All data previews will work via the datastore. The dataproxy will not be disabled but we won't use it any more.
>> * Will the datapusher be enabled by default? Will be a single central
>> instance that will be used by all instances? Will users need to deploy
>> their own?
>
> Yes, it will be enabled by default. There will be a config option
> "datapusher_url" which will by default point to our own instance. If
> the URL is empty, the datapusher is disabled. Of course people can
> deploy their own instance of the datapusher but they don't have to.
OK, I don't know how the datapushed works well enough, but I guess
that several instances using the same endpoint and sending hundreds of
CSV to push simultaneously might create some bottlenecks, but if this
happens we/them can deploy additional datapusher instances.

>> * How will CKAN talk to the datapusher (ie send the data)? is it a
>> separate extension or ckanext-datastorer?
>
> I started development in the datastorer extension so that we can
> easily backport it to 2.0. For 2.1+ it will become part of the
> datastore core extension. I think this makes sense because almost
> everyone wants the datapusher.
+1


>>
>>
>> Thanks
>>
>> Adrià
>>
>>
>> On 23 May 2013 22:41, Dominik Moritz <dominik.moritz at okfn.org> wrote:
>>> Hi everyone,
>>>
>>> After our discussions in the dev meeting and the (IMHO great) solution that we came up with, I updated the page at [1] to include what we decided. If you think something is not clear or should not be implemented that way, now is the time to do that ;-) I'm very happy that we had this discussion today and made a decision because this causes much confusion.
>>>
>>> I tried to break the spec down into individual issues. All issues have a new, temporal label which will be deleted once the consolidation is finished. Have a look at them at [2].
>>>
>>> There is a trello card for sprint 11 and we need to assign the issues in the next sprint. Even though there is a high number of issues for this consolidation, we probably won't need very much time to implement the spec because most issues are straightforward.
>>>
>>> Best wishes,
>>> Dominik
>>>
>>> [1] https://github.com/okfn/ckan/wiki/Spec:-DataStore-and-FileStore-Consolidation
>>> [2] https://github.com/okfn/ckan/issues?labels=Datastore+Filestore+consolidation
>>>
>>> ---
>>> Dominik Moritz
>>> CKAN developer  |  skype: d.moritz  |  @doobly_doo
>>> The Open Knowledge Foundation
>>> Empowering through Open Knowledge
>>> http://okfn.org/  |  @okfn  |  http://ckan.org  |  @CKANproject
>>>
>>>
>>> _______________________________________________
>>> Services-coord mailing list
>>> Services-coord at lists.okfn.org
>>> http://lists.okfn.org/mailman/listinfo/services-coord
>>> Unsubscribe: http://lists.okfn.org/mailman/options/services-coord