[ckan-dev] ckan-dev Digest, Vol 33, Issue 11

Thu Jul 4 13:21:12 UTC 2013

Hi Winny, no idea why the unsubscribe form wouldn't work for you but
I've removed you now.

Mark

On 04/07/2013, Winny de Jong <WdeJong at ncdo.nl> wrote:
> Could you please take me off the list; it looks like the unsubscribe form
> isn't working... Thanks! Winny
>
> Op 4 jul. 2013 om 13:32 heeft "ckan-dev-request at lists.okfn.org"
> <ckan-dev-request at lists.okfn.org> het volgende geschreven:
>
>> Send ckan-dev mailing list submissions to
>>    ckan-dev at lists.okfn.org
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>>    http://lists.okfn.org/mailman/listinfo/ckan-dev
>> or, via email, send a message with subject or body 'help' to
>>    ckan-dev-request at lists.okfn.org
>>
>> You can reach the person managing the list at
>>    ckan-dev-owner at lists.okfn.org
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of ckan-dev digest..."
>>
>>
>> Today's Topics:
>>
>>   1. Perfomance work, review requested (Ian Ward)
>>   2. Re: Perfomance work, review requested (Toby Dacre)
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Wed, 3 Jul 2013 17:41:02 -0400
>> From: Ian Ward <ian at excess.org>
>> Subject: [ckan-dev] Perfomance work, review requested
>> To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
>> Message-ID:
>>    <CAA3rUNexJZ3+VasUoL=usZh9TmGGj49mkn39ZAku-9UgpLDDYw at mail.gmail.com>
>> Content-Type: text/plain; charset=ISO-8859-1
>>
>> I would like to know how likely some changes are to being accepted
>> into CKAN. I'd also like some help from people that know the code
>> better in resolving some of the test failures.
>>
>> These changes are related to leaning on SOLR a little more for better
>> performance of package_show: http://github.com/okfn/ckan/pull/1079
>>
>> 53a4e5c (1.9x faster): use the data_dict in solr instead of dictizing
>> the models from the DB, when possible. I might not have the "when
>> possible" part correct here.
>>
>> 973bb8c (4.9x faster): store the package_show-schema validated version
>> in SOLR data_dict to reduce the work when calling package_show. This
>> moves some work to the when packages are updated and created, but I
>> expect that this penalty can be removed because we probably have
>> already just generated a validated version of the package (no
>> optimization has been done here yet).
>>
>> f2a4822 (8x faster): allow actions to return a json string instead of
>> decoded json data and pass that directly to the caller, skipping the
>> work decoding json just to re-encode it on the other end. This might
>> not be the best implementation, but it does offer an extra 60%
>> improvement, and could be useful for other API calls too.
>>
>>
>> One change to package_list is awating review:
>> https://github.com/okfn/ckan/pull/1042
>>
>> The commits currently on the PR (10x faster) have been merged
>>
>> The remaining change
>> https://github.com/okfn/ckan/pull/1042#issuecomment-20067963 (18x
>> faster) involves using raw SQL for the query. I'm happy to put this
>> code as a method on the model, but the discussion side-tracked onto
>> what sort of parameters that method should have. Would a patch that
>> has no parameters and just returns all the ids be acceptable for now?
>> I'd like to have the parameter discussion, but that seems like a
>> separate thing.
>>
>>
>> Getting some feedback would be really helpful. I'm going to be looking
>> at a number of other calls that are really slow on our system, and I'd
>> like to know what sorts of things I can get away with :-)
>>
>> Ian
>>
>>
>>
>> ------------------------------
>>
>> Message: 2
>> Date: Thu, 4 Jul 2013 10:12:50 +0100
>> From: Toby Dacre <toby.okfn at gmail.com>
>> Subject: Re: [ckan-dev] Perfomance work, review requested
>> To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
>> Message-ID:
>>    <CAKZ3V-NVnF8_b-TiQnOm4YKvZJWwHErPEzNUhgzSrHQLR==RzA at mail.gmail.com>
>> Content-Type: text/plain; charset=ISO-8859-1
>>
>> On 3 July 2013 22:41, Ian Ward <ian at excess.org> wrote:
>>> I would like to know how likely some changes are to being accepted
>>> into CKAN. I'd also like some help from people that know the code
>>> better in resolving some of the test failures.
>>>
>>> These changes are related to leaning on SOLR a little more for better
>>> performance of package_show: http://github.com/okfn/ckan/pull/1079
>>>
>>> 53a4e5c (1.9x faster): use the data_dict in solr instead of dictizing
>>> the models from the DB, when possible. I might not have the "when
>>> possible" part correct here.
>>>
>>> 973bb8c (4.9x faster): store the package_show-schema validated version
>>> in SOLR data_dict to reduce the work when calling package_show. This
>>> moves some work to the when packages are updated and created, but I
>>> expect that this penalty can be removed because we probably have
>>> already just generated a validated version of the package (no
>>> optimization has been done here yet).
>>>
>>> f2a4822 (8x faster): allow actions to return a json string instead of
>>> decoded json data and pass that directly to the caller, skipping the
>>> work decoding json just to re-encode it on the other end. This might
>>> not be the best implementation, but it does offer an extra 60%
>>> improvement, and could be useful for other API calls too.
>>
>> The main thing is backward compatibility to not break things.  Your PR
>> 1079 seems to do this - I've only had a quick skim.  One thing I might
>> change is `json_string` and use `return_type` as that feels like it
>> has more flexibility but others may disagree there
>>
>>
>>>
>>> One change to package_list is awating review:
>>> https://github.com/okfn/ckan/pull/1042
>>>
>>> The commits currently on the PR (10x faster) have been merged
>>>
>>> The remaining change
>>> https://github.com/okfn/ckan/pull/1042#issuecomment-20067963 (18x
>>> faster) involves using raw SQL for the query. I'm happy to put this
>>> code as a method on the model, but the discussion side-tracked onto
>>> what sort of parameters that method should have. Would a patch that
>>> has no parameters and just returns all the ids be acceptable for now?
>>> I'd like to have the parameter discussion, but that seems like a
>>> separate thing.
>>
>> the two things are independent so can be done separately imho
>>
>>>
>>> Getting some feedback would be really helpful. I'm going to be looking
>>> at a number of other calls that are really slow on our system, and I'd
>>> like to know what sorts of things I can get away with :-)
>>
>> I think the main thing is to make small pull requests as these are
>> quicker/easier for us to review and merge and less likely to get stuck
>> in the queue.  Stats showing improvements are really helpful too.
>>
>> I think anything that helps speed up the ckan beast will be
>> appreciated so long as it does not over-complicate the code.
>>
>> The tests can be a pain irc is probably the best place to request help
>> with them
>>
>> Thanks
>>
>>>
>>> Ian
>>>
>>> _______________________________________________
>>> ckan-dev mailing list
>>> ckan-dev at lists.okfn.org
>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>
>>
>>
>> --
>> Toby Dacre
>>
>> The Open Knowledge Foundation
>>
>> Empowering through Open Knowledge
>> http://okfn.org/  |  @okfn
>>
>>
>>
>> ------------------------------
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: http://lists.okfn.org/mailman/optionss/ckan-dev
>>
>>
>> End of ckan-dev Digest, Vol 33, Issue 11
>> ****************************************
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>

-- 
Business development and user engagement manager
The Open Knowledge Foundation
Empowering through Open Knowledge
http://okfn.org/  |  @okfn  |  http://ckan.org  |  @CKANproject