[ECODP-dev] Production database size

[John Glover]

Hi Bert,

No, purge_revision_history and purge_package_extra_revision (spelled with
underscores when accessed through the API) are two separate calls. The
former purges old data from the resource_revision database table, and the
latter removes old data from package_extra_revision. Both tables store old
versions of data relating to datasets, and so both should be run regularly
due to the large number of daily updates.

Regards,
John


On 26 September 2013 17:06, Bert Van Nuffelen <
bert.van.nuffelen at tenforce.com> wrote:

> Hi John,
>
> thanks for the investigation.
>
> Is the api call
>
> api.purge=http://localhost:8008/data/api/action/purge_revision_history
>
> equal to the purge-package-extra-revision paster command?
>
> kind regards,
>
> Bert
>
>
> 2013/9/26 John Glover <john.glover at okfn.org>
>
>> Hi Bert,
>>
>> I have had a look at the PO database dump. The sizes of the largest
>> tables are as follows:
>>
>> public.kombu_message                           | 15 GB
>> pg_toast.pg_toast_78586                        | 3662 MB
>> public.task_status                             | 1548 MB
>> public.package_extra_revision                  | 811 MB
>> public.task_status_entity_id_task_type_key_key | 670 MB
>> public.task_status_pkey                        | 458 MB
>>
>> Apart from package_extra_revision, the other large tables all refer to
>> tables used by the old CKAN extensions (such as ckanext-archiver,
>> ckanext-datastorer and ckanext-qa). These should not be installed any more.
>>
>> In the previous release, we supplied a paster command to purge the
>> task_status and kombu_message tables:
>>
>> paster --plugin=ckanext-ecportal ecportal purge-task-data -c <config>
>>
>> Running this should get rid of most of this unnecessary data (although
>> you will probably to do a VACUUM in postgres afterwards to reclaim space).
>> This should only have to be run once if the extensions have been correctly
>> uninstalled. If these tables continue to grow, then at least one of the
>> extensions is still installed. So, make sure that none of the following
>> appear in the ckan.plugins config list: archiver, qa, datastorer.
>>
>> From looking at the timestamps of the task_status entries, it would seem
>> at least 1 of these extensions is still installed, as the last write was by
>> the datastorer on the 4th September.
>>
>>  The package revision table is also quite large, but this is expected
>> due to the daily updating of all Eurostat packages. This should be
>> regularly cleared with the purge-package-extra-revision paster command as
>> in previous releases.
>>
>> Regards,
>> John
>>
>> _______________________________________________
>> Ecodp-dev mailing list
>> Ecodp-dev at lists.okfn.org
>> http://lists.okfn.org/mailman/listinfo/ecodp-dev
>>
>>
>
>
> --
> Bert Van Nuffelen
>
> Semantic Technologies Software Architect at TenForce
> www.tenforce.be
>
> Bert.Van.Nuffelen at tenforce.com
> Office: +32 (0)16 31 48 60
> Mobile:+32 479 06 24 26
> skype: bert.van.nuffelen
>
> _______________________________________________
> Ecodp-dev mailing list
> Ecodp-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ecodp-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.okfn.org/mailman/private/ecodp-dev/attachments/20130926/0d2bc3f8/attachment.html>