[ECODP-dev] Production database size

Mon Sep 30 13:06:04 UTC 2013

Hi Dimitrios,

This is covered in the first email in this thread. Most of the data (in the
dump that I looked at) is in the tables that are used by ckanext-archiver,
ckanext-datastorer and/or ckanext-qa. All of these processes produce data
for each resource update. Some unnecessary data is also in the resource
revision tables.

Yes, 28 GB is far too large for a site with that number of datasets.

Regards,
John

On 30 September 2013 14:53, Dimitrios Mexis <dimitrios.mexis at tenforce.com>wrote:

>  Hello John,
>
> We have a back up from PO two weeks almost, and it has 24GB of data, while
> now PO says the db is 28GB. That is 4GB difference obviously.
>
> What module or what functionality is the source of producing such amount
> of data?
>
> Regards
> Dimitrios
>
>
> On 26/09/2013 13:46, John Glover wrote:
>
> Hi Bert,
>
>  I have had a look at the PO database dump. The sizes of the largest
> tables are as follows:
>
>  public.kombu_message                           | 15 GB
> pg_toast.pg_toast_78586                        | 3662 MB
> public.task_status                             | 1548 MB
> public.package_extra_revision                  | 811 MB
> public.task_status_entity_id_task_type_key_key | 670 MB
> public.task_status_pkey                        | 458 MB
>
>  Apart from package_extra_revision, the other large tables all refer to
> tables used by the old CKAN extensions (such as ckanext-archiver,
> ckanext-datastorer and ckanext-qa). These should not be installed any more.
>
>  In the previous release, we supplied a paster command to purge the
> task_status and kombu_message tables:
>
>  paster --plugin=ckanext-ecportal ecportal purge-task-data -c <config>
>
>  Running this should get rid of most of this unnecessary data (although
> you will probably to do a VACUUM in postgres afterwards to reclaim space).
> This should only have to be run once if the extensions have been correctly
> uninstalled. If these tables continue to grow, then at least one of the
> extensions is still installed. So, make sure that none of the following
> appear in the ckan.plugins config list: archiver, qa, datastorer.
>
>  From looking at the timestamps of the task_status entries, it would seem
> at least 1 of these extensions is still installed, as the last write was by
> the datastorer on the 4th September.
>
>  The package revision table is also quite large, but this is expected due
> to the daily updating of all Eurostat packages. This should be regularly
> cleared with the purge-package-extra-revision paster command as in previous
> releases.
>
>  Regards,
> John
>
>
> _______________________________________________
> Ecodp-dev mailing listEcodp-dev at lists.okfn.orghttp://lists.okfn.org/mailman/listinfo/ecodp-dev
>
>
>
> _______________________________________________
> Ecodp-dev mailing list
> Ecodp-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ecodp-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.okfn.org/mailman/private/ecodp-dev/attachments/20130930/6e72d0d8/attachment.html>