[ECODP-dev] quick fix for the production database - Release 00.07.50.SP1

Thu Mar 7 16:05:22 UTC 2013

Hi Bert,

Thanks for this new update.

>From our side, the run of the 2 scripts on our TEST environment was also conclusive and we now are managing the deployment and run of those scripts also on PROD, after having restored the last backup of prod 00.07.05. restore is done but cleaning (scripts) is still on going for the moment.

See u tomorrow to doing a new status via telco.

Best regards,

Olivier ISOARD
Projets Informatiques
_________________

Office des publications de l'Union européenne
Direction A – Core Business Services
Unité A3 - Projets Informatiques

2985 Luxembourg • LUXEMBOURG
Tel: +352 292942360 • Fax: +352 292944604
olivier.isoard at publications.europa.eu<mailto:olivier.isoard at publications.europa.eu>
http://publications.europa.eu<http://publications.europa.eu/>

From: Bert Van Nuffelen [mailto:bert.van.nuffelen at tenforce.com]
Sent: Thursday, March 07, 2013 4:04 PM
To: ISOARD Olivier (OP)
Cc: Project list for EC ODP CKAN project; ZAJAC Agnieszka (OP); HOHN Norbert (OP); MEYER André (OP-EXT)
Subject: Re: quick fix for the production database - Release 00.07.50.SP1

Hi

as promised a quick update from our side w.r.t. the upload of eurostat.
The processing finished properly.

I will trigger an additional run with some other eurostat zips to see if the behaviour is reoccurring.

Bert
2013/3/7 Bert Van Nuffelen <bert.van.nuffelen at tenforce.com<mailto:bert.van.nuffelen at tenforce.com>>
Hi Olivier,

it does not matter the release version. It is a valid operation for all CKAN databases.

kind regards,

Bert
2013/3/7 ISOARD Olivier (OP) <Olivier.ISOARD at publications.europa.eu<mailto:Olivier.ISOARD at publications.europa.eu>>

Hello Bert,

Thanks for this update. We already got the 2 scripts and are about to run them on our TEST env first. As this is pure db cleaning, it does not matter if we run it on 00.07.50 ?

Please note that we named this delivery 00.07.50.SP1. (I did the update and tagging in SVN myself, preferring you’re still focusing on the complementary investigations :) )

Best regards,

Olivier ISOARD
Projets Informatiques
_________________

Office des publications de l'Union européenne
Direction A – Core Business Services
Unité A3 - Projets Informatiques

2985 Luxembourg • LUXEMBOURG
Tel: +352 2929<tel:%2B352%202929>42360 • Fax: +352 2929<tel:%2B352%202929>44604
olivier.isoard at publications.europa.eu<mailto:olivier.isoard at publications.europa.eu>
http://publications.europa.eu<http://publications.europa.eu/>

From: Bert Van Nuffelen [mailto:bert.van.nuffelen at tenforce.com<mailto:bert.van.nuffelen at tenforce.com>]
Sent: Thursday, March 07, 2013 12:43 PM
To: ISOARD Olivier (OP); ZAJAC Agnieszka (OP); HOHN Norbert (OP); MEYER André (OP-EXT)
Cc: Project list for EC ODP CKAN project
Subject: quick fix for the production database

Dear,

our investigations has lead to the first conclusion. The regular upload of (new) data increased the CKAN database table activity to a high number.
Very lickely this causes the instability of the CKAN system.

On our test system (a release 00.07.01) we activated the production dump. During the indexing (a process step required after changing the database in the back)
the CKAN system halted. After emptying the activity table the indexing was restarted and it finished properly. After that we have run rdf2ckan with the latest zip you have provided.
At this moment the EEA and SANCO zips passed. Currently a Eurostat zip is running.

In the svn you find a script to clean the appropriate tables. This is a quick-win step towards having back a stable system.

https://webgate.ec.europa.eu/publications/svn/ODP/trunk/ckan/management_scripts/ckan_clear_activity_table.sh

Copy this script to $USERHOME/ckan_management and complete the necessary parameters. These are the same as for a CKAN backup.
•         TARGETBACKUPDIR
•         CKAN_PWD
•         POSTGRES_HOST
The script will first backup the table and afterwards empty the tables.

The execution on our site costs +- 45 minutes and resulted in a file of 7.6 GB.

After the script execution, you have to trigger for this time the reindexing of the database.
In $USERHOME/ckan_management

     ./ckan_reindex.sh

(It  costs +- 1hour on our site)

We keep you informed about the outcome of the Eurostat test.

ps. I inserted the description of the new script also in the OM on section 7.10.

kind regards,

Bert

--
Bert Van Nuffelen

Semantic Technologies Software Architect at TenForce
www.tenforce.be<http://www.tenforce.be>

Bert.Van.Nuffelen at tenforce.com<mailto:Bert.Van.Nuffelen at tenforce.com>
Office: +32 (0)16 31 48 60<tel:%2B32%20%280%2916%2031%2048%2060>
Mobile:+32 479 06 24 26<tel:%2B32%20479%2006%2024%2026>
skype: bert.van.nuffelen

--
Bert Van Nuffelen

Semantic Technologies Software Architect at TenForce
www.tenforce.be<http://www.tenforce.be>

Bert.Van.Nuffelen at tenforce.com<mailto:Bert.Van.Nuffelen at tenforce.com>
Office: +32 (0)16 31 48 60<tel:%2B32%20%280%2916%2031%2048%2060>
Mobile:+32 479 06 24 26<tel:%2B32%20479%2006%2024%2026>
skype: bert.van.nuffelen

--
Bert Van Nuffelen

Semantic Technologies Software Architect at TenForce
www.tenforce.be<http://www.tenforce.be>

Bert.Van.Nuffelen at tenforce.com<mailto:Bert.Van.Nuffelen at tenforce.com>
Office: +32 (0)16 31 48 60
Mobile:+32 479 06 24 26
skype: bert.van.nuffelen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.okfn.org/mailman/private/ecodp-dev/attachments/20130307/327bd176/attachment.html>