[ECODP-dev] ODP : troubleshoot issues with production instance

[ISOARD Olivier (OP)]

Dear Bastiaan,

All the material (logs / dump / datasets) for the investigations have been collected and uploaded on TenForce FTP, under “/2013-03-06 Production issue material/”

I’m looking forward to receiving feedback from you.

PS: for info, we have decided to put on hold any new dataset we may receive for the daily ingestion process on production. The last import (yesterday) took more than 4 hours and finally crashed.

Best regards,

Olivier ISOARD
Projets Informatiques
_________________

Office des publications de l'Union européenne
Direction A – Core Business Services
Unité A3 - Projets Informatiques

2985 Luxembourg • LUXEMBOURG
Tel: +352 292942360 • Fax: +352 292944604
olivier.isoard at publications.europa.eu<mailto:olivier.isoard at publications.europa.eu>
http://publications.europa.eu<http://publications.europa.eu/>

From: Bastiaan Deblieck [mailto:bastiaan.deblieck at tenforce.com]
Sent: Wednesday, March 06, 2013 9:57 AM
To: ISOARD Olivier (OP)
Cc: Darwin Peltan; Bert Van Nuffelen (bert.van.nuffelen at tenforce.com); ZAJAC Agnieszka (OP); MEYER André (OP-EXT); Project list for EC ODP CKAN project; HOHN Norbert (OP); Dimitrios Mexis (Dimitrios.Mexis at tenforce.com); THUNUS François (OP); PAPPALARDO Roberto (OP)
Subject: Re: ODP : troubleshoot issues with production instance

Dear Olivier,

We have a virtual machine with an empty 07.01 ready for testing.

Best regards,
Bastiaan

On Wed, Mar 6, 2013 at 9:42 AM, ISOARD Olivier (OP) <Olivier.ISOARD at publications.europa.eu<mailto:Olivier.ISOARD at publications.europa.eu>> wrote:
Dear Tenforce, Okf,

I was finally a bit too optimistic yesterday afternoon when I reported that rolling back to 00.07.01 stabilized the production env. Since yesterday evening, OP is facing exactly the same troubles than with 00.07.05 (recurrent crashes and perf. Drop)

So it means that regarding the testing env to setup on 10F side, I propose to limit the software installation to 00.07.01, in order to be strictly identical to the current OP production. From our side, we’re going to directly collect the logs from PROD, also doing a dump of the PROD database (in 00.07.01). As agreed yesterday afternoon during our telco, I will also prepare a package with as several large dataset zip packages we recently received, for you to experiment the performances problems in RDF2CKAN and CKAN2TS.

I’ll keep you posted when the material for investigation will be ready.

Best regards,

Olivier ISOARD
Projets Informatiques
_________________

Office des publications de l'Union européenne
Direction A – Core Business Services
Unité A3 - Projets Informatiques

2985 Luxembourg • LUXEMBOURG
Tel: +352 292942360<tel:%2B352%20292942360> • Fax: +352 292944604<tel:%2B352%20292944604>
olivier.isoard at publications.europa.eu<mailto:olivier.isoard at publications.europa.eu>
http://publications.europa.eu<http://publications.europa.eu/>

From: ISOARD Olivier (OP)
Sent: Tuesday, March 05, 2013 11:04 AM
To: 'Darwin Peltan'
Cc: ZAJAC Agnieszka (OP); MEYER André (OP-EXT); Project list for EC ODP CKAN project; Bastiaan Deblieck (Bastiaan.Deblieck at tenforce.com<mailto:Bastiaan.Deblieck at tenforce.com>); Bert Van Nuffelen (bert.van.nuffelen at tenforce.com<mailto:bert.van.nuffelen at tenforce.com>); HOHN Norbert (OP)
Subject: ODP: troubleshoot issues with production instance

Hi Darwin,

Thanks for the proposal. We have anyway our weekly telco together with 10F this afternoon. Let’s discuss it there.

Just an update on the current situation:


>  The roll back to 00.07.01 is now completed. We applied then there the latest changed we managed last week: Load of DS and user account creation.

>  Yesterday after the roll back we still faced drop of prefs & troubles which made us doubting it may come directly from 00.07.05 patch.

>  We are still investigating other potential sources explaining the problem: VM, network, visitor traffic on the website…

>  What we noted already this morning is that Baidu, the Chinese search engine was crawling the ODP. We still checking whether or not this was also the case last week when the servers crashed several times.

>From you side, could you please have a look at what can be configured, for example in “robot.txt” in order to have a better fine tuning of what can be indexed by crawlers from what we can forbid access without preventing people finding ODP main pages.

Best regards,

Olivier ISOARD
Projets Informatiques
_________________

Office des publications de l'Union européenne
Direction A – Core Business Services
Unité A3 - Projets Informatiques

2985 Luxembourg • LUXEMBOURG
Tel: +352 2929<tel:%2B352%202929>42360 • Fax: +352 2929<tel:%2B352%202929>44604
olivier.isoard at publications.europa.eu<mailto:olivier.isoard at publications.europa.eu>
http://publications.europa.eu<http://publications.europa.eu/>

From: Darwin Peltan [mailto:darwin.peltan at okfn.org]
Sent: Tuesday, March 05, 2013 10:52 AM
To: ISOARD Olivier (OP); ZAJAC Agnieszka (OP); MEYER André (OP-EXT); Project list for EC ODP CKAN project
Subject: Call to troubleshoot issues with production instance

Dear Olivier

I just wanted to reiterate that David and I are available today to assist with troubleshooting.

We are available this morning or this afternoon apart from 15:30 to 17:30 CET.

A goto meeting session with your sysadmin may allow us to pinpoint the issue.

Best,

Darwin

Darwin Peltan
Project Manager

The Open Knowledge Foundation
http://www.okfn.org

Skype: darwinp
Twitter: @darwin



--
Bastiaan Deblieck
Semantic Technology Business Unit Manager
http://www.tenforce.com/
T: +32 16 31 48 60
M:+32 475 95 49 32
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.okfn.org/mailman/private/ecodp-dev/attachments/20130306/66ecd661/attachment.html>