[ECODP-dev] Fwd: FW: [ODP] 09.00 Installation update - general status - comments

[Bert Van Nuffelen]

Hi Darwin, John and David,

Can you have a look to the performance. Currentltly release 00.09.00 is 80
times slower on ingest (from 250ms to 20s).
That is a serious regression. Would it be possible to give a short status
for this afternoon (call with PO at 1400h Brussels time) on the
investigation and what is the action plan to get further improvements.

best regards,

Bert

---------- Forwarded message ----------
From: ZAJAC Agnieszka (OP) <Agnieszka.ZAJAC at publications.europa.eu>
Date: 2013/7/8
Subject: FW: [ODP] 09.00 Installation update - general status - comments
To: "Bert.Van.Nuffelen at tenforce.com" <Bert.Van.Nuffelen at tenforce.com>
Cc: "MEYER André (OP-EXT)" <Andre.MEYER at ext.publications.europa.eu>,
"SABETE Vafa (OP)" <Vafa.SABETE at publications.europa.eu>, "ISOARD Olivier
(OP)" <Olivier.ISOARD at publications.europa.eu>, "HOHN Norbert (OP)" <
Norbert.HOHN at publications.europa.eu>


 Hello Bert,****

** **

Could you please have a look at Andre's observation on the install and
provide us with the feedback?****

** **

I put in copy Vafa Sabete who is Olivier's backup for this project.****

** **

Thank you in advance.****

** **

Regards,****

Agnieszka****

** **

*From:* MEYER André (OP-EXT)
*Sent:* Monday, July 08, 2013 9:52 AM
*To:* ZAJAC Agnieszka (OP); SABETE Vafa (OP)
*Cc:* HOHN Norbert (OP); BOUSSERT Philippe (OP); ZADRA Julien (OP-EXT)
*Subject:* [ODP] 09.00 Installation update - general status - comments****

** **

Hello Agnieszka, Vafa,****

** **

*Installation status:*

The installation of 09.00 is officially *finished* and all the required
steps of the installation manual have been performed. An early check of the
functionalities looks ok for the basic functions but I do see already one
major problem. ****

** **

*Problems:*

**-          **The old rdf import-script rdf2ckan.sh takes around 2hrs on
the current production & test environment in version 08.04 for a standard
ESTAT file. The newly revised and supposedly improved version of said
script took over 14 hours to import the latest ESTAT file, gave a lot of
warnings for some datasets that were not properly validated and gave an
error for 33 datasets which were not imported at all.****

**-          **I noticed that the current postgres database on the newly
installed system with a single ESTAT ingestion has a size of double of that
of the production system which gets updates twice a day for months … If the
size increases this drastically we will have to revise the configuration of
the machines and dramatically increase disk storage.****

**-          **I thought this release was supposed to speed up the portal,
if this is still the case, then this goal was not achieved as it is still
awfully slow without the static caching of the BigIP. If no performance
increase was planned, than please ignore this.****

**-          **The configured cron jobs seem to produce logfiles called
batch_*.log in the home directory of the ecodp user. This is not
acceptable, log files should be created in an appropriate log folder and
not spam the root home user directory.****

**-          **The Tomcat6 seems to be unstable. Going to the data tab
often results in timeouts because the tomcat is not responding, restarting
the service doesn’t help. Sometimes it works, sometimes it doesn’t. No
indictator in the logfiles for any problems or errors on the tomcat side.
Due to this tomcat problem I haven’t been able to create a publisher
account so I have not yet tested anything related to user creation. *Additional
info*: Accessing the tab as an anonymous user works almost always,
accessing the data tab as logged in user api almost always results in a
timeout.****

** **

*Comments:*

The installation itself was sufficiently documented and track changes for
errors or unclear paragraphs have been added. I have uploaded the document
to the JIRA ticket. ****

** **

There is however one problem with the way configurations are handled. As of
this version the PO takes responsibility of certain middleware like the
apache webserver or the postgres database server. With this responsibility
we agreed that we are the ones configuring those software server but
TenForce required in their installation to replace our configurations of
said services by their own config. ****

This is not ok and needs to be changed before an installation in prod can
be done. TenForce needs to specify what has to be added or set in a
configuration file and we will make it happen, but not completely replace
ours. Preferably they should provide a file to append, e.g. for apache a
file that goes into the conf.d folder without touching our httpd.conf at
all. Same goes for nginx of course.****

** **

*André Meyer*

Application Team - Integration engineer****

*_**__**_**__**_**__**__**_**__**_**__***

* *

*Publications Office of the European Union *

*Unit A4 - Infrastructure and IT Security Systems***

* *

*Halian S.à.r.l.  (under contract with the Publications Office)*****

( (+352) 2929-42442****

+  andre.meyer at ext.publications.europa.eu****

** **



-- 
Bert Van Nuffelen

Semantic Technologies Software Architect at TenForce
www.tenforce.be

Bert.Van.Nuffelen at tenforce.com
Office: +32 (0)16 31 48 60
Mobile:+32 479 06 24 26
skype: bert.van.nuffelen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.okfn.org/mailman/private/ecodp-dev/attachments/20130709/3dca35c7/attachment.html>