[ECODP-dev] Fwd: FW: [ODP] 09.00 Installation update - general status - comments

Mon Jul 8 08:50:09 UTC 2013

Hi all,

a "formal reply" on the release 00.09.00

@ John,

can you provide insight on the database grow?
Also any idea on the tomcat issue?

ps. the loading of RDF2CKAN is slower due to more validation ...

best,

Bert

---------- Forwarded message ----------
From: ZAJAC Agnieszka (OP) <Agnieszka.ZAJAC at publications.europa.eu>
Date: 2013/7/8
Subject: FW: [ODP] 09.00 Installation update - general status - comments
To: "Bert.Van.Nuffelen at tenforce.com" <Bert.Van.Nuffelen at tenforce.com>
Cc: "MEYER André (OP-EXT)" <Andre.MEYER at ext.publications.europa.eu>,
"SABETE Vafa (OP)" <Vafa.SABETE at publications.europa.eu>, "ISOARD
Olivier (OP)" <Olivier.ISOARD at publications.europa.eu>, "HOHN Norbert
(OP)" <Norbert.HOHN at publications.europa.eu>

Hello Bert,

Could you please have a look at Andre's observation on the install and
provide us with the feedback?

I put in copy Vafa Sabete who is Olivier's backup for this project.

Thank you in advance.

Regards,

Agnieszka

From: MEYER André (OP-EXT)
Sent: Monday, July 08, 2013 9:52 AM
To: ZAJAC Agnieszka (OP); SABETE Vafa (OP)
Cc: HOHN Norbert (OP); BOUSSERT Philippe (OP); ZADRA Julien (OP-EXT)
Subject: [ODP] 09.00 Installation update - general status - comments

Hello Agnieszka, Vafa,

Installation status:

The installation of 09.00 is officially finished and all the required
steps of the installation manual have been performed. An early check
of the functionalities looks ok for the basic functions but I do see
already one major problem.

Problems:

-          The old rdf import-script rdf2ckan.sh takes around 2hrs on
the current production & test environment in version 08.04 for a
standard ESTAT file. The newly revised and supposedly improved version
of said script took over 14 hours to import the latest ESTAT file,
gave a lot of warnings for some datasets that were not properly
validated and gave an error for 33 datasets which were not imported at
all.

-          I noticed that the current postgres database on the newly
installed system with a single ESTAT ingestion has a size of double of
that of the production system which gets updates twice a day for
months … If the size increases this drastically we will have to revise
the configuration of the machines and dramatically increase disk
storage.

-          I thought this release was supposed to speed up the portal,
if this is still the case, then this goal was not achieved as it is
still awfully slow without the static caching of the BigIP. If no
performance increase was planned, than please ignore this.

-          The configured cron jobs seem to produce logfiles called
batch_*.log in the home directory of the ecodp user. This is not
acceptable, log files should be created in an appropriate log folder
and not spam the root home user directory.

-          The Tomcat6 seems to be unstable. Going to the data tab
often results in timeouts because the tomcat is not responding,
restarting the service doesn’t help. Sometimes it works, sometimes it
doesn’t. No indictator in the logfiles for any problems or errors on
the tomcat side. Due to this tomcat problem I haven’t been able to
create a publisher account so I have not yet tested anything related
to user creation. Additional info: Accessing the tab as an anonymous
user works almost always, accessing the data tab as logged in user api
almost always results in a timeout.

Comments:

The installation itself was sufficiently documented and track changes
for errors or unclear paragraphs have been added. I have uploaded the
document to the JIRA ticket.

There is however one problem with the way configurations are handled.
As of this version the PO takes responsibility of certain middleware
like the apache webserver or the postgres database server. With this
responsibility we agreed that we are the ones configuring those
software server but TenForce required in their installation to replace
our configurations of said services by their own config.

This is not ok and needs to be changed before an installation in prod
can be done. TenForce needs to specify what has to be added or set in
a configuration file and we will make it happen, but not completely
replace ours. Preferably they should provide a file to append, e.g.
for apache a file that goes into the conf.d folder without touching
our httpd.conf at all. Same goes for nginx of course.

André Meyer

Application Team - Integration engineer

_________________

Publications Office of the European Union

Unit A4 - Infrastructure and IT Security Systems

Halian S.à.r.l.  (under contract with the Publications Office)

( (+352) 2929-42442

+  andre.meyer at ext.publications.europa.eu

--
Bert Van Nuffelen

Semantic Technologies Software Architect at TenForce
www.tenforce.be

Bert.Van.Nuffelen at tenforce.com
Office: +32 (0)16 31 48 60
Mobile:+32 479 06 24 26
skype: bert.van.nuffelen