[ECODP-dev] Investigation of Estat ingestion problems

Thu Apr 25 09:33:11 UTC 2013

Bert,

I've updated Jira with the recent package and log.

Regards,
Agnieszka

From: ZAJAC Agnieszka (OP)
Sent: Thursday, April 25, 2013 11:14 AM
To: 'Bert Van Nuffelen'
Cc: HOHN Norbert (OP); MEYER André (OP-EXT); Project list for EC ODP CKAN project; ISOARD Olivier (OP); THUNUS François (OP)
Subject: RE: Investigation of Estat ingestion problems

Hi Bert,

Thank you very much for a quick response.

Just to let you know: André has just installed 00.08.04 without problems ☺ and is in the process of ingesting data.

Regarding Estat: we noticed it occurs only with morning packages, the ones from afternoon are ingesting without problems.
Regarding Comp: André managed to ingest on test the latest package so I'm a bit confused about your last remark saying that something is still not correct.

@ André: when you're done with ingestion could you provide Bert with the logs requested?

Thank you for looking closer at this issue.

Regards,
Agnieszka

From: Bert Van Nuffelen [mailto:bert.van.nuffelen at tenforce.com]
Sent: Thursday, April 25, 2013 11:01 AM
To: ZAJAC Agnieszka (OP); Project list for EC ODP CKAN project
Cc: HOHN Norbert (OP)
Subject: Re: Investigation of Estat issue

Hi Agnieszka,
my investigation (jira ODP-239) has led to the conclusion that the publisher and its package for eurostat is not the problem, but it is a pure system operational problem.
In the attached logs the error on which RDF2CKAN halts is the last line:

2013-04-24 07:40:06,080 main ERROR org.ckan.demo.ecportal.rdf2ckan.http.JsonAPI UPLOAD ERROR: No JSON output found.
This means that CKAN/apache did not returned a respons to the request. Consequently the process stopped.
I could reproduce this problem one time and found then the next error message in the next apache logs

/applications/ecodp/users/ecodp/logs/httpd/error.error.log

[Tue Apr 23 10:43:03 2013] [error] Exception KeyError: KeyError(139770906142688,) in <module 'threading' from '/usr/lib64/python2.6/threading.pyc'> ignored
[Tue Apr 23 10:43:03 2013] [error] 2013-04-23 10:43:03,948 ERROR [ckan.controllers.api] Validation error: "{'__type': 'Validation Error', 'name': [u'That URL is already in use.']}"
[Tue Apr 23 10:43:05 2013] [error] 2013-04-23 10:43:05,146 ERROR [ckan.controllers.api] Validation error: "{'__type': 'Validation Error', 'name': [u'That URL is already in use.']}"
[Tue Apr 23 10:43:05 2013] [error] Exception AttributeError: "'NoneType' object has no attribute 'clearing'" in
This indicates an exception created in the python container processes that handle the CKAN requests. This is outside the RDF2CKAN tool and has to do with a combination of process configuration and CKAN code. Which will be investigated.

To accomodate the problem:
If it happens, lets restart the apache service and start in ingestion process again for the same rdf2ckan package.

Secondly, I would like to recieve the content of the above log file that covers the period when it happened so that we have more information to pinpoint the problem.
Because reproducing it is not so straightforward as my tests showed. I could only reproduce it once.
kind regards,
Bert

2013/4/25 ZAJAC Agnieszka (OP) <Agnieszka.ZAJAC at publications.europa.eu<mailto:Agnieszka.ZAJAC at publications.europa.eu>>
Hello Bert,

In the view of recurring problems with Estat updates would you have some time today to have a look at it so that I could come back to them as soon as possible.

Thanks in advance,

Agnieszka

Agnieszka Zając
Open Data Portal Section
_________________
Publications Office of the European Union
Dissemination and Reuse Directorate
Unit C1 Common Portal and Open Data Portal

L-2985 Luxembourg • LUXEMBOURG
Tel: +352 2929.42834<tel:%2B352%202929.42834>
agnieszka.zajac at publications.europa.eu<mailto:agnieszka.zajac at publications.europa.eu>
http://publications.europa.eu<http://publications.europa.eu/>
http://open-data.europa.eu

--
Bert Van Nuffelen

Semantic Technologies Software Architect at TenForce
www.tenforce.be<http://www.tenforce.be>

Bert.Van.Nuffelen at tenforce.com<mailto:Bert.Van.Nuffelen at tenforce.com>
Office: +32 (0)16 31 48 60
Mobile:+32 479 06 24 26
skype: bert.van.nuffelen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.okfn.org/mailman/private/ecodp-dev/attachments/20130425/63375235/attachment.html>