[CKAN-support] [Open Knowledge Foundation] Update: ODP-383

Wed Mar 26 17:57:09 UTC 2014

##- Please type your reply above this line -##

[Open Knowledge Foundation] Update: ODP-383

You are registered as a CC on this support request (136). Reply to this email to add a comment to the request.

----------------------------------------------

CKAN Support, Mar 26 17:57

Hi Bert, I can't say based on the log file snippet that you provided which CKAN module the error comes from. I don't recognise the error message (and it doesn't look like a CKAN error to me). If CKAN has crashed, then the log file on the server should contain a full Python traceback which would tell you which lines of which files were on the stack when it crashed. From that, I could tell you which CKAN module crashed. So I would need the full log file.

----------------------------------------------

Bert Van Nuffelen, Mar 14 15:49

Hi Adam,

the ESTAT package is a RDF2CKAN package containing about 5500 datasets in
RDF format. The RDF2CKAN tool will interprete those RDF dataset
descriptions and turn them into json to be sent as an update to the EU ODP
CKAN.

So far my tests show on our test systems that the package content is not
the origin of the problem. I suspect that it might be related to the load
the RDF2CKAN execution creates on the wsgi processes.
Here is the default config:

    WSGIScriptAlias /data /applications/ecodp/users/ecodp/ckan/conf/wsgi.py
    WSGIDaemonProcess ecodp display-name=ecodp processes=6 threads=15
maximum-requests=2000
    WSGIProcessGroup ecodp

    # pass authorization info on (needed for rest api)
    WSGIPassAuthorization On

kind regards,

Bert

2014-03-14 16:37 GMT+01:00 Adam McGreggor <adam.mcgreggor at okfn.org>:

> Bert,
>
> I can confirm that we’ve received this ticket, and it does look like
> something the warranty period should cover.
>
> I’ve assigned the task over to the support team, who’ll hopefully have
> more information.
>
> Could you clarify what that ESTAT package is, please.
>
> Adam
>
> On 13 Mar 2014, at 22:17, Bert Van Nuffelen <
> bert.van.nuffelen at tenforce.com> wrote:
>
> > Dear OKF,
> >
> > We recieved the next ticket ODP-383. This ticket reports a CKAN crash on
> production.
> >
> > The problem description is the following:
> >
> > >> start_quote
> >
> > We just discovered that today’s ESTAT package from 11:00 causes the
> webserver to crash. After investigating the log files I noticed a new error
> message right before the first crash, to confirm my suspicion I gave the go
> ahead to Gregory Wassmann to relaunch the ingestion and it crashed again.
> >
> >
> > Here is a snippet from said log file for the 2nd crash:
> >
> >
> > [Tue Mar 11 14:36:49 2014] [error] Exception AttributeError: "'NoneType'
> object has no attribute 'clearing'" in <bound method
> PluginEnvironment.__del__ of  Services for Environment '<default>'
> >
> > [Tue Mar 11 14:36:49 2014] [error] > ignored
> >
> > [Tue Mar 11 14:36:49 2014] [error] Exception AttributeError: "'NoneType'
> object has no attribute 'clearing'" in <bound method
> PluginEnvironment.__del__ of  Services for Environment '<default>'
> >
> > [Tue Mar 11 14:36:49 2014] [error] > ignored
> >
> > [Tue Mar 11 14:36:50 2014] [error] Exception AttributeError: "'NoneType'
> object has no attribute 'clearing'" in <bound method
> PluginEnvironment.__del__ of  Services for Environment '<default>'
> >
> > [Tue Mar 11 14:36:50 2014] [error] > ignored
> >
> > [Tue Mar 11 14:36:50 2014] [error] Exception AttributeError: "'NoneType'
> object has no attribute 'clearing'" in <bound method
> PluginEnvironment.__del__ of  Services for Environment '<default>'
> >
> > [Tue Mar 11 14:36:50 2014] [error] > ignored
> >
> > [Tue Mar 11 14:36:50 2014] [error] Exception AttributeError: "'NoneType'
> object has no attribute 'clearing'" in <bound method
> PluginEnvironment.__del__ of  Services for Environment '<default>'
> >
> > [Tue Mar 11 14:36:50 2014] [error] > ignored
> >
> > [Tue Mar 11 14:36:50 2014] [error] Exception AttributeError: "'NoneType'
> object has no attribute 'clearing'" in <bound method
> PluginEnvironment.__del__ of  Services for Environment '<default>'
> >
> > [Tue Mar 11 14:36:50 2014] [error] > ignored
> >
> > [Tue Mar 11 14:36:52 2014] [error] [client 158.167.99.23] (2)No such
> file or directory: mod_wsgi (pid=21578): Unable to connect to WSGI daemon
> process 'ecodp' on '/var/run/wsgi.21367.0.1.sock' after multiple attempts.
> >
> >
> > As you can see, there are a couple of messages that indicate the usage
> of a ‘clearing’ attribute which seems to cause the problem as right after
> these messages pop up, the server crashes, as indicated by the last line in
> the snippet that I have included here in the email.
> >
> >
> > << end_quote
> >
> >
> > Can you indicate from which CKAN module the above exceptions come? Note
> that I find the above exceptions in older logs and so far they never
> crashed the wsgi daemon (logs dating back to 4 september 2013).  Loading
> the ESTAT on our test system does not result in a WSGI crash.
> >
> >
> >
> >
> >
> > kind regards,
> >
> >
> >
> > Bert
> >
>
> --
> Adam McGreggor
> General Manager  |  skype: adamamyl  |  @adamamyl
> The Open Knowledge Foundation
> Empowering through Open Knowledge
> http://okfn.org/  |  @okfn
>
>

-- 
Bert Van Nuffelen

Semantic Technologies Software Architect at TenForce
www.tenforce.be

Bert.Van.Nuffelen at tenforce.com
Office: +32 (0)16 31 48 60
Mobile:+32 479 06 24 26
skype: bert.van.nuffelen

----------------------------------------------

Bert Van Nuffelen, Mar 13 22:17

Dear OKF,

We recieved the next ticket ODP-383. This ticket reports a CKAN crash on
production.

The problem description is the following:

>> start_quote

We just discovered that today’s ESTAT package from 11:00 causes the
webserver to crash. After investigating the log files I noticed a new error
message right before the first crash, to confirm my suspicion I gave the go
ahead to Gregory Wassmann to relaunch the ingestion and it crashed again.

Here is a snippet from said log file for the 2nd crash:

[Tue Mar 11 14:36:49 2014] [error] Exception AttributeError: "'NoneType'
object has no attribute 'clearing'" in <bound method
PluginEnvironment.__del__ of  Services for Environment '<default>'

[Tue Mar 11 14:36:49 2014] [error] > ignored

[Tue Mar 11 14:36:49 2014] [error] Exception AttributeError: "'NoneType'
object has no attribute 'clearing'" in <bound method
PluginEnvironment.__del__ of  Services for Environment '<default>'

[Tue Mar 11 14:36:49 2014] [error] > ignored

[Tue Mar 11 14:36:50 2014] [error] Exception AttributeError: "'NoneType'
object has no attribute 'clearing'" in <bound method
PluginEnvironment.__del__ of  Services for Environment '<default>'

[Tue Mar 11 14:36:50 2014] [error] > ignored

[Tue Mar 11 14:36:50 2014] [error] Exception AttributeError: "'NoneType'
object has no attribute 'clearing'" in <bound method
PluginEnvironment.__del__ of  Services for Environment '<default>'

[Tue Mar 11 14:36:50 2014] [error] > ignored

[Tue Mar 11 14:36:50 2014] [error] Exception AttributeError: "'NoneType'
object has no attribute 'clearing'" in <bound method
PluginEnvironment.__del__ of  Services for Environment '<default>'

[Tue Mar 11 14:36:50 2014] [error] > ignored

[Tue Mar 11 14:36:50 2014] [error] Exception AttributeError: "'NoneType'
object has no attribute 'clearing'" in <bound method
PluginEnvironment.__del__ of  Services for Environment '<default>'

[Tue Mar 11 14:36:50 2014] [error] > ignored

[Tue Mar 11 14:36:52 2014] [error] [client 158.167.99.23] (2)No such file
or directory: mod_wsgi (pid=21578): Unable to connect to WSGI daemon
process 'ecodp' on '/var/run/wsgi.21367.0.1.sock' after multiple attempts.

As you can see, there are a couple of messages that indicate the usage of a
‘clearing’ attribute which seems to cause the problem as right after these
messages pop up, the server crashes, as indicated by the last line in the
snippet that I have included here in the email.

<< end_quote

Can you indicate from which CKAN module the above exceptions come? Note
that I find the above exceptions in older logs and so far they never
crashed the wsgi daemon (logs dating back to 4 september 2013).  Loading
the ESTAT on our test system does not result in a WSGI crash.

kind regards,

Bert

-- 
Bert Van Nuffelen

Semantic Technologies Software Architect at TenForce
www.tenforce.be

Bert.Van.Nuffelen at tenforce.com
Office: +32 (0)16 31 48 60
Mobile:+32 479 06 24 26
skype: bert.van.nuffelen

--------------------------------
This email is a service from Open Knowledge Foundation.

[FEE1-V4C0]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.okfn.org/mailman/private/ckan-support/attachments/20140326/34429d31/attachment-0002.html>