[CKAN-support] [Open Knowledge Foundation] Assignment: ODP-383

• Previous message: [CKAN-support] [Open Knowledge Foundation] Assignment: Extra field in resources
• Next message: [CKAN-support] User cannot see Organisations they are members of?
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

##- Please type your reply above this line -##

[Open Knowledge Foundation] Assignment: ODP-383

You have been assigned to this ticket (#136).

----------------------------------------------

Adam McGreggor, Mar 25 12:16 (private)

Support request with SLA -- could someone look at this today (Tuesday 25th March), please?

----------------------------------------------

Bert Van Nuffelen, Mar 14 15:49

Hi Adam,

the ESTAT package is a RDF2CKAN package containing about 5500 datasets in
RDF format. The RDF2CKAN tool will interprete those RDF dataset
descriptions and turn them into json to be sent as an update to the EU ODP
CKAN.

So far my tests show on our test systems that the package content is not
the origin of the problem. I suspect that it might be related to the load
the RDF2CKAN execution creates on the wsgi processes.
Here is the default config:

    WSGIScriptAlias /data /applications/ecodp/users/ecodp/ckan/conf/wsgi.py
    WSGIDaemonProcess ecodp display-name=ecodp processes=6 threads=15
maximum-requests=2000
    WSGIProcessGroup ecodp

    # pass authorization info on (needed for rest api)
    WSGIPassAuthorization On

kind regards,

Bert


2014-03-14 16:37 GMT+01:00 Adam McGreggor <adam.mcgreggor at okfn.org>:

> Bert,
>
> I can confirm that we’ve received this ticket, and it does look like
> something the warranty period should cover.
>
> I’ve assigned the task over to the support team, who’ll hopefully have
> more information.
>
> Could you clarify what that ESTAT package is, please.
>
> Adam
>
> On 13 Mar 2014, at 22:17, Bert Van Nuffelen <
> bert.van.nuffelen at tenforce.com> wrote:
>
> > Dear OKF,
> >
> > We recieved the next ticket ODP-383. This ticket reports a CKAN crash on
> production.
> >
> > The problem description is the following:
> >
> > >> start_quote
> >
> > We just discovered that today’s ESTAT package from 11:00 causes the
> webserver to crash. After investigating the log files I noticed a new error
> message right before the first crash, to confirm my suspicion I gave the go
> ahead to Gregory Wassmann to relaunch the ingestion and it crashed again.
> >
> >
> > Here is a snippet from said log file for the 2nd crash:
> >
> >
> > [Tue Mar 11 14:36:49 2014] [error] Exception AttributeError: "'NoneType'
> object has no attribute 'clearing'" in <bound method
> PluginEnvironment.__del__ of  Services for Environment '<default>'
> >
> > [Tue Mar 11 14:36:49 2014] [error] > ignored
> >
> > [Tue Mar 11 14:36:49 2014] [error] Exception AttributeError: "'NoneType'
> object has no attribute 'clearing'" in <bound method
> PluginEnvironment.__del__ of  Services for Environment '<default>'
> >
> > [Tue Mar 11 14:36:49 2014] [error] > ignored
> >
> > [Tue Mar 11 14:36:50 2014] [error] Exception AttributeError: "'NoneType'
> object has no attribute 'clearing'" in <bound method
> PluginEnvironment.__del__ of  Services for Environment '<default>'
> >
> > [Tue Mar 11 14:36:50 2014] [error] > ignored
> >
> > [Tue Mar 11 14:36:50 2014] [error] Exception AttributeError: "'NoneType'
> object has no attribute 'clearing'" in <bound method
> PluginEnvironment.__del__ of  Services for Environment '<default>'
> >
> > [Tue Mar 11 14:36:50 2014] [error] > ignored
> >
> > [Tue Mar 11 14:36:50 2014] [error] Exception AttributeError: "'NoneType'
> object has no attribute 'clearing'" in <bound method
> PluginEnvironment.__del__ of  Services for Environment '<default>'
> >
> > [Tue Mar 11 14:36:50 2014] [error] > ignored
> >
> > [Tue Mar 11 14:36:50 2014] [error] Exception AttributeError: "'NoneType'
> object has no attribute 'clearing'" in <bound method
> PluginEnvironment.__del__ of  Services for Environment '<default>'
> >
> > [Tue Mar 11 14:36:50 2014] [error] > ignored
> >
> > [Tue Mar 11 14:36:52 2014] [error] [client 158.167.99.23] (2)No such
> file or directory: mod_wsgi (pid=21578): Unable to connect to WSGI daemon
> process 'ecodp' on '/var/run/wsgi.21367.0.1.sock' after multiple attempts.
> >
> >
> > As you can see, there are a couple of messages that indicate the usage
> of a ‘clearing’ attribute which seems to cause the problem as right after
> these messages pop up, the server crashes, as indicated by the last line in
> the snippet that I have included here in the email.
> >
> >
> > << end_quote
> >
> >
> > Can you indicate from which CKAN module the above exceptions come? Note
> that I find the above exceptions in older logs and so far they never
> crashed the wsgi daemon (logs dating back to 4 september 2013).  Loading
> the ESTAT on our test system does not result in a WSGI crash.
> >
> >
> >
> >
> >
> > kind regards,
> >
> >
> >
> > Bert
> >
>
> --
> Adam McGreggor
> General Manager  |  skype: adamamyl  |  @adamamyl
> The Open Knowledge Foundation
> Empowering through Open Knowledge
> http://okfn.org/  |  @okfn
>
>


-- 
Bert Van Nuffelen

Semantic Technologies Software Architect at TenForce
www.tenforce.be

Bert.Van.Nuffelen at tenforce.com
Office: +32 (0)16 31 48 60
Mobile:+32 479 06 24 26
skype: bert.van.nuffelen

----------------------------------------------

Adam McGreggor, Mar 14 15:38 (private)

Bert,

I can confirm that we’ve received this ticket, and it does look like something the warranty period should cover.

I’ve assigned the task over to the support team, who’ll hopefully have more information.

Could you clarify what that ESTAT package is, please.

Adam

On 13 Mar 2014, at 22:17, Bert Van Nuffelen <bert.van.nuffelen at tenforce.com> wrote:

> Dear OKF,
> 
> We recieved the next ticket ODP-383. This ticket reports a CKAN crash on production. 
> 
> The problem description is the following:
> 
> >> start_quote
> 
> We just discovered that today’s ESTAT package from 11:00 causes the webserver to crash. After investigating the log files I noticed a new error message right before the first crash, to confirm my suspicion I gave the go ahead to Gregory Wassmann to relaunch the ingestion and it crashed again.
> 
>  
> Here is a snippet from said log file for the 2nd crash:
> 
>  
> [Tue Mar 11 14:36:49 2014] [error] Exception AttributeError: "'NoneType' object has no attribute 'clearing'" in <bound method PluginEnvironment.__del__ of  Services for Environment '<default>'
> 
> [Tue Mar 11 14:36:49 2014] [error] > ignored
> 
> [Tue Mar 11 14:36:49 2014] [error] Exception AttributeError: "'NoneType' object has no attribute 'clearing'" in <bound method PluginEnvironment.__del__ of  Services for Environment '<default>'
> 
> [Tue Mar 11 14:36:49 2014] [error] > ignored
> 
> [Tue Mar 11 14:36:50 2014] [error] Exception AttributeError: "'NoneType' object has no attribute 'clearing'" in <bound method PluginEnvironment.__del__ of  Services for Environment '<default>'
> 
> [Tue Mar 11 14:36:50 2014] [error] > ignored
> 
> [Tue Mar 11 14:36:50 2014] [error] Exception AttributeError: "'NoneType' object has no attribute 'clearing'" in <bound method PluginEnvironment.__del__ of  Services for Environment '<default>'
> 
> [Tue Mar 11 14:36:50 2014] [error] > ignored
> 
> [Tue Mar 11 14:36:50 2014] [error] Exception AttributeError: "'NoneType' object has no attribute 'clearing'" in <bound method PluginEnvironment.__del__ of  Services for Environment '<default>'
> 
> [Tue Mar 11 14:36:50 2014] [error] > ignored
> 
> [Tue Mar 11 14:36:50 2014] [error] Exception AttributeError: "'NoneType' object has no attribute 'clearing'" in <bound method PluginEnvironment.__del__ of  Services for Environment '<default>'
> 
> [Tue Mar 11 14:36:50 2014] [error] > ignored
> 
> [Tue Mar 11 14:36:52 2014] [error] [client 158.167.99.23] (2)No such file or directory: mod_wsgi (pid=21578): Unable to connect to WSGI daemon process 'ecodp' on '/var/run/wsgi.21367.0.1.sock' after multiple attempts.
> 
>  
> As you can see, there are a couple of messages that indicate the usage of a ‘clearing’ attribute which seems to cause the problem as right after these messages pop up, the server crashes, as indicated by the last line in the snippet that I have included here in the email.
> 
> 
> << end_quote
> 
> 
> Can you indicate from which CKAN module the above exceptions come? Note that I find the above exceptions in older logs and so far they never crashed the wsgi daemon (logs dating back to 4 september 2013).  Loading the ESTAT on our test system does not result in a WSGI crash. 
> 
> 
> 
> 
> 
> kind regards,
> 
> 
> 
> Bert
> 

-- 
Adam McGreggor
General Manager  |  skype: adamamyl  |  @adamamyl
The Open Knowledge Foundation
Empowering through Open Knowledge
http://okfn.org/  |  @okfn

----------------------------------------------

Adam McGreggor, Mar 14 15:32 (private)

Inclined to say this isn't our problem, as we don't supply the ESTAT package (or do we?) and they're responsible for the crash (keep ckan machines, for CKAN).

Apart from that, can we answer the pertinent question:

"Can you indicate from which CKAN module the above exceptions come?"

-- looks to me like we may need more info.

(FWIW, we're just in Warranty period with these people now; I don't want us to spend a lot of time on it).

----------------------------------------------

Bert Van Nuffelen, Mar 13 22:17

Dear OKF,

We recieved the next ticket ODP-383. This ticket reports a CKAN crash on
production.

The problem description is the following:

>> start_quote

We just discovered that today’s ESTAT package from 11:00 causes the
webserver to crash. After investigating the log files I noticed a new error
message right before the first crash, to confirm my suspicion I gave the go
ahead to Gregory Wassmann to relaunch the ingestion and it crashed again.



Here is a snippet from said log file for the 2nd crash:



[Tue Mar 11 14:36:49 2014] [error] Exception AttributeError: "'NoneType'
object has no attribute 'clearing'" in <bound method
PluginEnvironment.__del__ of  Services for Environment '<default>'

[Tue Mar 11 14:36:49 2014] [error] > ignored

[Tue Mar 11 14:36:49 2014] [error] Exception AttributeError: "'NoneType'
object has no attribute 'clearing'" in <bound method
PluginEnvironment.__del__ of  Services for Environment '<default>'

[Tue Mar 11 14:36:49 2014] [error] > ignored

[Tue Mar 11 14:36:50 2014] [error] Exception AttributeError: "'NoneType'
object has no attribute 'clearing'" in <bound method
PluginEnvironment.__del__ of  Services for Environment '<default>'

[Tue Mar 11 14:36:50 2014] [error] > ignored

[Tue Mar 11 14:36:50 2014] [error] Exception AttributeError: "'NoneType'
object has no attribute 'clearing'" in <bound method
PluginEnvironment.__del__ of  Services for Environment '<default>'

[Tue Mar 11 14:36:50 2014] [error] > ignored

[Tue Mar 11 14:36:50 2014] [error] Exception AttributeError: "'NoneType'
object has no attribute 'clearing'" in <bound method
PluginEnvironment.__del__ of  Services for Environment '<default>'

[Tue Mar 11 14:36:50 2014] [error] > ignored

[Tue Mar 11 14:36:50 2014] [error] Exception AttributeError: "'NoneType'
object has no attribute 'clearing'" in <bound method
PluginEnvironment.__del__ of  Services for Environment '<default>'

[Tue Mar 11 14:36:50 2014] [error] > ignored

[Tue Mar 11 14:36:52 2014] [error] [client 158.167.99.23] (2)No such file
or directory: mod_wsgi (pid=21578): Unable to connect to WSGI daemon
process 'ecodp' on '/var/run/wsgi.21367.0.1.sock' after multiple attempts.



As you can see, there are a couple of messages that indicate the usage of a
‘clearing’ attribute which seems to cause the problem as right after these
messages pop up, the server crashes, as indicated by the last line in the
snippet that I have included here in the email.

<< end_quote


Can you indicate from which CKAN module the above exceptions come? Note
that I find the above exceptions in older logs and so far they never
crashed the wsgi daemon (logs dating back to 4 september 2013).  Loading
the ESTAT on our test system does not result in a WSGI crash.



kind regards,


Bert






-- 
Bert Van Nuffelen

Semantic Technologies Software Architect at TenForce
www.tenforce.be

Bert.Van.Nuffelen at tenforce.com
Office: +32 (0)16 31 48 60
Mobile:+32 479 06 24 26
skype: bert.van.nuffelen

--------------------------------
This email is a service from Open Knowledge Foundation.









[FEE1-V4C0]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.okfn.org/mailman/private/ckan-support/attachments/20140325/f8932fe9/attachment-0002.html>

• Previous message: [CKAN-support] [Open Knowledge Foundation] Assignment: Extra field in resources
• Next message: [CKAN-support] User cannot see Organisations they are members of?
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]