[ckan-dev] ckan errors after package update on RHEL server
Koebrick, Andrew (MNIT)
andrew.koebrick at state.mn.us
Tue Dec 15 15:46:59 UTC 2015
Stefan,
Thanks for the advice. I eventually traced the problem to some conflict between monit and tomcat. After disabling monit, all errors stopped. I will dig a bit deeper to try and understand what the root cause was, but this gets us back to a stable production instances immediately.
Cheers,
Andrew
From: ckan-dev [mailto:ckan-dev-bounces at lists.okfn.org] On Behalf Of Stefan Oderbolz
Sent: Tuesday, December 15, 2015 2:56 AM
To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
Subject: Re: [ckan-dev] ckan errors after package update on RHEL server
Hi Andrew,
it looks like a problem with solr or rather the connection to solr.
Some things to check:
- Is solr running on the configured port? Try to curl the configured solr_url from your config.ini, e.g. curl http://127.0.0.1:8983/solr
- Are you using tomcat to run solr? Check its logfile for indicators of something going wrong, usually /var/log/tomcat6/catalina.<date>.log
- Are you using SELinux? If so, try to disable it and re-try, this might be a pointer to the source of the error
Hope it helps.
-Stefan
On Mon, Dec 14, 2015 at 10:57 PM, Koebrick, Andrew (MNIT) <andrew.koebrick at state.mn.us<mailto:andrew.koebrick at state.mn.us>> wrote:
Today I updated packages on our RedHat 6.7 system which runs CKAN. Although I did the package update on our dev and test servers first and tested a number of pages without detecting issues, after updating production, I started to get occasional errors of this type:
Module ckan.controllers.group:187 in read
>> c.group_dict = self._action('group_show')(context, data_dict)
Module ckan.logic:424 in wrapped
>> result = _action(context, data_dict, **kw)
Module ckan.logic.action.get:1228 in organization_show
>> return _group_or_org_show(context, data_dict, is_org=True)
Module ckan.logic.action.get:1167 in _group_or_org_show
>> packages_field=packages_field)
Module ckan.lib.dictization.model_dictize:429 in group_dictize
>> group, just_the_count=True)
Module ckan.lib.dictization.model_dictize:419 in get_packages_for_this_group
>> q)
Module ckan.logic:424 in wrapped
>> result = _action(context, data_dict, **kw)
Module ckan.logic.action.get:1701 in package_search
>> query.run(data_dict)
Module ckan.lib.search.query:361 in run
>> solr_response = conn.raw_query(**query)
Module solr.core:706 in raw_query
>> return self.select.raw(**params)
Module solr.core:823 in raw
>> rsp = conn._post(self.selector, request, conn.form_headers)
Module solr.core:646 in _post
>> self._reconnect()
Module solr.core:625 in _reconnect
>> self.conn.connect()
Module httplib:742 in connect
>> self.timeout)
Module socket:567 in create_connection
>> raise error, msg
error: [Errno 111] Connection refused
I restarted httpd, solr, and even rebooted the whole system, but the errors persist. When I go to the URI indicated in the WebApp error (for example gisdata.mn.gov/organization/fdd407b9-9779-4c1c-b53d-e56682f39fc0?organization=us-mn-state-mngeo&tags=ortho&res_format=gpk<http://gisdata.mn.gov/organization/fdd407b9-9779-4c1c-b53d-e56682f39fc0?organization=us-mn-state-mngeo&tags=ortho&res_format=gpk> the page loads correctly almost all the time. If I keep hitting reload perhaps one out of 10 requests will fail.
Any ideas as to what is wrong? The packages changed when upgrading were:
Removed:
kernel.x86_64 0:2.6.32-504.12.2.el6
Installed:
kernel.x86_64 0:2.6.32-573.8.1.el6
Updated:
SDL.x86_64 0:1.2.14-7.el6_7.1 bash.x86_64 0:4.1.2-33.el6_7.1
coreutils.x86_64 0:8.4-37.el6_7.3 coreutils-libs.x86_64 0:8.4-37.el6_7.3
cronie.x86_64 0:1.4.4-15.el6_7.1 cronie-anacron.x86_64 0:1.4.4-15.el6_7.1
db4.x86_64 0:4.7.25-20.el6_7 db4-cxx.x86_64 0:4.7.25-20.el6_7
db4-devel.x86_64 0:4.7.25-20.el6_7 db4-utils.x86_64 0:4.7.25-20.el6_7
device-mapper.x86_64 0:1.02.95-3.el6_7.3 device-mapper-event.x86_64 0:1.02.95-3.el6_7.3
device-mapper-event-libs.x86_64 0:1.02.95-3.el6_7.3 device-mapper-libs.x86_64 0:1.02.95-3.el6_7.3
drupal7.noarch 0:7.41-1.el6 drupal7-date.noarch 0:2.9-1.el6
drupal7-jquery_update.noarch 0:2.7-1.el6 firefox.x86_64 0:38.4.0-1.el6_7
gawk.x86_64 0:3.1.7-10.el6_7.3 glibc.i686 0:2.12-1.166.el6_7.3
glibc.x86_64 0:2.12-1.166.el6_7.3 glibc-common.x86_64 0:2.12-1.166.el6_7.3
glibc-devel.x86_64 0:2.12-1.166.el6_7.3 glibc-headers.x86_64 0:2.12-1.166.el6_7.3
gofer.noarch 0:2.6.6-2.el6_6sat grep.x86_64 0:2.20-3.el6_7.1
initscripts.x86_64 0:9.03.49-1.el6_7.2 jakarta-commons-collections.noarch 0:3.2.1-3.5.el6_7
java-1.6.0-openjdk.x86_64 1:1.6.0.37-1.13.9.4.el6_7 java-1.6.0-openjdk-devel.x86_64 1:1.6.0.37-1.13.9.4.el6_7
java-1.7.0-openjdk.x86_64 1:1.7.0.91-2.6.2.2.el6_7 kernel-firmware.noarch 0:2.6.32-573.8.1.el6
kernel-headers.x86_64 0:2.6.32-573.8.1.el6 libipa_hbac.x86_64 0:1.12.4-47.el6_7.4
libipa_hbac-python.x86_64 0:1.12.4-47.el6_7.4 libreport.x86_64 0:2.0.9-25.el6_7
libreport-cli.x86_64 0:2.0.9-25.el6_7 libreport-compat.x86_64 0:2.0.9-25.el6_7
libreport-filesystem.x86_64 0:2.0.9-25.el6_7 libreport-gtk.x86_64 0:2.0.9-25.el6_7
libreport-newt.x86_64 0:2.0.9-25.el6_7 libreport-plugin-kerneloops.x86_64 0:2.0.9-25.el6_7
libreport-plugin-logger.x86_64 0:2.0.9-25.el6_7 libreport-plugin-mailx.x86_64 0:2.0.9-25.el6_7
libreport-plugin-reportuploader.x86_64 0:2.0.9-25.el6_7 libreport-plugin-rhtsupport.x86_64 0:2.0.9-25.el6_7
libreport-plugin-ureport.x86_64 0:2.0.9-25.el6_7 libreport-python.x86_64 0:2.0.9-25.el6_7
libsss_idmap.x86_64 0:1.12.4-47.el6_7.4 logrotate.x86_64 0:3.7.8-25.el6_7
lvm2.x86_64 0:2.02.118-3.el6_7.3 lvm2-libs.x86_64 0:2.02.118-3.el6_7.3
monit.x86_64 0:5.14-1.el6 nspr.x86_64 0:4.10.8-2.el6_7
nss.x86_64 0:3.19.1-5.el6_7 nss-softokn.x86_64 0:3.14.3-23.el6_7
nss-softokn-freebl.i686 0:3.14.3-23.el6_7 nss-softokn-freebl.x86_64 0:3.14.3-23.el6_7
nss-sysinit.x86_64 0:3.19.1-5.el6_7 nss-tools.x86_64 0:3.19.1-5.el6_7
nss-util.x86_64 0:3.19.1-2.el6_7 nss_compat_ossl.x86_64 0:0.9.6-2.el6_7
ntp.x86_64 0:4.2.6p5-5.el6_7.2 ntpdate.x86_64 0:4.2.6p5-5.el6_7.2
openldap.x86_64 0:2.4.40-7.el6_7 perf.x86_64 0:2.6.32-573.8.1.el6
perl.x86_64 4:5.10.1-141.el6_7.1 perl-Archive-Extract.x86_64 1:0.38-141.el6_7.1
perl-Archive-Tar.x86_64 0:1.58-141.el6_7.1 perl-CGI.x86_64 0:3.51-141.el6_7.1
perl-CPAN.x86_64 0:1.9402-141.el6_7.1 perl-CPANPLUS.x86_64 0:0.88-141.el6_7.1
perl-Compress-Raw-Bzip2.x86_64 0:2.021-141.el6_7.1 perl-Compress-Raw-Zlib.x86_64 1:2.021-141.el6_7.1
perl-Compress-Zlib.x86_64 0:2.021-141.el6_7.1 perl-Digest-SHA.x86_64 1:5.47-141.el6_7.1
perl-ExtUtils-CBuilder.x86_64 1:0.27-141.el6_7.1 perl-ExtUtils-Embed.x86_64 0:1.28-141.el6_7.1
perl-ExtUtils-MakeMaker.x86_64 0:6.55-141.el6_7.1 perl-ExtUtils-ParseXS.x86_64 1:2.2003.0-141.el6_7.1
perl-File-Fetch.x86_64 0:0.26-141.el6_7.1 perl-IO-Compress-Base.x86_64 0:2.021-141.el6_7.1
perl-IO-Compress-Bzip2.x86_64 0:2.021-141.el6_7.1 perl-IO-Compress-Zlib.x86_64 0:2.021-141.el6_7.1
perl-IO-Zlib.x86_64 1:1.09-141.el6_7.1 perl-IPC-Cmd.x86_64 1:0.56-141.el6_7.1
perl-Locale-Maketext-Simple.x86_64 1:0.18-141.el6_7.1 perl-Log-Message.x86_64 1:0.02-141.el6_7.1
perl-Log-Message-Simple.x86_64 0:0.04-141.el6_7.1 perl-Module-Build.x86_64 1:0.3500-141.el6_7.1
perl-Module-CoreList.x86_64 0:2.18-141.el6_7.1 perl-Module-Load.x86_64 1:0.16-141.el6_7.1
perl-Module-Load-Conditional.x86_64 0:0.30-141.el6_7.1 perl-Module-Loaded.x86_64 1:0.02-141.el6_7.1
perl-Module-Pluggable.x86_64 1:3.90-141.el6_7.1 perl-Object-Accessor.x86_64 1:0.34-141.el6_7.1
perl-Package-Constants.x86_64 1:0.02-141.el6_7.1 perl-Params-Check.x86_64 1:0.26-141.el6_7.1
perl-Parse-CPAN-Meta.x86_64 1:1.40-141.el6_7.1 perl-Pod-Escapes.x86_64 1:1.04-141.el6_7.1
perl-Pod-Simple.x86_64 1:3.13-141.el6_7.1 perl-Term-UI.x86_64 0:0.20-141.el6_7.1
perl-Test-Harness.x86_64 0:3.17-141.el6_7.1 perl-Test-Simple.x86_64 0:0.92-141.el6_7.1
perl-Time-HiRes.x86_64 4:1.9721-141.el6_7.1 perl-Time-Piece.x86_64 0:1.15-141.el6_7.1
perl-core.x86_64 0:5.10.1-141.el6_7.1 perl-devel.x86_64 4:5.10.1-141.el6_7.1
perl-libs.x86_64 4:5.10.1-141.el6_7.1 perl-parent.x86_64 1:0.221-141.el6_7.1
perl-version.x86_64 3:0.77-141.el6_7.1 pgdg-redhat92.noarch 0:9.2-8
postfix.x86_64 2:2.6.6-6.el6_7.1 postgresql.x86_64 0:8.4.20-4.el6_7
postgresql-devel.x86_64 0:8.4.20-4.el6_7 postgresql-libs.x86_64 0:8.4.20-4.el6_7
procps.x86_64 0:3.2.8-34.el6_7 pulp-rpm-handlers.noarch 0:2.6.0.17-1.el6
python-argparse.noarch 0:1.2.1-3.rhel6 python-gofer.noarch 0:2.6.6-2.el6_6sat
python-gofer-proton.noarch 0:2.6.6-2.el6_6sat python-pulp-agent-lib.noarch 0:2.6.0.17-1.el6
python-pulp-common.noarch 0:2.6.0.17-1.el6 python-pulp-rpm-common.noarch 0:2.6.0.17-1.el6
python-qpid-proton.x86_64 0:0.10-2.el6 python-sssdconfig.noarch 0:1.12.4-47.el6_7.4
qpid-proton-c.x86_64 0:0.10-2.el6 rng-tools.x86_64 0:5-2.el6_7
selinux-policy.noarch 0:3.7.19-279.el6_7.7 selinux-policy-targeted.noarch 0:3.7.19-279.el6_7.7
sssd.x86_64 0:1.12.4-47.el6_7.4 sssd-ad.x86_64 0:1.12.4-47.el6_7.4
sssd-client.x86_64 0:1.12.4-47.el6_7.4 sssd-common.x86_64 0:1.12.4-47.el6_7.4
sssd-common-pac.x86_64 0:1.12.4-47.el6_7.4 sssd-ipa.x86_64 0:1.12.4-47.el6_7.4
sssd-krb5.x86_64 0:1.12.4-47.el6_7.4 sssd-krb5-common.x86_64 0:1.12.4-47.el6_7.4
sssd-ldap.x86_64 0:1.12.4-47.el6_7.4 sssd-proxy.x86_64 0:1.12.4-47.el6_7.4
system-config-lvm.noarch 0:1.1.12-17.el6_7.1 tmpwatch.x86_64 0:2.9.16-5.el6_7
tzdata.noarch 0:2015g-2.el6 tzdata-java.noarch 0:2015g-2.el6
zip.x86_64 0:3.0-1.el6_7.1
I hesitate to try to roll all of these back and would much rather get the updated system working correctly. Thanks for any assistance folk can give.
Andrew Koebrick | MINNESOTA GEOSPATIAL INFORMATION OFFICE
Web Coordinator / Systems administrator / Librarian
MN.IT<http://MN.IT> Services @ MNGEO
651-201-2465<tel:651-201-2465> (w) | 651-296-6398<tel:651-296-6398> (f) | andrew.koebrick at state.mn.us<mailto:andrew.koebrick at state.mn.us>
658 Cedar St., Room 300, St. Paul, MN 55155, www.mngeo.state.mn.us<http://www.mngeo.state.mn.us/>
[cid:image002.jpg at 01CE61F8.52552AE0]<http://www.mn.gov/oet>
Information Technology for Minnesota Government | mn.gov/mnit<http://www.mn.gov/oet>
_______________________________________________
ckan-dev mailing list
ckan-dev at lists.okfn.org<mailto:ckan-dev at lists.okfn.org>
https://lists.okfn.org/mailman/listinfo/ckan-dev
Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
--
Liip AG // Limmatstrasse 183 // CH-8005 Zürich
Tel +41 43 500 39 80 // GnuPG 0x7B588C67 // www.liip.ch<http://www.liip.ch/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20151215/54db3546/attachment-0003.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 1624 bytes
Desc: image001.jpg
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20151215/54db3546/attachment-0003.jpg>
More information about the ckan-dev
mailing list