[ckan-dev] ckan errors after package update on RHEL server

Koebrick, Andrew (MNIT) andrew.koebrick at state.mn.us
Tue Dec 15 15:46:59 UTC 2015


Stefan,

Thanks for the advice.  I eventually traced the problem to some conflict between monit and tomcat. After disabling monit, all errors stopped.  I will dig a bit deeper to try and understand what the root cause was, but this gets us back to a stable production instances immediately.

Cheers,

Andrew

From: ckan-dev [mailto:ckan-dev-bounces at lists.okfn.org] On Behalf Of Stefan Oderbolz
Sent: Tuesday, December 15, 2015 2:56 AM
To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
Subject: Re: [ckan-dev] ckan errors after package update on RHEL server

Hi Andrew,

it looks like a problem with solr or rather the connection to solr.

Some things to check:

- Is solr running on the configured port? Try to curl the configured solr_url from your config.ini, e.g. curl http://127.0.0.1:8983/solr
- Are you using tomcat to run solr? Check its logfile for indicators of something going wrong, usually /var/log/tomcat6/catalina.<date>.log
- Are you using SELinux? If so, try to disable it and re-try, this might be a pointer to the source of the error


Hope it helps.
-Stefan

On Mon, Dec 14, 2015 at 10:57 PM, Koebrick, Andrew (MNIT) <andrew.koebrick at state.mn.us<mailto:andrew.koebrick at state.mn.us>> wrote:
Today I updated packages on our RedHat 6.7 system which runs CKAN.  Although I did the package update on our dev and test servers first and tested a number of pages without detecting issues, after updating production, I started to get occasional errors of this type:

Module ckan.controllers.group:187 in read
>>  c.group_dict = self._action('group_show')(context, data_dict)
Module ckan.logic:424 in wrapped
>>  result = _action(context, data_dict, **kw)
Module ckan.logic.action.get:1228 in organization_show
>>  return _group_or_org_show(context, data_dict, is_org=True)
Module ckan.logic.action.get:1167 in _group_or_org_show
>>  packages_field=packages_field)
Module ckan.lib.dictization.model_dictize:429 in group_dictize
>>  group, just_the_count=True)
Module ckan.lib.dictization.model_dictize:419 in get_packages_for_this_group
>>  q)
Module ckan.logic:424 in wrapped
>>  result = _action(context, data_dict, **kw)
Module ckan.logic.action.get:1701 in package_search
>>  query.run(data_dict)
Module ckan.lib.search.query:361 in run
>>  solr_response = conn.raw_query(**query)
Module solr.core:706 in raw_query
>>  return self.select.raw(**params)
Module solr.core:823 in raw
>>  rsp = conn._post(self.selector, request, conn.form_headers)
Module solr.core:646 in _post
>>  self._reconnect()
Module solr.core:625 in _reconnect
>>  self.conn.connect()
Module httplib:742 in connect
>>  self.timeout)
Module socket:567 in create_connection
>>  raise error, msg
error: [Errno 111] Connection refused

I restarted httpd, solr, and even rebooted the whole system, but the errors persist.  When I go to the URI indicated in the WebApp error (for example gisdata.mn.gov/organization/fdd407b9-9779-4c1c-b53d-e56682f39fc0?organization=us-mn-state-mngeo&tags=ortho&res_format=gpk<http://gisdata.mn.gov/organization/fdd407b9-9779-4c1c-b53d-e56682f39fc0?organization=us-mn-state-mngeo&tags=ortho&res_format=gpk> the page loads correctly almost all the time.  If I keep hitting reload perhaps one out of 10 requests will fail.

Any ideas as to what is wrong?  The packages changed when upgrading were:

Removed:
  kernel.x86_64 0:2.6.32-504.12.2.el6

Installed:
  kernel.x86_64 0:2.6.32-573.8.1.el6

Updated:
  SDL.x86_64 0:1.2.14-7.el6_7.1                                                bash.x86_64 0:4.1.2-33.el6_7.1
  coreutils.x86_64 0:8.4-37.el6_7.3                                            coreutils-libs.x86_64 0:8.4-37.el6_7.3
  cronie.x86_64 0:1.4.4-15.el6_7.1                                             cronie-anacron.x86_64 0:1.4.4-15.el6_7.1
  db4.x86_64 0:4.7.25-20.el6_7                                                 db4-cxx.x86_64 0:4.7.25-20.el6_7
  db4-devel.x86_64 0:4.7.25-20.el6_7                                           db4-utils.x86_64 0:4.7.25-20.el6_7
  device-mapper.x86_64 0:1.02.95-3.el6_7.3                                     device-mapper-event.x86_64 0:1.02.95-3.el6_7.3
  device-mapper-event-libs.x86_64 0:1.02.95-3.el6_7.3                          device-mapper-libs.x86_64 0:1.02.95-3.el6_7.3
  drupal7.noarch 0:7.41-1.el6                                                  drupal7-date.noarch 0:2.9-1.el6
  drupal7-jquery_update.noarch 0:2.7-1.el6                                     firefox.x86_64 0:38.4.0-1.el6_7
  gawk.x86_64 0:3.1.7-10.el6_7.3                                               glibc.i686 0:2.12-1.166.el6_7.3
  glibc.x86_64 0:2.12-1.166.el6_7.3                                            glibc-common.x86_64 0:2.12-1.166.el6_7.3
  glibc-devel.x86_64 0:2.12-1.166.el6_7.3                                      glibc-headers.x86_64 0:2.12-1.166.el6_7.3
  gofer.noarch 0:2.6.6-2.el6_6sat                                              grep.x86_64 0:2.20-3.el6_7.1
  initscripts.x86_64 0:9.03.49-1.el6_7.2                                       jakarta-commons-collections.noarch 0:3.2.1-3.5.el6_7
  java-1.6.0-openjdk.x86_64 1:1.6.0.37-1.13.9.4.el6_7                          java-1.6.0-openjdk-devel.x86_64 1:1.6.0.37-1.13.9.4.el6_7
  java-1.7.0-openjdk.x86_64 1:1.7.0.91-2.6.2.2.el6_7                           kernel-firmware.noarch 0:2.6.32-573.8.1.el6
  kernel-headers.x86_64 0:2.6.32-573.8.1.el6                                   libipa_hbac.x86_64 0:1.12.4-47.el6_7.4
  libipa_hbac-python.x86_64 0:1.12.4-47.el6_7.4                                libreport.x86_64 0:2.0.9-25.el6_7
  libreport-cli.x86_64 0:2.0.9-25.el6_7                                        libreport-compat.x86_64 0:2.0.9-25.el6_7
  libreport-filesystem.x86_64 0:2.0.9-25.el6_7                                 libreport-gtk.x86_64 0:2.0.9-25.el6_7
  libreport-newt.x86_64 0:2.0.9-25.el6_7                                       libreport-plugin-kerneloops.x86_64 0:2.0.9-25.el6_7
  libreport-plugin-logger.x86_64 0:2.0.9-25.el6_7                              libreport-plugin-mailx.x86_64 0:2.0.9-25.el6_7
  libreport-plugin-reportuploader.x86_64 0:2.0.9-25.el6_7                      libreport-plugin-rhtsupport.x86_64 0:2.0.9-25.el6_7
  libreport-plugin-ureport.x86_64 0:2.0.9-25.el6_7                             libreport-python.x86_64 0:2.0.9-25.el6_7
  libsss_idmap.x86_64 0:1.12.4-47.el6_7.4                                      logrotate.x86_64 0:3.7.8-25.el6_7
  lvm2.x86_64 0:2.02.118-3.el6_7.3                                             lvm2-libs.x86_64 0:2.02.118-3.el6_7.3
  monit.x86_64 0:5.14-1.el6                                                    nspr.x86_64 0:4.10.8-2.el6_7
  nss.x86_64 0:3.19.1-5.el6_7                                                  nss-softokn.x86_64 0:3.14.3-23.el6_7
  nss-softokn-freebl.i686 0:3.14.3-23.el6_7                                    nss-softokn-freebl.x86_64 0:3.14.3-23.el6_7
  nss-sysinit.x86_64 0:3.19.1-5.el6_7                                          nss-tools.x86_64 0:3.19.1-5.el6_7
  nss-util.x86_64 0:3.19.1-2.el6_7                                             nss_compat_ossl.x86_64 0:0.9.6-2.el6_7
  ntp.x86_64 0:4.2.6p5-5.el6_7.2                                               ntpdate.x86_64 0:4.2.6p5-5.el6_7.2
  openldap.x86_64 0:2.4.40-7.el6_7                                             perf.x86_64 0:2.6.32-573.8.1.el6
  perl.x86_64 4:5.10.1-141.el6_7.1                                             perl-Archive-Extract.x86_64 1:0.38-141.el6_7.1
  perl-Archive-Tar.x86_64 0:1.58-141.el6_7.1                                   perl-CGI.x86_64 0:3.51-141.el6_7.1
  perl-CPAN.x86_64 0:1.9402-141.el6_7.1                                        perl-CPANPLUS.x86_64 0:0.88-141.el6_7.1
  perl-Compress-Raw-Bzip2.x86_64 0:2.021-141.el6_7.1                           perl-Compress-Raw-Zlib.x86_64 1:2.021-141.el6_7.1
  perl-Compress-Zlib.x86_64 0:2.021-141.el6_7.1                                perl-Digest-SHA.x86_64 1:5.47-141.el6_7.1
  perl-ExtUtils-CBuilder.x86_64 1:0.27-141.el6_7.1                             perl-ExtUtils-Embed.x86_64 0:1.28-141.el6_7.1
  perl-ExtUtils-MakeMaker.x86_64 0:6.55-141.el6_7.1                            perl-ExtUtils-ParseXS.x86_64 1:2.2003.0-141.el6_7.1
  perl-File-Fetch.x86_64 0:0.26-141.el6_7.1                                    perl-IO-Compress-Base.x86_64 0:2.021-141.el6_7.1
  perl-IO-Compress-Bzip2.x86_64 0:2.021-141.el6_7.1                            perl-IO-Compress-Zlib.x86_64 0:2.021-141.el6_7.1
  perl-IO-Zlib.x86_64 1:1.09-141.el6_7.1                                       perl-IPC-Cmd.x86_64 1:0.56-141.el6_7.1
  perl-Locale-Maketext-Simple.x86_64 1:0.18-141.el6_7.1                        perl-Log-Message.x86_64 1:0.02-141.el6_7.1
  perl-Log-Message-Simple.x86_64 0:0.04-141.el6_7.1                            perl-Module-Build.x86_64 1:0.3500-141.el6_7.1
  perl-Module-CoreList.x86_64 0:2.18-141.el6_7.1                               perl-Module-Load.x86_64 1:0.16-141.el6_7.1
  perl-Module-Load-Conditional.x86_64 0:0.30-141.el6_7.1                       perl-Module-Loaded.x86_64 1:0.02-141.el6_7.1
  perl-Module-Pluggable.x86_64 1:3.90-141.el6_7.1                              perl-Object-Accessor.x86_64 1:0.34-141.el6_7.1
  perl-Package-Constants.x86_64 1:0.02-141.el6_7.1                             perl-Params-Check.x86_64 1:0.26-141.el6_7.1
  perl-Parse-CPAN-Meta.x86_64 1:1.40-141.el6_7.1                               perl-Pod-Escapes.x86_64 1:1.04-141.el6_7.1
  perl-Pod-Simple.x86_64 1:3.13-141.el6_7.1                                    perl-Term-UI.x86_64 0:0.20-141.el6_7.1
  perl-Test-Harness.x86_64 0:3.17-141.el6_7.1                                  perl-Test-Simple.x86_64 0:0.92-141.el6_7.1
  perl-Time-HiRes.x86_64 4:1.9721-141.el6_7.1                                  perl-Time-Piece.x86_64 0:1.15-141.el6_7.1
  perl-core.x86_64 0:5.10.1-141.el6_7.1                                        perl-devel.x86_64 4:5.10.1-141.el6_7.1
  perl-libs.x86_64 4:5.10.1-141.el6_7.1                                        perl-parent.x86_64 1:0.221-141.el6_7.1
  perl-version.x86_64 3:0.77-141.el6_7.1                                       pgdg-redhat92.noarch 0:9.2-8
  postfix.x86_64 2:2.6.6-6.el6_7.1                                             postgresql.x86_64 0:8.4.20-4.el6_7
  postgresql-devel.x86_64 0:8.4.20-4.el6_7                                     postgresql-libs.x86_64 0:8.4.20-4.el6_7
  procps.x86_64 0:3.2.8-34.el6_7                                               pulp-rpm-handlers.noarch 0:2.6.0.17-1.el6
  python-argparse.noarch 0:1.2.1-3.rhel6                                       python-gofer.noarch 0:2.6.6-2.el6_6sat
  python-gofer-proton.noarch 0:2.6.6-2.el6_6sat                                python-pulp-agent-lib.noarch 0:2.6.0.17-1.el6
  python-pulp-common.noarch 0:2.6.0.17-1.el6                                   python-pulp-rpm-common.noarch 0:2.6.0.17-1.el6
  python-qpid-proton.x86_64 0:0.10-2.el6                                       python-sssdconfig.noarch 0:1.12.4-47.el6_7.4
  qpid-proton-c.x86_64 0:0.10-2.el6                                            rng-tools.x86_64 0:5-2.el6_7
  selinux-policy.noarch 0:3.7.19-279.el6_7.7                                   selinux-policy-targeted.noarch 0:3.7.19-279.el6_7.7
  sssd.x86_64 0:1.12.4-47.el6_7.4                                              sssd-ad.x86_64 0:1.12.4-47.el6_7.4
  sssd-client.x86_64 0:1.12.4-47.el6_7.4                                       sssd-common.x86_64 0:1.12.4-47.el6_7.4
  sssd-common-pac.x86_64 0:1.12.4-47.el6_7.4                                   sssd-ipa.x86_64 0:1.12.4-47.el6_7.4
  sssd-krb5.x86_64 0:1.12.4-47.el6_7.4                                         sssd-krb5-common.x86_64 0:1.12.4-47.el6_7.4
  sssd-ldap.x86_64 0:1.12.4-47.el6_7.4                                         sssd-proxy.x86_64 0:1.12.4-47.el6_7.4
  system-config-lvm.noarch 0:1.1.12-17.el6_7.1                                 tmpwatch.x86_64 0:2.9.16-5.el6_7
  tzdata.noarch 0:2015g-2.el6                                                  tzdata-java.noarch 0:2015g-2.el6
  zip.x86_64 0:3.0-1.el6_7.1


I hesitate to try to roll all of these back and would much rather get the updated system working correctly.  Thanks for any assistance folk can give.


Andrew Koebrick  |  MINNESOTA GEOSPATIAL INFORMATION OFFICE
Web Coordinator / Systems administrator / Librarian
MN.IT<http://MN.IT> Services @ MNGEO
651-201-2465<tel:651-201-2465> (w)  |   651-296-6398<tel:651-296-6398> (f) |  andrew.koebrick at state.mn.us<mailto:andrew.koebrick at state.mn.us>
658 Cedar St., Room 300, St. Paul, MN 55155, www.mngeo.state.mn.us<http://www.mngeo.state.mn.us/>

[cid:image002.jpg at 01CE61F8.52552AE0]<http://www.mn.gov/oet>

Information Technology for Minnesota Government   |   mn.gov/mnit<http://www.mn.gov/oet>




_______________________________________________
ckan-dev mailing list
ckan-dev at lists.okfn.org<mailto:ckan-dev at lists.okfn.org>
https://lists.okfn.org/mailman/listinfo/ckan-dev
Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev



--
Liip AG  // Limmatstrasse 183 //  CH-8005 Zürich
Tel +41 43 500 39 80 // GnuPG 0x7B588C67 // www.liip.ch<http://www.liip.ch/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20151215/54db3546/attachment-0003.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 1624 bytes
Desc: image001.jpg
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20151215/54db3546/attachment-0003.jpg>


More information about the ckan-dev mailing list