[ckan-dev] CKAN 2.5.2 test suite errors

Wed Aug 10 04:25:12 UTC 2016

Hi Jim,

Very glad to hear this worked.

On a base install, with no extensions installed, and final configuration
incomplete, CKAN functions despite errors in the the test suite. I suspect
there are people on the dev-list who are familiar with the errors you've
listed, and would know what their relevance is.

As we refine the public stack we're releasing, I would envision that
passing the test-suite would be one of the benchmarks we'd use to determine
that the stack is ready for release 1.

Our internal situation is possibly a little unique in that we have at least
four key teams within the organisation that pump out CKAN builds.
We've got Orchestration (which is my arena), Dev, Operations and Sysadmin.
On that basis I'm doing the initial infrastructure build of CKAN, but the
Dev and Ops teams are configuring, theming and putting all the finishing
touches on it. I'm in the process of expanding my overall understanding of
the application so I can go far deeper into the stack.

Eliminating 194 odd errors sounds like a big win in the mean time.
Hopefully the others aren't too difficult to eliminate.

Shane.

------------------------------------------------------------------
*SHANE DAVIS*  |  *SERVICE DELIVERY MANAGER*
Link Digital
www.linkdigital.com.au

On Wed, Aug 10, 2016 at 5:20 AM, Espeland, James <james.espeland at montana.edu
> wrote:

> Wow Shane!
>
>
>
> Thank you so much for your assistance!  By following your instructions, we
> were able to cut the number of errors we were seeing from the CKAN 2.5.2
> test suite significantly!
>
>
>
> We didn’t move /var/solr to /data/solr, but if you do, we should point out
> to other readers that /etc/default/solr.in.sh would have to be adjusted
> accordingly.
>
>
>
> Your conf directory contents supplied via github were definitely most
> useful!
>
>
>
> Before we were seeing around 200 errors when running the CKAN 2.5.2 test
> suite.  Now we are seeing 4 errors and 6 failures.  Here is a filtered
> version of the test output (warnings and debug statements removed):
>
>
>
> **********
>
>
>
> ERROR: ckan.tests.config.test_environment.TestUpdateConfig.
> test_update_config_db_url_precedence
>
> ERROR: ckan.tests.config.test_environment.TestUpdateConfig.
> test_update_config_db_url_precedence
>
> ERROR: ckan.tests.config.test_environment.TestUpdateConfig.
> test_update_config_env_vars
>
> ERROR: ckan.tests.config.test_environment.TestUpdateConfig.
> test_update_config_env_vars
>
> FAIL: ckan.tests.legacy.functional.api.test_activity.
> TestActivity.test_add_tag
>
> FAIL: ckan.tests.legacy.functional.api.test_activity.
> TestActivity.test_delete_extras
>
> FAIL: ckan.tests.legacy.functional.api.test_util.TestUtil.test_status
>
> FAIL: ckan.tests.legacy.logic.test_action.TestAction.test_03_
> create_update_package
>
> FAIL: ckan.tests.legacy.logic.test_action.TestAction.test_26_resource_show
>
> FAIL: ckan.tests.logic.action.test_get.TestPackageShow.test_
> package_show_is_lazy
>
>
>
> Ran 1923 tests in 1062.937s
>
>
>
> FAILED (SKIP=12, errors=4, failures=6)
>
>
>
> **********
>
>
>
> Do you see any errors or failures when you run the test suite?
>
>
>
> Are our results at an acceptable level of success?
>
>
>
> If not, do you have any ideas concerning corrections we can make to
> achieve better results?
>
>
>
> Thank you again, Shane!  Your advice and configuration files are most
> graciously appreciated!
>
>
>
> Best regards,
>
>
>
> --- Jim Espeland
>
> james.espeland at montana.edu
>
>
>
>
>
> *From:* Shane Davis [mailto:shane.davis at linkdigital.com.au]
> *Sent:* Monday, August 8, 2016 5:49 PM
> *To:* Espeland, James <james.espeland at montana.edu>
> *Cc:* CKAN Development Discussions <ckan-dev at lists.okfn.org>
> *Subject:* Re: [ckan-dev] CKAN 2.5.2 test suite errors
>
>
>
> Hi Jim,
>
>
>
> If you download https://github.com/DataShades/opswx-ckan-
> cookbook/blob/master/files/default/ckan_solr_conf.zip and extract it into
> a /conf where your solr core in the solr data directory is, this schema and
> solr config works.
>
>
>
> The setup we execute which will work in a simple set up as well is;
>
> Download and extract Solr 5.2.2
>
> Run the setup script that comes with Solr.
>
> We move the /var/solr directory to /data/solr so it's picked up by our
> daily backup process.
>
> We create a core in /data/solr/data. Typical naming would be dev_ckan.
>
> We create a core.properties file in /data/solr/data/dev_ckan with a single
> name=dev_ckan
>
> We extract the archive I linked to above to  /data/solr/data/dev_ckan/conf
> and create a /data/solr/data/dev_ckan/data directory.
>
>
>
> It's so long since we created this config for Solr 5 that I can't remember
> the specific changes we made. From memory it was changing the data types
> Solr uses. When Solr changed from 4 to 5, a couple of the data types
> changed. That at least is my foggy recollection.
>
>
>
> I'm fairly certain the process outlined above should get you up and
> running. No doubt adding this core to the Solr install you have should
> suffice after you restart Solr and point CKAN at it.
>
>
>
>
>
>
>
>
> Shane.
>
>
>
> ------------------------------------------------------------------
> *SHANE DAVIS*  |  *SERVICE DELIVERY MANAGER*
> Link Digital
>
>
>
>
>
>
>
>
>
> On Tue, Aug 9, 2016 at 8:31 AM, Espeland, James <
> james.espeland at montana.edu> wrote:
>
> Hi Shane,
>
>
>
> I tried Solr 5.5.2 and while Solr seems to run just fine, I run into the
> same types of Solr errors (as Solr 6.0.x) when I run the CKAN test suite
> (undefined field, field can’t be found, unknown field).  I also tried Solr
> 4.10.4, but ran into numerous problems configuring and running that version
> of Solr.
>
>
>
> I saw your message regarding the work you’ve done on an enterprise stack
> for AWS, but at this time, we’re mostly interested in setting up a local
> instance of CKAN.
>
>
>
> I remember that you mentioned that you had seen similar errors to the ones
> I am seeing.  Can you recall what you did to correct those errors?  I
> suspect that it will require a configuration change of some sort.
>
>
>
> Thank you again for any assistance you can offer to me!
>
>
>
> Best regards,
>
>
>
> --- Jim Espeland
>
> james.espeland at montana.edu
>
>
>
>
>
> *From:* Shane Davis [mailto:shane.davis at linkdigital.com.au]
> *Sent:* Wednesday, August 3, 2016 3:42 PM
> *To:* CKAN Development Discussions <ckan-dev at lists.okfn.org>
> *Cc:* Espeland, James <james.espeland at montana.edu>
> *Subject:* Re: [ckan-dev] CKAN 2.5.2 test suite errors
>
>
>
> Hi Jim,
>
>
>
> Solr 5.5.2 definitely works. We're using that as a standard right now. If
> I get the opportunity over the next week, I'll try a Solr 6.1 install. When
> we moved from Solr 4 to Solr 5 I remember running into issues not
> dissimilar to the one's you're reporting. We're in final stages of UAT with
> Datashades which is a CKAN 2.5.2 build using Solr 5.5.2. At an absolute
> minimum that will provide an insight into the Solr config we're using.
>
>
>
> I'm extremely keen to get a release up on GitHub even if it's not a
> version one. There's a couple of known issues I need to fix beforehand, but
> a stable release is very close. I'll notify the list when it's actually
> published.
>
>
>
>
> Shane.
>
>
>
> ------------------------------------------------------------------
> *SHANE DAVIS*  |  *SERVICE DELIVERY MANAGER*
> Link Digital
>
> www.linkdigital.com.au
> p: *02 6111 2907* <02%206111%202907> | f: 02 6248 5582 <02%206248%205582>
> m: 0400 611 428
>
> t: @link_sdm
>
> s: link_sdm
> PO Box 199 Canberra ACT 2601
> 32 Lonsdale Street Braddon ACT 2612
>
> [image: Image removed by sender.] [image: Image removed by sender.] [image:
> Image removed by sender.]
>
>
>
> [image: Image removed by sender.] [image: Image removed by sender.] [image:
> Image removed by sender.]
>
>
>
>
>
> On Thu, Aug 4, 2016 at 4:51 AM, Espeland, James <
> james.espeland at montana.edu> wrote:
>
> Hello Adri?,
>
> Thank you very much for your reply to my request for assistance.
>
> Yes, it does appear that my Solr instance is not picking up the schema
> file.
>
> I'm not committed to using Solr 6.x, but I would like to use the most
> recent version that is known to work with the schema.xml file provided by
> CKAN.  I'm trying to set up CKAN 2.5.2.
>
> Can you recommend to me the latest known version of Solr that will work
> with the schema.xml file provided by CKAN?
>
> Thank you again,
>
> --- James Espeland
>
>
> > Date: Wed, 3 Aug 2016 14:26:09 +0100
> > From: Adri? Mercader <adria.mercader at okfn.org>
> > To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
> > Subject: Re: [ckan-dev] CKAN 2.5.2 test suite errors
> > Message-ID:
> >       <CAGJR8iJje3jFBG-
> > bcRJBJgLqkZxRSwGP1ieueGUw18f3L_4XYg at mail.gmail.com>
> > Content-Type: text/plain; charset=UTF-8
> >
> > Hi James,
> >
> > These errors seem to suggest that Solr is not using the right schema
> file.
> >
> > I'm not familiar with Solr 6 but I seem to recall from other messages on
> the list
> > that it required a special setup to use file based schemas.
> >
> > Maybe someone with more experience can provide more details.
> >
> > Adri?
> >
> >
> >
> > On 21 July 2016 at 20:34, Espeland, James <james.espeland at montana.edu>
> > wrote:
> > > Hello Everyone,
> > >
> > >
> > >
> > > I have been setting up a pilot instance of CKAN 2.5.2 following the
> > > CKAN documentation for Release 2.5.2.  I can run the Paste development
> > > server from the command line and see the CKAN front page.  The
> > > documentation then says to ?run CKAN?s tests to make sure that
> everything?s
> > working?.
> > >
> > >
> > >
> > > I'm still getting a number of errors when I run the tests.
> > >
> > >
> > >
> > > My pilot environment includes:
> > >
> > >
> > >
> > > centos 6.7
> > >
> > > ckan 2.5.2
> > >
> > > solr 6.0.0
> > >
> > > java 1.8.0
> > >
> > > python 2.7.11
> > >
> > > postgres 9.4.8
> > >
> > > jetty 9.3.8
> > >
> > >
> > >
> > > I have created a solr core named ckan1.  I set the solr_url entry in
> > > test-core.ini to:
> > >
> > >
> > >
> > > http://127.0.0.1:8393/solr/ckan1
> > >
> > >
> > >
> > > I soft-linked the ckan src schema.xml into my core config directory as
> > > follows:
> > >
> > >
> > >
> > > ln -s /usr/lib/ckan/default/src/ckan/ckan/config/solr/schema.xml
> > > /var/solr/data/ckan1/conf/schema.xml
> > >
> > >
> > >
> > > When I run the tests, I get the following types of errors in the
> > > solr.log
> > > file:
> > >
> > >
> > >
> > > **********
> > >
> > >
> > >
> > > 2016-07-21 19:05:02.647 ERROR (qtp1634198-20) [   x:ckan1]
> > > o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException:
> > > undefined field site_id
> > >
> > >
> > >
> > > 2016-07-21 19:05:09.544 ERROR (qtp1634198-15) [   x:ckan1]
> > > o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: s
> > >
> > > ort param field can't be found: metadata_modified
> > >
> > >
> > >
> > > 2016-07-21 19:05:21.759 ERROR (qtp1634198-13) [   x:ckan1]
> > > o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: E
> > >
> > > RROR: [doc=812b3b0a-30e4-4aa6-bcd6-9e1fe2072d22] unknown field
> > 'data_dict'
> > >
> > >
> > >
> > > 2016-07-21 19:05:40.053 ERROR (qtp1634198-13) [   x:ckan1]
> > > o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: E
> > >
> > > RROR: [doc=9315d7ca-7c3e-4e59-92f6-8d494e1d374d] unknown field
> > > 'res_description'
> > >
> > >
> > >
> > > 2016-07-21 19:05:56.431 ERROR (qtp1634198-15) [   x:ckan1]
> > > o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: u
> > >
> > > ndefined field capacity
> > >
> > >
> > >
> > > 016-07-21 19:06:56.184 ERROR (qtp1634198-15) [   x:ckan1]
> > > o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: E
> > >
> > > RROR: [doc=abed426b-cfa4-45b1-8403-cbf23c3fa899] unknown field
> > 'maintainer'
> > >
> > >
> > >
> > > 2016-07-21 19:07:38.205 ERROR (qtp1634198-15) [   x:ckan1]
> > > o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: E
> > >
> > > RROR: [doc=0cdcd6d7-8cf6-4767-bb3a-f9ed82141a6f] unknown field
> > 'owner_org'
> > >
> > >
> > >
> > > 2016-07-21 19:08:51.554 ERROR (qtp1634198-17) [   x:ckan1]
> > > o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: E
> > >
> > > RROR: [doc=d9567b82-d3f0-4c17-b222-d9a7499f7940] unknown field
> > 'capacity'
> > >
> > >
> > >
> > > 2016-07-21 19:08:51.638 ERROR (qtp1634198-20) [   x:ckan1]
> > > o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: E
> > >
> > > RROR: [doc=test-index] unknown field 'views_recent'
> > >
> > >
> > >
> > > 2016-07-21 19:08:51.727 ERROR (qtp1634198-16) [   x:ckan1]
> > > o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException:
> > > undefined field entity_type
> > >
> > >
> > >
> > > **********
> > >
> > >
> > >
> > > I suspect that something isn?t configured correctly.
> > >
> > >
> > >
> > > Any help or advice regarding how to correct this would be greatly
> > > appreciated.
> > >
> > >
> > >
> > > Best regards,
> > >
> > >
> > >
> > > --- Jim Espeland
> > >
> > > james.espeland at montana.edu
> > >
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20160810/f196324f/attachment-0003.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 524 bytes
Desc: not available
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20160810/f196324f/attachment-0003.jpg>