[ckan-dev] CKAN 2.5.2 test suite errors

Wed Aug 10 04:42:55 UTC 2016

Hi Shane,

Your solution works beautifully with Solr 5.5.2!  As you said, even with a few errors in the test suite, the CKAN system seems to be functioning.

Just as an experiment, I tried this with Solr 6.1.0 and it does not work (the Solr admin interface says there is a problem with the core).

I cannot thank you enough for your assistance with this last hurdle and thank you to the many other people who answered questions for me along the way.

Best regards,

--- Jim Espeland
james.espeland at montana.edu

From: Shane Davis [mailto:shane.davis at linkdigital.com.au]
Sent: Tuesday, August 9, 2016 10:25 PM
To: Espeland, James <james.espeland at montana.edu>
Cc: CKAN Development Discussions <ckan-dev at lists.okfn.org>
Subject: Re: [ckan-dev] CKAN 2.5.2 test suite errors

Hi Jim,

Very glad to hear this worked.

On a base install, with no extensions installed, and final configuration incomplete, CKAN functions despite errors in the the test suite. I suspect there are people on the dev-list who are familiar with the errors you've listed, and would know what their relevance is.

As we refine the public stack we're releasing, I would envision that passing the test-suite would be one of the benchmarks we'd use to determine that the stack is ready for release 1.

Our internal situation is possibly a little unique in that we have at least four key teams within the organisation that pump out CKAN builds.
We've got Orchestration (which is my arena), Dev, Operations and Sysadmin. On that basis I'm doing the initial infrastructure build of CKAN, but the Dev and Ops teams are configuring, theming and putting all the finishing touches on it. I'm in the process of expanding my overall understanding of the application so I can go far deeper into the stack.

Eliminating 194 odd errors sounds like a big win in the mean time. Hopefully the others aren't too difficult to eliminate.

Shane.

------------------------------------------------------------------
SHANE DAVIS  |  SERVICE DELIVERY MANAGER
Link Digital
www.linkdigital.com.au<http://www.linkdigital.com.au/>

On Wed, Aug 10, 2016 at 5:20 AM, Espeland, James <james.espeland at montana.edu<mailto:james.espeland at montana.edu>> wrote:
Wow Shane!

Thank you so much for your assistance!  By following your instructions, we were able to cut the number of errors we were seeing from the CKAN 2.5.2 test suite significantly!

We didn’t move /var/solr to /data/solr, but if you do, we should point out to other readers that /etc/default/solr.in.sh<http://solr.in.sh> would have to be adjusted accordingly.

Your conf directory contents supplied via github were definitely most useful!

Before we were seeing around 200 errors when running the CKAN 2.5.2 test suite.  Now we are seeing 4 errors and 6 failures.  Here is a filtered version of the test output (warnings and debug statements removed):

**********

ERROR: ckan.tests.config.test_environment.TestUpdateConfig.test_update_config_db_url_precedence
ERROR: ckan.tests.config.test_environment.TestUpdateConfig.test_update_config_db_url_precedence
ERROR: ckan.tests.config.test_environment.TestUpdateConfig.test_update_config_env_vars
ERROR: ckan.tests.config.test_environment.TestUpdateConfig.test_update_config_env_vars
FAIL: ckan.tests.legacy.functional.api.test_activity.TestActivity.test_add_tag
FAIL: ckan.tests.legacy.functional.api.test_activity.TestActivity.test_delete_extras
FAIL: ckan.tests.legacy.functional.api.test_util.TestUtil.test_status
FAIL: ckan.tests.legacy.logic.test_action.TestAction.test_03_create_update_package
FAIL: ckan.tests.legacy.logic.test_action.TestAction.test_26_resource_show
FAIL: ckan.tests.logic.action.test_get.TestPackageShow.test_package_show_is_lazy

Ran 1923 tests in 1062.937s

FAILED (SKIP=12, errors=4, failures=6)

**********

Do you see any errors or failures when you run the test suite?

Are our results at an acceptable level of success?

If not, do you have any ideas concerning corrections we can make to achieve better results?

Thank you again, Shane!  Your advice and configuration files are most graciously appreciated!

Best regards,

--- Jim Espeland
james.espeland at montana.edu<mailto:james.espeland at montana.edu>

From: Shane Davis [mailto:shane.davis at linkdigital.com.au<mailto:shane.davis at linkdigital.com.au>]
Sent: Monday, August 8, 2016 5:49 PM
To: Espeland, James <james.espeland at montana.edu<mailto:james.espeland at montana.edu>>
Cc: CKAN Development Discussions <ckan-dev at lists.okfn.org<mailto:ckan-dev at lists.okfn.org>>
Subject: Re: [ckan-dev] CKAN 2.5.2 test suite errors

Hi Jim,

If you download https://github.com/DataShades/opswx-ckan-cookbook/blob/master/files/default/ckan_solr_conf.zip and extract it into a /conf where your solr core in the solr data directory is, this schema and solr config works.

The setup we execute which will work in a simple set up as well is;
Download and extract Solr 5.2.2
Run the setup script that comes with Solr.
We move the /var/solr directory to /data/solr so it's picked up by our daily backup process.
We create a core in /data/solr/data. Typical naming would be dev_ckan.
We create a core.properties file in /data/solr/data/dev_ckan with a single name=dev_ckan
We extract the archive I linked to above to  /data/solr/data/dev_ckan/conf and create a /data/solr/data/dev_ckan/data directory.

It's so long since we created this config for Solr 5 that I can't remember the specific changes we made. From memory it was changing the data types Solr uses. When Solr changed from 4 to 5, a couple of the data types changed. That at least is my foggy recollection.

I'm fairly certain the process outlined above should get you up and running. No doubt adding this core to the Solr install you have should suffice after you restart Solr and point CKAN at it.

Shane.

------------------------------------------------------------------
SHANE DAVIS  |  SERVICE DELIVERY MANAGER
Link Digital

On Tue, Aug 9, 2016 at 8:31 AM, Espeland, James <james.espeland at montana.edu<mailto:james.espeland at montana.edu>> wrote:
Hi Shane,

I tried Solr 5.5.2 and while Solr seems to run just fine, I run into the same types of Solr errors (as Solr 6.0.x) when I run the CKAN test suite (undefined field, field can’t be found, unknown field).  I also tried Solr 4.10.4, but ran into numerous problems configuring and running that version of Solr.

I saw your message regarding the work you’ve done on an enterprise stack for AWS, but at this time, we’re mostly interested in setting up a local instance of CKAN.

I remember that you mentioned that you had seen similar errors to the ones I am seeing.  Can you recall what you did to correct those errors?  I suspect that it will require a configuration change of some sort.

Thank you again for any assistance you can offer to me!

Best regards,

--- Jim Espeland
james.espeland at montana.edu<mailto:james.espeland at montana.edu>

From: Shane Davis [mailto:shane.davis at linkdigital.com.au<mailto:shane.davis at linkdigital.com.au>]
Sent: Wednesday, August 3, 2016 3:42 PM
To: CKAN Development Discussions <ckan-dev at lists.okfn.org<mailto:ckan-dev at lists.okfn.org>>
Cc: Espeland, James <james.espeland at montana.edu<mailto:james.espeland at montana.edu>>
Subject: Re: [ckan-dev] CKAN 2.5.2 test suite errors

Hi Jim,

Solr 5.5.2 definitely works. We're using that as a standard right now. If I get the opportunity over the next week, I'll try a Solr 6.1 install. When we moved from Solr 4 to Solr 5 I remember running into issues not dissimilar to the one's you're reporting. We're in final stages of UAT with Datashades which is a CKAN 2.5.2 build using Solr 5.5.2. At an absolute minimum that will provide an insight into the Solr config we're using.

I'm extremely keen to get a release up on GitHub even if it's not a version one. There's a couple of known issues I need to fix beforehand, but a stable release is very close. I'll notify the list when it's actually published.

Shane.

------------------------------------------------------------------
SHANE DAVIS  |  SERVICE DELIVERY MANAGER
Link Digital
www.linkdigital.com.au<http://www.linkdigital.com.au>
p: 02 6111 2907<tel:02%206111%202907> | f: 02 6248 5582<tel:02%206248%205582>
m: 0400 611 428<tel:0400%20611%20428>
t: @link_sdm
s: link_sdm
PO Box 199 Canberra ACT 2601
32 Lonsdale Street Braddon ACT 2612
[Image removed by sender.] [Image removed by sender.]  [Image removed by sender.]

[Image removed by sender.] [Image removed by sender.]  [Image removed by sender.]

On Thu, Aug 4, 2016 at 4:51 AM, Espeland, James <james.espeland at montana.edu<mailto:james.espeland at montana.edu>> wrote:
Hello Adri?,

Thank you very much for your reply to my request for assistance.

Yes, it does appear that my Solr instance is not picking up the schema file.

I'm not committed to using Solr 6.x, but I would like to use the most recent version that is known to work with the schema.xml file provided by CKAN.  I'm trying to set up CKAN 2.5.2.

Can you recommend to me the latest known version of Solr that will work with the schema.xml file provided by CKAN?

Thank you again,

--- James Espeland

> Date: Wed, 3 Aug 2016 14:26:09 +0100
> From: Adri? Mercader <adria.mercader at okfn.org<mailto:adria.mercader at okfn.org>>
> To: CKAN Development Discussions <ckan-dev at lists.okfn.org<mailto:ckan-dev at lists.okfn.org>>
> Subject: Re: [ckan-dev] CKAN 2.5.2 test suite errors
> Message-ID:
>       <CAGJR8iJje3jFBG-
> bcRJBJgLqkZxRSwGP1ieueGUw18f3L_4XYg at mail.gmail.com<mailto:bcRJBJgLqkZxRSwGP1ieueGUw18f3L_4XYg at mail.gmail.com>>
> Content-Type: text/plain; charset=UTF-8
>
> Hi James,
>
> These errors seem to suggest that Solr is not using the right schema file.
>
> I'm not familiar with Solr 6 but I seem to recall from other messages on the list
> that it required a special setup to use file based schemas.
>
> Maybe someone with more experience can provide more details.
>
> Adri?
>
>
>
> On 21 July 2016 at 20:34, Espeland, James <james.espeland at montana.edu<mailto:james.espeland at montana.edu>>
> wrote:
> > Hello Everyone,
> >
> >
> >
> > I have been setting up a pilot instance of CKAN 2.5.2 following the
> > CKAN documentation for Release 2.5.2.  I can run the Paste development
> > server from the command line and see the CKAN front page.  The
> > documentation then says to ?run CKAN?s tests to make sure that everything?s
> working?.
> >
> >
> >
> > I'm still getting a number of errors when I run the tests.
> >
> >
> >
> > My pilot environment includes:
> >
> >
> >
> > centos 6.7
> >
> > ckan 2.5.2
> >
> > solr 6.0.0
> >
> > java 1.8.0
> >
> > python 2.7.11
> >
> > postgres 9.4.8
> >
> > jetty 9.3.8
> >
> >
> >
> > I have created a solr core named ckan1.  I set the solr_url entry in
> > test-core.ini to:
> >
> >
> >
> > http://127.0.0.1:8393/solr/ckan1
> >
> >
> >
> > I soft-linked the ckan src schema.xml into my core config directory as
> > follows:
> >
> >
> >
> > ln -s /usr/lib/ckan/default/src/ckan/ckan/config/solr/schema.xml
> > /var/solr/data/ckan1/conf/schema.xml
> >
> >
> >
> > When I run the tests, I get the following types of errors in the
> > solr.log
> > file:
> >
> >
> >
> > **********
> >
> >
> >
> > 2016-07-21 19:05:02.647 ERROR (qtp1634198-20) [   x:ckan1]
> > o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException:
> > undefined field site_id
> >
> >
> >
> > 2016-07-21 19:05:09.544 ERROR (qtp1634198-15) [   x:ckan1]
> > o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: s
> >
> > ort param field can't be found: metadata_modified
> >
> >
> >
> > 2016-07-21 19:05:21.759 ERROR (qtp1634198-13) [   x:ckan1]
> > o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: E
> >
> > RROR: [doc=812b3b0a-30e4-4aa6-bcd6-9e1fe2072d22] unknown field
> 'data_dict'
> >
> >
> >
> > 2016-07-21 19:05:40.053 ERROR (qtp1634198-13) [   x:ckan1]
> > o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: E
> >
> > RROR: [doc=9315d7ca-7c3e-4e59-92f6-8d494e1d374d] unknown field
> > 'res_description'
> >
> >
> >
> > 2016-07-21 19:05:56.431 ERROR (qtp1634198-15) [   x:ckan1]
> > o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: u
> >
> > ndefined field capacity
> >
> >
> >
> > 016-07-21 19:06:56.184 ERROR (qtp1634198-15) [   x:ckan1]
> > o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: E
> >
> > RROR: [doc=abed426b-cfa4-45b1-8403-cbf23c3fa899] unknown field
> 'maintainer'
> >
> >
> >
> > 2016-07-21 19:07:38.205 ERROR (qtp1634198-15) [   x:ckan1]
> > o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: E
> >
> > RROR: [doc=0cdcd6d7-8cf6-4767-bb3a-f9ed82141a6f] unknown field
> 'owner_org'
> >
> >
> >
> > 2016-07-21 19:08:51.554 ERROR (qtp1634198-17) [   x:ckan1]
> > o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: E
> >
> > RROR: [doc=d9567b82-d3f0-4c17-b222-d9a7499f7940] unknown field
> 'capacity'
> >
> >
> >
> > 2016-07-21 19:08:51.638 ERROR (qtp1634198-20) [   x:ckan1]
> > o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: E
> >
> > RROR: [doc=test-index] unknown field 'views_recent'
> >
> >
> >
> > 2016-07-21 19:08:51.727 ERROR (qtp1634198-16) [   x:ckan1]
> > o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException:
> > undefined field entity_type
> >
> >
> >
> > **********
> >
> >
> >
> > I suspect that something isn?t configured correctly.
> >
> >
> >
> > Any help or advice regarding how to correct this would be greatly
> > appreciated.
> >
> >
> >
> > Best regards,
> >
> >
> >
> > --- Jim Espeland
> >
> > james.espeland at montana.edu<mailto:james.espeland at montana.edu>
> >

_______________________________________________
ckan-dev mailing list
ckan-dev at lists.okfn.org<mailto:ckan-dev at lists.okfn.org>
https://lists.okfn.org/mailman/listinfo/ckan-dev
Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20160810/2212589e/attachment-0003.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 524 bytes
Desc: image001.jpg
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20160810/2212589e/attachment-0003.jpg>