[ckan-discuss] Search

[Friedrich Lindenberg]

Hi David,

the same thing has been haunting me with regards to another Pylons app, but I finally figured it out: pastedeploy won't by default read the config file to configure the logger when run from mod_wsgi. In order for this to happen, we'll have to amend the launcher script for CKAN:

[...]
from paste.deploy import loadapp
config_filepath = os.path.join(instance_dir, config_file)
from paste.script.util.logging_config import fileConfig
fileConfig(config_filepath)
application = loadapp('config:%s' % config_filepath)

That will allow us to configure a logger like this:

# Logging configuration
[loggers]
keys = root, ckan

[handlers]
keys = file

[formatters]
keys = generic

[logger_root]
level = INFO
handlers = file

[logger_ckan]
level = DEBUG
handlers = file
qualname = ckan

[handler_file]
class = handlers.RotatingFileHandler
formatter = generic
level = NOTSET
args = ('/var/log/ckan/de.log', 'a', 2048, 3)

[formatter_generic]
format = %(asctime)s %(levelname)-5.5s [%(name)s] %(message)s

I'll take a few minutes tomorrow to adapt the other ckan instances. That should give us a clearer picture, especially with regards to common API errors - and maybe wrt out current performance problems.

Friedrich 

On Aug 16, 2010, at 10:46 PM, David Read wrote:

> Friedrich,
> 
> Many thanks for looking into this.
> 
> I noticed you did some stuff related to loggers. Did you get logging
> working in CKAN? Rufus and I tried a while ago with the ini settings
> and never managed to get it to produce a log file.
> 
> If so, I'd like to log in CKAN when these notifications are sent, to
> see if this is a problem on the CKAN side. I can reproduce creating
> these packages, and hopefully the problem.
> 
> David
> 
> On 16 August 2010 13:57, Friedrich Lindenberg <friedrich at pudo.org> wrote:
>> Hi David,
>> 
>> thanks for the notice - this is obviously a problem with the Solr index [1]. I have no theory on what caused these packages not to be indexed other than the really bad performance on eu3 this morning. The indexer's log [2] is clean, and CKAN only has the postgres-related crashes.
>> 
>> If you notice any pattern in this, please let me know. I'll try to write some consistency tests for the indexer that can be run against a live instance to check for the validity of its results.
>> 
>> Friedrich
>> 
>> [1] http://solr.okfn.org/solr/ckan/select/?q=tags:hesa&version=2.2&start=0&rows=10&indent=on&fq=site_id:test.ckan.net
>> [2] /home/okfn/var/ckanext/ckanext/solrindexer/stdout.test.log on eu4
>> 
>> On Aug 16, 2010, at 1:31 PM, David Read wrote:
>> 
>>> Hi Friedrich,
>>> 
>>> I'm getting confused with the new search running on test.ckan.net...
>>> This morning I've loaded in 5 packages via the API which all have the
>>> hesa tag. When I search for hesa it only comes up with one of those
>>> packages. It's like only one of the five is indexed. I see from the
>>> config that it's using SOLR search. Is it me or is there something not
>>> quite right here?
>>> 
>>> Dave
>> 
>>