[ckan-dev] Fetcher failing to start - empty routing key

Mon Jul 3 06:21:20 UTC 2017

Hi

Has anyone had trouble with the fetcher failing to start and keep exiting?

I put in some debugging and it appeared to be due to the RedisConsumer being initialised with an empty routing key.

I couldn’t work out a good fix. A couple of hours later it just seemed to clear it self. I guess something expired out of the cache. 

I do not know what originally caused it. I looked through the logs and checked the harvest jobs. There were two jobs that were left in running state. I aborted them one at a time checking to see if the fetcher came back.  
I restarted all the ckan jobs fetcher etc, and restarted redis.  

Luckily this was on the staging server. I wish I knew what caused the issue in the first instance.  I reran both harvests and they finished.

Anyway I just wondered if anyone else had come across an issue like this.

Cheers
  Karen

trace:
2017-07-03 13:10:34,853 DEBUG [ckanext.harvest.queue] Fetch queue consumer registered

Traceback (most recent call last):
  File "/usr/lib/ckan/dia/bin/paster", line 11, in <module>
    load_entry_point('PasteScript==2.0.2', 'console_scripts', 'paster')()
  File "/usr/lib/ckan/dia/lib/python2.7/site-packages/PasteScript-2.0.2-py2.7.egg/paste/script/command.py", line 102, in run
    invoke(command, command_name, options, args[1:])
  File "/usr/lib/ckan/dia/lib/python2.7/site-packages/PasteScript-2.0.2-py2.7.egg/paste/script/command.py", line 141, in invoke
    exit_code = runner.run(args)
  File "/usr/lib/ckan/dia/lib/python2.7/site-packages/PasteScript-2.0.2-py2.7.egg/paste/script/command.py", line 236, in run
    result = self.command()
  File "/usr/lib/ckan/dia/src/ckanext-harvest/ckanext/harvest/commands/harvester.py", line 190, in command
    for method, header, body in consumer.consume(queue=get_fetch_queue_name()):
  File "/usr/lib/ckan/dia/src/ckanext-harvest/ckanext/harvest/queue.py", line 203, in consume
    self.redis.set(self.persistance_key(body),
  File "/usr/lib/ckan/dia/src/ckanext-harvest/ckanext/harvest/queue.py", line 210, in persistance_key
    return self.routing_key + ':' + message[self.message_key]
TypeError: cannot concatenate 'str' and 'NoneType' objects