[ckan-dev] paster index rebuild failure - ASCII

[David Read]

Michael,
Interesting - some unicode encoding problem in the indexing. I suggest
you put use print or pdb to poke about the referenced lines in
ckan/lib/search/index.py and see if you can find out if it's one
particular dataset with the problem, what character in the dataset is
causing the problem, and that will help us reproduce the issue.
David

On 22 August 2017 at 22:26, Michael Speth <SpethM at landcareresearch.co.nz> wrote:
> Greetings,
>
>   I am trying to rebuild the index on a CKAN 2.6.2 instance.  I have
> imported the database and resources from a CKAN 2.5 instance.  Many of the
> indexes report this error:
>
> 'UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 19723:
> ordinal not in range(128)'
>
>
> Of course, the byte and position are different for each index.  But the
> UnicodeDecodeError 'ascii' codec can't decode is the same.
>
>
> Any ideas of how to solve this problem?
>
>
> Stack Trace:
>
>
> Traceback (most recent call last):
>   File "/usr/lib/ckan/default/bin/paster", line 11, in <module>
>     sys.exit(run())
>   File
> "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py",
> line 102, in run
>     invoke(command, command_name, options, args[1:])
>   File
> "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py",
> line 141, in invoke
>     exit_code = runner.run(args)
>   File
> "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py",
> line 236, in run
>     result = self.command()
>   File "/usr/lib/ckan/default/src/ckan/ckan/lib/cli.py", line 439, in
> command
>     self.rebuild()
>   File "/usr/lib/ckan/default/src/ckan/ckan/lib/cli.py", line 456, in
> rebuild
>     rebuild(self.args[1])
>   File "/usr/lib/ckan/default/src/ckan/ckan/lib/search/__init__.py", line
> 162, in rebuild
>     package_index.insert_dict(pkg_dict)
>   File "/usr/lib/ckan/default/src/ckan/ckan/lib/search/index.py", line 76,
> in insert_dict
>     return self.update_dict(data)
>   File "/usr/lib/ckan/default/src/ckan/ckan/lib/search/index.py", line 101,
> in update_dict
>     self.index_package(pkg_dict, defer_commit)
>   File "/usr/lib/ckan/default/src/ckan/ckan/lib/search/index.py", line 288,
> in index_package
>     conn.add(docs=[pkg_dict], commit=commit)
>   File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/pysolr.py",
> line 875, in add
>     overwrite=overwrite, handler=handler)
>   File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/pysolr.py",
> line 471, in _update
>     return self._send_request('post', path, message, {'Content-type':
> 'text/xml; charset=utf-8'})
>   File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/pysolr.py",
> line 361, in _send_request
>     timeout=self.timeout)
>   File
> "/usr/lib/ckan/default/local/lib/python2.7/site-packages/requests/sessions.py",
> line 340, in post
>     return self.request('POST', url, data=data, **kwargs)
>   File
> "/usr/lib/ckan/default/local/lib/python2.7/site-packages/requests/sessions.py",
> line 279, in request
>     resp = self.send(prep, stream=stream, timeout=timeout, verify=verify,
> cert=cert, proxies=proxies)
>   File
> "/usr/lib/ckan/default/local/lib/python2.7/site-packages/requests/sessions.py",
> line 374, in send
>     r = adapter.send(request, **kwargs)
>   File
> "/usr/lib/ckan/default/local/lib/python2.7/site-packages/requests/adapters.py",
> line 174, in send
>     timeout=timeout
>   File
> "/usr/lib/ckan/default/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py",
> line 422, in urlopen
>     body=body, headers=headers)
>   File
> "/usr/lib/ckan/default/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py",
> line 274, in _make_request
>     conn.request(method, url, **httplib_request_kw)
>   File "/usr/lib/python2.7/httplib.py", line 1017, in request
>     self._send_request(method, url, body, headers)
>   File "/usr/lib/python2.7/httplib.py", line 1051, in _send_request
>     self.endheaders(body)
>   File "/usr/lib/python2.7/httplib.py", line 1013, in endheaders
>     self._send_output(message_body)
>   File "/usr/lib/python2.7/httplib.py", line 862, in _send_output
>     msg += message_body
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 19723:
> ordinal not in range(128)
>
>
>
> --
> Michael Speth
> DevOps Computer Engineer
> Landcare Research Consultant
>
> ________________________________
>
> Please consider the environment before printing this email
> Warning: This electronic message together with any attachments is
> confidential. If you receive it in error: (i) you must not read, use,
> disclose, copy or retain it; (ii) please contact the sender immediately by
> reply email and then delete the emails.
> The views expressed in this email may not be those of Landcare Research New
> Zealand Limited. http://www.landcareresearch.co.nz
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>