[ckan-dev] Updating Solr to support point search and confusion with Spatial Search Widget

Wed Jul 22 11:56:32 UTC 2015

Hi Tanmay,

Thanks for your response. I have tried to update search index but it
failed. I get the following error:

Unable to update search index.('Solr returned an error: 400 Bad Request -
<?xml version="1.0" encoding="UTF-8"?>\n<response>\n<lst
name="responseHeader"><int name="status">400</int><int
name="QTime">5</int></lst><lst name="error"><str name="msg">ERROR:
[doc=46f47f41e48f6135af991236e300c6d6] Error adding field
\'spatial_geom\'=\'POINT (13.383333 52.516666)\' msg=Unable to read: POINT
(13.383333 52.516666)</str><int
name="code">400</int></lst>\n</response>\n',)

The same error happens when using CKAN in the browser when trying to add or
update a dataset with spatial attribute.
I did not have that error before I updated Solr. Does anyone know what
might be the problem? I've trying to find a solution the whole morning but
failed.

Thanks.

--------
Filip Radulovic
www.filip.milstan.net

On Wed, Jul 22, 2015 at 1:44 AM, Tanmay Thakur <tthakur at accela.com> wrote:

>   Hi Filip,
> You might want to try rebuilding the search-index.
>
> paster --plugin=ckan search-index rebuild --config=/etc/ckan/std/std.ini
>
>  For more details, see:
>
> http://docs.ckan.org/en/1117-start-new-test-suite/paster.html#search-index-rebuild-search-index
>
>
>  Regards,
> Tanmay Thakur
> Software Engineer
>  <http://www.accela.com/>
>
>   From: ckan-dev <ckan-dev-bounces at lists.okfn.org> on behalf of Filip
> Radulovic <filiprd at gmail.com>
> Reply-To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
> Date: Tuesday, July 21, 2015 at 3:22 PM
> To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
> Subject: Re: [ckan-dev] Updating Solr to support point search and
> confusion with Spatial Search Widget
>
>   Hi all,
>
>  I have managed to install another version of Solr (4.0.0) and to put
> CKAN to work with it.
> From what I can see it works fine, the only problem is that CKAN now does
> not recognise in search any datasets that I previously uploaded (only the
> new ones). Anyone has an idea about why is that?
>
>  In my user profile I can see and access all datasets, including the old
> ones.
>
>
>  For future reference, what I did is this:
> - uninstall solr-jetty (apt-get command)
> - install jetty only (apt-get command)
> - manually downloading Solr and install it (see for example here
> <http://www.kingstonlabs.com/blog/how-to-install-solr-36-on-ubuntu-1204/>)
> - setting up the schema.xml via symlink to CKAN file (similarly as in the
> CKAN installation instructions)
> - changing solr_url in development.ini to 127.0.0.1:8983/solr/sitename
>
>
>
>
>  --------
> Filip Radulovic
> www.filip.milstan.net
>
> On Tue, Jul 21, 2015 at 8:00 PM, Filip Radulovic <filiprd at gmail.com>
> wrote:
>
>> Hi all,
>>
>>  I have installed CKAN on Ubuntu 14.04, and I've added ckanext-spatial
>> extension with the Spatial Search Widget and Dataset Extent Map.
>> I have added a few datasets with extra attribute with 'spatial' key and
>> both points and polygons.
>>
>>  Now, both are shown very nice in Dataset Extend Map, but when I try to
>> filter datasets in search, only those that have a polygon associated with
>> them are shown in results.
>>
>>  Form the installation instructions, it says that polygons and points
>> are supported in solr-spatial-field backend (which requires solr 4.x+), but
>> I am using solr backend (with solr 3.6.2 that came with running apt-get as
>> specified in installation instructions). The thing is that first I tried to
>> update solr to 4.0.0 but I failed, and only then I realised that search
>> with polygons is working, but not with points.
>>
>>  So I am confused. How come the search is working with polygons at all
>> if I don't have the right solr version and I haven't even put JTS, and how
>> to enable the search to take into account datasets with point.
>>
>>  Maybe another question is how to update solr - the repository does not
>> have other versions, and I couldn't do it manually because I could not find
>> related folders and files in solr and jetty?
>> Anyone has a clue how to proceed?
>>
>>  If it helps, I list here the output with an error in the console that I
>> get when adding a dataset with spatial attribute which is a point (I don't
>> see the same error when adding a polygon).
>>
>>  2015-07-21 19:49:29,922 DEBUG [ckanext.spatial.plugin] Received: u'{
>> "type": "Point", "coordinates": [-3.716667,40.383335] }'
>> 2015-07-21 19:49:29,956 DEBUG [ckanext.spatial.lib] Extent for package
>> 10f6c260-76d8-481e-8f75-a7bc3e9735d2 unchanged
>> 2015-07-21 19:49:31,470 INFO  [ckan.lib.base]
>>  /api/3/action/resource_show render time 0.037 seconds
>> 2015-07-21 19:49:31,942 DEBUG [ckanext.spatial.plugin] Received: u'{
>> "type": "Point", "coordinates": [-3.716667,40.383335] }'
>> 2015-07-21 19:49:32,000 DEBUG [ckanext.spatial.lib] Extent for package
>> 10f6c260-76d8-481e-8f75-a7bc3e9735d2 unchanged
>> 2015-07-21 19:49:32,633 ERROR [ckanext.spatial.plugin] Solr backend only
>> supports bboxes, ignoring geometry { "type": "Point", "coordinates":
>> [-3.716667,40.383335] }
>> 2015-07-21 19:49:33,393 INFO  [ckan.lib.base]
>>  /dataset/new_resource/bristol-energy-in-buildings render time 3.790 seconds
>> 2015-07-21 19:49:33,403 INFO  [ckan.lib.base]
>>  /api/3/action/resource_show render time 0.231 seconds
>> /usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py:459:
>> SAWarning: Unicode type received non-unicode bind param value.
>>   param.append(processors[key](compiled_params[key]))
>> Fetching from:
>> http://127.0.0.1:5000/dataset/10f6c260-76d8-481e-8f75-a7bc3e9735d2/resource/faaacf90-e43a-4c34-892e-5c770d41e863/download/Bristol.csv
>> 2015-07-21 19:49:33,966 WARNI [ckan.lib.maintain] Function
>> _resource_preview() in module ckan.controllers.package has been deprecated
>> and will be removed in a later release of ckan. Resource preview is
>> deprecated. Please use the new resource views
>> 2015-07-21 19:49:34,464 INFO  [ckan.lib.base]
>>  /dataset/10f6c260-76d8-481e-8f75-a7bc3e9735d2/resource/faaacf90-e43a-4c34-892e-5c770d41e863/download/Bristol.csv
>> render time 0.196 seconds
>> 2015-07-21 19:49:34,620 WARNI [ckan.lib.maintain] Function get_action()
>> in module ckan.lib.helpers has been deprecated and will be removed in a
>> later release of ckan. helpers.get_action() is deprecated and will be
>> removed in a future version of CKAN. Instead, please use the extra_vars
>> param to render() in your controller to pass results from action functions
>> to your templates.
>> 2015-07-21 19:49:35,513 WARNI [ckan.lib.maintain] Function get_action()
>> in module ckan.lib.helpers has been deprecated and will be removed in a
>> later release of ckan. helpers.get_action() is deprecated and will be
>> removed in a future version of CKAN. Instead, please use the extra_vars
>> param to render() in your controller to pass results from action functions
>> to your templates.
>> Deleting "faaacf90-e43a-4c34-892e-5c770d41e863" from datastore.
>> 2015-07-21 19:49:36,837 INFO  [ckan.lib.base]
>>  /api/3/action/datastore_delete render time 0.063 seconds
>> Determined headers and types: [{'type': u'timestamp', 'id':
>> u'date_time'}, {'type': u'numeric', 'id': u'consumption'}, {'type':
>> u'text', 'id': u'Intervention'}, {'type': u'text', 'id': u'Building'},
>> {'type': u'text', 'id': u'Energy'}, {'type': u'text', 'id': u'City'}]
>> Saving chunk 0
>> 2015-07-21 19:49:37,919 INFO  [ckan.lib.base]
>>  /api/3/action/resource_show render time 0.425 seconds
>> Fetching from:
>> http://127.0.0.1:5000/dataset/10f6c260-76d8-481e-8f75-a7bc3e9735d2/resource/faaacf90-e43a-4c34-892e-5c770d41e863/download/Bristol.csv
>> 2015-07-21 19:49:38,150 INFO  [ckan.lib.base]
>>  /dataset/bristol-energy-in-buildings render time 4.359 seconds
>> 2015-07-21 19:49:39,227 INFO  [ckan.lib.base]
>>  /dataset/10f6c260-76d8-481e-8f75-a7bc3e9735d2/resource/faaacf90-e43a-4c34-892e-5c770d41e863/download/Bristol.csv
>> render time 0.412 seconds
>> Deleting "faaacf90-e43a-4c34-892e-5c770d41e863" from datastore.
>> 2015-07-21 19:49:40,568 INFO  [ckan.lib.base]
>>  /api/3/action/datastore_create render time 2.245 seconds
>> Saving chunk 1
>> 2015-07-21 19:49:41,267 INFO  [ckan.lib.base]
>>  /api/3/action/datastore_delete render time 0.313 seconds
>> Determined headers and types: [{'type': u'timestamp', 'id':
>> u'date_time'}, {'type': u'numeric', 'id': u'consumption'}, {'type':
>> u'text', 'id': u'Intervention'}, {'type': u'text', 'id': u'Building'},
>> {'type': u'text', 'id': u'Energy'}, {'type': u'text', 'id': u'City'}]
>> 2015-07-21 19:49:41,629 INFO  [ckan.lib.base]
>>  /api/3/action/datastore_create render time 0.394 seconds
>> Successfully pushed 274 entries to "faaacf90-e43a-4c34-892e-5c770d41e863".
>> Saving chunk 0
>> 2015-07-21 19:49:42,479 INFO  [ckan.lib.base]
>>  /api/3/action/datapusher_hook render time 0.338 seconds
>> 2015-07-21 19:49:42,946 INFO  [ckan.lib.base]
>>  /api/3/action/datastore_create render time 0.396 seconds
>> Saving chunk 1
>> 2015-07-21 19:49:43,975 INFO  [ckan.lib.base]
>>  /api/3/action/datastore_create render time 0.692 seconds
>> Successfully pushed 274 entries to "faaacf90-e43a-4c34-892e-5c770d41e863".
>> 2015-07-21 19:49:44,235 INFO  [ckan.lib.base]  /api/i18n/en render time
>> 0.003 seconds
>> 2015-07-21 19:49:44,652 INFO  [ckan.lib.base]
>>  /api/3/action/datapusher_hook render time 0.169 seconds
>>
>>
>>  Cheers
>>
>>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20150722/9d957c8f/attachment-0003.html>