[ckan-dev] Updating Solr to support point search and confusion with Spatial Search Widget

Wed Jul 22 13:38:03 UTC 2015

Small update: Updating indexes failed when doing it with solr-spatial-field
backend (my previous email).
If I go back to the state before breaking the index (I am doing everything
in VM and have backups) and update the index with solr as backend, it just
states the error with points and polygons, but it works and all datasets
show in search after.
Still, when doing search with spatial widget (solr backend) after updating
search index the following new error appears:

2015-07-22 15:29:57,597 ERROR [ckan.controllers.package] Dataset search
error: ('SOLR returned an error running query: {\'sort\': u\'score desc,
metadata_modified desc\', \'fq\': [u\'+dataset_type:dataset
capacity:"public" +site_id:"oeg-otalex" +state:active\', \'{!frange
incl=false l=0
u=1}div(mul(mul(max(0,sub(min(125.859375,maxx),max(-35.15625,minx))),max(0,sub(min(63.2336274123,maxy),max(-21.2893743559,miny)))),2),add(13609.5239566,mul(sub(maxy,miny),sub(maxx,minx))))\'],
\'bf\':
\'div(mul(mul(max(0,sub(min(125.859375,maxx),max(-35.15625,minx))),max(0,sub(min(63.2336274123,maxy),max(-21.2893743559,miny)))),2),add(13609.5239566,mul(sub(maxy,miny),sub(maxx,minx))))\',
\'rows\': 21, \'facet.field\': [\'organization\', \'groups\', \'tags\',
\'res_format\', \'license_id\'], \'facet.limit\': \'50\', \'mm\': \'2<-1
5<80%\', \'defType\': \'edismax\', \'facet.mincount\': 1, \'q\': \'*:*\',
\'start\': 0, \'wt\': \'json\', \'qf\': \'name^4 title^4 tags^2 groups^2
text\', \'tie\': \'0.1\', \'facet\': \'true\', \'fl\': \'id
validated_data_dict\', \'fq_list\': [\'{!frange incl=false l=0
u=1}div(mul(mul(max(0,sub(min(125.859375,maxx),max(-35.15625,minx))),max(0,sub(min(63.2336274123,maxy),max(-21.2893743559,miny)))),2),add(13609.5239566,mul(sub(maxy,miny),sub(maxx,minx))))\']}
Error: \'Internal Server Error\'',)

I am doing my best to figure it out and fix widget search to work properly
with solr-spatial-field and points. If I manage I will share the
experience, but I tried a bunch of stuff and I am afraid I don't have any
more ideas.

--------
Filip Radulovic
www.filip.milstan.net

On Wed, Jul 22, 2015 at 1:56 PM, Filip Radulovic <filiprd at gmail.com> wrote:

> Hi Tanmay,
>
> Thanks for your response. I have tried to update search index but it
> failed. I get the following error:
>
> Unable to update search index.('Solr returned an error: 400 Bad Request -
> <?xml version="1.0" encoding="UTF-8"?>\n<response>\n<lst
> name="responseHeader"><int name="status">400</int><int
> name="QTime">5</int></lst><lst name="error"><str name="msg">ERROR:
> [doc=46f47f41e48f6135af991236e300c6d6] Error adding field
> \'spatial_geom\'=\'POINT (13.383333 52.516666)\' msg=Unable to read: POINT
> (13.383333 52.516666)</str><int
> name="code">400</int></lst>\n</response>\n',)
>
> The same error happens when using CKAN in the browser when trying to add
> or update a dataset with spatial attribute.
> I did not have that error before I updated Solr. Does anyone know what
> might be the problem? I've trying to find a solution the whole morning but
> failed.
>
> Thanks.
>
>
>
> --------
> Filip Radulovic
> www.filip.milstan.net
>
> On Wed, Jul 22, 2015 at 1:44 AM, Tanmay Thakur <tthakur at accela.com> wrote:
>
>>   Hi Filip,
>> You might want to try rebuilding the search-index.
>>
>> paster --plugin=ckan search-index rebuild --config=/etc/ckan/std/std.ini
>>
>>  For more details, see:
>>
>> http://docs.ckan.org/en/1117-start-new-test-suite/paster.html#search-index-rebuild-search-index
>>
>>
>>  Regards,
>> Tanmay Thakur
>> Software Engineer
>>  <http://www.accela.com/>
>>
>>   From: ckan-dev <ckan-dev-bounces at lists.okfn.org> on behalf of Filip
>> Radulovic <filiprd at gmail.com>
>> Reply-To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
>> Date: Tuesday, July 21, 2015 at 3:22 PM
>> To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
>> Subject: Re: [ckan-dev] Updating Solr to support point search and
>> confusion with Spatial Search Widget
>>
>>   Hi all,
>>
>>  I have managed to install another version of Solr (4.0.0) and to put
>> CKAN to work with it.
>> From what I can see it works fine, the only problem is that CKAN now does
>> not recognise in search any datasets that I previously uploaded (only the
>> new ones). Anyone has an idea about why is that?
>>
>>  In my user profile I can see and access all datasets, including the old
>> ones.
>>
>>
>>  For future reference, what I did is this:
>> - uninstall solr-jetty (apt-get command)
>> - install jetty only (apt-get command)
>> - manually downloading Solr and install it (see for example here
>> <http://www.kingstonlabs.com/blog/how-to-install-solr-36-on-ubuntu-1204/>
>> )
>> - setting up the schema.xml via symlink to CKAN file (similarly as in the
>> CKAN installation instructions)
>> - changing solr_url in development.ini to 127.0.0.1:8983/solr/sitename
>>
>>
>>
>>
>>  --------
>> Filip Radulovic
>> www.filip.milstan.net
>>
>> On Tue, Jul 21, 2015 at 8:00 PM, Filip Radulovic <filiprd at gmail.com>
>> wrote:
>>
>>> Hi all,
>>>
>>>  I have installed CKAN on Ubuntu 14.04, and I've added ckanext-spatial
>>> extension with the Spatial Search Widget and Dataset Extent Map.
>>> I have added a few datasets with extra attribute with 'spatial' key and
>>> both points and polygons.
>>>
>>>  Now, both are shown very nice in Dataset Extend Map, but when I try to
>>> filter datasets in search, only those that have a polygon associated with
>>> them are shown in results.
>>>
>>>  Form the installation instructions, it says that polygons and points
>>> are supported in solr-spatial-field backend (which requires solr 4.x+), but
>>> I am using solr backend (with solr 3.6.2 that came with running apt-get as
>>> specified in installation instructions). The thing is that first I tried to
>>> update solr to 4.0.0 but I failed, and only then I realised that search
>>> with polygons is working, but not with points.
>>>
>>>  So I am confused. How come the search is working with polygons at all
>>> if I don't have the right solr version and I haven't even put JTS, and how
>>> to enable the search to take into account datasets with point.
>>>
>>>  Maybe another question is how to update solr - the repository does not
>>> have other versions, and I couldn't do it manually because I could not find
>>> related folders and files in solr and jetty?
>>> Anyone has a clue how to proceed?
>>>
>>>  If it helps, I list here the output with an error in the console that
>>> I get when adding a dataset with spatial attribute which is a point (I
>>> don't see the same error when adding a polygon).
>>>
>>>  2015-07-21 19:49:29,922 DEBUG [ckanext.spatial.plugin] Received: u'{
>>> "type": "Point", "coordinates": [-3.716667,40.383335] }'
>>> 2015-07-21 19:49:29,956 DEBUG [ckanext.spatial.lib] Extent for package
>>> 10f6c260-76d8-481e-8f75-a7bc3e9735d2 unchanged
>>> 2015-07-21 19:49:31,470 INFO  [ckan.lib.base]
>>>  /api/3/action/resource_show render time 0.037 seconds
>>> 2015-07-21 19:49:31,942 DEBUG [ckanext.spatial.plugin] Received: u'{
>>> "type": "Point", "coordinates": [-3.716667,40.383335] }'
>>> 2015-07-21 19:49:32,000 DEBUG [ckanext.spatial.lib] Extent for package
>>> 10f6c260-76d8-481e-8f75-a7bc3e9735d2 unchanged
>>> 2015-07-21 19:49:32,633 ERROR [ckanext.spatial.plugin] Solr backend only
>>> supports bboxes, ignoring geometry { "type": "Point", "coordinates":
>>> [-3.716667,40.383335] }
>>> 2015-07-21 19:49:33,393 INFO  [ckan.lib.base]
>>>  /dataset/new_resource/bristol-energy-in-buildings render time 3.790 seconds
>>> 2015-07-21 19:49:33,403 INFO  [ckan.lib.base]
>>>  /api/3/action/resource_show render time 0.231 seconds
>>> /usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py:459:
>>> SAWarning: Unicode type received non-unicode bind param value.
>>>   param.append(processors[key](compiled_params[key]))
>>> Fetching from:
>>> http://127.0.0.1:5000/dataset/10f6c260-76d8-481e-8f75-a7bc3e9735d2/resource/faaacf90-e43a-4c34-892e-5c770d41e863/download/Bristol.csv
>>> 2015-07-21 19:49:33,966 WARNI [ckan.lib.maintain] Function
>>> _resource_preview() in module ckan.controllers.package has been deprecated
>>> and will be removed in a later release of ckan. Resource preview is
>>> deprecated. Please use the new resource views
>>> 2015-07-21 19:49:34,464 INFO  [ckan.lib.base]
>>>  /dataset/10f6c260-76d8-481e-8f75-a7bc3e9735d2/resource/faaacf90-e43a-4c34-892e-5c770d41e863/download/Bristol.csv
>>> render time 0.196 seconds
>>> 2015-07-21 19:49:34,620 WARNI [ckan.lib.maintain] Function get_action()
>>> in module ckan.lib.helpers has been deprecated and will be removed in a
>>> later release of ckan. helpers.get_action() is deprecated and will be
>>> removed in a future version of CKAN. Instead, please use the extra_vars
>>> param to render() in your controller to pass results from action functions
>>> to your templates.
>>> 2015-07-21 19:49:35,513 WARNI [ckan.lib.maintain] Function get_action()
>>> in module ckan.lib.helpers has been deprecated and will be removed in a
>>> later release of ckan. helpers.get_action() is deprecated and will be
>>> removed in a future version of CKAN. Instead, please use the extra_vars
>>> param to render() in your controller to pass results from action functions
>>> to your templates.
>>> Deleting "faaacf90-e43a-4c34-892e-5c770d41e863" from datastore.
>>> 2015-07-21 19:49:36,837 INFO  [ckan.lib.base]
>>>  /api/3/action/datastore_delete render time 0.063 seconds
>>> Determined headers and types: [{'type': u'timestamp', 'id':
>>> u'date_time'}, {'type': u'numeric', 'id': u'consumption'}, {'type':
>>> u'text', 'id': u'Intervention'}, {'type': u'text', 'id': u'Building'},
>>> {'type': u'text', 'id': u'Energy'}, {'type': u'text', 'id': u'City'}]
>>> Saving chunk 0
>>> 2015-07-21 19:49:37,919 INFO  [ckan.lib.base]
>>>  /api/3/action/resource_show render time 0.425 seconds
>>> Fetching from:
>>> http://127.0.0.1:5000/dataset/10f6c260-76d8-481e-8f75-a7bc3e9735d2/resource/faaacf90-e43a-4c34-892e-5c770d41e863/download/Bristol.csv
>>> 2015-07-21 19:49:38,150 INFO  [ckan.lib.base]
>>>  /dataset/bristol-energy-in-buildings render time 4.359 seconds
>>> 2015-07-21 19:49:39,227 INFO  [ckan.lib.base]
>>>  /dataset/10f6c260-76d8-481e-8f75-a7bc3e9735d2/resource/faaacf90-e43a-4c34-892e-5c770d41e863/download/Bristol.csv
>>> render time 0.412 seconds
>>> Deleting "faaacf90-e43a-4c34-892e-5c770d41e863" from datastore.
>>> 2015-07-21 19:49:40,568 INFO  [ckan.lib.base]
>>>  /api/3/action/datastore_create render time 2.245 seconds
>>> Saving chunk 1
>>> 2015-07-21 19:49:41,267 INFO  [ckan.lib.base]
>>>  /api/3/action/datastore_delete render time 0.313 seconds
>>> Determined headers and types: [{'type': u'timestamp', 'id':
>>> u'date_time'}, {'type': u'numeric', 'id': u'consumption'}, {'type':
>>> u'text', 'id': u'Intervention'}, {'type': u'text', 'id': u'Building'},
>>> {'type': u'text', 'id': u'Energy'}, {'type': u'text', 'id': u'City'}]
>>> 2015-07-21 19:49:41,629 INFO  [ckan.lib.base]
>>>  /api/3/action/datastore_create render time 0.394 seconds
>>> Successfully pushed 274 entries to
>>> "faaacf90-e43a-4c34-892e-5c770d41e863".
>>> Saving chunk 0
>>> 2015-07-21 19:49:42,479 INFO  [ckan.lib.base]
>>>  /api/3/action/datapusher_hook render time 0.338 seconds
>>> 2015-07-21 19:49:42,946 INFO  [ckan.lib.base]
>>>  /api/3/action/datastore_create render time 0.396 seconds
>>> Saving chunk 1
>>> 2015-07-21 19:49:43,975 INFO  [ckan.lib.base]
>>>  /api/3/action/datastore_create render time 0.692 seconds
>>> Successfully pushed 274 entries to
>>> "faaacf90-e43a-4c34-892e-5c770d41e863".
>>> 2015-07-21 19:49:44,235 INFO  [ckan.lib.base]  /api/i18n/en render time
>>> 0.003 seconds
>>> 2015-07-21 19:49:44,652 INFO  [ckan.lib.base]
>>>  /api/3/action/datapusher_hook render time 0.169 seconds
>>>
>>>
>>>  Cheers
>>>
>>>
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20150722/e3d00e95/attachment-0003.html>