[ckan-dev] Updating Solr to support point search and confusion with Spatial Search Widget

Thu Jul 23 10:25:19 UTC 2015

Hi Sol,

I have followed your instructions and now the spatial search widget works
perfectly.
Thank you very much for the response!

Cheers

--------
Filip Radulovic
www.filip.milstan.net

On Thu, Jul 23, 2015 at 8:30 AM, Filip Radulovic <filiprd at gmail.com> wrote:

> Sure, I just wanted to check.
> Thanks again, I will try to fix the spatial search today and let you know
> what happened.
>
> Sent from my mobile device
> On Jul 23, 2015 08:06, "李承錱[Sol]" <u103133.u103135 at gmail.com> wrote:
>
>> Hi Filip:
>>
>> I installed PostgreSQL 9.3.9 and postgresql-9.3-postgis-2.1 with apt-get
>> just like what you have stated. But I thought your problem would not be
>> related with postgis since you used the solr-spatial-field backend.
>>
>>
>> Cheers,
>>
>> Sol
>>
>> 2015-07-23 13:51 GMT+08:00 Filip Radulovic <filiprd at gmail.com>:
>>
>>> Hi Sol,
>>>
>>> Thanks a lot for the comment. Since I haven't made it work yet, I
>>> decided to install spatial extension from scratch today so your comment is
>>> very helpful and came in the right moment. I used Solr 4.0.0 but I'll try
>>> with 5.1.0.
>>>
>>> Just one thing, which version of PostgreSQL and PostGIS you have? Mine
>>> PostgreSQL is installed with apt-get as suggested in CKAN installation
>>> instruction, and I did the same with PostGIS but I had to add version
>>> because the command from the instructions did not work (sudo apt-get
>>> install postgresql-9.3-postgis-2.1, note "-2.1" at the end).
>>>
>>> I will update when I finish reinstallation.
>>>
>>> Cheers
>>>
>>> Sent from my mobile device
>>> On Jul 23, 2015 06:19, "李承錱[Sol]" <u103133.u103135 at gmail.com> wrote:
>>>
>>>> Hi Filip:
>>>>
>>>> For debugging purposes, my suggestion is to check the solr log directly
>>>> since it is much easier to read. The log is located at
>>>> http://YOUR-SERVER-IP/solr/#/~logging.
>>>>
>>>> Not sure what caused your problem, but I just finished a CKAN 2.3 +
>>>> Solr 5.1.0 + ckanext-spatial with solr-spatial-field backend installation
>>>> on a Ubuntu 14.04 machine. Both indexing and searching (point, bbox and
>>>> polygon) have worked well so far.
>>>>
>>>> The following links are some installation notes:
>>>>
>>>> 1. Install Solr 5 for CKAN 2.3:
>>>> https://gist.github.com/u10313335/1cd42194992c501fe5c6
>>>> 2. ckanext-spatial w/ solr-spatial-field backend
>>>> https://gist.github.com/u10313335/5649488bc3a4037e1357
>>>>
>>>> (Note: I did not use the latest 5.2.1 version because it may fail when
>>>> restarting. For details, see https://github.com/ckan/ckan/issues/2524.)
>>>>
>>>> Hope that helps.
>>>>
>>>>
>>>> Cheers,
>>>>
>>>> Sol
>>>>
>>>> 2015-07-22 23:11 GMT+08:00 Filip Radulovic <filiprd at gmail.com>:
>>>>
>>>>> Hi all,
>>>>>
>>>>> So, another update: Instead of updating search index with
>>>>> solr-spatial-field backend in development.ini, I have first done an update
>>>>> with solr backend and then another search index update with
>>>>> solr-spatial-field. The search index was updated without errors, all
>>>>> datasets are listed in search, and adding points/polygons to spatial
>>>>> attribute in dataset does not generate any errors.
>>>>>
>>>>> However, the spatial search widget still does not show any results
>>>>> when performing search (with solr-spatial-field configuration). Did anyone
>>>>> have the same problem? Does spatial search work in your case?
>>>>>
>>>>> Cheers
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --------
>>>>> Filip Radulovic
>>>>> www.filip.milstan.net
>>>>>
>>>>> On Wed, Jul 22, 2015 at 3:38 PM, Filip Radulovic <filiprd at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Small update: Updating indexes failed when doing it with
>>>>>> solr-spatial-field backend (my previous email).
>>>>>> If I go back to the state before breaking the index (I am doing
>>>>>> everything in VM and have backups) and update the index with solr as
>>>>>> backend, it just states the error with points and polygons, but it works
>>>>>> and all datasets show in search after.
>>>>>> Still, when doing search with spatial widget (solr backend) after
>>>>>> updating search index the following new error appears:
>>>>>>
>>>>>> 2015-07-22 15:29:57,597 ERROR [ckan.controllers.package] Dataset
>>>>>> search error: ('SOLR returned an error running query: {\'sort\': u\'score
>>>>>> desc, metadata_modified desc\', \'fq\': [u\'+dataset_type:dataset
>>>>>> capacity:"public" +site_id:"oeg-otalex" +state:active\', \'{!frange
>>>>>> incl=false l=0
>>>>>> u=1}div(mul(mul(max(0,sub(min(125.859375,maxx),max(-35.15625,minx))),max(0,sub(min(63.2336274123,maxy),max(-21.2893743559,miny)))),2),add(13609.5239566,mul(sub(maxy,miny),sub(maxx,minx))))\'],
>>>>>> \'bf\':
>>>>>> \'div(mul(mul(max(0,sub(min(125.859375,maxx),max(-35.15625,minx))),max(0,sub(min(63.2336274123,maxy),max(-21.2893743559,miny)))),2),add(13609.5239566,mul(sub(maxy,miny),sub(maxx,minx))))\',
>>>>>> \'rows\': 21, \'facet.field\': [\'organization\', \'groups\', \'tags\',
>>>>>> \'res_format\', \'license_id\'], \'facet.limit\': \'50\', \'mm\': \'2<-1
>>>>>> 5<80%\', \'defType\': \'edismax\', \'facet.mincount\': 1, \'q\': \'*:*\',
>>>>>> \'start\': 0, \'wt\': \'json\', \'qf\': \'name^4 title^4 tags^2 groups^2
>>>>>> text\', \'tie\': \'0.1\', \'facet\': \'true\', \'fl\': \'id
>>>>>> validated_data_dict\', \'fq_list\': [\'{!frange incl=false l=0
>>>>>> u=1}div(mul(mul(max(0,sub(min(125.859375,maxx),max(-35.15625,minx))),max(0,sub(min(63.2336274123,maxy),max(-21.2893743559,miny)))),2),add(13609.5239566,mul(sub(maxy,miny),sub(maxx,minx))))\']}
>>>>>> Error: \'Internal Server Error\'',)
>>>>>>
>>>>>> I am doing my best to figure it out and fix widget search to work
>>>>>> properly with solr-spatial-field and points. If I manage I will share the
>>>>>> experience, but I tried a bunch of stuff and I am afraid I don't have any
>>>>>> more ideas.
>>>>>>
>>>>>>
>>>>>>
>>>>>> --------
>>>>>> Filip Radulovic
>>>>>> www.filip.milstan.net
>>>>>>
>>>>>> On Wed, Jul 22, 2015 at 1:56 PM, Filip Radulovic <filiprd at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Tanmay,
>>>>>>>
>>>>>>> Thanks for your response. I have tried to update search index but it
>>>>>>> failed. I get the following error:
>>>>>>>
>>>>>>> Unable to update search index.('Solr returned an error: 400 Bad
>>>>>>> Request - <?xml version="1.0" encoding="UTF-8"?>\n<response>\n<lst
>>>>>>> name="responseHeader"><int name="status">400</int><int
>>>>>>> name="QTime">5</int></lst><lst name="error"><str name="msg">ERROR:
>>>>>>> [doc=46f47f41e48f6135af991236e300c6d6] Error adding field
>>>>>>> \'spatial_geom\'=\'POINT (13.383333 52.516666)\' msg=Unable to read: POINT
>>>>>>> (13.383333 52.516666)</str><int
>>>>>>> name="code">400</int></lst>\n</response>\n',)
>>>>>>>
>>>>>>> The same error happens when using CKAN in the browser when trying to
>>>>>>> add or update a dataset with spatial attribute.
>>>>>>> I did not have that error before I updated Solr. Does anyone know
>>>>>>> what might be the problem? I've trying to find a solution the whole morning
>>>>>>> but failed.
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --------
>>>>>>> Filip Radulovic
>>>>>>> www.filip.milstan.net
>>>>>>>
>>>>>>> On Wed, Jul 22, 2015 at 1:44 AM, Tanmay Thakur <tthakur at accela.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>>   Hi Filip,
>>>>>>>> You might want to try rebuilding the search-index.
>>>>>>>>
>>>>>>>> paster --plugin=ckan search-index rebuild --config=/etc/ckan/std/std.ini
>>>>>>>>
>>>>>>>>  For more details, see:
>>>>>>>>
>>>>>>>> http://docs.ckan.org/en/1117-start-new-test-suite/paster.html#search-index-rebuild-search-index
>>>>>>>>
>>>>>>>>
>>>>>>>>  Regards,
>>>>>>>> Tanmay Thakur
>>>>>>>> Software Engineer
>>>>>>>>  <http://www.accela.com/>
>>>>>>>>
>>>>>>>>   From: ckan-dev <ckan-dev-bounces at lists.okfn.org> on behalf of
>>>>>>>> Filip Radulovic <filiprd at gmail.com>
>>>>>>>> Reply-To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
>>>>>>>> Date: Tuesday, July 21, 2015 at 3:22 PM
>>>>>>>> To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
>>>>>>>> Subject: Re: [ckan-dev] Updating Solr to support point search and
>>>>>>>> confusion with Spatial Search Widget
>>>>>>>>
>>>>>>>>   Hi all,
>>>>>>>>
>>>>>>>>  I have managed to install another version of Solr (4.0.0) and to
>>>>>>>> put CKAN to work with it.
>>>>>>>> From what I can see it works fine, the only problem is that CKAN
>>>>>>>> now does not recognise in search any datasets that I previously uploaded
>>>>>>>> (only the new ones). Anyone has an idea about why is that?
>>>>>>>>
>>>>>>>>  In my user profile I can see and access all datasets, including
>>>>>>>> the old ones.
>>>>>>>>
>>>>>>>>
>>>>>>>>  For future reference, what I did is this:
>>>>>>>> - uninstall solr-jetty (apt-get command)
>>>>>>>> - install jetty only (apt-get command)
>>>>>>>> - manually downloading Solr and install it (see for example here
>>>>>>>> <http://www.kingstonlabs.com/blog/how-to-install-solr-36-on-ubuntu-1204/>
>>>>>>>> )
>>>>>>>> - setting up the schema.xml via symlink to CKAN file (similarly as
>>>>>>>> in the CKAN installation instructions)
>>>>>>>> - changing solr_url in development.ini to
>>>>>>>> 127.0.0.1:8983/solr/sitename
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>  --------
>>>>>>>> Filip Radulovic
>>>>>>>> www.filip.milstan.net
>>>>>>>>
>>>>>>>> On Tue, Jul 21, 2015 at 8:00 PM, Filip Radulovic <filiprd at gmail.com
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>>  I have installed CKAN on Ubuntu 14.04, and I've added
>>>>>>>>> ckanext-spatial extension with the Spatial Search Widget and Dataset Extent
>>>>>>>>> Map.
>>>>>>>>> I have added a few datasets with extra attribute with 'spatial'
>>>>>>>>> key and both points and polygons.
>>>>>>>>>
>>>>>>>>>  Now, both are shown very nice in Dataset Extend Map, but when I
>>>>>>>>> try to filter datasets in search, only those that have a polygon associated
>>>>>>>>> with them are shown in results.
>>>>>>>>>
>>>>>>>>>  Form the installation instructions, it says that polygons and
>>>>>>>>> points are supported in solr-spatial-field backend (which requires solr
>>>>>>>>> 4.x+), but I am using solr backend (with solr 3.6.2 that came with running
>>>>>>>>> apt-get as specified in installation instructions). The thing is that first
>>>>>>>>> I tried to update solr to 4.0.0 but I failed, and only then I realised that
>>>>>>>>> search with polygons is working, but not with points.
>>>>>>>>>
>>>>>>>>>  So I am confused. How come the search is working with polygons
>>>>>>>>> at all if I don't have the right solr version and I haven't even put JTS,
>>>>>>>>> and how to enable the search to take into account datasets with point.
>>>>>>>>>
>>>>>>>>>  Maybe another question is how to update solr - the repository
>>>>>>>>> does not have other versions, and I couldn't do it manually because I could
>>>>>>>>> not find related folders and files in solr and jetty?
>>>>>>>>> Anyone has a clue how to proceed?
>>>>>>>>>
>>>>>>>>>  If it helps, I list here the output with an error in the console
>>>>>>>>> that I get when adding a dataset with spatial attribute which is a point (I
>>>>>>>>> don't see the same error when adding a polygon).
>>>>>>>>>
>>>>>>>>>  2015-07-21 19:49:29,922 DEBUG [ckanext.spatial.plugin] Received:
>>>>>>>>> u'{ "type": "Point", "coordinates": [-3.716667,40.383335] }'
>>>>>>>>> 2015-07-21 19:49:29,956 DEBUG [ckanext.spatial.lib] Extent for
>>>>>>>>> package 10f6c260-76d8-481e-8f75-a7bc3e9735d2 unchanged
>>>>>>>>> 2015-07-21 19:49:31,470 INFO  [ckan.lib.base]
>>>>>>>>>  /api/3/action/resource_show render time 0.037 seconds
>>>>>>>>> 2015-07-21 19:49:31,942 DEBUG [ckanext.spatial.plugin] Received:
>>>>>>>>> u'{ "type": "Point", "coordinates": [-3.716667,40.383335] }'
>>>>>>>>> 2015-07-21 19:49:32,000 DEBUG [ckanext.spatial.lib] Extent for
>>>>>>>>> package 10f6c260-76d8-481e-8f75-a7bc3e9735d2 unchanged
>>>>>>>>> 2015-07-21 19:49:32,633 ERROR [ckanext.spatial.plugin] Solr
>>>>>>>>> backend only supports bboxes, ignoring geometry { "type": "Point",
>>>>>>>>> "coordinates": [-3.716667,40.383335] }
>>>>>>>>> 2015-07-21 19:49:33,393 INFO  [ckan.lib.base]
>>>>>>>>>  /dataset/new_resource/bristol-energy-in-buildings render time 3.790 seconds
>>>>>>>>> 2015-07-21 19:49:33,403 INFO  [ckan.lib.base]
>>>>>>>>>  /api/3/action/resource_show render time 0.231 seconds
>>>>>>>>> /usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py:459:
>>>>>>>>> SAWarning: Unicode type received non-unicode bind param value.
>>>>>>>>>   param.append(processors[key](compiled_params[key]))
>>>>>>>>> Fetching from:
>>>>>>>>> http://127.0.0.1:5000/dataset/10f6c260-76d8-481e-8f75-a7bc3e9735d2/resource/faaacf90-e43a-4c34-892e-5c770d41e863/download/Bristol.csv
>>>>>>>>> 2015-07-21 19:49:33,966 WARNI [ckan.lib.maintain] Function
>>>>>>>>> _resource_preview() in module ckan.controllers.package has been deprecated
>>>>>>>>> and will be removed in a later release of ckan. Resource preview is
>>>>>>>>> deprecated. Please use the new resource views
>>>>>>>>> 2015-07-21 19:49:34,464 INFO  [ckan.lib.base]
>>>>>>>>>  /dataset/10f6c260-76d8-481e-8f75-a7bc3e9735d2/resource/faaacf90-e43a-4c34-892e-5c770d41e863/download/Bristol.csv
>>>>>>>>> render time 0.196 seconds
>>>>>>>>> 2015-07-21 19:49:34,620 WARNI [ckan.lib.maintain] Function
>>>>>>>>> get_action() in module ckan.lib.helpers has been deprecated and will be
>>>>>>>>> removed in a later release of ckan. helpers.get_action() is deprecated and
>>>>>>>>> will be removed in a future version of CKAN. Instead, please use the
>>>>>>>>> extra_vars param to render() in your controller to pass results from action
>>>>>>>>> functions to your templates.
>>>>>>>>> 2015-07-21 19:49:35,513 WARNI [ckan.lib.maintain] Function
>>>>>>>>> get_action() in module ckan.lib.helpers has been deprecated and will be
>>>>>>>>> removed in a later release of ckan. helpers.get_action() is deprecated and
>>>>>>>>> will be removed in a future version of CKAN. Instead, please use the
>>>>>>>>> extra_vars param to render() in your controller to pass results from action
>>>>>>>>> functions to your templates.
>>>>>>>>> Deleting "faaacf90-e43a-4c34-892e-5c770d41e863" from datastore.
>>>>>>>>> 2015-07-21 19:49:36,837 INFO  [ckan.lib.base]
>>>>>>>>>  /api/3/action/datastore_delete render time 0.063 seconds
>>>>>>>>> Determined headers and types: [{'type': u'timestamp', 'id':
>>>>>>>>> u'date_time'}, {'type': u'numeric', 'id': u'consumption'}, {'type':
>>>>>>>>> u'text', 'id': u'Intervention'}, {'type': u'text', 'id': u'Building'},
>>>>>>>>> {'type': u'text', 'id': u'Energy'}, {'type': u'text', 'id': u'City'}]
>>>>>>>>> Saving chunk 0
>>>>>>>>> 2015-07-21 19:49:37,919 INFO  [ckan.lib.base]
>>>>>>>>>  /api/3/action/resource_show render time 0.425 seconds
>>>>>>>>> Fetching from:
>>>>>>>>> http://127.0.0.1:5000/dataset/10f6c260-76d8-481e-8f75-a7bc3e9735d2/resource/faaacf90-e43a-4c34-892e-5c770d41e863/download/Bristol.csv
>>>>>>>>> 2015-07-21 19:49:38,150 INFO  [ckan.lib.base]
>>>>>>>>>  /dataset/bristol-energy-in-buildings render time 4.359 seconds
>>>>>>>>> 2015-07-21 19:49:39,227 INFO  [ckan.lib.base]
>>>>>>>>>  /dataset/10f6c260-76d8-481e-8f75-a7bc3e9735d2/resource/faaacf90-e43a-4c34-892e-5c770d41e863/download/Bristol.csv
>>>>>>>>> render time 0.412 seconds
>>>>>>>>> Deleting "faaacf90-e43a-4c34-892e-5c770d41e863" from datastore.
>>>>>>>>> 2015-07-21 19:49:40,568 INFO  [ckan.lib.base]
>>>>>>>>>  /api/3/action/datastore_create render time 2.245 seconds
>>>>>>>>> Saving chunk 1
>>>>>>>>> 2015-07-21 19:49:41,267 INFO  [ckan.lib.base]
>>>>>>>>>  /api/3/action/datastore_delete render time 0.313 seconds
>>>>>>>>> Determined headers and types: [{'type': u'timestamp', 'id':
>>>>>>>>> u'date_time'}, {'type': u'numeric', 'id': u'consumption'}, {'type':
>>>>>>>>> u'text', 'id': u'Intervention'}, {'type': u'text', 'id': u'Building'},
>>>>>>>>> {'type': u'text', 'id': u'Energy'}, {'type': u'text', 'id': u'City'}]
>>>>>>>>> 2015-07-21 19:49:41,629 INFO  [ckan.lib.base]
>>>>>>>>>  /api/3/action/datastore_create render time 0.394 seconds
>>>>>>>>> Successfully pushed 274 entries to
>>>>>>>>> "faaacf90-e43a-4c34-892e-5c770d41e863".
>>>>>>>>> Saving chunk 0
>>>>>>>>> 2015-07-21 19:49:42,479 INFO  [ckan.lib.base]
>>>>>>>>>  /api/3/action/datapusher_hook render time 0.338 seconds
>>>>>>>>> 2015-07-21 19:49:42,946 INFO  [ckan.lib.base]
>>>>>>>>>  /api/3/action/datastore_create render time 0.396 seconds
>>>>>>>>> Saving chunk 1
>>>>>>>>> 2015-07-21 19:49:43,975 INFO  [ckan.lib.base]
>>>>>>>>>  /api/3/action/datastore_create render time 0.692 seconds
>>>>>>>>> Successfully pushed 274 entries to
>>>>>>>>> "faaacf90-e43a-4c34-892e-5c770d41e863".
>>>>>>>>> 2015-07-21 19:49:44,235 INFO  [ckan.lib.base]  /api/i18n/en render
>>>>>>>>> time 0.003 seconds
>>>>>>>>> 2015-07-21 19:49:44,652 INFO  [ckan.lib.base]
>>>>>>>>>  /api/3/action/datapusher_hook render time 0.169 seconds
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>  Cheers
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> ckan-dev mailing list
>>>>>>>> ckan-dev at lists.okfn.org
>>>>>>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>>>>>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> ckan-dev mailing list
>>>>> ckan-dev at lists.okfn.org
>>>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> ckan-dev mailing list
>>>> ckan-dev at lists.okfn.org
>>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>>
>>>>
>>> _______________________________________________
>>> ckan-dev mailing list
>>> ckan-dev at lists.okfn.org
>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>
>>>
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20150723/731bd43a/attachment-0003.html>