[ckan-dev] Updating Solr to support point search and confusion with Spatial Search Widget

Tue Jul 21 23:44:24 UTC 2015

Hi Filip,
You might want to try rebuilding the search-index.

paster --plugin=ckan search-index rebuild --config=/etc/ckan/std/std.ini

For more details, see:
http://docs.ckan.org/en/1117-start-new-test-suite/paster.html#search-index-rebuild-search-index

Regards,
Tanmay Thakur
Software Engineer
[http://www.accela.com/ext/logo_email.png]<http://www.accela.com/>

From: ckan-dev <ckan-dev-bounces at lists.okfn.org<mailto:ckan-dev-bounces at lists.okfn.org>> on behalf of Filip Radulovic <filiprd at gmail.com<mailto:filiprd at gmail.com>>
Reply-To: CKAN Development Discussions <ckan-dev at lists.okfn.org<mailto:ckan-dev at lists.okfn.org>>
Date: Tuesday, July 21, 2015 at 3:22 PM
To: CKAN Development Discussions <ckan-dev at lists.okfn.org<mailto:ckan-dev at lists.okfn.org>>
Subject: Re: [ckan-dev] Updating Solr to support point search and confusion with Spatial Search Widget

Hi all,

I have managed to install another version of Solr (4.0.0) and to put CKAN to work with it.
>From what I can see it works fine, the only problem is that CKAN now does not recognise in search any datasets that I previously uploaded (only the new ones). Anyone has an idea about why is that?

In my user profile I can see and access all datasets, including the old ones.

For future reference, what I did is this:
- uninstall solr-jetty (apt-get command)
- install jetty only (apt-get command)
- manually downloading Solr and install it (see for example here<http://www.kingstonlabs.com/blog/how-to-install-solr-36-on-ubuntu-1204/>)
- setting up the schema.xml via symlink to CKAN file (similarly as in the CKAN installation instructions)
- changing solr_url in development.ini to 127.0.0.1:8983/solr/sitename<http://127.0.0.1:8983/solr/sitename>

--------
Filip Radulovic
www.filip.milstan.net<http://www.filip.milstan.net>

On Tue, Jul 21, 2015 at 8:00 PM, Filip Radulovic <filiprd at gmail.com<mailto:filiprd at gmail.com>> wrote:
Hi all,

I have installed CKAN on Ubuntu 14.04, and I've added ckanext-spatial extension with the Spatial Search Widget and Dataset Extent Map.
I have added a few datasets with extra attribute with 'spatial' key and both points and polygons.

Now, both are shown very nice in Dataset Extend Map, but when I try to filter datasets in search, only those that have a polygon associated with them are shown in results.

Form the installation instructions, it says that polygons and points are supported in solr-spatial-field backend (which requires solr 4.x+), but I am using solr backend (with solr 3.6.2 that came with running apt-get as specified in installation instructions). The thing is that first I tried to update solr to 4.0.0 but I failed, and only then I realised that search with polygons is working, but not with points.

So I am confused. How come the search is working with polygons at all if I don't have the right solr version and I haven't even put JTS, and how to enable the search to take into account datasets with point.

Maybe another question is how to update solr - the repository does not have other versions, and I couldn't do it manually because I could not find related folders and files in solr and jetty?
Anyone has a clue how to proceed?

If it helps, I list here the output with an error in the console that I get when adding a dataset with spatial attribute which is a point (I don't see the same error when adding a polygon).

2015-07-21 19:49:29,922 DEBUG [ckanext.spatial.plugin] Received: u'{ "type": "Point", "coordinates": [-3.716667,40.383335] }'
2015-07-21 19:49:29,956 DEBUG [ckanext.spatial.lib] Extent for package 10f6c260-76d8-481e-8f75-a7bc3e9735d2 unchanged
2015-07-21 19:49:31,470 INFO  [ckan.lib.base]  /api/3/action/resource_show render time 0.037 seconds
2015-07-21 19:49:31,942 DEBUG [ckanext.spatial.plugin] Received: u'{ "type": "Point", "coordinates": [-3.716667,40.383335] }'
2015-07-21 19:49:32,000 DEBUG [ckanext.spatial.lib] Extent for package 10f6c260-76d8-481e-8f75-a7bc3e9735d2 unchanged
2015-07-21 19:49:32,633 ERROR [ckanext.spatial.plugin] Solr backend only supports bboxes, ignoring geometry { "type": "Point", "coordinates": [-3.716667,40.383335] }
2015-07-21 19:49:33,393 INFO  [ckan.lib.base]  /dataset/new_resource/bristol-energy-in-buildings render time 3.790 seconds
2015-07-21 19:49:33,403 INFO  [ckan.lib.base]  /api/3/action/resource_show render time 0.231 seconds
/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py:459: SAWarning: Unicode type received non-unicode bind param value.
  param.append(processors[key](compiled_params[key]))
Fetching from: http://127.0.0.1:5000/dataset/10f6c260-76d8-481e-8f75-a7bc3e9735d2/resource/faaacf90-e43a-4c34-892e-5c770d41e863/download/Bristol.csv
2015-07-21 19:49:33,966 WARNI [ckan.lib.maintain] Function _resource_preview() in module ckan.controllers.package has been deprecated and will be removed in a later release of ckan. Resource preview is deprecated. Please use the new resource views
2015-07-21 19:49:34,464 INFO  [ckan.lib.base]  /dataset/10f6c260-76d8-481e-8f75-a7bc3e9735d2/resource/faaacf90-e43a-4c34-892e-5c770d41e863/download/Bristol.csv render time 0.196 seconds
2015-07-21 19:49:34,620 WARNI [ckan.lib.maintain] Function get_action() in module ckan.lib.helpers has been deprecated and will be removed in a later release of ckan. helpers.get_action() is deprecated and will be removed in a future version of CKAN. Instead, please use the extra_vars param to render() in your controller to pass results from action functions to your templates.
2015-07-21 19:49:35,513 WARNI [ckan.lib.maintain] Function get_action() in module ckan.lib.helpers has been deprecated and will be removed in a later release of ckan. helpers.get_action() is deprecated and will be removed in a future version of CKAN. Instead, please use the extra_vars param to render() in your controller to pass results from action functions to your templates.
Deleting "faaacf90-e43a-4c34-892e-5c770d41e863" from datastore.
2015-07-21 19:49:36,837 INFO  [ckan.lib.base]  /api/3/action/datastore_delete render time 0.063 seconds
Determined headers and types: [{'type': u'timestamp', 'id': u'date_time'}, {'type': u'numeric', 'id': u'consumption'}, {'type': u'text', 'id': u'Intervention'}, {'type': u'text', 'id': u'Building'}, {'type': u'text', 'id': u'Energy'}, {'type': u'text', 'id': u'City'}]
Saving chunk 0
2015-07-21 19:49:37,919 INFO  [ckan.lib.base]  /api/3/action/resource_show render time 0.425 seconds
Fetching from: http://127.0.0.1:5000/dataset/10f6c260-76d8-481e-8f75-a7bc3e9735d2/resource/faaacf90-e43a-4c34-892e-5c770d41e863/download/Bristol.csv
2015-07-21 19:49:38,150 INFO  [ckan.lib.base]  /dataset/bristol-energy-in-buildings render time 4.359 seconds
2015-07-21 19:49:39,227 INFO  [ckan.lib.base]  /dataset/10f6c260-76d8-481e-8f75-a7bc3e9735d2/resource/faaacf90-e43a-4c34-892e-5c770d41e863/download/Bristol.csv render time 0.412 seconds
Deleting "faaacf90-e43a-4c34-892e-5c770d41e863" from datastore.
2015-07-21 19:49:40,568 INFO  [ckan.lib.base]  /api/3/action/datastore_create render time 2.245 seconds
Saving chunk 1
2015-07-21 19:49:41,267 INFO  [ckan.lib.base]  /api/3/action/datastore_delete render time 0.313 seconds
Determined headers and types: [{'type': u'timestamp', 'id': u'date_time'}, {'type': u'numeric', 'id': u'consumption'}, {'type': u'text', 'id': u'Intervention'}, {'type': u'text', 'id': u'Building'}, {'type': u'text', 'id': u'Energy'}, {'type': u'text', 'id': u'City'}]
2015-07-21 19:49:41,629 INFO  [ckan.lib.base]  /api/3/action/datastore_create render time 0.394 seconds
Successfully pushed 274 entries to "faaacf90-e43a-4c34-892e-5c770d41e863".
Saving chunk 0
2015-07-21 19:49:42,479 INFO  [ckan.lib.base]  /api/3/action/datapusher_hook render time 0.338 seconds
2015-07-21 19:49:42,946 INFO  [ckan.lib.base]  /api/3/action/datastore_create render time 0.396 seconds
Saving chunk 1
2015-07-21 19:49:43,975 INFO  [ckan.lib.base]  /api/3/action/datastore_create render time 0.692 seconds
Successfully pushed 274 entries to "faaacf90-e43a-4c34-892e-5c770d41e863".
2015-07-21 19:49:44,235 INFO  [ckan.lib.base]  /api/i18n/en render time 0.003 seconds
2015-07-21 19:49:44,652 INFO  [ckan.lib.base]  /api/3/action/datapusher_hook render time 0.169 seconds

Cheers

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20150721/6cbbc450/attachment-0003.html>