[ckan-dev] Updating Solr to support point search and confusion with Spatial Search Widget

Tue Jul 21 22:22:54 UTC 2015

Hi all,

I have managed to install another version of Solr (4.0.0) and to put CKAN
to work with it.
>From what I can see it works fine, the only problem is that CKAN now does
not recognise in search any datasets that I previously uploaded (only the
new ones). Anyone has an idea about why is that?

In my user profile I can see and access all datasets, including the old
ones.

For future reference, what I did is this:
- uninstall solr-jetty (apt-get command)
- install jetty only (apt-get command)
- manually downloading Solr and install it (see for example here
<http://www.kingstonlabs.com/blog/how-to-install-solr-36-on-ubuntu-1204/>)
- setting up the schema.xml via symlink to CKAN file (similarly as in the
CKAN installation instructions)
- changing solr_url in development.ini to 127.0.0.1:8983/solr/sitename

--------
Filip Radulovic
www.filip.milstan.net

On Tue, Jul 21, 2015 at 8:00 PM, Filip Radulovic <filiprd at gmail.com> wrote:

> Hi all,
>
> I have installed CKAN on Ubuntu 14.04, and I've added ckanext-spatial
> extension with the Spatial Search Widget and Dataset Extent Map.
> I have added a few datasets with extra attribute with 'spatial' key and
> both points and polygons.
>
> Now, both are shown very nice in Dataset Extend Map, but when I try to
> filter datasets in search, only those that have a polygon associated with
> them are shown in results.
>
> Form the installation instructions, it says that polygons and points are
> supported in solr-spatial-field backend (which requires solr 4.x+), but I
> am using solr backend (with solr 3.6.2 that came with running apt-get as
> specified in installation instructions). The thing is that first I tried to
> update solr to 4.0.0 but I failed, and only then I realised that search
> with polygons is working, but not with points.
>
> So I am confused. How come the search is working with polygons at all if I
> don't have the right solr version and I haven't even put JTS, and how to
> enable the search to take into account datasets with point.
>
> Maybe another question is how to update solr - the repository does not
> have other versions, and I couldn't do it manually because I could not find
> related folders and files in solr and jetty?
> Anyone has a clue how to proceed?
>
> If it helps, I list here the output with an error in the console that I
> get when adding a dataset with spatial attribute which is a point (I don't
> see the same error when adding a polygon).
>
> 2015-07-21 19:49:29,922 DEBUG [ckanext.spatial.plugin] Received: u'{
> "type": "Point", "coordinates": [-3.716667,40.383335] }'
> 2015-07-21 19:49:29,956 DEBUG [ckanext.spatial.lib] Extent for package
> 10f6c260-76d8-481e-8f75-a7bc3e9735d2 unchanged
> 2015-07-21 19:49:31,470 INFO  [ckan.lib.base]  /api/3/action/resource_show
> render time 0.037 seconds
> 2015-07-21 19:49:31,942 DEBUG [ckanext.spatial.plugin] Received: u'{
> "type": "Point", "coordinates": [-3.716667,40.383335] }'
> 2015-07-21 19:49:32,000 DEBUG [ckanext.spatial.lib] Extent for package
> 10f6c260-76d8-481e-8f75-a7bc3e9735d2 unchanged
> 2015-07-21 19:49:32,633 ERROR [ckanext.spatial.plugin] Solr backend only
> supports bboxes, ignoring geometry { "type": "Point", "coordinates":
> [-3.716667,40.383335] }
> 2015-07-21 19:49:33,393 INFO  [ckan.lib.base]
>  /dataset/new_resource/bristol-energy-in-buildings render time 3.790 seconds
> 2015-07-21 19:49:33,403 INFO  [ckan.lib.base]  /api/3/action/resource_show
> render time 0.231 seconds
> /usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py:459:
> SAWarning: Unicode type received non-unicode bind param value.
>   param.append(processors[key](compiled_params[key]))
> Fetching from:
> http://127.0.0.1:5000/dataset/10f6c260-76d8-481e-8f75-a7bc3e9735d2/resource/faaacf90-e43a-4c34-892e-5c770d41e863/download/Bristol.csv
> 2015-07-21 19:49:33,966 WARNI [ckan.lib.maintain] Function
> _resource_preview() in module ckan.controllers.package has been deprecated
> and will be removed in a later release of ckan. Resource preview is
> deprecated. Please use the new resource views
> 2015-07-21 19:49:34,464 INFO  [ckan.lib.base]
>  /dataset/10f6c260-76d8-481e-8f75-a7bc3e9735d2/resource/faaacf90-e43a-4c34-892e-5c770d41e863/download/Bristol.csv
> render time 0.196 seconds
> 2015-07-21 19:49:34,620 WARNI [ckan.lib.maintain] Function get_action() in
> module ckan.lib.helpers has been deprecated and will be removed in a later
> release of ckan. helpers.get_action() is deprecated and will be removed in
> a future version of CKAN. Instead, please use the extra_vars param to
> render() in your controller to pass results from action functions to your
> templates.
> 2015-07-21 19:49:35,513 WARNI [ckan.lib.maintain] Function get_action() in
> module ckan.lib.helpers has been deprecated and will be removed in a later
> release of ckan. helpers.get_action() is deprecated and will be removed in
> a future version of CKAN. Instead, please use the extra_vars param to
> render() in your controller to pass results from action functions to your
> templates.
> Deleting "faaacf90-e43a-4c34-892e-5c770d41e863" from datastore.
> 2015-07-21 19:49:36,837 INFO  [ckan.lib.base]
>  /api/3/action/datastore_delete render time 0.063 seconds
> Determined headers and types: [{'type': u'timestamp', 'id': u'date_time'},
> {'type': u'numeric', 'id': u'consumption'}, {'type': u'text', 'id':
> u'Intervention'}, {'type': u'text', 'id': u'Building'}, {'type': u'text',
> 'id': u'Energy'}, {'type': u'text', 'id': u'City'}]
> Saving chunk 0
> 2015-07-21 19:49:37,919 INFO  [ckan.lib.base]  /api/3/action/resource_show
> render time 0.425 seconds
> Fetching from:
> http://127.0.0.1:5000/dataset/10f6c260-76d8-481e-8f75-a7bc3e9735d2/resource/faaacf90-e43a-4c34-892e-5c770d41e863/download/Bristol.csv
> 2015-07-21 19:49:38,150 INFO  [ckan.lib.base]
>  /dataset/bristol-energy-in-buildings render time 4.359 seconds
> 2015-07-21 19:49:39,227 INFO  [ckan.lib.base]
>  /dataset/10f6c260-76d8-481e-8f75-a7bc3e9735d2/resource/faaacf90-e43a-4c34-892e-5c770d41e863/download/Bristol.csv
> render time 0.412 seconds
> Deleting "faaacf90-e43a-4c34-892e-5c770d41e863" from datastore.
> 2015-07-21 19:49:40,568 INFO  [ckan.lib.base]
>  /api/3/action/datastore_create render time 2.245 seconds
> Saving chunk 1
> 2015-07-21 19:49:41,267 INFO  [ckan.lib.base]
>  /api/3/action/datastore_delete render time 0.313 seconds
> Determined headers and types: [{'type': u'timestamp', 'id': u'date_time'},
> {'type': u'numeric', 'id': u'consumption'}, {'type': u'text', 'id':
> u'Intervention'}, {'type': u'text', 'id': u'Building'}, {'type': u'text',
> 'id': u'Energy'}, {'type': u'text', 'id': u'City'}]
> 2015-07-21 19:49:41,629 INFO  [ckan.lib.base]
>  /api/3/action/datastore_create render time 0.394 seconds
> Successfully pushed 274 entries to "faaacf90-e43a-4c34-892e-5c770d41e863".
> Saving chunk 0
> 2015-07-21 19:49:42,479 INFO  [ckan.lib.base]
>  /api/3/action/datapusher_hook render time 0.338 seconds
> 2015-07-21 19:49:42,946 INFO  [ckan.lib.base]
>  /api/3/action/datastore_create render time 0.396 seconds
> Saving chunk 1
> 2015-07-21 19:49:43,975 INFO  [ckan.lib.base]
>  /api/3/action/datastore_create render time 0.692 seconds
> Successfully pushed 274 entries to "faaacf90-e43a-4c34-892e-5c770d41e863".
> 2015-07-21 19:49:44,235 INFO  [ckan.lib.base]  /api/i18n/en render time
> 0.003 seconds
> 2015-07-21 19:49:44,652 INFO  [ckan.lib.base]
>  /api/3/action/datapusher_hook render time 0.169 seconds
>
>
> Cheers
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20150722/1b2098d8/attachment-0003.html>