[ckan-dev] Spatial search limits

Wed Apr 18 14:18:48 UTC 2012

Just to update on this: the limit we're hitting is exactly 1024
dataset results. The SOLR logs provide the full error:

too many boolean clauses
Caused by: org.apache.lucene.search.BooleanQuery$TooManyClauses:
maxClauseCount is set to 1024

So I've changed the value of maxBooleanClauses in solrconfig.xml from
1024 to a higher number. But having restarted Jetty, the error remains
and still reports 1024, so I'm trying to find out why.

Maybe I'll hit the 64kb Jetty request limit once this is solved...

Dave

On 17 April 2012 21:05, David Read <david.read at hackneyworkshop.com> wrote:
> I've hit a limit with the spatial search provided in ckanext-spatial.
> If you do a search over a geographical area that has too many results
> then I get a SOLR exception.
>
> Dataset search error: ('SOLR returned an error running
> query: {\'sort\': \'score desc, name asc\', \'fq\': \'
> +site_id:"dgu-shazam" +state:active\', \'facet.mincount\'
> : 1, \'rows\': 11, \'facet.field\': [\'groups\', \'tags\',
> \'res_format\', \'license\', \'resource-type\', \'UKL
> P\', \'license_id-is-ogl\', \'publisher\'], \'wt\': \'json\',
> \'facet.limit\': \'50\', \'facet\': \'true\', \'q\
> ': u\'(id:ff4cc143-c00a-46b7-81bb-5095372847b6 OR
> id:2e279200-0ec7-4f15-a2a5-cdc68e277944 OR id:5a705e99-5425-43
> 37-abbc-195516e32ae4 OR id:0fe66473-8eca-4114-9642-20fb8cc11391 OR ...snip...
>
> Jetty has a limit of the size of request that is sent to SOLR. It can
> be increased to 64k http://drupal.org/node/443980 but by my
> calculations that sets the limit at about 1500 results, which is still
> too low for our site.
>
> Have you experienced this and do you have any thoughts of how to overcome this?
>
> (BTW the relevant ckanext-spatial tests are currently broken due to
> the new API schema check process. I strongly believe that CKAN should
> only check that every parameter is provided when submitting by form -
> it is way too inconvenient in the API and logic layer. This is a form
> specific thing.)
>
> David