[ckan-dev] solr

[David Read]

On 19 May 2011 11:10, Seb Bacon <seb.bacon at okfn.org> wrote:
> On 19 May 2011 11:03, David Read <david.read at okfn.org> wrote:
>> On 19 May 2011 10:29, Friedrich Lindenberg
>> <friedrich.lindenberg at okfn.org> wrote:
>>> Hi Seb,
>>>
>>> On Thu, May 19, 2011 at 11:19 AM, Seb Bacon <seb.bacon at okfn.org> wrote:
>>>> I was wondering, because supporting *both* postgres FTS *and* solr is
>>>> a bit tricky (in fact, we're effectively not supporting both, as the
>>>> faceting stuff has only been implemented for installations with solr)
>>
>> Ok, usual devil's advocate stuff...
>>
>> It feels like a fair bit of work to get the whole dev team and all
>> clients onto SOLR.
>>
>> What's the problem with the status quo of having faceting as an
>> optional feature?
>
> I don't think there's a big problem right now.  I can just see it
> becoming a problem when we start using solr features more extensively.
>  IIRC *not* having solr, at least relatively recently, wasn't handled
> elegantly in the code.  My fear is that this is likely to happen more
> over time.
>
> There's also a kind of branding/perception issue: over time, with
> postgres, you're getting a relatively scrappier user experience
> compared with solr.  We don't need to remove postgres support, but we
> could officially deprecate it, i.e. all new installs and documentation
> assume solr installed.
>
>> Seb and Friedrich, do you have we got a new feature or requirement
>> that you have in mind? If there is no other agenda or plan here then I
>> see no benefit to this proposed work.
>
> I don't see it as a feature but as refactoring to reduce technical debt.

I'll try and summarize what I see as the pluses & minuses of dropping postgres:

benefits:
* we don't have to explain any more that faceting is only provided if
you use SOLR
* future search features don't have to be implemented in both SOLR and postgres.
* we could refactor to remove the code to allow different search
backends (but maybe this is useful to keep as it is) which could be
seen as reducing technical debt

drawbacks:
* we have to write tests for solr search (we should do this anyway)
* all our team have to learn solr admin & schema writing
* there are some question marks in my mind about solr's schema
flexibility to cope with extras. I'd like to see the schema to be
automatically derived from the CKAN package schema, yet isn't it right
that the SOLR schema is designed to live in /etc/ and changes mean
restarting SOLR? I'm sure we could get round this, but starts to feel
inflexible. Especially for writing tests. Friedrich can you enlighten
me on this?

David

> As I say, I would be happy with officially deprecating non-solr
> installs rather than migrating or similar.  But we owe our new users
> clarity over what is recommended, I think.
>
> Seb
>
>
>>>> We're in a funny half-way place right now as the Solr stuff needs more
>>>> hand-holding and documentation to allow people to easily run with it,
>>>> but with postgres you don't get our full functionality.
>>>
>>> Thats very true and my opinion on it would be to drop Postgres FTS
>>> support. Solr search is excellent and since search is a core feature
>>> of CKAN (until we've all figured out what "data collaboration" really
>>> means its not inaccurate to say it is *the* core) I think the
>>> additional overhead of deploying a Java-based service might be
>>> justified. Removing Psql-FTS would drive down the complexity of the
>>> search code significantly.
>>>
>>> This certainly needs some more discussion but I'd like this thread to
>>> end in voting or a BDFL judgement.
>>> - Friedrich
>>>
>>> _______________________________________________
>>> ckan-dev mailing list
>>> ckan-dev at lists.okfn.org
>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>>
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
>