[ckan-dev] solr

[Seb Bacon]

On 24 May 2011 16:13, Rufus Pollock <rufus.pollock at okfn.org> wrote:
> On 24 May 2011 13:52, David Read <david.read at okfn.org> wrote:
> [...]
>> > I don't see it as a feature but as refactoring to reduce technical debt.
>>
>> I'll try and summarize what I see as the pluses & minuses of dropping postgres:
>
> I think dropping postgres is a really major step. I'd personally argue
> in the other direction: doing a bit of work to finish off the
> abstraction of the search interface that we started. At the moment the
> only thing we are really using that isn't in postgres is facets and
> that shouldn't be that hard to do. Requiring SOLR by default is a big
> ask -- it's a significant extra bit of kit to maintain.

But then, if it's not hard to do facets in postgres, why are we using
solr?  I understood this was the main motivation?

The way I see it, we should focus our development efforts on one
search backend.  I don't think we have time to maintain two.  If we
need nice stopwords and the like, and it's hard to do them in
postgres, then we should go with Solr.

Does abstracting the search backend away imply a search API that
supports facets (with a postgres implementation)?  How about stop
words?

Sorry for asking annoying questions without solutions :)  I shall try
to be more constructive soon...

Seb

>> benefits:
>> * we don't have to explain any more that faceting is only provided if
>> you use SOLR
>> * future search features don't have to be implemented in both SOLR and postgres.
>> * we could refactor to remove the code to allow different search
>> backends (but maybe this is useful to keep as it is) which could be
>> seen as reducing technical debt
>>
>> drawbacks:
>> * we have to write tests for solr search (we should do this anyway)
>> * all our team have to learn solr admin & schema writing
>> * there are some question marks in my mind about solr's schema
>> flexibility to cope with extras. I'd like to see the schema to be
>> automatically derived from the CKAN package schema, yet isn't it right
>> that the SOLR schema is designed to live in /etc/ and changes mean
>> restarting SOLR? I'm sure we could get round this, but starts to feel
>> inflexible. Especially for writing tests. Friedrich can you enlighten
>> me on this?
>>
>> David
>>
>> > As I say, I would be happy with officially deprecating non-solr
>> > installs rather than migrating or similar.  But we owe our new users
>> > clarity over what is recommended, I think.
>> >
>> > Seb
>> >
>> >
>> >>>> We're in a funny half-way place right now as the Solr stuff needs more
>> >>>> hand-holding and documentation to allow people to easily run with it,
>> >>>> but with postgres you don't get our full functionality.
>> >>>
>> >>> Thats very true and my opinion on it would be to drop Postgres FTS
>> >>> support. Solr search is excellent and since search is a core feature
>> >>> of CKAN (until we've all figured out what "data collaboration" really
>> >>> means its not inaccurate to say it is *the* core) I think the
>> >>> additional overhead of deploying a Java-based service might be
>> >>> justified. Removing Psql-FTS would drive down the complexity of the
>> >>> search code significantly.
>> >>>
>> >>> This certainly needs some more discussion but I'd like this thread to
>> >>> end in voting or a BDFL judgement.
>> >>> - Friedrich
>> >>>
>> >>> _______________________________________________
>> >>> ckan-dev mailing list
>> >>> ckan-dev at lists.okfn.org
>> >>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>> >>>
>> >>
>> >> _______________________________________________
>> >> ckan-dev mailing list
>> >> ckan-dev at lists.okfn.org
>> >> http://lists.okfn.org/mailman/listinfo/ckan-dev
>> >>
>> >
>> > _______________________________________________
>> > ckan-dev mailing list
>> > ckan-dev at lists.okfn.org
>> > http://lists.okfn.org/mailman/listinfo/ckan-dev
>> >
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>
>
>
> --
> Co-Founder, Open Knowledge Foundation
> Promoting Open Knowledge in a Digital Age
> http://www.okfn.org/ - http://blog.okfn.org/
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
>