[ckan-changes] [ckan/ckan] 1210eb: [#1838] Always create full-text search indexes

Tue Jul 22 14:38:37 UTC 2014

  Branch: refs/heads/1792-filterable-resource-views
  Home:   https://github.com/ckan/ckan
  Commit: 1210ebe3b1d520e1af357dbc130736891935a71b
      https://github.com/ckan/ckan/commit/1210ebe3b1d520e1af357dbc130736891935a71b
  Author: Vitor Baptista <vitor at vitorbaptista.com>
  Date:   2014-07-20 (Sun, 20 Jul 2014)

  Changed paths:
    M ckanext/datastore/db.py
    M ckanext/datastore/tests/test_create.py

  Log Message:
  -----------
  [#1838] Always create full-text search indexes

Before this, we only created FTS indexes when the user sent a `indexes` or
`primary_key` parameter to the `datastore_create` call, which only happens when
you're directly calling it (i.e. not uploading a file through CKAN's
interface).

This makes it faster to upload something to the datastore, but creates problems
as queries aren't on indexed fields. So this commit changes it to always create
at least the full-text search indexes.

  Commit: cbb08202f18f3079c12eb01ee84def52b2621723
      https://github.com/ckan/ckan/commit/cbb08202f18f3079c12eb01ee84def52b2621723
  Author: Vitor Baptista <vitor at vitorbaptista.com>
  Date:   2014-07-20 (Sun, 20 Jul 2014)

  Changed paths:
    M ckan/config/solr/schema.xml
    M ckan/lib/search/index.py
    M ckan/new_tests/lib/search/test_index.py
    M ckan/tests/functional/api/test_package_search.py
    M ckanext/datastore/db.py
    M ckanext/datastore/interfaces.py
    M ckanext/datastore/logic/action.py
    M ckanext/datastore/logic/schema.py
    M ckanext/datastore/plugin.py
    M ckanext/datastore/tests/test_search.py
    M doc/api/index.rst
    M doc/contributing/reviewing.rst

  Log Message:
  -----------
  Merge branch 'master' into 1838-fts-on-specific-columns

Conflicts:
	ckanext/datastore/plugin.py

  Commit: 27f54220cc8ada5df39e20edf848a497a6fbb84a
      https://github.com/ckan/ckan/commit/27f54220cc8ada5df39e20edf848a497a6fbb84a
  Author: Vitor Baptista <vitor at vitorbaptista.com>
  Date:   2014-07-20 (Sun, 20 Jul 2014)

  Changed paths:
    M ckanext/datastore/tests/test_create.py

  Log Message:
  -----------
  [#1838] Use assert_equal to facilitate test debugging

  Commit: ed9ece0fadb4891b8012fbd2e9ff4682b9c94f1c
      https://github.com/ckan/ckan/commit/ed9ece0fadb4891b8012fbd2e9ff4682b9c94f1c
  Author: Vitor Baptista <vitor at vitorbaptista.com>
  Date:   2014-07-21 (Mon, 21 Jul 2014)

  Changed paths:
    M ckanext/datastore/plugin.py
    M ckanext/datastore/tests/test_search.py

  Log Message:
  -----------
  [#1838] Calculate FTS rankings only on specific columns

The bug was that when we've done a full-text query on a specific column, the
query was OK, but the rank was calculated using all text from the row. It
messed up when trying to get DISTINCT values.

  Commit: d949a018e03c65998620ab510a255aa65904fd83
      https://github.com/ckan/ckan/commit/d949a018e03c65998620ab510a255aa65904fd83
  Author: Vitor Baptista <vitor at vitorbaptista.com>
  Date:   2014-07-21 (Mon, 21 Jul 2014)

  Changed paths:
    M ckanext/datastore/tests/test_create.py

  Log Message:
  -----------
  [#1838] Fix tests where we were asserting a assert_equals

  Commit: b50452da76170d77f41baef675fa71a16733f15f
      https://github.com/ckan/ckan/commit/b50452da76170d77f41baef675fa71a16733f15f
  Author: Vitor Baptista <vitor at vitorbaptista.com>
  Date:   2014-07-21 (Mon, 21 Jul 2014)

  Changed paths:
    M ckanext/datastore/db.py
    M ckanext/datastore/plugin.py
    A ckanext/datastore/tests/test_db.py
    M ckanext/datastore/tests/test_plugin.py

  Log Message:
  -----------
  [#1838] Set default FTS language using ckan.datastore.default_fts_lang

If you set `ckan.datastore.default_fts_lang` on the INI file, it'll be used as
the default language to be used when creating full-text search indexes and
queries. It can be overwritten on a per-request basis by using the `lang`
parameter. If none exist (neither `lang` or `ckan.datastore.default_fts_lang`),
"english" is used as a default.

  Commit: 8e365f5568f8f05c3c3c798565555e0537195f5f
      https://github.com/ckan/ckan/commit/8e365f5568f8f05c3c3c798565555e0537195f5f
  Author: Vitor Baptista <vitor at vitorbaptista.com>
  Date:   2014-07-22 (Tue, 22 Jul 2014)

  Changed paths:
    M ckan/new_tests/helpers.py
    M ckanext/datastore/tests/test_create.py

  Log Message:
  -----------
  [#1838] Fix test on FTS indexes creation

It's really bad that I'm now testing the number of indexes created, which is
quite flaky. The problem is that we change the index name depending on if we're
working on Postgres < or > 9. This works for now, but can be improved later.

  Commit: 3d8ba39e56c12b3a3f70fc8e6f609141c07fd99e
      https://github.com/ckan/ckan/commit/3d8ba39e56c12b3a3f70fc8e6f609141c07fd99e
  Author: Vitor Baptista <vitor at vitorbaptista.com>
  Date:   2014-07-22 (Tue, 22 Jul 2014)

  Changed paths:
    M ckanext/datastore/db.py

  Log Message:
  -----------
  [#1838] Refactor FTS index creation

  Commit: a9d2186a05ab0ad66bcfc412a0be739e118b4521
      https://github.com/ckan/ckan/commit/a9d2186a05ab0ad66bcfc412a0be739e118b4521
  Author: Vitor Baptista <vitor at vitorbaptista.com>
  Date:   2014-07-22 (Tue, 22 Jul 2014)

  Changed paths:
    M ckanext/datastore/db.py
    M ckanext/datastore/tests/test_create.py
    M ckanext/datastore/tests/test_db.py

  Log Message:
  -----------
  [#1838] Create index only if it doesn't exist

We do so by creating a unique index name that's specific to each pair
resource_id + field_name. Before trying to create an index, we check if an
index with this same name exist, and only create it if it doesn't.

  Commit: f8e8976d9fb418dc35d7ef3f1c422832a68a1cb8
      https://github.com/ckan/ckan/commit/f8e8976d9fb418dc35d7ef3f1c422832a68a1cb8
  Author: Vitor Baptista <vitor at vitorbaptista.com>
  Date:   2014-07-22 (Tue, 22 Jul 2014)

  Changed paths:
    M ckanext/datastore/tests/test_create.py

  Log Message:
  -----------
  [#1838] Refactor tests to be less flaky

Instead of asserting the number of indexes created, we now assert on the actual
index names.

  Commit: d2a6c4b5534cb17ab890e0d6c683a662a10ffbf0
      https://github.com/ckan/ckan/commit/d2a6c4b5534cb17ab890e0d6c683a662a10ffbf0
  Author: Vitor Baptista <vitor at vitorbaptista.com>
  Date:   2014-07-22 (Tue, 22 Jul 2014)

  Changed paths:
    M ckan/config/deployment.ini_tmpl
    M ckanext/datastore/logic/action.py
    M doc/maintaining/configuration.rst

  Log Message:
  -----------
  [#1838] Adds docs

  Commit: c6739d2652cc4358bb4441cd0c71e8ee967e726d
      https://github.com/ckan/ckan/commit/c6739d2652cc4358bb4441cd0c71e8ee967e726d
  Author: Vitor Baptista <vitor at vitorbaptista.com>
  Date:   2014-07-22 (Tue, 22 Jul 2014)

  Changed paths:
    M ckan/public/base/javascript/modules/resource-view-filters-form.js
    M ckan/public/base/javascript/modules/resource-view-filters.js

  Log Message:
  -----------
  [#1792] Refactoring resource views filters' dropdowns

I've cleaned the detection if there's more data than what's displayed, removed
some useless code toremove repeated values (now we have the DISTINCT
parameter), and made the dropdown at the resource view manage page just load 20
results, the same as the other dropdowns.

  Commit: 86c295b8561198bb333b90fee86e7b1222059fe9
      https://github.com/ckan/ckan/commit/86c295b8561198bb333b90fee86e7b1222059fe9
  Author: Vitor Baptista <vitor at vitorbaptista.com>
  Date:   2014-07-22 (Tue, 22 Jul 2014)

  Changed paths:
    M ckan/config/deployment.ini_tmpl
    M ckan/new_tests/helpers.py
    M ckanext/datastore/db.py
    M ckanext/datastore/logic/action.py
    M ckanext/datastore/plugin.py
    M ckanext/datastore/tests/test_create.py
    A ckanext/datastore/tests/test_db.py
    M ckanext/datastore/tests/test_plugin.py
    M ckanext/datastore/tests/test_search.py
    M doc/maintaining/configuration.rst

  Log Message:
  -----------
  Merge branch '1838-fts-on-specific-columns' into 1792-filterable-resource-views

Conflicts:
	ckanext/datastore/plugin.py

Compare: https://github.com/ckan/ckan/compare/2b37ef98382f...86c295b85611