[ckan-changes] [ckan/ckan] 766dbe: [#1838] Small refactoring on plugin's datastore_se...

Mon Jul 21 08:34:08 UTC 2014

  Branch: refs/heads/1792-filterable-resource-views
  Home:   https://github.com/ckan/ckan
  Commit: 766dbe441e07bd60c4c04fb4093584493bbe4995
      https://github.com/ckan/ckan/commit/766dbe441e07bd60c4c04fb4093584493bbe4995
  Author: Vitor Baptista <vitor at vitorbaptista.com>
  Date:   2014-07-10 (Thu, 10 Jul 2014)

  Changed paths:
    M ckanext/datastore/plugin.py

  Log Message:
  -----------
  [#1838] Small refactoring on plugin's datastore_search()

I removed some unused variables, and stopped validating on _where(), as we're
already validating on datastore_validate().

  Commit: 30c6de01e3c050c32c1e84e454c86d6a2c8aa569
      https://github.com/ckan/ckan/commit/30c6de01e3c050c32c1e84e454c86d6a2c8aa569
  Author: Vitor Baptista <vitor at vitorbaptista.com>
  Date:   2014-07-10 (Thu, 10 Jul 2014)

  Changed paths:
    M ckanext/datastore/logic/schema.py
    M ckanext/datastore/plugin.py
    M ckanext/datastore/tests/test_search.py

  Log Message:
  -----------
  [#1838] Add full-text searching on specific fields on datastore_search

You still can send queries to `datastore_search` adding a `q` parameter as a
string, and they'll work as they always had (i.e. we're backwards compatible).
But now you're able to send `q` as a dict with a string value, as in:

```json
"q": {
    "title": "CKAN"
}
```

That would do a full-text search only on the "title" field, and return the
results. I haven't created the indexes yet, so these searches can be pretty
slow.

  Commit: 029ee11bf72acf488cdafdd1926c19cefc1549b3
      https://github.com/ckan/ckan/commit/029ee11bf72acf488cdafdd1926c19cefc1549b3
  Author: Vitor Baptista <vitor at vitorbaptista.com>
  Date:   2014-07-10 (Thu, 10 Jul 2014)

  Changed paths:
    M ckanext/datastore/plugin.py

  Log Message:
  -----------
  Revert "[#1838] Small refactoring on plugin's datastore_search()"

This reverts commit 766dbe441e07bd60c4c04fb4093584493bbe4995.

  Commit: 50c6fdefa202d1ed03b63f77f4337f6ea9d0fe8a
      https://github.com/ckan/ckan/commit/50c6fdefa202d1ed03b63f77f4337f6ea9d0fe8a
  Author: Vitor Baptista <vitor at vitorbaptista.com>
  Date:   2014-07-10 (Thu, 10 Jul 2014)

  Changed paths:
    M ckanext/datastore/plugin.py

  Log Message:
  -----------
  [#1838] Don't add "q" filters on fields that don't exist

We simply ignore them. We can't raise errors, because there might be another
extension that does understand them, so we can't say they're actually invalid.

  Commit: c74a9b27a8a358f252f9a030c9f9684f1e206718
      https://github.com/ckan/ckan/commit/c74a9b27a8a358f252f9a030c9f9684f1e206718
  Author: Vitor Baptista <vitor at vitorbaptista.com>
  Date:   2014-07-10 (Thu, 10 Jul 2014)

  Changed paths:
    M ckanext/datastore/plugin.py

  Log Message:
  -----------
  [#1838] Fix bg where we were limiting filters to columns on the fields param

The problem was that `_where` and `_sort` needed a list of all columns, but if
the user sent a field list using the `fields` parameter, they actually received
the list of fields in that paramter.

This fixes that.

  Commit: 9f9f4114d7bb4a663cbccad6ef1dae5fa14b26c9
      https://github.com/ckan/ckan/commit/9f9f4114d7bb4a663cbccad6ef1dae5fa14b26c9
  Author: Vitor Baptista <vitor at vitorbaptista.com>
  Date:   2014-07-16 (Wed, 16 Jul 2014)

  Changed paths:
    M ckanext/datastore/tests/test_create.py

  Log Message:
  -----------
  [#1838] Add tests for current index creation code

  Commit: 9072a9413ee3188c64e4700da823fa4d64e365f2
      https://github.com/ckan/ckan/commit/9072a9413ee3188c64e4700da823fa4d64e365f2
  Author: Vitor Baptista <vitor at vitorbaptista.com>
  Date:   2014-07-16 (Wed, 16 Jul 2014)

  Changed paths:
    M ckanext/datastore/db.py
    M ckanext/datastore/tests/test_create.py

  Log Message:
  -----------
  [#1838] Create FTS indexes on each textual field

We create those indexes at the same time we're indexing `_full_text`. This is
needed to allow full-text searches on specific columns, and not only on the
entire row.

For the test, unfortunately I couldn't test that a FTS index was created for
each specific field, but instead rely on the number of indexes, because as the
index is not on the column itself, but on the return value of `to_tsvector`, we
don't have that information available.

There're still issues with this code as we're using English for default (and
there's no way to overwrite that).

  Commit: 48c3b55ab713a9bb9267f1490569d64691075fb8
      https://github.com/ckan/ckan/commit/48c3b55ab713a9bb9267f1490569d64691075fb8
  Author: Vitor Baptista <vitor at vitorbaptista.com>
  Date:   2014-07-17 (Thu, 17 Jul 2014)

  Changed paths:
    M ckanext/datastore/db.py

  Log Message:
  -----------
  [#1838] Use the default Postgres' language to create the FTS index

  Commit: 0383138e9abdfed1513cdcf585bd407e892efb0a
      https://github.com/ckan/ckan/commit/0383138e9abdfed1513cdcf585bd407e892efb0a
  Author: Vitor Baptista <vitor at vitorbaptista.com>
  Date:   2014-07-17 (Thu, 17 Jul 2014)

  Changed paths:
    M ckanext/datastore/db.py

  Log Message:
  -----------
  [#1838] Escape % from all strings that may have them

The problem we faced when doing format() twice was that if someone passed a
query with "{}", it'll look confuse the second format(). For example, the
query:

```json
{
    filters: {
  "country": "{test}"
    }
}
```

It'll confuse the second format(). So, instead of doing two formats to escape
%, we escape % before running it.

  Commit: 2ab9ccce189e7496ad57eb478685685b17b41376
      https://github.com/ckan/ckan/commit/2ab9ccce189e7496ad57eb478685685b17b41376
  Author: Vitor Baptista <vitor at vitorbaptista.com>
  Date:   2014-07-18 (Fri, 18 Jul 2014)

  Changed paths:
    M ckanext/datastore/logic/schema.py
    M ckanext/datastore/tests/test_search.py

  Log Message:
  -----------
  [#1838] Fix bug where we weren't converting JSON to dicts

  Commit: 21e2fc79d541c813af09890e39e1c8929b68448d
      https://github.com/ckan/ckan/commit/21e2fc79d541c813af09890e39e1c8929b68448d
  Author: Vitor Baptista <vitor at vitorbaptista.com>
  Date:   2014-07-18 (Fri, 18 Jul 2014)

  Changed paths:
    M ckanext/datastore/db.py
    M ckanext/datastore/logic/schema.py
    M ckanext/datastore/plugin.py
    M ckanext/datastore/tests/test_create.py
    M ckanext/datastore/tests/test_search.py

  Log Message:
  -----------
  Merge branch '1838-fts-on-specific-columns' into 1792-filterable-resource-views

Conflicts:
	ckanext/datastore/plugin.py

  Commit: 94c2abf0f52426f4a37d7cf512257ac07980a660
      https://github.com/ckan/ckan/commit/94c2abf0f52426f4a37d7cf512257ac07980a660
  Author: Vitor Baptista <vitor at vitorbaptista.com>
  Date:   2014-07-21 (Mon, 21 Jul 2014)

  Changed paths:
    M ckan/public/base/javascript/modules/resource-view-filters-form.js
    M ckan/public/base/javascript/modules/resource-view-filters.js

  Log Message:
  -----------
  [#1792] Autocomplete using the specific field, and not all text

For this, we now depend on #1838 being merged before.

Compare: https://github.com/ckan/ckan/compare/7d3cfb2b1bb5...94c2abf0f524