[ckan-changes] [ckan/ckan] 607182: [#1815] Add "distinct" param to datastore_search

Wed Jul 16 15:30:54 UTC 2014

  Branch: refs/heads/master
  Home:   https://github.com/ckan/ckan
  Commit: 6071824b9ed90d96d45a4564d041c5c228a33622
      https://github.com/ckan/ckan/commit/6071824b9ed90d96d45a4564d041c5c228a33622
  Author: Vitor Baptista <vitor at vitorbaptista.com>
  Date:   2014-07-03 (Thu, 03 Jul 2014)

  Changed paths:
    M ckanext/datastore/db.py
    M ckanext/datastore/logic/action.py
    M ckanext/datastore/logic/schema.py
    M ckanext/datastore/plugin.py
    M ckanext/datastore/tests/test_search.py

  Log Message:
  -----------
  [#1815] Add "distinct" param to datastore_search

This allows the users to get only the distinct values of the query. The "total"
field on the returned data is related to all fields, not only the distincts.
For example, in our test we have the data:

```json
[
    {"author": "tolstoy", "title": "Anna Karenina"},
    {"author": "tolstoy", "title": "War and Peace"}
]
```

We want to get a list with all authors on the datastore resource. Then we query
the "datastore_search" with:

```json
{
    "resource_id": "the_resource_id",
    "fields": ["author"],
    "distinct": True
}
```

The result is;

```json
{
    "success": True,
    "total": 2,
    "records": [{"author": "tolstoy"}]
}
```

Note that even though there's only one returned record, the "total" field is 2. This is because "total" doesn't take into account the distinct values, but all rows that match our query.

  Commit: 06e731a974263261920b51d528e8f0520962ad49
      https://github.com/ckan/ckan/commit/06e731a974263261920b51d528e8f0520962ad49
  Author: Vitor Baptista <vitor at vitorbaptista.com>
  Date:   2014-07-03 (Thu, 03 Jul 2014)

  Changed paths:
    M ckanext/datastore/db.py
    M ckanext/datastore/interfaces.py
    M ckanext/datastore/plugin.py
    M ckanext/datastore/tests/test_search.py

  Log Message:
  -----------
  [#1830] datastore_search accepts multiple filters values as array

With this patch, if you query the datastore_search with filters:

```json
{
    "filters": {
  "country": ["Brazil", "Argentina"]
    }
}
```

It'll return the rows that have `country IN ("Brazil", "Argentina")`. I had to
change the `datastore_search` extension point on `IDatastore` because I needed
the fields' types. I needed the fields' types because the Datastore accept
fields with array type (e.g. `_text`). When filtering a field that has an array
type, I want to query literally.

In the sample query above, the Datastore can create two different queries,
depending on the type of the `country` field. If it's not an array type, it'll
do as I described above, querying `country IN ("Brazil", "Argentina")`. If it's
of an array type, it'll query literally `country = ["Brazil", "Argentina"]`.

  Commit: 4827d4603d7e91bc6b865c0a93df8d0751e962ae
      https://github.com/ckan/ckan/commit/4827d4603d7e91bc6b865c0a93df8d0751e962ae
  Author: Vitor Baptista <vitor at vitorbaptista.com>
  Date:   2014-07-03 (Thu, 03 Jul 2014)

  Changed paths:
    M ckanext/datastore/db.py

  Log Message:
  -----------
  [#1830] Keep the fields' order

  Commit: 8266afc881f87836744bbe1c7cd291425c905ec2
      https://github.com/ckan/ckan/commit/8266afc881f87836744bbe1c7cd291425c905ec2
  Author: Vitor Baptista <vitor at vitorbaptista.com>
  Date:   2014-07-03 (Thu, 03 Jul 2014)

  Changed paths:
    M ckanext/datastore/db.py
    M ckanext/datastore/interfaces.py
    M ckanext/datastore/plugin.py

  Log Message:
  -----------
  [#1830] IDatastore.datastore_delete() receives fields_types

This is to keep it consistent with .datastore_search(), and make the code
started in d3761f8 work.

  Commit: b46b1af5ea80eed3691e46007998b5b218ff852b
      https://github.com/ckan/ckan/commit/b46b1af5ea80eed3691e46007998b5b218ff852b
  Author: Vitor Baptista <vitor at vitorbaptista.com>
  Date:   2014-07-03 (Thu, 03 Jul 2014)

  Changed paths:
    M ckanext/datastore/db.py
    M ckanext/datastore/interfaces.py
    M ckanext/datastore/plugin.py

  Log Message:
  -----------
  [#1830] IDatastore.datastore_validate() receives fields_types

This is to keep it consistent with .datastore_search(), and make the code
started in d3761f8 work.

  Commit: 2dbf0c5bb93db80800ab69654b161bd5cce9a3b7
      https://github.com/ckan/ckan/commit/2dbf0c5bb93db80800ab69654b161bd5cce9a3b7
  Author: Vitor Baptista <vitor at vitorbaptista.com>
  Date:   2014-07-04 (Fri, 04 Jul 2014)

  Changed paths:
    M requirements.in
    M requirements.txt

  Log Message:
  -----------
  [#1830] Fix python2.6 support

To do so, we need to add the "ordereddict" requirement.

  Commit: 01cad43619a38646995873c38b7536b10cfd15c9
      https://github.com/ckan/ckan/commit/01cad43619a38646995873c38b7536b10cfd15c9
  Author: Vitor Baptista <vitor at vitorbaptista.com>
  Date:   2014-07-04 (Fri, 04 Jul 2014)

  Changed paths:
    M ckanext/datastore/db.py
    M requirements.in
    M requirements.txt

  Log Message:
  -----------
  [#1830] Use OrderedDict from ckan.common to fix python2.6 support

  Commit: ce890420afa10118f6e903cbe139afce7b059bf1
      https://github.com/ckan/ckan/commit/ce890420afa10118f6e903cbe139afce7b059bf1
  Author: amercader <amercadero at gmail.com>
  Date:   2014-07-16 (Wed, 16 Jul 2014)

  Changed paths:
    M ckanext/datastore/db.py
    M ckanext/datastore/logic/action.py
    M ckanext/datastore/logic/schema.py
    M ckanext/datastore/plugin.py
    M ckanext/datastore/tests/test_search.py

  Log Message:
  -----------
  Merge branch '1815-distinct-datastore-search'

  Commit: 4c4e8b5e91fd831cee5435d90b65856dd1e5eb7a
      https://github.com/ckan/ckan/commit/4c4e8b5e91fd831cee5435d90b65856dd1e5eb7a
  Author: Vitor Baptista <vitor at vitorbaptista.com>
  Date:   2014-07-16 (Wed, 16 Jul 2014)

  Changed paths:
    M ckan/config/solr/schema.xml
    M ckan/lib/search/index.py
    M ckan/new_tests/lib/search/test_index.py
    M doc/api/index.rst
    M doc/contributing/reviewing.rst

  Log Message:
  -----------
  Merge branch 'master' into 1830

  Commit: 16f1c06f6ad2c5b14fc40fc390c4c7e84c84068d
      https://github.com/ckan/ckan/commit/16f1c06f6ad2c5b14fc40fc390c4c7e84c84068d
  Author: Vitor Baptista <vitor at vitorbaptista.com>
  Date:   2014-07-16 (Wed, 16 Jul 2014)

  Changed paths:
    M ckanext/datastore/tests/test_search.py

  Log Message:
  -----------
  [#1830] Add tests for multiple filters on same field

  Commit: 9a28b1dbabe274fd15f103b4cb5185db3115d216
      https://github.com/ckan/ckan/commit/9a28b1dbabe274fd15f103b4cb5185db3115d216
  Author: amercader <amercadero at gmail.com>
  Date:   2014-07-16 (Wed, 16 Jul 2014)

  Changed paths:
    M ckanext/datastore/db.py
    M ckanext/datastore/interfaces.py
    M ckanext/datastore/plugin.py
    M ckanext/datastore/tests/test_search.py

  Log Message:
  -----------
  Merge branch '1830-querying-multiple-values-on-the-datastore'

Compare: https://github.com/ckan/ckan/compare/fa0294aa89fb...9a28b1dbabe2