[ckan-changes] [okfn/ckan] 5c65c6: [doc/using-data-api][s]: add python example and do...
GitHub
noreply at github.com
Tue Apr 24 02:23:58 UTC 2012
Branch: refs/heads/master
Home: https://github.com/okfn/ckan
Commit: 5c65c6e99717940fcc4932dcba19b7f2a7545aff
https://github.com/okfn/ckan/commit/5c65c6e99717940fcc4932dcba19b7f2a7545aff
Author: Rufus Pollock <rufus.pollock at okfn.org>
Date: 2012-04-23 (Mon, 23 Apr 2012)
Changed paths:
M doc/api.rst
M doc/using-data-api.rst
Log Message:
-----------
[doc/using-data-api][s]: add python example and do some minor refactoring.
diff --git a/doc/api.rst b/doc/api.rst
index 98d84dc..51deb4a 100644
--- a/doc/api.rst
+++ b/doc/api.rst
@@ -91,6 +91,8 @@ key is not authorized for the operation, or the header is somehow malformed,
then the requested operation will not be carried out and the CKAN API will
respond with status code 403.
+.. _get-api-key:
+
Obtaining an API key
--------------------
diff --git a/doc/using-data-api.rst b/doc/using-data-api.rst
index 3c6a6fe..ed340e3 100644
--- a/doc/using-data-api.rst
+++ b/doc/using-data-api.rst
@@ -17,16 +17,16 @@ Furthermore, it means that what is presented below is essentially a tutorial in
Quickstart
==========
-``endpoint`` refers to the data API endpoint (or ElasticSearch index / table).
+``{{endpoint}}`` refers to the data API endpoint (or ElasticSearch index / table).
Key urls:
-* Query: ``{endpoint}/_search`` (in ElasticSearch < 0.19 this will return an
+* Query: ``{{endpoint}}/_search`` (in ElasticSearch < 0.19 this will return an
error if visited without a query parameter)
- * Query example: ``{endpoint}/_search?size=5&pretty=true``
+ * Query example: ``{{endpoint}}/_search?size=5&pretty=true``
-* Schema (Mapping): ``{endpoint}/_mapping``
+* Schema (Mapping): ``{{endpoint}}/_mapping``
Examples
--------
@@ -34,15 +34,26 @@ Examples
cURL (or Browser)
~~~~~~~~~~~~~~~~~
-The following examples utilize the <a href="http://curl.haxx.se/">cURL</a>
-command line utility. If you prefer, you you can just open the relevant urls in
-your browser::
+The following examples utilize the cURL_ command line utility. If you prefer,
+you you can just open the relevant urls in your browser::
+ // query for documents / rows with title field containing 'jones'
// added pretty=true to get the json results pretty printed
- curl {endpoint}/_search?q=title:jones&size=5&pretty=true</pre>
+ curl {{endpoint}}/_search?q=title:jones&size=5&pretty=true
+
+Adding some data (requires an :ref:`API Key <get-api-key>`)::
+
+ // requires an API key
+ // Data (argument to -d) should be a JSON document
+ curl -X POST -H "Authorization: {{YOUR-API-KEY}}" {{endpoint}} -d '{
+ "title": "jones",
+ "amount": 5.7
+ }'
+
+.. _cURL: http://curl.haxx.se/
Javascript
-~~~~~~~~~~~
+~~~~~~~~~~
A simple ajax (JSONP) request to the data API using jQuery::
@@ -51,13 +62,71 @@ A simple ajax (JSONP) request to the data API using jQuery::
q: 'title:jones' // query on the title field for 'jones'
};
$.ajax({
- url: {endpoint}/_search,
+ url: {{endpoint}}/_search,
dataType: 'jsonp',
success: function(data) {
alert('Total results found: ' + data.hits.total)
}
});
+The Data API supports CORs so you can also write to it (this requires the json2_ library for ``JSON.stringify``)::
+
+ var data = {
+ title: 'jones',
+ amount: 5.7
+ };
+ $.ajax({
+ url: {{endpoint}},
+ type: 'POST',
+ data: JSON.stringify(data),
+ success: function(data) {
+ alert('Uploaded ok')
+ }
+ });
+
+.. _json2: https://github.com/douglascrockford/JSON-js/blob/master/json2.js
+
+Python
+~~~~~~
+
+.. note:: You can also use the `DataStore Python client library`_.
+
+.. _DataStore Python client library: http://github.com/okfn/datastore-client
+
+::
+
+ import urllib2
+ import json
+
+ # =================================
+ # Store data in the DataStore table
+
+ url = '{{endpoint}}'
+ data = {
+ 'title': 'jones',
+ 'amount': 5.7
+ }
+ # have to send the data as JSON
+ data = json.dumps(data)
+ # need to add your API key (and have authorization to write to this endpoint)
+ headers = {'Authorization': 'YOUR-API-KEY'}
+
+ req = urllib2.Request(url, data, headers)
+ out = urllib2.urlopen(req)
+ print out.read()
+
+ # =========================
+ # Query the DataStore table
+
+ url = '{{endpoint}}/_search?q=title:jones&size=5'
+ req = urllib2.Request(url)
+ out = urllib2.urlopen(req)
+ data = out.read()
+ print data
+ # returned data is JSON
+ data = json.loads(data)
+ # total number of results
+ print data['hits']['total']
Querying
========
@@ -69,7 +138,7 @@ Basic queries can be done using only query string parameters in the URL. For
example, the following searches for text 'hello' in any field in any document
and returns at most 5 results::
- {endpoint}/_search?q=hello&size=5
+ {{endpoint}}/_search?q=hello&size=5
Basic queries like this have the advantage that they only involve accessing a
URL and thus, for example, can be performed just using any web browser.
@@ -99,15 +168,15 @@ options for how a query is sent to the search endpoint:
1. Either as the value of a source query parameter e.g.::
- {endpoint}/_search?source={Query-as-JSON}
+ {{endpoint}}/_search?source={Query-as-JSON}
2. Or in the request body, e.g.::
- curl -XGET {endpoint}/_search -d 'Query-as-JSON'
+ curl -XGET {{endpoint}}/_search -d 'Query-as-JSON'
For example::
- curl -XGET {endpoint}/_search -d '{
+ curl -XGET {{endpoint}}/_search -d '{
"query" : {
"term" : { "user": "kimchy" }
}
================================================================
Commit: 0385ff944c1702b23471d14b29486a73879e783c
https://github.com/okfn/ckan/commit/0385ff944c1702b23471d14b29486a73879e783c
Author: Rufus Pollock <rufus.pollock at okfn.org>
Date: 2012-04-23 (Mon, 23 Apr 2012)
Changed paths:
M doc/using-data-api.rst
Log Message:
-----------
[doc/using-data-api][s]: add info for adding, updating deleting data via API plus section on facets.
diff --git a/doc/using-data-api.rst b/doc/using-data-api.rst
index ed340e3..09caf21 100644
--- a/doc/using-data-api.rst
+++ b/doc/using-data-api.rst
@@ -433,13 +433,54 @@ result here using a `bool query`_.
}
}
+
Facets
------
Facets provide a way to get summary information about then data in an
elasticsearch table, for example counts of distinct values.
-TODO: complete
+ElasticSearch (and hence the Data API) provides rich faceting capabilities:
+http://www.elasticsearch.org/guide/reference/api/search/facets/
+
+There are various kinds of facets available, for example (full list on the facets page):
+
+* Terms_ - counts by distinct terms (values) in a field
+* Range_ - counts for a given set of ranges in a field
+* Histogram_ and `Date Histogram`_ - counts by constant interval ranges
+* Statistical_ - statistical summary of a field (mean, sum etc)
+* `Terms Stats`_ - statistical summary on one field (stats field) for distinct
+ terms in another field. For example, spending stats per department or per
+ region.
+* `Geo Distance`_: counts by distance ranges from a given point
+
+Note that you can apply multiple facets per query.
+
+.. _Terms: http://www.elasticsearch.org/guide/reference/api/search/facets/terms-facet.html
+.. _Range: http://www.elasticsearch.org/guide/reference/api/search/facets/range-facet.html
+.. _Histogram: http://www.elasticsearch.org/guide/reference/api/search/facets/histogram-facet.html
+.. _Date Histogram: http://www.elasticsearch.org/guide/reference/api/search/facets/date-histogram-facet.html
+.. _Statistical: http://www.elasticsearch.org/guide/reference/api/search/facets/statistical-facet.html
+.. _Terms Stats: http://www.elasticsearch.org/guide/reference/api/search/facets/terms-stats-facet.html
+.. _Geo Distance: http://www.elasticsearch.org/guide/reference/api/search/facets/geo-distance-facet.html
+
+
+Adding, Updating and Deleting Data
+==================================
+
+ElasticSeach, and hence the Data API, have a standard RESTful API. Thus::
+
+ POST {{endpoint}} : INSERT
+ PUT/POST {{endpoint}}/{{id}} : UPDATE (or INSERT)
+ DELETE {{endpoint}}/{{id}} : DELETE
+
+For more on INSERT and UPDATE see the `Index API`_ documentation.
+
+.. _Index API: http://elasticsearch.org/guide/reference/api/index_.html
+
+There is also support bulk insert and updates via the `Bulk API`_.
+
+.. _Bulk API: http://elasticsearch.org/guide/reference/api/bulk.html
Schema Mapping
================================================================
Compare: https://github.com/okfn/ckan/compare/cdefa5e...0385ff9
More information about the ckan-changes
mailing list