[ckan-dev] Is it possible to get all datasets AND metadata in a single RESTful API call ?

Wed Apr 9 11:50:54 UTC 2014

On 25 March 2014 13:07, Martin Stephenson <stephenson.martin at gmail.com> wrote:
> Is there a way to do this in a single call ? Otherwise I will really be
> hammering the CKAN server with possibly 1000's of calls to get all the
> metadata

It's relatively easy to use pagination to get all datasets metadata
using the package_search action, which uses the search index making it
relatively fast. This is a Python snippet but it should be trivial to
adapt to any need:

import requests

n = 500
page = 1
datasets = []

while True:
    search_params = {
        'q': '*:*',
        'fq': 'dataset_type:dataset',
        'sort': 'metadata_modified desc',
        'rows': n,
        'start': n * (page - 1),
    }

    query = requests.get('http://demo.ckan.org/api/action/package_search',
params=search_params).json()
    results = query.get('result', []).get('results', [])
    if len(results):
        datasets.extend(results)
        page = page + 1
    else:
        break

Hope this helps,

Adrià