[ckan-dev] How is the extras field supposed to be used?

Tue Dec 17 17:50:37 UTC 2013

Hi.
I've read previous posts (7-8 Nov, 5-10 Dec) about 'extras' field but it's still bit unclear to me how to handle arbitrary extra fields of a package. I am harvesting ddi data and want to put unused fields (as flattened xpath/value pairs) to package_extras table.

Should I use converters like 'ckan.logic.converters.convert_to_extras' in my custom schema to create ('extras',) array of dicts like:

    # Ex.1
    data_dict = {
        ('name',): 'myvalue1',
        ('title',): 'myvalue2,
        ...
        ('extras',): [{'key': 'my/arbitrary/key_1', 'value': 'val1'}, {}, ...]
    }

or should I just put my arbitrary keys to 'extras' subdict of key/value pairs in data_dict:

    # Ex.2
    data_dict = {
        'name': 'myvalue1',
        'title': 'myvalue2,
        ...
        'extras': {
            'my/arbitrary/key_1': 'val1',
            'my/arbitrary/key_2': 'val2',
            ...
        }
    }

The first example shows the package dict after 'ckan.lib.navl.dictization_functions._validate' but before 'ckan.lib.navl.dictization_functions.unflatten' when using default 'package_create_rest'. The second example shows the package dict that is given as a parameter for 'package_create_rest'. They are from alternative implementations.

I am now using my own converter to move fields from 'xpaths' subdict to ('extras',) array of dicts (see. Ex.1):

    def xpath_to_extras(key, data, errors, context):
        extras = data.get(('extras',), [])
        if not extras:
            data[('extras',)] = extras
        for k, v in data[key].iteritems():
            extras.append({'key': k, 'value': v})

    pakcage_dict = {
        'name': 'myvalue1',
        'title': 'myvalue2,
        'my_field1': 'myvalue3',
        ...
        'xpaths': {
            'my/arbitrary/key_1': 'val1',
            'my/arbitrary/key_2': 'val2',
            ...
        }
    }

    schema = create_package_schema()
    schema['xpaths'] = [xpath_to_extras]
    result = self._create_or_update_package(package_dict, harvest_object, schema)

This is because some of my extra fields like 'my_field1' are already converted similarly to convert_to_extras(). 'unflatten' doesn't like if there are converted fields in ('extras',) and ('extra', 56, 'key') types of keys flattened from 'extras' subdict (see Ex.2). Now I'd like to know which is the recommended way.

Best regards

Juho Lehtonen