[ckan-dev] issues in convert_to_extras and convert_from_extras

Thu Nov 27 00:22:16 UTC 2014

Hello everyone,

Ckan convert_to_extras and convert_from_extras don’t work properly with custom fields for resources or any multi_level schemas.
For example, when adding multiple resources on creating a new dataset, the values of the first resource is always overwritten by the correspondent fields from the last added resource.

Suppose we add a custom field to the resources schema:

	Create and update schema :
	schema['resources'].update({
                                    'storage_location': [ignore_missing, convert_to_extras]
                                 })

	Show schema :
	schema['resources'].update({
                                    'storage_location': [convert_from_extras, ignore_missing]
                                 })

And suppose we are adding two resource for a new dataset. Then the key for the custom field of the first resource will be ('resources', 0, 'storage_location') and for the second resources it will be :
('resources', 1, 'storage_location’).

Below is the code for convert_to_extras and convert_from_extras :

def convert_to_extras(key, data, errors, context):

    # Get the current extras index
    current_indexes = [k[1] for k in data.keys()
                       if len(k) > 1 and k[0] == 'extras']

    new_index = max(current_indexes) + 1 if current_indexes else 0

    data[('extras', new_index, 'key')] = key[-1]
    data[('extras', new_index, 'value')] = data[key]

def convert_from_extras(key, data, errors, context):

    def remove_from_extras(data, key):
        to_remove = []
        for data_key, data_value in data.iteritems():
            if (data_key[0] == 'extras'
                and data_key[1] == key):
                to_remove.append(data_key)
        for item in to_remove:
            del data[item]

    for data_key, data_value in data.iteritems():
        if (data_key[0] == 'extras'
            and data_key[-1] == 'key'
            and data_value == key[-1]):
            data[key] = data[('extras', data_key[1], 'value')]
            break
    else:
        return
    remove_from_extras(data, data_key[1])

Now, the problem is that no matter which resource you add, the new field key will always be added to the extras as ('extras', 0, 'storage_location’), which means there will be multiple values in the extras with the same key.
That’s why the custom field of the first record is overwritten when adding multiple resources.

The other issue is in convert_from extras at the end of the function where it says :
return remove_from_extras(data, data_key[1])

It could give an index out of bound exception if data_key is not a compound key because data_key has only one element.

I fixed the problem by creating a unique key out of the three components(something like ‘resources:index:storage_location’) when adding fields to extras and then retrieving them from the extras by reconstructing the key out of the key string. But I am not sure that is the best solution.

-Khalegh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20141126/6f397219/attachment-0002.html>