[ckan-dev] issues in convert_to_extras and convert_from_extras

[Ian Ward]

Try simply not using convert_from/to_extras for resource fields. Resources
and datasets store their extras differently.
On 26 Nov 2014 19:22, "Khalegh Mamakani" <khalegh at highwaythreesolutions.com>
wrote:

> Hello everyone,
>
> Ckan convert_to_extras and convert_from_extras don’t work properly with
> custom fields for resources or any multi_level schemas.
> For example, when adding multiple resources on creating a new dataset, the
> values of the first resource is always overwritten by the correspondent
> fields from the last added resource.
>
> Suppose we add a custom field to the resources schema:
>
> Create and update schema :
> schema['resources'].update({
>                                     'storage_location': [ignore_missing,
> convert_to_extras]
>                                  })
>
>
> Show schema :
> schema['resources'].update({
>                                     'storage_location':
> [convert_from_extras, ignore_missing]
>                                  })
>
>
> And suppose we are adding two resource for a new dataset. Then the key for
> the custom field of the first resource will be ('resources', 0,
> 'storage_location') and for the second resources it will be :
> ('resources', 1, 'storage_location’).
>
> Below is the code for convert_to_extras and convert_from_extras :
>
> def convert_to_extras(key, data, errors, context):
>
>     # Get the current extras index
>     current_indexes = [k[1] for k in data.keys()
>                        if len(k) > 1 and k[0] == 'extras']
>
>     new_index = max(current_indexes) + 1 if current_indexes else 0
>
>     data[('extras', new_index, 'key')] = key[-1]
>     data[('extras', new_index, 'value')] = data[key]
>
>
> def convert_from_extras(key, data, errors, context):
>
>     def remove_from_extras(data, key):
>         to_remove = []
>         for data_key, data_value in data.iteritems():
>             if (data_key[0] == 'extras'
>                 and data_key[1] == key):
>                 to_remove.append(data_key)
>         for item in to_remove:
>             del data[item]
>
>     for data_key, data_value in data.iteritems():
>         if (data_key[0] == 'extras'
>             and data_key[-1] == 'key'
>             and data_value == key[-1]):
>             data[key] = data[('extras', data_key[1], 'value')]
>             break
>     else:
>         return
>     remove_from_extras(data, data_key[1])
>
>
> Now, the problem is that no matter which resource you add, the new field
> key will always be added to the extras as ('extras', 0, 'storage_location’),
> which means there will be multiple values in the extras with the same key.
> That’s why the custom field of the first record is overwritten when adding
> multiple resources.
>
>
> The other issue is in convert_from extras at the end of the function where
> it says :
> return remove_from_extras(data, data_key[1])
>
> It could give an index out of bound exception if data_key is not a
> compound key because data_key has only one element.
>
>
> I fixed the problem by creating a unique key out of the three
> components(something like ‘resources:index:storage_location’) when adding
> fields to extras and then retrieving them from the extras by reconstructing
> the key out of the key string. But I am not sure that is the best solution.
>
>
> -Khalegh
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20141126/9dbaefe8/attachment-0003.html>