[ckan-dev] Repeatable metadata elements

Wed Apr 2 18:38:24 UTC 2014

Hello Pabitra,

I asked the same question a while ago but I didn’t get any answers. I had to add multiple values for a group of fields like contact name, contact email, etc. I checked ckan src for schema definition and came up with the following solution which I am not sure if it is the best way of doing it.

In my dataset form class I am overriding create_package_schema, update_package_schema, show_package_schema. Both  create_package_schema and update_package_schema use the same schema definition. Here is part of the schema containing the multi value fields :

       schema.update({
			... ,
                        'contacts' : contacts_db_schema(),
                        'dates' : dates_to_db_schema(),
			... ,
                      })

def contacts_db_schema():
    schema = {
              'name' : [check_empty, convert_to_extras],
              'organization' : [check_empty, convert_to_extras],
              'email' : [check_empty, valid_email, convert_to_extras],
              'role' : [check_empty, convert_to_extras],
              'private' : [ignore_missing, convert_to_extras],
              'delete' :[ignore_missing, convert_to_extras]
              }
    return schema

Ckan uses this schema when creating or updating  a dataset. It generates keys like  (contacts, 0, name) or (contacts, 0, email) which are used by convert_to_extras convertor function.

I tried to do the same for show_package_schema, but ckan doesn’t give me the same key for fields so that I can use them to get the corresponding values from extras. The other problem is that ckan’s default convert_to_extras function takes the last part of the key when storing the data in extras which obviously doesn’t work when having multiple values.

For show_package_schema I am simply updating the schema as follow :
    def show_package_schema(self):
        schema = super(DatasetFormClass, self).show_package_schema()

        schema['tags']['__extras'].append(toolkit.get_converter('free_tags_only'))
        schema.update({
                        'contacts' : [convert_from_extras, ignore_missing],
                        'dates' : [convert_from_extras, ignore_missing],
                       })

Then I had to override both convert_to_extras and convert_from_extras. Be careful you cannot use the same converters for resources, you need to use ckan’s default if you have extra fields for resources.
def convert_to_extras(key, data, errors, context):
#    print "key :====> ", key, "data : ====>", data[key] 
    extras = data.get(('extras',), [])
    if not extras:
        data[('extras',)] = extras

    keyStr = ':'.join([str(x) for x in key])
    extras.append({'key': keyStr, 'value': data[key]})

def convert_from_extras(key, data, errors, context):

    print "key : <====", key, "\n"
    def remove_from_extras(data, keyList):
        to_remove = []
        for data_key, data_value in data.iteritems():
            if (data_key[0] == 'extras' and data_key[1] in keyList) :
                to_remove.append(data_key)
        for item in to_remove:
            del data[item]

    indexList = [] # A list containing the index of items in extras to be removed.
    new_data = {}  #A new dictionary for data stored in extras with the given key
    for data_key, data_value in data.iteritems():
        if (data_key[0] == 'extras'
            and data_key[-1] == 'key'):
            #Extract the key components eparated by ':'
            keyList = data_value.split(':')
            #Check for multiple value inputs and convert the list item index to integer
            if (len(keyList) > 1):
                keyList[1] = int(keyList[1])
            #Construct the key for the stored value(s)
            newKey = tuple(keyList)
            if (key[-1] == newKey[0]):
                #Retrieve data from extras and add it to new_data so it can be added to the data dictionary. 
                new_data[newKey] = data[('extras', data_key[1], 'value')]
                #Add the data index in extras to the list of items to be removed. 
                indexList.append(data_key[1])

    #print new_data
    #Remove all data from extras with the given index
    remove_from_extras(data, indexList)

    #Remove previous data stored under the given key
    del data[key]

    deleteIndex = []
    for data_key, data_value in new_data.iteritems():
        #If this is a deleted record then add it to the deleted list to be removed from data later.  
        if ('delete' in data_key
            and data_value == '1'):
            deleteIndex.append(data_key[1])
    deleted = []        
    for data_key, data_value in new_data.iteritems():
            if (len(data_key) > 1 
                and data_key[1] in deleteIndex):
                deleted.append(data_key)
    for item in deleted :
        del new_data[item]

    #Add data extracted from extras to the data dictionary
    for data_key, data_value in new_data.iteritems():
        data[data_key] = data_value

I am attaching my template and javascript for contacts here as well. I hope this helps. Please let me know if you have any problem with the code.

Best,
Khalegh

On Apr 2, 2014, at 10:47 AM, Pabitra Dash <pkdash_reena at hotmail.com> wrote:

> Hi All,
>  
> I have implemented a custom metadata schema for dataset. We have another requirement to make some of the metadata elements repeatable. For example, I have creator element and creator_email element which I want to make repeatable. So in the UI when the user selects add a creator, I should be able to store name and email for another creator. Not sure how I define a schema for these kind of repeatable elements. 
>  
> Would appreciate any help on this.
>  
> Pabitra
> 
> 
> 
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20140402/588a6212/attachment-0012.html>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20140402/588a6212/attachment-0013.html>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20140402/588a6212/attachment-0014.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: contacts.js
Type: text/javascript
Size: 3369 bytes
Desc: not available
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20140402/588a6212/attachment-0003.bin>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20140402/588a6212/attachment-0015.html>