[ckan-dev] Issues with adding a custom input field

Kirstein, Fabian fabian.kirstein at fokus.fraunhofer.de
Sat Jan 12 11:28:19 UTC 2013


Hi Adrià,

thanks again!
I applied your changes and the original error is gone, but I got two new errors.
When I try to edit and save a dataset I receive the following error message: "junk: The input field __junk was not expected."
I doesn't matter if I fill the new field.
Creating a new dataset leads in the second step to a Statement error. I attached the traceback.

Have you tried my extension on your own? Probably there is something wrong with my CKAN installation.

Have a nice weekend!
Fabian



URL: http://ubuntu:5000/dataset/new
File '/home/fki/pyenv/local/lib/python2.7/site-packages/weberror/evalexception.py', line 431 in respond
  app_iter = self.application(environ, detect_start_response)
File '/home/fki/pyenv/local/lib/python2.7/site-packages/webob/dec.py', line 147 in __call__
  resp = self.call_func(req, *args, **self.kwargs)
File '/home/fki/pyenv/local/lib/python2.7/site-packages/webob/dec.py', line 208 in call_func
  return self.func(req, *args, **kwargs)
File '/home/fki/pyenv/local/lib/python2.7/site-packages/fanstatic/publisher.py', line 234 in __call__
  return request.get_response(self.app)
File '/home/fki/pyenv/local/lib/python2.7/site-packages/webob/request.py', line 1053 in get_response
  application, catch_exc_info=False)
File '/home/fki/pyenv/local/lib/python2.7/site-packages/webob/request.py', line 1022 in call_application
  app_iter = application(self.environ, start_response)
File '/home/fki/pyenv/local/lib/python2.7/site-packages/webob/dec.py', line 147 in __call__
  resp = self.call_func(req, *args, **self.kwargs)
File '/home/fki/pyenv/local/lib/python2.7/site-packages/webob/dec.py', line 208 in call_func
  return self.func(req, *args, **kwargs)
File '/home/fki/pyenv/local/lib/python2.7/site-packages/fanstatic/injector.py', line 54 in __call__
  response = request.get_response(self.app)
File '/home/fki/pyenv/local/lib/python2.7/site-packages/webob/request.py', line 1053 in get_response
  application, catch_exc_info=False)
File '/home/fki/pyenv/local/lib/python2.7/site-packages/webob/request.py', line 1022 in call_application
  app_iter = application(self.environ, start_response)
File '/home/fki/pyenv/local/lib/python2.7/site-packages/beaker/middleware.py', line 73 in __call__
  return self.app(environ, start_response)
File '/home/fki/pyenv/local/lib/python2.7/site-packages/beaker/middleware.py', line 155 in __call__
  return self.wrap_app(environ, session_start_response)
File '/home/fki/pyenv/local/lib/python2.7/site-packages/routes/middleware.py', line 131 in __call__
  response = self.app(environ, start_response)
File '/home/fki/pyenv/local/lib/python2.7/site-packages/pylons/wsgiapp.py', line 125 in __call__
  response = self.dispatch(controller, environ, start_response)
File '/home/fki/pyenv/local/lib/python2.7/site-packages/pylons/wsgiapp.py', line 324 in dispatch
  return controller(environ, start_response)
File '/home/fki/pyenv/src/ckan/ckan/lib/base.py', line 291 in __call__
  res = WSGIController.__call__(self, environ, start_response)
File '/home/fki/pyenv/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 221 in __call__
  response = self._dispatch_call()
File '/home/fki/pyenv/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 172 in _dispatch_call
  response = self._inspect_call(func)
File '/home/fki/pyenv/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 107 in _inspect_call
  result = self._perform_call(func, args)
File '/home/fki/pyenv/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 60 in _perform_call
  return func(**args)
File '/home/fki/pyenv/src/ckan/ckan/controllers/package.py', line 447 in new
  return self._save_new(context)
File '/home/fki/pyenv/src/ckan/ckan/controllers/package.py', line 888 in _save_new
  pkg_dict = get_action('package_create')(context, data_dict)
File '/home/fki/pyenv/src/ckan/ckan/logic/action/create.py', line 153 in package_create
  model.Session.flush()
File '/home/fki/pyenv/local/lib/python2.7/site-packages/sqlalchemy/orm/scoping.py', line 114 in do
  return getattr(self.registry(), name)(*args, **kwargs)
File '/home/fki/pyenv/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py', line 1583 in flush
  self._flush(objects)
File '/home/fki/pyenv/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py', line 1654 in _flush
  flush_context.execute()
File '/home/fki/pyenv/local/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py', line 331 in execute
  rec.execute(self)
File '/home/fki/pyenv/local/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py', line 475 in execute
  uow
File '/home/fki/pyenv/local/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py', line 64 in save_obj
  table, insert)
File '/home/fki/pyenv/local/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py', line 558 in _emit_insert_statements
  execute(statement, params)
File '/home/fki/pyenv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py', line 1449 in execute
  params)
File '/home/fki/pyenv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py', line 1584 in _execute_clauseelement
  compiled_sql, distilled_params
File '/home/fki/pyenv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py', line 1651 in _execute_context
  None, None)
File '/home/fki/pyenv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py', line 1647 in _execute_context
  context = constructor(dialect, self, conn, *args)
File '/home/fki/pyenv/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py', line 475 in _init_compiled
  processors[key](compiled_params[key])
File '/home/fki/pyenv/local/lib/python2.7/site-packages/sqlalchemy/types.py', line 622 in process
  return impl_processor(process_param(value, dialect))
File '/home/fki/pyenv/src/ckan/ckan/model/types.py', line 50 in process_bind_param
  return unicode(json.dumps(value, ensure_ascii=False))
File '/home/fki/pyenv/local/lib/python2.7/site-packages/simplejson/__init__.py', line 334 in dumps
  **kw).encode(obj)
File '/home/fki/pyenv/local/lib/python2.7/site-packages/simplejson/encoder.py', line 237 in encode
  chunks = self.iterencode(o, _one_shot=True)
File '/home/fki/pyenv/local/lib/python2.7/site-packages/simplejson/encoder.py', line 311 in iterencode
  return _iterencode(o, 0)
File '/home/fki/pyenv/local/lib/python2.7/site-packages/simplejson/encoder.py', line 213 in default
  raise TypeError(repr(o) + " is not JSON serializable")
StatementError: <ckan.lib.navl.dictization_functions.Missing object at 0x3dd6cd0> is not JSON serializable (original cause: TypeError: <ckan.lib.navl.dictization_functions.Missing object at 0x3dd6cd0> is not JSON serializable) 'INSERT INTO package_extra (id, package_id, key, value, state, revision_id) VALUES (%(id)s, %(package_id)s, %(key)s, %(value)s, %(state)s, %(revision_id)s)' [{'package_id': '44c71b57-0347-4aaf-b8f1-31d83380221f', 'revision_id': u'e1cae3da-8339-466c-abe0-ec33061280f6', 'value': <ckan.lib.navl.dictization_functions.Missing object at 0x3dd6cd0>, 'key': 'release', 'state': 'active'}]






-----Ursprüngliche Nachricht-----
Von: ckan-dev-bounces at lists.okfn.org [mailto:ckan-dev-bounces at lists.okfn.org] Im Auftrag von Adrià Mercader
Gesendet: Freitag, 11. Januar 2013 10:59
An: CKAN Development Discussions
Betreff: Re: [ckan-dev] Issues with adding a custom input field

Hi Fabian,

I managed to get some time to look at your extension, please see attached diff. This is an area where development is being made so your feedback is much appreciated.

There were a couple of issues:

* form_to_db_schema and db_to_form_schema were using the same schema (and an outdated one)
* The current implementation of IDatasetForm means that if your plugin is extending DefaultDatasetForm you need to also include form_to_db_schema_options, which gets called first (If you don't add it, the default will be used and your custom schema will be ignored.
(Maybe we need to review this logic as it is a bit confusing)

This will correctly store and retrieve your custom field from the database, but I'm afraid you'll find a new issue. As your field ends with "_date", there is a known bug with the Solr indexing that raises an error when saving the dataset. There is a fix under way, but on the meantime you'll need to either rename your field or remove the *_date dynamic field from the Solr schema.

https://github.com/okfn/ckan/issues/267


Hope this helps,

Adrià





On 8 January 2013 14:09, Kirstein, Fabian <fabian.kirstein at fokus.fraunhofer.de> wrote:
> Hi Adrià,
>
> I've corrected the typo, but it makes no difference.
> I attached the traceback.
>
> Thanks again.
>
> URL: http://ubuntu:5000/dataset/amazon-s3
> File '/home/fki/pyenv/local/lib/python2.7/site-packages/weberror/evalexception.py', line 431 in respond
>   app_iter = self.application(environ, detect_start_response) File 
> '/home/fki/pyenv/local/lib/python2.7/site-packages/webob/dec.py', line 147 in __call__
>   resp = self.call_func(req, *args, **self.kwargs) File 
> '/home/fki/pyenv/local/lib/python2.7/site-packages/webob/dec.py', line 208 in call_func
>   return self.func(req, *args, **kwargs) File 
> '/home/fki/pyenv/local/lib/python2.7/site-packages/fanstatic/publisher.py', line 234 in __call__
>   return request.get_response(self.app) File 
> '/home/fki/pyenv/local/lib/python2.7/site-packages/webob/request.py', line 1053 in get_response
>   application, catch_exc_info=False)
> File '/home/fki/pyenv/local/lib/python2.7/site-packages/webob/request.py', line 1022 in call_application
>   app_iter = application(self.environ, start_response) File 
> '/home/fki/pyenv/local/lib/python2.7/site-packages/webob/dec.py', line 147 in __call__
>   resp = self.call_func(req, *args, **self.kwargs) File 
> '/home/fki/pyenv/local/lib/python2.7/site-packages/webob/dec.py', line 208 in call_func
>   return self.func(req, *args, **kwargs) File 
> '/home/fki/pyenv/local/lib/python2.7/site-packages/fanstatic/injector.py', line 54 in __call__
>   response = request.get_response(self.app) File 
> '/home/fki/pyenv/local/lib/python2.7/site-packages/webob/request.py', line 1053 in get_response
>   application, catch_exc_info=False)
> File '/home/fki/pyenv/local/lib/python2.7/site-packages/webob/request.py', line 1022 in call_application
>   app_iter = application(self.environ, start_response) File 
> '/home/fki/pyenv/local/lib/python2.7/site-packages/beaker/middleware.py', line 73 in __call__
>   return self.app(environ, start_response) File 
> '/home/fki/pyenv/local/lib/python2.7/site-packages/beaker/middleware.py', line 155 in __call__
>   return self.wrap_app(environ, session_start_response) File 
> '/home/fki/pyenv/local/lib/python2.7/site-packages/routes/middleware.py', line 131 in __call__
>   response = self.app(environ, start_response) File 
> '/home/fki/pyenv/local/lib/python2.7/site-packages/pylons/wsgiapp.py', line 125 in __call__
>   response = self.dispatch(controller, environ, start_response) File 
> '/home/fki/pyenv/local/lib/python2.7/site-packages/pylons/wsgiapp.py', line 324 in dispatch
>   return controller(environ, start_response) File 
> '/home/fki/pyenv/src/ckan/ckan/lib/base.py', line 291 in __call__
>   res = WSGIController.__call__(self, environ, start_response) File 
> '/home/fki/pyenv/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 221 in __call__
>   response = self._dispatch_call()
> File '/home/fki/pyenv/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 172 in _dispatch_call
>   response = self._inspect_call(func)
> File '/home/fki/pyenv/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 107 in _inspect_call
>   result = self._perform_call(func, args) File 
> '/home/fki/pyenv/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 60 in _perform_call
>   return func(**args)
> File '/home/fki/pyenv/src/ckan/ckan/controllers/package.py', line 327 in read
>   return render(template, loader_class=loader) File 
> '/home/fki/pyenv/src/ckan/ckan/lib/base.py', line 194 in render
>   loader_class=loader_class)
> File '/home/fki/pyenv/local/lib/python2.7/site-packages/pylons/templating.py', line 249 in cached_template
>   return render_func()
> File '/home/fki/pyenv/src/ckan/ckan/lib/base.py', line 132 in render_template
>   return render_jinja2(template_name, globs) File 
> '/home/fki/pyenv/src/ckan/ckan/lib/base.py', line 85 in render_jinja2
>   return template.render(**extra_vars) File 
> '/home/fki/pyenv/local/lib/python2.7/site-packages/jinja2/environment.py', line 894 in render
>   return self.environment.handle_exception(exc_info, True) File 
> '/home/fki/pyenv/src/ckanext-addfield/ckanext/field/templates/package/read.html', line 3 in top-level template code
>   {% set pkg = c.pkg_dict %}
> File '/home/fki/pyenv/src/ckan/ckan/templates/page.html', line 1 in top-level template code
>   {% extends "base.html" %}
> File '/home/fki/pyenv/src/ckan/ckan/templates/base.html', line 104 in top-level template code
>   {%- block page %}{% endblock -%}
> File '/home/fki/pyenv/src/ckan/ckan/templates/page.html', line 15 in block "page"
>   {%- block content %}
> File '/home/fki/pyenv/src/ckan/ckan/templates/page.html', line 18 in block "content"
>   {% block main_content %}
> File '/home/fki/pyenv/src/ckan/ckan/templates/page.html', line 30 in block "main_content"
>   {% block toolbar %}
> File '/home/fki/pyenv/src/ckan/ckan/templates/page.html', line 40 in block "toolbar"
>   {% block actions %}
> File '/home/fki/pyenv/src/ckan/ckan/templates/page.html', line 42 in block "actions"
>   {% block actions_content %}{% endblock %} File 
> '/home/fki/pyenv/src/ckanext-addfield/ckanext/field/templates/package/read.html', line 16 in block "actions_content"
>   {% if h.check_access('package_update', {'id':pkg.id }) %} File 
> '/home/fki/pyenv/src/ckan/ckan/lib/helpers.py', line 527 in check_access
>   check_access_logic(action, context, data_dict) File 
> '/home/fki/pyenv/src/ckan/ckan/logic/__init__.py', line 207 in check_access
>   logic_authorization = is_authorized(action, context, data_dict) File 
> '/home/fki/pyenv/src/ckan/ckan/new_authz.py', line 17 in is_authorized
>   return auth_function(context, data_dict) File 
> '/home/fki/pyenv/src/ckan/ckan/logic/auth/update.py', line 15 in package_update
>   package = get_package_object(context, data_dict) File 
> '/home/fki/pyenv/src/ckan/ckan/logic/auth/__init__.py', line 24 in get_package_object
>   return _get_object(context, data_dict, 'package', 'Package') File 
> '/home/fki/pyenv/src/ckan/ckan/logic/auth/__init__.py', line 13 in _get_object
>   obj = getattr(model, class_name).get(id) File 
> '/home/fki/pyenv/src/ckan/ckan/model/package.py', line 76 in get
>   pkg = query.first()
> File '/home/fki/pyenv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py', line 2156 in first
>   ret = list(self[0:1])
> File '/home/fki/pyenv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py', line 2023 in __getitem__
>   return list(res)
> File '/home/fki/pyenv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py', line 2227 in __iter__
>   return self._execute_and_instances(context)
> File '/home/fki/pyenv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py', line 2242 in _execute_and_instances
>   result = conn.execute(querycontext.statement, self._params) File 
> '/home/fki/pyenv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py', line 1449 in execute
>   params)
> File '/home/fki/pyenv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py', line 1584 in _execute_clauseelement
>   compiled_sql, distilled_params
> File '/home/fki/pyenv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py', line 1698 in _execute_context
>   context)
> File '/home/fki/pyenv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py', line 1691 in _execute_context
>   context)
> File '/home/fki/pyenv/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py', line 331 in do_execute
>   cursor.execute(statement, parameters)
> ProgrammingError: (ProgrammingError) can't adapt type 'Undefined' 
> 'SELECT package.id AS package_id, package.name AS package_name, 
> package.title AS package_title, package.version AS package_version, 
> package.url AS package_url, package.author AS package_author, 
> package.author_email AS package_author_email, package.maintainer AS 
> package_maintainer, package.maintainer_email AS 
> package_maintainer_email, package.notes AS package_notes, 
> package.license_id AS package_license_id, package.type AS 
> package_type, package.state AS package_state, package.revision_id AS 
> package_revision_id \nFROM package \nWHERE package.id = %(id_1)s ORDER 
> BY package.name \n LIMIT %(param_1)s' {'id_1': Undefined, 'param_1': 
> 1}
>
>
> -----Ursprüngliche Nachricht-----
> Von: ckan-dev-bounces at lists.okfn.org 
> [mailto:ckan-dev-bounces at lists.okfn.org] Im Auftrag von Adrià Mercader
> Gesendet: Dienstag, 8. Januar 2013 10:43
> An: CKAN Development Discussions
> Betreff: Re: [ckan-dev] Issues with adding a custom input field
>
> Hi Fabian,
>
> You use "release_data" on form_to_db_schema and "release_date" on db_to_form_schema. Let's get this out of the way to make sure it's not affecting things.
>
> What ProgrammingError are you getting? Can you paste the full traceback?
>
>
> Adrià
>
> On 8 January 2013 08:07, Kirstein, Fabian <fabian.kirstein at fokus.fraunhofer.de> wrote:
>> Hi Adrià,
>>
>> thank you very much for having a look into my problem.
>> The solution you proposed is the same I've tried, but it just leads 
>> me to an error message. I implemented just the basic idea and pushed 
>> it to GitHub: https://github.com/FabiApfelkern/ckanext-addfield
>> If I want to view or edit a dataset I just get a " ProgrammingError".
>>
>> It would be absolutely great if you can have another look into it.
>> Fabian
>>
>>
>>
>> -----Ursprüngliche Nachricht-----
>> Von: ckan-dev-bounces at lists.okfn.org
>> [mailto:ckan-dev-bounces at lists.okfn.org] Im Auftrag von Adrià 
>> Mercader
>> Gesendet: Montag, 7. Januar 2013 17:17
>> An: CKAN Development Discussions
>> Betreff: Re: [ckan-dev] Issues with adding a custom input field
>>
>> Hi Fabian,
>>
>> When storing extra fields against the default CKAN schema, these get internally stored as extras, which can hold arbitrary key-value pairs.
>> You need to add the following converters to your custom schema on your form_to_db_schema and db_to_form_schema methods:
>>
>>
>> from ckan.logic.converters import convert_to_extras, 
>> convert_from_extras
>>
>> class YourPlugin(IDatasetForm):
>>
>> def form_to_db_schema(self):
>>
>>  // get the default package schema
>>  //...
>>
>>    schema['release_date'] = [convert_to_extras] //You can add more 
>> validators if needed
>>
>>    return schema
>>
>> def db_to_form_schema(self):
>>
>>  // get the default package schema
>>  //...
>>    schema['release_date'] = [convert_from_extras]
>>
>>    return schema
>>
>> After doing this, on additional_info.html, pkg_dict should have your extra fields (pkg_dict.release_date).
>>
>> Here is a real example of a custom schema that stores new fields (look for those with convert_to_extras):
>>
>> https://github.com/okfn/ckanext-harvest/blob/2.0-dataset-sources/ckan
>> e
>> xt/harvest/logic/schema.py
>>
>>
>> Hope this helps,
>>
>> Adrià
>>
>> On 7 January 2013 10:57, Kirstein, Fabian <fabian.kirstein at fokus.fraunhofer.de> wrote:
>>> Hello,
>>>
>>>
>>>
>>> do you need any additional information to answer my question? I 
>>> can’t solve it by my own.
>>>
>>> I tried to save the inputted value with schema.update and 
>>> convert_to_extras methods, but I
>>>
>>> am not sure how to apply it.
>>>
>>>
>>>
>>> Thanks!
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Von: ckan-dev-bounces at lists.okfn.org 
>>> [mailto:ckan-dev-bounces at lists.okfn.org] Im Auftrag von Kirstein, 
>>> Fabian
>>> Gesendet: Donnerstag, 3. Januar 2013 12:25
>>> An: ckan-dev at lists.okfn.org
>>> Betreff: [ckan-dev] Issues with adding a custom input field
>>>
>>>
>>>
>>> Hello everyone,
>>>
>>>
>>>
>>> this thread follows my „Customizing Forms in 2.0a“  thread. I have 
>>> still issues adding some custom input fields, store the input and display it.
>>>
>>> I try to explain what to do. I’ve started to write my own custom 
>>> extension implementing IDatasetForm . I’ve overwritten the 
>>> package_metadata_fields.html and add
>>>
>>> an additional field, like this:
>>>
>>> {{ form.input('release_date', label=_('Release Date'), 
>>> id='field-release-date', placeholder=_('2011-06-13'), 
>>> value=data.release_date, error=errors.release_date,
>>> classes=['control-medium']) }}
>>>
>>> In the form_to_db_schema() method, I would like to store the value 
>>> of the field as an extra field. I am not pretty sure how to access 
>>> the inputted value.
>>>
>>> Then I want to override additional_info.html to display the stored value.
>>> But I’m not really sure how to pass data to such a snippet. The 
>>> c-object which
>>>
>>> I can manipulate setup_template_variables() is not available in the snippet.
>>>
>>>
>>>
>>> I would really appreciate any help. I can’t be so hard to add a 
>>> single input field, I guess I just have some misunderstandings.
>>>
>>> Thanks a lot.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> ckan-dev mailing list
>>> ckan-dev at lists.okfn.org
>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>>
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev


More information about the ckan-dev mailing list