[ckan-dev] Forms in CKAN

Fri Jan 14 18:38:37 UTC 2011

I see two main main (very much related) sections to the form question:
1. the inner mechanics of the form and 2. their controllers. Here's a
brain dump of where I see things how they are.

1. Inner mechanics of the form

All the excitement in the form is around the 'extra' fields, with
every customer wanting a different set of them. So I'd say we need to
be able to have the different variants of the extra fields - text,
select box, date, date range, etc. as common 'widgets' which can be
reused for every form that we do.

A form widget encapsulates:
* html/javascript template
* methods to convert/move data between most combinations of: model,
html form template, html preview, POSTed params and API format
* validation
* stored configuration (i.e. name, label, options etc)

The key part is the 2nd point - making sure these data interchange
methods all work well. In our current setup with formalchemy there are
converters grouped in each formalchemy field to convert to/from almost
all these different formats:
* database via raw_value / sync
* html template via render
* html preview via render_readonly
* params via bind
* API format via bind
The one that is missing is converting from the field to the API format
(although it could be added).
One idea Rufus is making the package dictionary the 'central' format
to build html from, and bypass the field value. I'm not convinced this
is saving us much for all the effort it would be.

As Rufus says, the principal thing formalchemy gives for free is
automatically detecting sqlalchemy field properties. We mostly don't
use this, and much else in formalchemy we override anyway, whilst
sticking to its basic structure. So we could dump FA without too much
pain, but changing the structure to become package-dict-centric would
be harder and I don't see much benefit.

The Fields in common.py could do with a bit of a tidy up, but do they
encapsulate most things about a form widget in one place in the code
and is pretty logical.

Rufus says it is a mistake to "merge too much model/validation/form
generation into one thing" but I quite like encapsulation.

The configuration using the form builder needs a bit of tightening up
to avoid some gotchas, but basically putting together a form is pretty
simple.

I've bitched plenty about formalchemy in the past, but actually the
more I understand it, the less I wrestle and more I appreciate its
structure.

2. Form controllers

The controllers for the forms for the web and model API has grown in
purpose immensely and is long overdue for a refactor to bring in
commonality.

The tests are extensive, and I believe they need to be, because there
are plenty of gotchas, what with the previews being complicated with
sessions.

Rufus and I took a decision last Spring to render a package using a
template that reads a Package object. (The alternatives include a) the
template reading a package dictionary, or b) the template calling
formalchemy's render_readonly functions to render each widget and
splicing them together) But when we want to render a preview of a
package, we haven't yet got the modified Package object (since it's
not saved). So to solve this we do save the package but in a detached
session, so it doesn't save it to the db. I'm slightly tempted to
change this complexity to a) or b) but none of these are without
problems.

(I noticed you put some question marks on this topic into the
controller and package_saver code in your recent work, so I've
explained things, and refactored somewhat to improve things, currently
in my branch feature-dgu-757, if you want to take a look.)

I hope this helps set the scene. Plenty needs explaining. I can
understand some frustration getting up to speed with the structure,
and there is much to improve, but I don't see a reason to change it
fundamentally. I'll certainly be interested though to see what better
ways there are these days, and if one is justified us changing to.

David

On 14 January 2011 14:47, Rufus Pollock <rufus.pollock at okfn.org> wrote:
> On 14 January 2011 13:44, Seb Bacon <seb.bacon at gmail.com> wrote:
>> Hi,
>>
>> Next week I plan to review the way we currently generate forms in
>> CKAN, with a view to improving them.
>>
>> In preparation for that, I would love to know what people think about
>> the current situation: what works, and what doesn't? What are the
>> properties of a decent form generating system?
>
> Libraries I have used: FormEncode, FormAlchemy (what we are currently
> using, before that formencode).
>
> Neither seemed perfect but I think the form issue is a 'hard' problem
> (perhaps with no perfect answer) [1]. FormAlchemy, in retrospect, was
> probably a mistake as it merges too much model/validation/form
> generation into one thing.
>
> At least 3 functions involved:
>
> 1. Generating (or just filling) a form template with 'form data' (and errors)
> 2. Converting model data to form data (also happens for APIs in fact)
> -- let's call this 'dict-ization'
> 3. Converting form data to model data (and validating) (inverse of
> previous step)
>
> I think one can and should separate 1 from 2+3 (and one of problems
> with formalchemy is it doesn't -- the attraction being you don't
> repeat yourself as forms get generated from model but I think this is
> actually a false economy in medium-term).
>
> I'm not specifically recommending the following as I haven't used them
> but I've looked through docs, they are active and reasonably mature:
>
> 1. Flatland: http://discorporate.us/projects/flatland/docs/tip/
>  * Only does 2+3 which is a good thing IMO
>
> 2. WTForms: http://wtforms.simplecodes.com/
>  * Used in standard flask docs: <http://flask.pocoo.org/docs/patterns/wtforms/>
>
> Regards,
>
> Rufus
>
> [1]: http://blog.ianbicking.org/on-form-libraries.html
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
>