[ckan-dev] Thoughts on CKAN development

[Sean Hammond]

Hi Salman, I think you make a lot of good points here...

>  *   Installation and Setup
> 
> Installing CKAN on the supported version of Ubuntu is pretty easy
> (kudos). But beyond that, installing it on another distro can be quite
> cumbersome. There are contributed instructions for installing on
> CentOS. These could benefit from being organized better and included
> and maintained in the official docs.

Is it really difficult to install it on another distro? A lot of work
went into simplifying the source install instructions for CKAN 1.8 and 2.0,
installing on other distros should now be pretty much the same as
intalling on Ubuntu, except that you have to figure out how to install
the 8 debian packages at the start on your own distro (and maybe the
solr setup would be different too).

I still think it's great to have people contribute source install
instructions for different distros, although since wiki.ckan.org is
deprecated now the right place to put them would either be on CKAN's
GitHub wiki or on the CKAN tag on StackOverflow.

I don't think we can realistically put install instructions for other
distros in the docs though, I just don't think we have the resources to
maintain several sets of install instructions, so I prefer this stuff
out on wikis or SO where it's quicker and easier for the community to
jump in and help out. (Anyone can submit a pull request for the CKAN
docs of course, but it's a bit more time consuming.)

> The ckanext-datastore plugin seems to be quite popular but also a
> cause of many troubleshooting inquiries on the mailing list (although
> not as much of late).

I believe ckanext-datastore is currently being replaced by a core
ckanext/datastore plugin, so this should get much easier.

> Perhaps a fabric or salt script can be made available for source
> installations?

I think automating the source install would be a great contribution.
Previously we thought about using Ansibel for this, which we thought
looked better than Fabric, easier to maintain in the long run.

>  *   Interface classes lack documentation (other than function doc
>  strings).

I think the docstrings for these could be a lot better. I also think
it's better to have working, example extensions rather than extensive
documentation though. But I think these examples need to be in the core
git repo so that they get maintained and kept up to date. For example we
could add a ckanext/example_idataset form extension, etc.  Working code
is better because it can have tests, so if a ckan dev breaks the plugin
interface they will know about it, whereas with just docs we might not
find out until someone tries to use it.

> The IDatasetForm and IPackageController interfaces in particular are
> important examples. Almost every non-trivial extension will need to
> implement these interfaces but there is insufficient documentation
> about them. Further, these interfaces have a large number of methods
> and they seem very complex.

I believe IDatasetForm is meant to replace IPackageController going
forward, could be wrong about that. I have actually been working on
better documentation and example extensions with tests for IDatasetForm
and IGroupForm on a branch, but it is very hard to steal the time to
finish this off:

https://github.com/okfn/ckan/tree/2750-add-docs-and-examples-for-idatasetform-and-igroupform

>  *   Request data flow is not clear from documentation.
> 
> Often, I found myself reading the core CKAN code to figure out when a
> certain interface method gets called, (eg:
> IDatasetForm.setup_template_variables). Manually tracing the call
> graph gets very aggravating quickly.

I think this is a case of the interface methods needing better
docstrings.

>  *   Interaction between forms and models is complex.
> 
> There are several dictize and dict save methods in the core that could
> use better documentation.

Absolutely agree

> Modifying the package package form even the slightest requires almost
> a full re-implementation of several package templates - case in point,
> the Organization plugin. A combination of better documentation and
> simpler logic will be helpful.

Not sure I understand how modifying the form a little bit requires the
templates to be completely rewritten.

The templates in CKAN 2.0 are much improved, you can just override or
add to one part of a template file, you don't have to copy the entire
file into your extension.

>  *   'context' dictionary is used in several places but it is not
>  documented.
> 
> Several controller methods construct an arbitrary 'context' dictionary
> and pass it around. There is no documentation about which attributes
> are expected or optional. It also smells of code repetition because
> almost every controller action method creates one of these.

Yeah I agree this aspect of the code needs better documenting. I still
think I'd prefer multiple function and method arguments as well rather
than passing it all in a dictionary, doesn't seem pythonic and makes
dependencies and parameters difficult to trace. People have tried to
explain the point of this to me a couple of times but I'm not sure it
sunk in.

> The package_formalchemy and auth vs new_authz modules can be really
> confusing. I'm sure there are other examples of such things which can
> stand to be removed from the core once and for all.

I think both of these are on their way out at some point