[ckan-discuss] Plugins for CKAN: overview + framework

Thu Aug 12 13:02:23 BST 2010

Hi all, 

just a quick update on the integration of plugins into CKAN. This essentially has two parts: (1) internal plugins and (2) external workers. 

(1) Internal Plugins

With lots of advice from Rufus I've coded a rudimentary plugin architecture [1] based on entry points thats already a part of CKAN 1.1. This only exposes very few extension points and needs to be refactored using a component architecture similar to what trac uses [2]. To make this really useful, we should also answer the question: which parts of the current CKAN core can be turned into components? 

IMHO this could include:
* anything related to RDF generation/export, including the talis code etc. 
* search indexing (partially done) 
* the admin interface
* change distribution (don't understand enough of it to have an opinion)

While we don't need to refactor all of this right now, it'd be helpful to collect ideas in order to consider those as requirements for the plugin system. 

At the moment, we have the following plugins ready:

* Disqus (http://bitbucket.org/pudo/ckandisqus) will add commenting to package pages (running on ckan.net now).
* Deliverance (http://bitbucket.org/pudo/ckanextdeliverance) will allow proxied delivery of external CMS pages via CKAN. This means you can edit text pages in a CMS like Drupal and then have CKAN deliver those pages with CKANs layout. In the medium term, we hope this can simplify some of the Drupal/CKAN setups running at the moment. 

(2) Workers

CKANext Workers [3] is a little package that will attach itself to the CKAN queue and allow clients to do processing whenever packages or resources change. It exposes a simple API that can be implemented by workers and it has a command line utility that can be used by supervisord to execute one or multiple plugins. 

As a little reference, I've built CKANext Archive [5], a downloading daemon that will retrieve all resources after package changes and attempt to store them via OFS (to S3, at the moment). 

We should now be able to build similar services to generate package statistics for RDF, WFS, WMS etc. content. 

Since all of this is pretty unstable I'd be very grateful for any feedback or bug reports :-) 

 Friedrich 

[1]: http://knowledgeforge.net/ckan/doc/ckan/plugins.html
[2]: http://trac.edgewall.org/wiki/TracDev/ComponentArchitecture
[3]: http://bitbucket.org/pudo/ckanextworker
[4]: http://bitbucket.org/pudo/ckanextworker/src/tip/README.md
[5]: http://bitbucket.org/pudo/ckanextarchive