[kforge-dev] Taking stock of our current development strategy

Tue Oct 17 14:44:54 UTC 2006

Rufus Pollock wrote:

> Dear All,
>
> Recently I've been thinking hard about the relationship of 
> domainmodel/kforge/ckan to the existing python webapp frameworks. We 
> started work on KForge (and its core which has now become domainmodel) 
> in April 2005 several months before either django or turbogears had 
> even appeared on the scene let alone got close to 1.0. Other notable 
> developments since we began work are the inclusion of (some) support 
> for inheritance in sqlobject (v0.7) and the maturation of the wsgi 
> effort (wsgiref is now part of python2.5).
>
> Below are some rough thoughts on where we are and where we can go. Let 
> me know what you think.

If we want to take stock of our current development strategy, I think I 
would prefer to see thought and discussion about the design of the 
KForge software system continue, pretty much, along the same lines on 
which it was begun, and which have enabled us to achieve what we have so 
far.

As I understand things, our current development strategy, if it 
resembles our past development strategy, is not to select from various 
degrees of usage of a framework, it is not to have a "core", and it 
isn't to produce a framework.

If we want to take stock of our current development strategy, we should 
remember that we have sustained development and produced working, 
well-factored, software applications because we have followed an agile 
approach, because we have applied patterns of enterprise application 
architecture, because there is a clear requirement clearly identified 
and recorded within the KForge and CKAN user stories, and because there 
is a little resource to cover some of the costs of the work.

If we want to take stock of our current development position, we should 
note the several aspects in the code which aren't well factored, the 
several aspects which don't conform to object-oriented norms (being 
object-oriented is a lot more than being "object-based", as Python is 
throughout), and other aspects which aren't well named. There are also a 
number of aspects which aren't under test.

> ## The Current Situation and its Origins ##

In the first instance, surely the most important thing about the current 
situation is that we have a working KForge system, and a running 
KnowledgeForge service. Whilst it's true that we have factored out many 
things into the domainmodel package, unless we plan to release this as a 
framework for use by others (which would require more work than is 
required for our own purpose of having well-factored code across a 
number of systems) we should think of it as minor to the actual systems, 
which are the major concerns of the development, and which stand in 
second place only to the user stories/acceptance tests (which are the 
most important objects of development, and which in turn standing in 
minor relation to the major objects of analysis: the working events, and 
working responses to those events).

>
> We now have our own 'mini-framework' in the form of the dm 
> (domainmodel) package which provides:
>
>   0. Standard plumbing including:
>      * system dictionary (+ config loading)
>      * logging
>      * some testing extras
>      * exceptions

Let's not propagate the phrase "standard plumbing". We don't have such 
an intermediate organising element in the code, so it's probably 
misleading and counter productive to have one in our discussions.

Instead, I would much prefer to employ the vocabulary contained within 
the catalogue of patterns of enterprise application architecture:

http://www.martinfowler.com/eaaCatalog/

>   1. persistence layer (using sqlobject)

Actually, the dm package doesn't provide a persistence layer, but does 
provide a domain model meta data-driven data mapper layer (dm/db.py):
http://www.martinfowler.com/eaaCatalog/dataMapper.html

It is only the KForge and the CKAN system that have persistence layers 
(formed from dm.db + sqlobject + RDBMS).

But dm just provides a mapping layer. Specifically, it provides KForge 
with Concrete Table Inheritance and Metadata Mapping, and uses sqlobject 
to do that with postgres.

http://www.martinfowler.com/eaaCatalog/concreteTableInheritance.html
http://www.martinfowler.com/eaaCatalog/metadataMapping.html

>   2. domain model (objects + registers)

+ domain logic

Domain Model:
http://www.martinfowler.com/eaaCatalog/domainModel.html

Registry:
http://www.martinfowler.com/eaaCatalog/registry.html

>      * as well as some very basic objects (NamedDomainObject, 
> DatedDomainObject etc)

I don't think those class names actually exist: we have 
SimpleNamedObject, StandardObject, etc.

>      * and some non-standard ones (e.g. Person, access control stuff)

Person normally extends StandardObject, so it doesn't really work to say 
they are non-standard. It may be better to say they are non-simple, or 
standard extensions, or that there is a default, basic Person object 
that can easily be extended or replaced in a dm application.

>   3. command layer (i'm still not sure what the prognosis for this is 
> -- do we really need it?)

We don't have a command layer. We have a (broken) service layer 
(consisting of an incomplete, misunderstood, and now neglected set of 
commands).

http://www.martinfowler.com/eaaCatalog/serviceLayer.html

>   4. common webapp code (including an admin interface)
>      * we do use django (in a very old version) to:
>        * provide a web server
>        * do url mapping,
>        * do templating
>        * help out with our manipulators

Also: html decoding, mod_python event handling... common web app 
presentation layer stuff, as you say. We decided to use Django to cover 
these concerns.

>   5. A plugin system with a limited form of event handling and 
> notification

It probably makes more sense to talk about it as a way to extend the 
domain model, where plugins are notified of, and may handle, domain 
model object lifetime events. It isn't an abstract publish-subscribe 
framework, it doesn't make use of one, and it doesn't propagate events 
that aren't domain model object lifetime events. But that's because it 
wasn't written to.

I suppose it isn't that the form of event handling and notification is 
limited, but rather that the concerns of extending the domain model at 
run time don't call for a more sophisticated event system. And because 
we don't have other event notification/handling aspects, we didn't hook 
up a more general event-notification system.

>
> The key question is how much of what we have done is specific to our 
> projects and how much we could get from using an external framework.

I disagree: I'm not even sure if it's a good question: even if there is 
a good and easily obtainable answer to that question, it still wouldn't 
even begin to consitute an agile, patterned, development 
approach/strategy. I suspect there isn't one key question, but rather a 
number of aparallel concerns that have more or less adequate answers 
within the agile (diachronic) and pattern language (synchronic) strategies.

http://www.answers.com/topic/diachronic
http://www.answers.com/topic/synchronic

>
> ## The Options ##
>
> 1. Stay as we are in the main

I would agree with: "continue to develop as we have been developing".

>
>   * If we do I suggest replacing some parts of django (e.g. templating)

It would be really simple to use a different template processor. It 
would be much less simple to port all the templates. I think Django 
templates are fine. Why do you suggest replacing Django templating? We 
decided to use Django for the KForge and CKAN presentation layers.

>   * might be able to increase reuse in certain areas by using e.g. 
> wsgi more heavily

Yes. I think we should continue by increasing reuse, made possible 
through test-driven development and merciless refactoring. That's the 
way we'll easily be able to do wsgi things, and other things that may 
turn up and be desirable.

>   * could we pare back our domain model to be closer to sqlobject (or 
> if not articulate clearly what we are gaining from having our own 
> domain model base objects)

No, we can't pare our domain model back to be closer to sqlobject, 
because we aren't trying to implement a domain object to relational 
database mapper, but rather an effective domain model (a "domain logic" 
pattern, not an "object-relational structural" pattern).

Sqlobject isn't a domain model, even with support for inheritance (which 
would simply allow for class table inheritance - an "object-relational 
structural" pattern). We use sqlobject to map values between domain 
model objects and database table records, to form queries, to hold and 
iterate over query results, to maintain a single instance of any object 
in memory, and so on. These are data mapping concerns. Domain model 
concerns are constructed to be separate from mapping concerns. I say 
constructed to be separate because it is quite possible (and quite easy) 
to mix the two, and produce a working confusion. I often feel quite 
anxious about this, because we don't seem to be on the same page at all 
with making and maintaining a consensus of conceptual distinction. I 
feel my utterances are simple: I repeat Fowler because Fowler appears to 
repeat things that others have found to work. I trust Fowler not to say 
things he isn't sure about without declaring that, and in turn I try not 
to repeat things I'm not sure about without such declaration. But I then 
hope (and to some extent I expect) you will repeat my utterances, which 
aren't properly mine, or Fowlers', but of the fourth person singular. So 
what happens?

But with sqlobject, there's too much internal method renaming, calls to 
__init__, and so on to be a clean domain object base class. As a domain 
object isn't concerned about the mapping to RDBMS, a domain object 
should not inherit from sqlobject, but could instead make use of 
sqlobject by delgation. This is indeed what happens in dm/db.py.

However, the advantages of creating a domain model are many, have 
nothing whatsoever to do with data mapping, persistence, or relational 
tables, but about getting the conceptions to develop on an abstract 
plane: a model of the problem domain.

http://domaindrivendesign.org/books/foreword.html

Please read these patterns (perhaps buy the book I lent you, as the full 
descriptions aren't published on the wiki):
http://www.martinfowler.com/eaaCatalog/domainModel.html
http://www.martinfowler.com/eaaCatalog/concreteTableInheritance.html
http://www.martinfowler.com/eaaCatalog/singleTableInheritance.html
http://www.martinfowler.com/eaaCatalog/classTableInheritance.html
http://www.martinfowler.com/eaaCatalog/associationTableMapping.html
http://www.martinfowler.com/eaaCatalog/inheritanceMappers.html
http://www.martinfowler.com/eaaCatalog/metadataMapping.html
http://www.martinfowler.com/eaaCatalog/repository.html
http://www.martinfowler.com/eaaCatalog/foreignKeyMapping.html

Like some of the other issues here, we've talked about this several 
times in similar ways.

In general, I don't want to discuss enterprise software application 
architecture without reference to these patterns. In particular, I don't 
enjoy discussing KForge software without reference to previous 
discussions about the application of these patterns to the support of 
the User Stories of the Open Knowledge Foundation.

>
> 2. Move fully to turbogears or django or ...
>
> There are large benefits to using as much of a framework such as tg or 
> django as possible in terms of sharing in the combined work of many 
> other hands (and in contributing to that).

As I've said previously on this list, I'm not convinced that KForge is 
simply a "web application". That KForge integrates other web 
applications indicates that the concerns of KForge go beyond the 
concerns of a normal web application, and so trying "fully" to use a web 
application framework will be frustrating, because (as we found with 
Django) opportunities to extend the objects of the framework are likely 
to be inadequate where the system doesn't follow the patterns of 
enterprise application architecture observed, recorded, and published by 
Martin Fowler. And Django doesn't even flirt with these patterns (except 
for MVC, but then there's dicussion about how Django isn't really MVC 
anyway...). We've gone over all this already.

>
> The question is to what extent can we do so? To really get much more 
> integration we need to be able to plug more directly into the domain 
> model. The big question then is whether we can do inheritance.

I don't understand "to plug more directly into the domain model", and I 
don't see any connection between on the one hand variations of table 
inheritance and data mapping, and on the other hand elements of 
presentation layer in Django depending on their domain model interface 
(if that's what you're talking about). The two concerns appear to me to 
be separate.

>
> Furthermore while e.g. turbogears provides some kind of access control 
> model it is not clear how suitable it is to our situation (e.g. to 
> kforge).

Sure.

> It might well turn out that adapting tg to our purposes would actually 
> entail more work that it has taken to get our system working in the 
> first place.

I agree with this last point: I expect it isn't a productive move. The 
recent history of software development is that code reuse has failed, 
that the limiting factor in software development is the formation of 
concepts (it's astonishing how limiting this can be), and so reuse moved 
up to the conceptual level, where the form of conceptual reuse, and the 
basis for communication amongst developers, is the named pattern. I've 
said this several times to you now, no?

http://appropriatesoftware.net/wiki?NamedPattern

Applying some of the patterns, like building a good object-relational 
mapper takes lots of time, and the recommendation (in PoEAA) is to use a 
third party component (like sqlobject). But most of the others are very 
straightforward, and are perhaps best implemented freshly each time.

Best regards,

John.

PS This email has taken me several hours to write. It repeats many 
things that we've spent similar amount of time going over. So what 
happens? Do you think I'm mistaken, or do you not understand what I'm 
saying, or suspect it somehow of a hidden agenda? Or do you forget? I'm 
really not sure where the discussion is breaking down, and why it's 
going round in this circle. But it's kinda hard work. I don't want to 
die trying to explain these things. If we can't form a consensus, and 
align our thinking, perhaps we should try operating different roles 
where I act as developer (decide technical concerns, and develop code 
adequate for your user stories) and you act as product owner (decide 
business concerns, develop user stories and acceptance tests, and 
determine whether code adequates to your user stories). I would prefer 
to form an adequate concensus about software development practice, but 
if we can't achieve that I feel it would better to focus each of us 
exclusively on one role.