[kforge-dev] domainmodel API design suggestions

[John Bywater]

Dear Jo,

>Is gloomy rhetoric like this really necessary?
>
I was only extending the "eat your own dog food" metaphor! That's why I 
used speach marks.

> In an open process 
>there are always going to be incidental situations where developers
>commit changes that break tests for others (such as forgetting to 'svn
>add' a file which does often happen.) The codebase isn't "poisoned" at
>this point, it just needs getting back in line with its tests or its
>test coverage improved. This happens. 
>  
>

That's why the KForge development process is a controlled process. And I 
don't mean that in a civilian police sense, but rather in a factory 
production sense. I will continue to speak up when I think there has 
been a departure from the basic quality procedures of the agile approach 
(to a first approximation: test first).

This diagram contains everything I want to say:
http://www.agiledata.org/essays/tdd.html#Figure1StepsOfTDD

>
>Well, on the positive side there is MySQL support now, which opens up
>to more potential users.
>
Sure. Thanks for setting up the db.type dictionary word, user 
configurable an' all.

We must fix those 'postgres' string escaping calls, and find the full 
sets of database keywords.

> A quite functional interim fix happened in a
>few minutes.
>
You may think so, and I may not, but the evidence is that the 
domainmodel tests were failing the day after this 'interim fix' was 
committed. I don't think the KForge code was still running after 
session.key was changed to session.sessionKey, because the Person class 
was broken. I'm sure ScanBooker was broken for the same reason. Are you 
really sure CKAN was working, new database an' all? I doubt it. I don't 
think anything was 'fixed' until the next day when I debugged the 
change. But I consider that the 'interim fix' wasn't fixed until the 
domain model names were restored.

> There was no necessity to take time at this point 
>implementing a map for reserved words;
>

By 'this point', I'm not sure whether you mean in this iteration, or at 
the point where you wanted to develop against MySQL? For the later: 
sure, it may have been most preferable to use your distribution's 
postgres package. For the later: under the banner of "fix broken 
windows", if uncontrolled changes are being made to core library classes 
because of name collisions, then to me it seems like a very good time to 
implement some name collision avoidance, to make sure the motivation 
doesn't arise again.

Let's look on the bright side! At least we've got the basis for 
model-database name collision avoidance support. We just need to add all 
the keywords for the different databases.

> I think an issue saying "just
>don't use reserved words / fix this one day" and moving on with what
>is in the plan, would have been a perfectly adequate response. 
>  
>
I understand what you're saying. I happen to think getting Postgres 
working also would have been perfectly fine.

>Sigh, i mean i totally accept we were at fault in not making sure we'd
>run the domainmodel tests as well as the ckan ones before committing
>changes to domainmodel, i hoped that was clear last week.
>  
>
Yes, that much was clear. I remember you saying that you ran the [ckan] 
tests, but they didn't catch the error.

In speaking for the agile approach, I'm also speaking against the ad hoc 
approach, which can characterise some open processes of software 
development.

You aren't advocating ad hoc-ery, are you?

>
>Why do i need a name for it? What do you mean by 'context' and 'forces'?
>  
>
I'm not telling you that you need a name for it. I'm just asking whether 
it has a name. If it had a name we could refer to it as something other 
than 'your idiom'.

Likewise, I'm not telling you that you must communicate the context and 
forces that are resolved by the idiom, but it would help me to 
understand what problem you're solving with your solution.

I thought you would recognise the terms context and forces from the 
pattern language texts. Here's one:

"So, the various writing styles, from labeled sections of a standard 
template to running paragraphs of a more narrative style, have less to 
do with the success of a pattern than some more basic elements. These 
include establishing a problem and its context, analyzing the forces 
bearing on the solution and, most important, offering a concrete solution."

http://c2.com/doc/plopd95.html

>I intend this as a candidate for the style of thing i mean, not a
>'this is how it should be' pronouncement. I'm puzzled by how broad
>these questions are - i thought my suggestion was quite constrained.
>  
>
My questions pertain only to your suggestion, and share it's breadth.

>Why it pertains to the domainmodel package? because this is about
>dm.application.Application's interface and how applications are built
>on the not-a-framework?
>  
>
Ironically enough, the Application class *is* the framework that 
assembles features from what would otherwise be a toolkit. It's 
attributes are the major features (more or less). It's methods assemble 
an application from specified features (by directing an application 
BUILDER).

>But right, what i wanted to do and was struggling to see how was
>getting lists of DomainObjects right from the registry, rather than 
>having to ask for a register of DomainClasses and start asking them
>what instances they know about, i was just looking for the wrong thing.
>  
>
This is one of the things that has been documented (on KForge at least):
http://project.knowledgeforge.net/kforge/trac/browser/trunk/docs/cli_shell.txt

I'm sure I mentioned it to you when you were here. I said "we normally 
do this..." but you were sure you were looking for meta data, so we 
misunderstood each other. Isn't meta data "data about data"? If so, then 
you really were looking for registered data. The meta data would 
describe the registered data? What isn't meta data for you?

>>Does it make sense to have the type of an instance be also responsible 
>>for collecting the set of instances of that type? I think it can make 
>>sense, but doesn't in our case:
>>    
>>
>
>So the code snippet Rufus sent earlier cleared this right up for me. 
>
>    >>> app = ckan.getA()
>    >>> tags = app.registry.tags  # or app.registry.projects or ...
>    >>> tags_i_want = tags.filter(...)
>  
>

OK. But please note that this short cut to the application object is 
fine in the shell but for documation's sake in system code please write:

from ckan.soleInstance import application

>>We have a need to support two distinct responsibilities, and have 
>>written two distinct system objects for this: the DomainObject class, 
>>and the DomainObjectRegister class. Having one fall back onto the other 
>>means that a DomainObject instance can't instantiate a collection of 
>>aggregate domain objects without somehow producing an objective 
>>confusion (which Spinoza defines as mixing two distinct concepts, such 
>>as 'triangle' and 'square'; in our case 'type' and 'collection').
>>    
>>
>
>It would be a lot easier for me if you could illustrate what you mean
>in code or pseudocode.
>

We have three different concepts: type, instance, and collection. 
Consider the following:

# an example of a type: string
class String(object):
def __init__(self, data='');
self.data = data

# an example of an instance of a type: a
a = string('a')

# another example of an instance of a type: b
b = string('b')

# an example of a collection of instances: c
c = list( (a, b) )


That's just what we've got. We don't collect DomainObject instances with 
the type. You'd want a collection for that. In our domain model, 
persistent collections are called registers.

Registers aren't domain types, but do know what type of domain object 
they collect.

Otherwise, when you asked an object for an aggregated collection, you'd 
get a class rather than an object, which wouldn't make sense.

Please tell me whether this does or doesn't make sense.

> E.g. an example of a DomainObject instance
>trying or failing to instantiate a collection of aggregate domain
>objects...?
>

One example is in KForge where a project.members, or a 
person.memberships, or a project.services, or a plugin.services, are 
register objects, and not Python classes.

The registry.persons object is also a register object.

The failure would be if registry.persons['person name'].members was a 
subclass of DomainObject, or if the subclasses of DomainObject somehow 
hold all their domain object instances.

That would just be confusing.

> Is this saying that DomainObjects can be objects or they can be
>sets, where 'instances can hold subsets of instances'?
>  
>

No. A DomainObject instance is not a set. It is an instances of a 
subclass of DomainObject, which is modelled type, corresponds to a 
category in the domain. Instances of the model type are collected in the 
registers of the model.

Registers are not DomainObjects, but rather DomainObjectRegisters.

>How does this relate to what filter() is doing above?
>  
>
It doesn't related to filter(). It relates to your earlier suggestion to 
have the domain model types be responsible for collecting domain object 
instances. But it turns our that wasn't what you wanted to achieve anyway.

Test-driven development eliminates this sort of wrongly directed 
discussion (amongst several other things that make you go faster).

What is it that you want filter() to do?

>>The requirements of the KnowledgeForge 
>>service (with things such as members of a project) led to the 
>>distinction I describe.
>>    
>>
>
>I don't see why that requires members to be a set though?
>
Rather than continuing with the term 'set', can we just call things like 
project.members a 'register', and the items in a register a collection 
(of registered domain objects)?

> (Forgive me
>if this has been thrashed out and is documented in the archives) Say,
>if this were at the SQLObject level directly i would use a
>many-to-many mapping between projects and their members.
>  
>

But then there wouldn't be an association object with which to persist 
and access the role of the member on that project.

>>Summary: the domainmodel code makes a distinction between type and 
>>collection, so that instances can hold subsets of instances without need 
>>for further conceptual complexity. But we didn't implement delegation by 
>>types to the collections of all instances of that type. But we can do that.
>>    
>>
>
>Okay, you have lost me now :) 
>  
>

It's just this: we've got some abstract domain model classes, which 
model different kinds of things in the problem domain (the emerging 
working processes of open knowledge, at least). Then, to manage distinct 
collections of instances, we've got a domain register class. Then, to 
register a particular thing in the problem domain we create an instance 
of the modelled type, that is an instance of the class, and we ask a 
register to do that, so that we can have a recurring structure to the 
model 
(registry-collection-instance-collection-instance-collection-instance....).

There are a number of advantages: one is that if you ask sub-collection 
to do things, it's parent domain object is assumed.

This is all basic DOMAIN MODEL stuff. I didn't attempt to do anything 
difficult.

>>I wouldn't necessarily say that there was an API at the moment. There is 
>>a collection classes, each having responsibilities and collaborations, 
>>and therefore holding several attributes and methods. And so there are a 
>>number of de facto interfaces, but no API as such has been designed or 
>>documented.
>>    
>>
>
>Right, well that is why i am interested in this discussion, as a
>prospective application programmer i would like an application
>programmer's interface, and would enjoy the opportunity to help design
>it.  
>  
>

Well, great. That's why I'm emphasising testing, because the best way to 
design a class is to write a test case for all these reasons, and more:

http://www.agiledata.org/essays/tdd.html

>>That having been said, the notes you made on "how to query for instances 
>>of an object in the domain model" were based on my answering your 
>>question of how to access what you referred to as "meta" data. That's 
>>why I was steering you to the register of domain classes, each of which 
>>hold a meta object that describes a type of domain object.
>>
>>As you really want the model data as such, rather than the model meta 
>>data, I would suggest taking a slightly different approach: use the 
>>domain object registers that have been instantiated on the registry 
>>directly (e.g. registry.projects).
>>    
>>
>
>Right, well i want both the model metadata and the instance data which
>is metadata about other things (like geospatial data sets).
>  
>

No problem. Write a test expressing something of what you want to 
achieve, and we can go (incrementally) from there.

>>I have do desire to discuss improvements to software systems without 
>>reference to what that improvement is to adequate to, expressed normally 
>>as a unit test, 
>>    
>>
>
>You don't want to have any software feature / interface discussions
>without unit test up front expressing a suggestion?
>

Wrong: Feature discussions start when the Product Owner (normally) is 
writing a User Story, and continue during the iteration when that User 
Story is scheduled for development. Implementing the User Story starts 
with a simple unit test that begins the process of adding. It continues 
with short alternations of adding and refactoring, in collaboration with 
the Product Owner, and until the Product Owner thinks the User Story is 
solved.

In this case discussing the API has been invoked by your wanting to add 
support for meta data exchange protocols (wasn't it?).

Let's pop back up to that original requirement, and write a test. Then 
we can go forward without all these mad diversions.

To be clear: "As Ron Jeffries likes to say, the goal of TDD is to write 
clean code that works."

The domainmodel/kforge/ckan code works and is quite clean.

As a general comment, we didn't achieve what we did by asserting that 
unamed idioms are 'more idiomatic' than an idiomatic thing.

> But there's always
>going to be topic drift.
>
I've got no idea what this means. Test-first is the way to stay on topic!

> In this case there was already a quite
>adequate way of doing what i needed and i quite quickly found out
>about it.
>  
>
So what's your complaint? I suspect that if you'd written a test, we 
could have gone straight for the ready made registry registers, and 
avoided much of this discussion.

>I think this is too much enforcement; too high a bar.
>

What is? Test-driven development? It isn't a high bar, it's a complete 
reversal of the normal way of writing software. It's a change of 
direction (away from running on hope, long debugging sessions, pointless 
schedules). It rocks. It's by far the easiest way to do things. Not 
doing this is a straight route down to the software crisis. There 
couldn't be a bar high enough to prevent that. Think about it as a 
safety barrier down the sides of the project, rather than a road block 
of some sort?

> It's a nice
>ideal
>
No it isn't a nice ideal. It's an accepted pragmatic programming skill. 
It's what loosers don't do.

"It's not about 'Testing'" it's about design:

http://www.pragmaticprogrammer.com/starter_kit/ut/index.html

"Shall we design something? Very well then, let's write a test case..."

> and i definitely agree that it is fastest and most fun to fight
>with code not with words.
>
I don't want to fight. I want to establish the basics of quality, 
sustainable user-driven software development, and achieve a harmonious 
process that becomes a sort of second nature due to familiarity. A bit 
like living in your home. That, so we can move swiftly on to the real 
issues, things like the NHS (and other things) being destroyed by iSoft 
(and other idiocies).

"Gentlemen, our time is short, and if we live, we live to tread on Kings!"

> I would cheerfully write doctest into an
>email and wonder whether that would be acceptable in order to persuade
>you to discuss software suggestions and changes.
>  
>
The best thing (read: please do this!) is to decide what you're trying 
to add to CKAN, and to write it down as a User Story. Then, normally I 
would hope you would check this with the Product Owner, but I assume you 
have in this case.

Then for implementation, I suggest these steps: to name the concern, to 
write an empty unit test after that name, to add that unit test to the 
repository but without including it in the CKAN test suite at this time 
(do that when your User Story works).

Then: look and think again at what you're trying to add to CKAN, write 
it down the first thing that you know needs to be achieved directly as 
an assertion in a test method in a new TestCase (don't worry about the 
TestCase class name at this point, it can be renamed shortly), add the 
TestCase to the unit's suite of tests (they are at the top of the file), 
and run that unit test's suite to verify that the test indeed fails 
(use: '.../ckan-test your.testunit'). Then: think of the SIMPLEST THING 
THAT COULD POSSIBLY WORK combined with YOU AREN'T GOING TO NEED IT to 
satisfy the test. Make small changes untill your test passes. Then 
repeat this paragraph until your User Story works.

This is the way to excrete "clean code that works".

To be clear: renaming variables, methods, and classes is a refactoring, 
which assumes a passing test suite, which importantly is run each time a 
small change is made.

>>because the process doesn't complete, and is therefore 
>>bound to be exhausting. I would like this to apply the idea of 
>>adequation to discussion of the qualities of a software system: you say 
>>it's baroque (ie overly complex?) but you don't say what you measure and 
>>judge its complexity against.
>>    
>>
>
>Well, the judgement is personal about the heavy use of inheritance
>hierarchy and multiple inheritance.
>
No, I'm sorry there isn't heavy use of multiple inheritance. There is 
*some* use of multiple inheritance, but only where necessary to solve 
particular multiple inheritance shaped problems. The views have quite a 
lot of multiple inheritance for this reason, but I expect most that will 
go away when we make the views COMPOSITE, but not all will go away for 
very good reasons (which I can explain). The domain model layer uses it 
very lightly to avoid some repetition. It isn't used anywhere else.

What do you mean by "heavy use of inheritance heirarchy"?

I'd say, overall, you're comments are consistent with what I 'always' 
hear from developers who haven't done object oriented development very 
much, which includes not having had much opportunity to develop an agile 
approach, most notably those who didn't already get into test-driven 
development.

> I have shown bits of the design and
>code to a couple of other programmers and they both had similar 
>reactions to mine, and always conclude 'well i am sure it does work
>very well'.
>
Frankly, your friends don't need to well and be sure it does work very 
well. We have (at least when it isn't broken by people making 
undisciplined changes) a system which passes its test suite and 
therefore demonstrably works very well in its own terms. In addition, 
the tests largely correspond to a series of user stories that a Product 
Owner has determined, and the system is in production on a number of sites.

Compare it to GForge, which is what we started with. After that, I'm not 
bothered what unamed individuals happen 'always' to think.

To repeat: we care about "clean working code".

Show me something that's a mess, and I'll clean it up. Show me something 
that's complex, and I'll tell you how we got there, and then ask you 
what you think would be simpler.

Talking on the "molar aggregate" level (ie "the 'system' is 
overcomplex", when it's made up of some simple things, some new things, 
some old things, some complex things, more than likely one or two overly 
complex things) is I would suggest to depart from productive lines of 
reasoning.

> I'm quite familiar with the application domain and have
>built several similar 'not-quite-frameworks' along similar lines.
>  
>
Are they online somewhere? Can I have a look?

>I know there has been a lot of roundabout discussion of the role of
>the meta-model mapping layer and to what extent it is really necessary
>to have one, and it will be really positive if that can be made to
>support a range of ORMs and OODBs as well as SQLObject in the future.
>  
>
Sure. But the value of DOMAIN MODEL is there regardless of how many 
different means of persistence are supported.

>And it is tempting to stay on the path of something that looks or
>feels right even when that activity is yakshaving with regards to the
>applications that one originally set out to get built.
>  
>
Who is shaving yaks? What does that mean?

What is the temptation? What are you saying looks or feels right?

I'm sorry, I don't know what you're trying to say.

>Okay i will look at least at http://www.martinfowler.com/eaaCatalog/
>  
>
Thank you.

>But the patterns aren't an endpoint; they are a midpoint.
>
I feel they are one startpoint of producing a conceptual component, and 
an endpoint of a number of system developments. They get you going at 
the start of a piece of design, and they get produced at the end of many 
pieces of work. That's why I asked you about the "idiom" you invoked. I 
was asking whether you were in the process of introducing a pattern that 
I didn't know.

I hope this project is still beginning. At least I'm just getting 
started. we're still doing gathering work, it would seem..... And we're 
still on major version 0....

Please remember that nobody has been working on this fulltime, and it's 
only been running for 18 months total elapsed time. I think we've been 
highly productive overall, but we didn't peak, or middle out yet, I 
don't feel. Perhaps when we've got two or three pairs of developers up 
to speed, a version 1.0 released, a greater install base, some more 
users providing feature requests, we'd be on the way?

> I don't
>think it's possible to take a set of patterns and mandate that they
>are fine and therefore the complete system design is fine. The system
>is as much about the people who interact with it, about the
>interaction of different subsystems within it, more like in
>http://en.wikipedia.org/wiki/Actor-network_theory
>  
>
If this were a general comment, I would entirely agree. As comment on 
this software project, which it must be, it's quite mistaken, because we 
have focussed everything from the outset on interactions with the 
system. That's what User Stories are for. User Stories don't equal 
usability, but we didn't really get to doing much usability work yet 
(Nick Stenning did a great job with the CSS and Javascript though), 
because we only just got the core User Stories working. If you feel the 
system isn't about the people who interact with it, please say more. 
This needs discussing with the Product Owner, who I think has made good 
decisions about priorities so far, and has steered a fine course. There 
are a few of things which I guessed perhaps would be wanted more (such 
as supporting somebody applying to join a project, listing project 
vacancies, etc). We talked about it and didn't do that yet. But if you 
think there is a problem with the way people interact with the system, 
let's talk about the User Stories (with the Product Owner).

Complementary to this, we have used published patterns of design to 
separate concerns within the system, and to address a number of other 
common problems. So there are already fairly well designed sub-systems.

As there is no value in forming a vulgarisation of the achievements, nor 
in attacking vulgarisations, so there must be little value in countering 
such attacks. I would simply like to point out that nowhere do I talk 
about the "complete system design" because, like the "API" there isn't 
one. I certainly don't equate the design of the system with the patterns 
that were used to produce the concepts which were coded for. I have said 
that the design is an application of Fowler's patterns. If you make an 
application of XML, you still have to create you schema. If you make an 
application of Python, you still have to write .py files. If you make an 
application of design patterns, you still have to think about the design.

Production happens on a molecular level, step by step, piece by piece, 
as you know. Show me where I have been talking about patterns and I will 
show you where I have been trying to initiate a sane discussion based on 
common terms, marking off the main aspects of the design in a common 
way, and with common names. I'm sorry if you never came across this 
approach before. But please don't make me out to be some kind of 
ham-fisted, template-applying, usability-destroying, 
monolith-integrating, cookie stamper.

We have some minimal architectural and process inclinations that I have 
set out in earlier emails, and which contributors are probably expected 
intelligently to follow (that, probably instead of project style guides, 
process manuals, other pre-emptive management misery): we apply design 
patterns and patterns of enterprise application artchitecture to our 
thinking, we're following the agile approach, and so on. Together, the 
six tendencies that Grady Booch reports are making software appropriate, 
after he emailed the whole software industry top brass (including Bill 
Gates, he said) to find out. I heard him speak at the Computer Science 
lab in Oxford.

>See also http://frot.org/zool/alexander.txt ... not having learned
>your disciple i can't shed it, but perhaps it is something you might
>bear in mind? ;P
>  
>
What, from that, would you like me to bear in mind? Shedding something?

Who cares about beauty? Not the net artists: for them Keats' "truth is 
beauty and beauty truth" has been replaced by "truth is information", 
and "the world is my interface". Not the software developers: for them 
high elegance has been lost within "less is more", or it's modern 
version "worse is better". In other words it's more elegant for things 
to be less elegant--hence your complaint?

The Timeless Way talks of things becoming second nature, not things 
becoming a distant memory. It's a point about producing an immanent 
understanding, rather than gaining and then getting rid of an autism of 
representation.

Christopher Alexander didn't depart from all the usual things he found 
insufferable, only to "shed" everything and to return back to wanting 
and being able to talk about nothing other than ECONOMY OF SCALE, 
CONCRETE EVERYWHERE, OUT OF TOWN HYPERMARKET, DEAD SUBURBS, VERTICAL 
DRINKING, NETWORKS OF LOATHING, BILLBOARDS WITH LIES, BOWLING ALONE. He 
continues on his way, with the vocabularies enhanced, having constituted 
a new surface on which to continue making steps (his new research 
group), to include refining and extending vocabulary in an exchange with 
others (one of its tasks).

I know because I've talked about this with him, and with Michael 
Mehaffy. As far as I understand, the point initially is to transform 
your own ways of designing, so that you can get over yourself, and stop 
making "expressive intentions of the designer". I'm a member of 
Christopher Alexander's new international research group, which has this 
aim:

"We want to help to transform what it is to design, by developing (or 
further developing) new kinds of analytic and generative tools. We want 
to help shift the emphasis from the expressive intentions of the 
designer, to the adaptive processes that make a design most successful, 
and the tools needed to do that. Progress has already been made (and in 
some cases can be extended further in our work) on evidence-based 
design, generative codes, Space Syntax, pattern languages and related 
tools."

From: http://www.esrg.blogspot.com/

The first symposium was a couple of weeks ago. One project of the group 
is updating the patterns of Pattern Language, making them Open Source 
(Open Knowledge?). I made the suggestion to have architects adopt 
software developer's steering metaphor, rather than the building 
metaphor. That was very well received. The research group may begin to 
work in just the way we have been on KForge.

The point, perhaps, is that we can only aquire useful design 
capabilities if we hold and develop that capability in common with 
others. That's what I'm trying to breakdown for the list here. We need 
to follow, adopt, and then remember the vocabularies for patterns that 
have emerged and been named in a number of areas of concern. We of 
course can join in with the process of identifying and naming patterns. 
But I haven't found that easy so far. The patterns for doing that are 
still being worked out. But shedding the vocabulary is absolutely wrong: 
that's what you're supposed to develop a close familiarity with, so you 
can then think most fully about the context you are actually in. You 
make it your second nature so you can deal with the matter in hand.

If you shed anything you shed your (old) self, but that doesn't mean you 
get over being capable, or of having words for things, or talking about 
and making use of elements of successful design, of using words for 
those elements in conversations with others, but just the reverse: it's 
when you get over yourselves that you are able to follow things that 
work without resisting so much you damage your chances. Particularly 
with language. If there is a Timeless Way, then the Way To Language 
(Heidegger) is certainly en route. But its a dangerous affair, and 
resisting is a natural response (otherwise you'd probably be a sucker 
for all sorts of nonsense! :-)).

See also AntiOedipus for much the same discussion. They talk of 
bodies-without-organs (e.g. bodies of practice, of knowledge, social 
bodies, etc). I explained the main propositions of AntiOedipus to 
Christopher Alexander at Kings College Cambridge on evening and seemed 
to agreed with them. (I talked about steps making up a walk: the walk is 
always there as a whole walk that lacks nothing, but making steps, and 
thus extending the walk doesn't exclude stepping further, and neither 
does the walk have a need for any further steps to be taken. And where a 
step is when we learn to do something, and the walk is our body of 
practice. "The first step is always the hardest...")

On this project, we've made some steps. It isn't very nice to state that 
our walk is lacking something (simplicity, you say?), especially when 
you don't first familiarise with how we are stepping, or what steps we 
made so far. Given that the way we are stepping is clearly indicated 
("we're following the agile approach...") and the agile approach is well 
described in the Web, and the currenly functionality is there to be seen 
(we're "eating our own dog food" yum yum), I'm not sure what you're 
trying to achieve; how you're trying to achieve it; what you're common 
propositions are.

I would prefer you join us on the agile way. As I said, at first it can 
help to be introduced to the project's ways for an iteration (or 
functional equivalent).

>Yes well perhaps this has the advantage that we *could* exchange more
>code, even proper doctests here rather than veering off into
>meta-level discussions which can take up a lot of time too :)
>  
>
This is the whole point, and exactly the sort of thing I'm trying to 
avoid. I'm prepared to go off the beaten track to find and to persuade 
others to move to where to ground is firmest, but I'm not going to stay 
where it is soft for very long (ie writing long emails about all this 
stuff).

What is a doctest, and how does it compare to the common term "unit test"?

Is a doctest something you write in WordUnit:

http://www.waterfall2006.com/beck.html :-)

[after checking what doctest is:]

The Python module doctest is not a unit testing framework. Following the 
Python system test suite: "It is preferred that tests for the [Python] 
test package use the unittest module." That's our policy too. Please 
don't use doctest on this project.

I prefer to keep process vocabulary non-language specific. So even 
though there is a Python module called unittest, I prefer to think, 
write, say that we are writing a unit test, and not a unittest.

Why can't we write, and talk about, unit tests? Why do you introduce the 
doctest module when you can see that we've been using the unittest 
module everywhere? If unittest isn't good enough, please say why.

Why can't just connect better with the established style of this 
project, and go from there? We have 10,000 lines of Python in the domain 
model package, 10,000 in the KForge package, 5,000 in ScanBooker, and 
3,000 in CKAN. Introducing different things for the sake of it isn't 
really helpful. I find it quite concerning, disconcerting even. If there 
are better tools to use, then we can plan to substitute. Doctest isn't a 
unit test framework, so let's not use it or talk about it anymore 
(unless something changes). We're using the Python unittest module to 
write unit tests. If that's a mistake, then we can address it a 
controlled manner. If not, let's not waste time, or confuse people.

I assure you that what we have done is (on the whole) not stupid, or 
baroque, and your spending a little time, perhaps one iteration as I 
suggested when you were up here, being shown things, connecting by 
connecting, rather than trying to connect by a process of belittling 
present achievements, will be most productive. The first thing to locate 
is the test suites for each package, but I think we did that by now.

But, having spent so much time writing about all these general issues, I 
would prefer not to give priority to any further discussion, meta or 
otherwise, that isn't along the basic lines I have indicated (yes, 
please read the archives, there isn't very much in them). If you want to 
argue about the role, nature or affects of agile, or test-driven, or 
pattern language, then I suspect this isn't the place. This is the place 
for kforge developments, which have assumed the agile approach. It would 
be polite if that was actively respected, rather than being actively 
disrespected. If you look carefully, you will find that there isn't a 
high barrier to entry but an appropriate one, that we have the 
slenderest of process, only a basic design, and are iterating our way 
forward without "believing" in what we're doing (in a religious, 
temperamental sense). Valid complaints would include: not enough tests, 
not enough refactoring, not enough pairing, not enough planning, not 
enough feedback, not enough courage, not enough listening, etc.

As a last word, I must point out that my indications so far are really 
only starting points. And the code is a work in progress. But if we 
can't begin, we can't continue. It is a measure of the youth of the 
project that we're still discussing which way is Up. I hope we can get 
onto more advanced things after the basic elements are in place. But I 
won't be writing at this length on this subject on this list again.

Let's move on? We could always declare this a success, and then move on. 
That's what iSoft seem to do. :-)

Bye bye,

John.