Prev: multimethod + multiple inheritance
Next: Calling same method in inherited classes
From: jasongorman on 18 Mar 2006 05:00
Hi

I've been working with a small cluster of organisations on
metrics-driven approaches to software process improvement. The work is
based on the theory that, regardless of what methods, practices and
tools teams use, if you measure some aspect of their performance (e.g.,
lines of code per iteration) then that's what you can expect to
improve.

The adage "be careful what you wish for" seems to apply! I've been
running workshops, both with clients and in public sessions here in
London, on metrics design for evolutionary/agile approaches to process
improvement. The workshop format has also been used by a handful of
people who attended the ones I've run, with interesting results.

You can find the workshop material and some metrics resources -
arguably of varying merit - at
http://www.parlezuml.com/metrics/index.htm

The way the workshop works is simple:

* Split into small groups of 2-4 people (depending on how many people
there are - there must be at least two groups to play the game)

* Choose one performance goal (either from the example provided, or
from your own goals)

* Design a metric that you think will be a good indicator of progress
towards that goal

* Swap your metric design with another group (you can go around
clockwise or anti-clockwise if that helps)

* Take the metrics design the other group has given you, and ask
yourself this question: "exactly what is this metric telling us to do?"
Think of it as a sort of of test-driven development - only for process
improvement, and the tests are measurements. In TDD, we are compelled
to write the code that the test tells us to write. If it's the wrong
code, then we wrote the wrong test :-) For example, if the goal is
"increase developer productivity", and the metric design measures the
much-maligned "lines of code per unit of time per developer", then the
simplest thing you can do - the thing the metric is telling you to do -
is to generate a lot of redundant code. Copying and pasting a file with
10,000 lines of "System.out.println('All code an no function makes
Jack.com a dull web site');" should yield impressive productivity
results with very little extra effort.

* If the results don't match the original goal (or the spirit of the
goal), redesign the metric to make it harder to game in the ways you've
explored.

* Pass the redesigned metric on to the next group (or back to the
orginal designers), and repeat the exercise until either:

a. All the goals are covered and all the metrics are a good match for
their goals
b. You run out of time

Pascal Van Cauwenberghe summarises the version of the workshop he ran
recently at http://blog.nayima.be/blog/Entry20060215.html

I'll be running it again with the help of Duncan Pierce at SPA 2006 in
about a week's time, and plan to run more workshops over the coming
months if suitable venues and guinea pigs can be found :-)

I want to encourage people to try both the metrics design workshop AND
Agile Software Process Improvement, and I'd be very interested to hear
how you get on. Please feel free to email me with your feedback and
experiences.

Jason Gorman
http://www.parlezuml.com

From: H. S. Lahman on 18 Mar 2006 11:36
Responding to Jasongorman...

> For example, if the goal is
> "increase developer productivity", and the metric design measures the
> much-maligned "lines of code per unit of time per developer", then the
> simplest thing you can do - the thing the metric is telling you to do -
> is to generate a lot of redundant code. Copying and pasting a file with
> 10,000 lines of "System.out.println('All code an no function makes
> Jack.com a dull web site');" should yield impressive productivity
> results with very little extra effort.

Alas, this is a common mistake in developing metrics programs. It mixes
the goals of process monitoring with those of process improvement.
Metrics simply provide <hopefully useful> observations about the
development process. They do not in any way change the process, nor are
they intended to foster change directly. IOW, the goal of a metric is
never to improve anything.

What the developers do in response to data collection is something quite
different and that is related to process improvement. One does not
improve processes by simply making the metrics look better and the
formal process improvement paradigms have built-in safeguards to prevent
that from happening. Thus it is inconceivable that any of the standard
process improvement approaches (TQM, 6-Sigma, Baldridge, etc.) would
lead to the conclusion that writing redundant code was the proper way to
improve productivity.

Bottom line: the "goal" of any metric is simply to provide the best
measurements of something you need to know to make good decisions about
the development process. IOW, if you define what you need to know to
make a process decision, you can define a useful metric. Thus the goal
of a productivity metric is not to improve productivity; it is simply to
provide the best information available so that someone can make informed
decisions about productivity.

There is a second implied misunderstanding of metrics here, at least
with respect to conventional usage for metrics systems in software
development. A metric is conventionally viewed as a system of
individual measurements, usually periodic over time, where the
interpretation is based on /differences/ between the measurements. (SPC
envelopes are a common example of triggering process change based on
changes in an ongoing metric.)

In fact, most metrics are developed after the fact of process
improvement. The goal of the metric is to ensure that whatever
improvement was defined is actually having the desired effect and that
the improvement is being practiced properly on an ongoing basis (e.g.,
that the developers have not back-slid into old habits from before the
improvement). So the question your metrics designers should be
answering is: Given productivity improvement X, how can we best measure
its effect and whether it is continuing to work?

In contrast, process improvement decisions are usually based primarily
on fresh, one-time data collection rather than ongoing metrics. Once
the process improvement has been identified based on analysis of that
data, one designs an ongoing metric to monitor it over time. If the
metric deviates from the expected course, then that triggers an
additional round of process improvement data collection and analysis.


*************
There is nothing wrong with me that could
not be cured by a capful of Drano.

H. S. Lahman
hsl(a)pathfindermda.com
Pathfinder Solutions -- Put MDA to Work
http://www.pathfindermda.com
blog: http://pathfinderpeople.blogs.com/hslahman
(888)OOA-PATH



From: AndyW on 18 Mar 2006 04:54
On Sat, 18 Mar 2006 16:36:25 GMT, "H. S. Lahman"
<h.lahman(a)verizon.net> wrote:

>Responding to Jasongorman...
>
>> For example, if the goal is
>> "increase developer productivity", and the metric design measures the
>> much-maligned "lines of code per unit of time per developer", then the
>> simplest thing you can do - the thing the metric is telling you to do -
>> is to generate a lot of redundant code. Copying and pasting a file with
>> 10,000 lines of "System.out.println('All code an no function makes
>> Jack.com a dull web site');" should yield impressive productivity
>> results with very little extra effort.
>
>Alas, this is a common mistake in developing metrics programs. It mixes
>the goals of process monitoring with those of process improvement.
>Metrics simply provide <hopefully useful> observations about the
>development process. They do not in any way change the process, nor are
>they intended to foster change directly. IOW, the goal of a metric is
>never to improve anything.
>
>What the developers do in response to data collection is something quite
>different and that is related to process improvement. One does not
>improve processes by simply making the metrics look better and the
>formal process improvement paradigms have built-in safeguards to prevent
>that from happening. Thus it is inconceivable that any of the standard
>process improvement approaches (TQM, 6-Sigma, Baldridge, etc.) would
>lead to the conclusion that writing redundant code was the proper way to
>improve productivity.
>
>Bottom line: the "goal" of any metric is simply to provide the best
>measurements of something you need to know to make good decisions about
>the development process. IOW, if you define what you need to know to
>make a process decision, you can define a useful metric. Thus the goal
>of a productivity metric is not to improve productivity; it is simply to
>provide the best information available so that someone can make informed
>decisions about productivity.
>
>There is a second implied misunderstanding of metrics here, at least
>with respect to conventional usage for metrics systems in software
>development. A metric is conventionally viewed as a system of
>individual measurements, usually periodic over time, where the
>interpretation is based on /differences/ between the measurements. (SPC
>envelopes are a common example of triggering process change based on
>changes in an ongoing metric.)
>
>In fact, most metrics are developed after the fact of process
>improvement. The goal of the metric is to ensure that whatever
>improvement was defined is actually having the desired effect and that
>the improvement is being practiced properly on an ongoing basis (e.g.,
>that the developers have not back-slid into old habits from before the
>improvement). So the question your metrics designers should be
>answering is: Given productivity improvement X, how can we best measure
>its effect and whether it is continuing to work?
>
>In contrast, process improvement decisions are usually based primarily
>on fresh, one-time data collection rather than ongoing metrics. Once
>the process improvement has been identified based on analysis of that
>data, one designs an ongoing metric to monitor it over time. If the
>metric deviates from the expected course, then that triggers an
>additional round of process improvement data collection and analysis.
>
I would agree with that.

In general, always ask the question first, then develop the metrics to
get you the answer. (never the other way round).

Also, in general with metrics you are looking for change you can only
find change if you have stability first (and this is what the CMM is
about). So consistency is good as it means stability with your
project. If he metrics change or deviate, then it indicates something
has occured that needs investigation.

If you instigate a process change, then the deviation should show up
in the metrics. Although I've found often that many people instigate
the process change then implement metrics to try and see if its
working - which is rather pointless as you have nothing to compare it
with.



From: Ilja Preu? on 19 Mar 2006 01:54
> Also, in general with metrics you are looking for change you can only
> find change if you have stability first (and this is what the CMM is
> about). So consistency is good as it means stability with your
> project.

This sounds interesting - could you please elaborate a little bit, perhaps
with an example?

Thanks!

Cheers, Ilja


From: AndyW on 18 Mar 2006 16:42
On Sun, 19 Mar 2006 07:54:07 +0100, "Ilja Preu?" <it(a)iljapreuss.de>
wrote:

>> Also, in general with metrics you are looking for change you can only
>> find change if you have stability first (and this is what the CMM is
>> about). So consistency is good as it means stability with your
>> project.
>
>This sounds interesting - could you please elaborate a little bit, perhaps
>with an example?
>

In CMM terms, level 2 is the repeatable level - pretty much what I
would suggest the majority of s/w teams are trying to achieve (based
on my probably incorrect observation).

Level 3 is probably the holy grail of teams that focus only on
methodologies as being the savior of everything [this is where I have
a poke at the agile and eXtreme folks :) ]. It means that you have
not only managed to repeat stuff for every project, but you'll be
using some baselining tool so that every project uses the same base
process tailored for whatever constraints have been identified. You
can think of this level as being the equiv for ISO9001 for Software.

So think of it at this level - eveything is consistent and nice and
stable. Your not trying to invent solutions to as many problems.
Change happens, but its unpredictable, but we have solutions in place
(part of the method) that allows us to easily handle it. (its still
reactive). Metrics at this level is trying to be historical but
because of the level of change, its often reactive (after the fact).

The example here would be - "what is the volume of LOC we produced for
the last three months - has it increased" ?

Level four is the managed level. Its here that metrics really kick
in. You have a nice stable process, what you are looking for is when
it may become unstable (change). Also, we are looking to predict
change rather than react to it.

The example here is - "what is the volume of LOC that was produced
each month for the last 6 months. Has it it varied this month - if
so, by how much and what is the root cause reason for it. Did a
developer have issues - what where they and how can we refine them out
so they dont happen again".

Its at this level when we are looking at historical metrics to see our
past performance (our stable process) and current metrics to identify
deviations to that (where change may occur), then predictive metrics
to try and identify the risk of change occuring in the future.

(I used to have a joke on one project that I could forecast defects 6
months before they occured - the test team wondered why I hadnt fixed
the defects, I used to respond - because the code hasnt been written
yet]. (it gets worse - wait til you see the expression on a designers
face when you tell them how many defects they will produce)

At level 4 its important to realise here that you are dealing with the
same process used across the organisation - so the metrics may in
general not be limited to one individual project (as people in this
forum often talk about), but many projects - although in smaller
companies its likely to just be one project (I've found maintenance
projects often get close to this level).

Level 5
Optimising is the holy grail. Its where you are looking at your
organisation and the industry as a whole and changing to compete and
also improve your process. Using metrics to reflect and refine etc.
I've found this level to be very condusive for creating the
environment for the high performance team.

This is the level I like to work at, so thats often why I am at odds
with some of the discussions. Its for this reason I dont like working
with Ego's that think they can run good projects by force of will and
self experience :)


just my thoughts.
 |  Next  |  Last
Pages: 1 2 3 4 5 6 7
Prev: multimethod + multiple inheritance
Next: Calling same method in inherited classes