Do You Get What You Measure?
in [OOP]
|
From: Alvin Ryder on 19 Mar 2006 20:52 jasongorman(a)blueyonder.co.uk wrote: > Hi > > I've been working with a small cluster of organisations on > metrics-driven approaches to software process improvement. The work is > based on the theory that, regardless of what methods, practices and > tools teams use, if you measure some aspect of their performance (e.g., > lines of code per iteration) then that's what you can expect to > improve. > A top level manager from many jobs ago use to say "a quick and easy way to improve a process is to measure it" ... "once people know they are measured by metric 'x' they'll improve 'x' .... or at least make it appear so." He had some references but I never followed them up I could just observe it in practice. We noticed it was like sugar for energy, you get a quick blast but if you want to do a better job you need something far more substantial. People just did whatever it took to make the 'x' results appear good but it was often at the expense of something else. > The adage "be careful what you wish for" seems to apply! > A more powerful adage is "put your heart into it" - if things don't change at that level people still only do what they were willing to do originally but now they just dress up that same chunk of real work differently, mysteriously it now has more 'x'. It wasn't entirely useless but on subsequent and deeper inspection the improvements weren't as great as 'x' apparently suggested. More powerful methods were to elimination waste especially in the form of variations and outright duplication ... people still did more or less the same work but it wasn't wasted work. Cheers.
From: H. S. Lahman on 20 Mar 2006 10:44 Responding to Bossavit... >>Bottom line: it doesn't matter very much which way one does development >>but it is extremely important that there is only One Way that everyone >>practices. > > > Assuming the process improvement benefits of standardizing on One Way > (monoculture approach) largely outweigh the penalties. I spent two decades doing ad hoc development and two decades doing <at least somewhat> repeatable development. There are no penalties; just benefits. The more repeatable, the better. >>That's because a fundamental thesis of process improvement >>is that it is self-correcting and will converge on the optimal process >>for the environment regardless of where one starts out. > > > Assuming there is one optimal process, which equates to the assumption > that there is one and only one "risk profile" for software projects. There will always be an optimal process for a particular development environment. What constitutes 'optimal', though, will vary considerably from one environment to another. ************* There is nothing wrong with me that could not be cured by a capful of Drano. H. S. Lahman hsl(a)pathfindermda.com Pathfinder Solutions -- Put MDA to Work http://www.pathfindermda.com blog: http://pathfinderpeople.blogs.com/hslahman (888)OOA-PATH
From: H. S. Lahman on 20 Mar 2006 11:00 Responding to Phlip... > I thought CMM[I] Level 5 specified that all teams were aware of each others' > practices, not necessarily obeying them. The CMM applies to an organizational unit and that is a quite flexible notion. In addition, processes have different levels of abstraction, just like objects. So at the level of, say, a 6-person team one could have a quite detailed suite of processes that everyone on that team practices (e.g., such as those XP dictates). Similarly, a 6-person team down the hall would have a detailed suite of processes that everyone on that team needs to follow. But those two teams might have quite different processes in detail because of differing development environments. However, at a higher level of abstraction the organization containing both those teams, say a Product Group, would also have a suite of processes defined at a higher level of abstraction. Both teams would follow those processes but their local processes would provide the detail. What you are referring to is, as AndyW points out, really L3 where infrastructure is provided to ensure different groups can coordinate. For example, if the two teams above are building different subsystems in a larger application, they need to negotiate interfaces between those subsystems and the requirements need to be clearly allocated to those subsystems. To do that one needs a process infrastructure at a higher level (Product Group) than the detailed processes the individual teams use to implement a particular subsystem. CMM L3 defines what the higher level organization needs to do to ensure that coordination takes place properly. >>>That's because a fundamental thesis of process improvement >>>is that it is self-correcting and will converge on the optimal process >>>for the environment regardless of where one starts out. >> >>Assuming there is one optimal process, which equates to the assumption >>that there is one and only one "risk profile" for software projects. > > > And that's why not all teams converge on the same spot. Right. What is an optimal development environment for one team may not be optimal for another. But within a given environment at a given level of abstraction the processes should be the same for everyone. ************* There is nothing wrong with me that could not be cured by a capful of Drano. H. S. Lahman hsl(a)pathfindermda.com Pathfinder Solutions -- Put MDA to Work http://www.pathfindermda.com blog: http://pathfinderpeople.blogs.com/hslahman (888)OOA-PATH
From: Nick Malik [Microsoft] on 21 Mar 2006 02:24 <jasongorman(a)blueyonder.co.uk> wrote in message news:1142676050.189389.39190(a)u72g2000cwu.googlegroups.com... > Hi > > I've been working with a small cluster of organisations on > metrics-driven approaches to software process improvement. The work is > based on the theory that, regardless of what methods, practices and > tools teams use, if you measure some aspect of their performance (e.g., > lines of code per iteration) then that's what you can expect to > improve. > Hello Jason, Even though I started the other thread about measurement on this forum (With Agile Methods, we are measuring the right things), and even though my initial rant was against using LOC as a measurement of productivity, I am concerned about your primary assumption. You appear to be saying "the act of measurement begets change on that measurement" I would say that the "act of measurement" PLUS the premise that "change equals improvement" PLUS visibility of the numeric changes (the deltas) will beget behavior changes. Perhaps I am picking nits. > * Take the metrics design the other group has given you, and ask > yourself this question: "exactly what is this metric telling us to do?" <snip> > For example, if the goal is > "increase developer productivity", and the metric design measures the > much-maligned "lines of code per unit of time per developer", then the > simplest thing you can do - the thing the metric is telling you to do - > is to generate a lot of redundant code. I like that... the "simplest thing you can do." Funny. When was the last time you ever saw a team of talented and creative developers choose the simplest path? This goes back to suggesting that the existence of the metric leads change. Even if the other aspects are present (premise or hypothesis, and visibility), there is no guarantee that change will occur. It is a LOT harder to change things than that. Also, you made a comparison to TDD. The advantage of TDD is that the compiler constrains the test. In other words, you have to code your test according to the rules of the system, then actually measure the system (the functionality). In your workshop, you are taking developers out of the real environment (their coding teams) and you are placing them in an unconstrained space. While it is interesting, and perhaps even fun, I would have a hard time calling the results valid. The point is that placing measurements on a team doesn't drive behavior, and having humans judge the effects of a measurement outside of a development environment is a meaningless exercise. It is true that introducing a measurement will have an effect on that which is measured, even if nothing else is changed. However, the effect will be fairly small and will stabilize fairly quickly to a baseline number. As others have pointed out, you have to understand the problem, collect data, analyze the data, and hypothesize a root cause FIRST. Then create a metric that should exhibit the change in behavior if your hypothesis is right. In other words, if X happens more, then Y should happen less. You can only validate this by measuring BOTH X AND Y. That is because the relationship between X and Y is your hypothesis. It is not proven to exist! Then, make a change. If the relationship exists that was hypothesized, you can often realize a benefit. Sometimes X does change, but Y does not. (Think: KLOC per function point went down but defects per function point did not). In that case, the relationship is proven only to be more complex than the hypothesis accounts for, if it exists at all. I was a bit sneaky. I mentioned all five stages of DMAIC without using their names. Define-Measure-Analyze-Improve-Control. These are the five steps of the six-sigma process. Add on top of that the Theory of Constraints, and you not only know what to change, but in what way. Theory of Constraints produces "Lean" manufacturing processes. These process improvements are based on the notion that waste is bad, and mechanisms to reduce waste are good. If you define, measure, and analyze, you will get some idea of what to improve, and what relationship between your measurements you expect to see. Then, changes in the relationships between the numbers is closely watched, not just right away, but also over time (the "control" part of DMAIC). So while I agree that "productivity measured by LOC/hour" is a bad thing on its face, I do not agree that you will get a valuable critique by asking developers in an off-site exercise. -- --- Nick Malik [Microsoft] MCSD, CFPS, Certified Scrummaster http://blogs.msdn.com/nickmalik Disclaimer: Opinions expressed in this forum are my own, and not representative of my employer. I do not answer questions on behalf of my employer. I'm just a programmer helping programmers. --
From: Ilja Preu? on 21 Mar 2006 08:50
AndyW wrote: > On Sun, 19 Mar 2006 07:54:07 +0100, "Ilja Preu?" <it(a)iljapreuss.de> > wrote: > >>> Also, in general with metrics you are looking for change you can >>> only find change if you have stability first (and this is what the >>> CMM is about). So consistency is good as it means stability with >>> your project. >> >> This sounds interesting - could you please elaborate a little bit, >> perhaps with an example? >> > > In CMM terms, level 2 is the repeatable level - pretty much what I > would suggest the majority of s/w teams are trying to achieve (based > on my probably incorrect observation). Repeatable in which sense? I could imagine that for an Agile team, adaptability and reliability are more important... > Level 3 is probably the holy grail of teams that focus only on > methodologies as being the savior of everything [this is where I have > a poke at the agile and eXtreme folks :) ]. I don't get the point about Agility/XP here. > So think of it at this level - eveything is consistent and nice and > stable. As far as I can tell, consistency and stability are only nice as long as you don't need creativity. If you do, the "chaordic edge" might be more effective? > Your not trying to invent solutions to as many problems. > Change happens, but its unpredictable, but we have solutions in place > (part of the method) that allows us to easily handle it. (its still > reactive). Metrics at this level is trying to be historical but > because of the level of change, its often reactive (after the fact). > > The example here would be - "what is the volume of LOC we produced for > the last three months - has it increased" ? The difference I see to the next example is that we are not looking for the root cause - or is there more to it? > Level four is the managed level. Its here that metrics really kick > in. You have a nice stable process, what you are looking for is when > it may become unstable (change). Also, we are looking to predict > change rather than react to it. > > The example here is - "what is the volume of LOC that was produced > each month for the last 6 months. Has it it varied this month - if > so, by how much and what is the root cause reason for it. Did a > developer have issues - what where they and how can we refine them out > so they dont happen again". > > Its at this level when we are looking at historical metrics to see our > past performance (our stable process) and current metrics to identify > deviations to that (where change may occur), then predictive metrics > to try and identify the risk of change occuring in the future. The above isn't an example of that last part - "predictive metrics", is it? How does that work, and what kind of stability do we need to be able to do it? > (I used to have a joke on one project that I could forecast defects 6 > months before they occured - the test team wondered why I hadnt fixed > the defects, I used to respond - because the code hasnt been written > yet]. I assume you mean the amount of defects, not the actual defect details...? > (it gets worse - wait til you see the expression on a designers > face when you tell them how many defects they will produce) That also doesn't sound very productive. ;) I guess you also could tell him what to do to produce less of them (or why the defect rate actually was acceptable to you)? > Level 5 > Optimising is the holy grail. Its where you are looking at your > organisation and the industry as a whole and changing to compete and > also improve your process. Using metrics to reflect and refine etc. > I've found this level to be very condusive for creating the > environment for the high performance team. > > This is the level I like to work at, so thats often why I am at odds > with some of the discussions. Its for this reason I dont like working > with Ego's that think they can run good projects by force of will and > self experience :) Well, ok. I'm not sure what part of your response was meant as a response to my actual answer, but it was at least interesting... ;) Cheers, Ilja |