Do You Get What You Measure?
in [OOP]
|
From: AndyW on 22 Mar 2006 17:16 On Thu, 23 Mar 2006 08:39:36 +0100, "Ilja Preu?" <it(a)iljapreuss.de> wrote: >Nick Malik [Microsoft] wrote: > >>>> In CMM terms, level 2 is the repeatable level - pretty much what I >>>> would suggest the majority of s/w teams are trying to achieve (based >>>> on my probably incorrect observation). >>> >>> Repeatable in which sense? >>> >>> I could imagine that for an Agile team, adaptability and reliability >>> are more important... >>> >> >> I think you misunderstand. Using Scrum or XP, there are very >> specific rules about what you do, and when, and who. Scrum has daily >> "stand up" meetings and monthly sprints. XP has pair programming and >> the planning game. These are rules that are not to be broken. > >That's a common misunderstanding - at least about XP: >http://www.xprogramming.com/xpmag/jatNotRules.htm > >> By >> following them, you provide repeatability to the development PROCESS >> so that creativity can flow properly to solving the business problem. > >How does a repeatable process allow creativity to "flow"? Serious question. Yes - I'm not sure why you think human creativity is restricted within the bounds of a given process.
From: H. S. Lahman on 23 Mar 2006 13:37 Responding to Preu?... >>The point here, though, is quite different. If the process is ad hoc, >>then one will tend to have high scatter in the metric because the >>developers are not doing things the same way from one application (or >>product or increment or whatever) to the next. IOW, the practices >>underlying one metric observation and the next will vary and the >>metric doesn't tell you which practices affected the observations. >>[Another example below] > > > I see. > > I'm not convinced, though, that you can stabilize a process enough to have > some metrics clearly pointing at the practice that needs improvement, that > it typically is a single practice that is responsible, nor that it would be > the best way to find out where to improve. I could even imagine that it > would do more harm than good, even if it worked. You'll have to trust me on this, but I've been there and done that. For a decade prior to my retiring our shop was militantly process-oriented. Because of all the process work already done things were pretty repeatable and the metrics were quite stable. However, you really don't need a lot of metrics for a given level of abstraction. For example, at the Division level we monitored four things across all products: reliability, schedule variance, actual feature set (features scheduled but not released), and process quality (number of failures to follow the process properly). The company had about a dozen divisions. Those like ours that had been process-oriented for years were quite stable; analysis was triggered for a few percent change in the metric. For other divisions that were very new to process monitoring the metrics had a lot of volatility because their processes were not very repeatable -- signaled in part by the fact that their process quality metric was numerically far larger than ours. Of course when one of those metrics blew its SPC envelope, one didn't really have a clue why and would have to investigate. As it happens, in our Division we tracked the same metrics on a product basis. Since different functional groups did different products, that usually provided the first level of diagnostic isolation. Then one would look for clues in other, lower level metrics. But in the end one usually had to collect some data to identify the root cause. <anecdotal evidence> Back in '84 we had a software release that was 24 months late on a 9 month schedule with a lot of overtime and the mean time to system crash was about 40 minutes. We had excuses like the application was too big for the machine (we broke the DEC linker for RSX because of too many overlays and too many entry points). But it still reflected on as professionals. Our process was completely ad hoc where the only common process was, literally, design-code-test. So we embarked on a grass roots process improvement initiative. Unfortunately we didn't know anything about Process engineering, so it was slow going. However, some problems were so big we couldn't help but improve by addressing them systematically. In '89 the entire Corporation bought into TQm and that gave us a formal process improvement infrastructure that sped things up a great deal. In '99 the Corporation bought into the SW-CMM for all software development, which gave us a formal process model as a framework for development. When I retired in '00 we had just gotten an award from the USN for 600 systems that had been deployed for nearly a decade with MTTF of 2.7 /years/ -- and those were almost all hardware failures. We were hitting 6-18 month schedules with -5%/+15% variance routinely. Perhaps most important of all, in my entire last year I think I only went in on 1-2 weekends for overtime. And we did all that while maintaining productivity that was integer factors greater than published data for some Major Players in the software arena doing the same sort of software. So repeatable processes and incremental process improvement do work. (Another statistic: in the first decade TQM was used the Corporate measured $2b added directly to the bottom line over that time.) </anecdotal evidence> >>>>Note that consistency is not equivalent to lack of creativity. The >>>>process of software development is mostly about things that are >>>>peripheral to the intellectual content of design. >>> >>> >>>I'm not convinced. I'd think that the process can have a significant >>>impact on creativity. >> >>There may be a terminology disconnect here. I use 'methodology' to >>describe the intellectual practices that go into software design >>while I use 'process' to describe the supporting infrastructure like >>IID, which, once defined, is essentially done by rote. IOW, it >>require creativity to practice a methodology but it doesn't to follow a >>process. (Though >>it takes substantial creativity to design a process.) > > > The creativity is not in following the process, but in adapting it to the > current situation. Blindly following a process is counter to Agility, the > way I understand it. I agree with the first sentence. As I said, designing a process requires creativity. The corollary is that changing a process requires creativity. But Process Engineering provides methodologies for guidance that are equivalent to software design methodologies. However, until one knows that a process is broken, there is no reason not to follow it. One just doesn't do it blindly; one has process metrics, reviews, etc. to monitor the process quality in the same way that one monitors the product quality. There are also ongoing process improvement activities that are orthogonal to the routine development (i.e., have independent goals) that will inevitably result in process changes. But that only affects developers when the process change is institutionalized. Until then, one just follows the process. [Process change is usually grass roots-based. So developers participate in the process improvement activities. (In our shop everyone, up to and including the CEO, was /required/ to spend 10% of their time on process improvement activities as a salary review item.) But that is outside the routine development processes.] >>>>For example XP is >>>>one of the most rigidly defined (repeatable) development processes >>>>to come down the 'pike. >>> >>>It comes with a very *specific* set of practices, yes. I wouldn't >>>say that it is exactly *rigid*, as you are supposed to adapt it to >>>your situation, nor would I actually say that it is *defined* by the >>>practices. And I wouldn't at all call it "repeatable", although I'm >>>not sure I'm understanding that the same way you do. It might have a >>>meaning in the process community I didn't quite grasp yet. >> >>Apostles like Martin have been on record several times over the years >>saying that if one does not do a practice or does the practices in a >>different order than XP prescribes, one is doing something other than >>XP. > > > I think we already discussed this in the past, but I can't remember the > outcome. Anyway, I don't hear Robert C. Martin say exactly that. And I hear > Ron Jeffries say that he thinks XP isn't something you either do or not do. > And Kent Beck writes in his second edition book (pg. 35): > > "Practices are situation depend. If the situation changes, you choose > different practices to meet those conditions. Your values do not have to > change in order to adapt a new situation. Some new principles may be called > for when you change domains. > [...] > Applaying a practice is a choice. I think the practices make programming > more effective. This is a collection of practices that work and work even > better together. [...] Experiment with XP using these practices as your > hypotheses. [...]" I don't have the 2nd edition, but that subject was in Chapt 27 of the 1st edition, albeit not the same words. There the context was the /mechanics/ of the key practices. IOW, you can change the length of an increment, but you can't not do IID. (His other examples were even more cosmetic.) The Party Line for XP has changed significantly over the years. Maybe the current view is that the 12 Key Practices are no longer sacred. But they certainly were 5-6 years ago when I was paying attention. ************* There is nothing wrong with me that could not be cured by a capful of Drano. H. S. Lahman hsl(a)pathfindermda.com Pathfinder Solutions -- Put MDA to Work http://www.pathfindermda.com blog: http://pathfinderpeople.blogs.com/hslahman (888)OOA-PATH
From: jasongorman on 11 Apr 2006 04:35 Thanks for all your interesting replies. Some useful thoughts in there. I should probably reply to a couple of general points that have been made. Firstly, about the focus on process vs. outcomes. Personally I'm very outcome-oriented. I don't care what process you follow as long as I get what I want. In that sense, I need to very carefully express what that is. Metrics - if they are well-designed - can be a testable way of describing a goal (i.e., "I want EXACTLY that"). Secondly, there's a mountain of evidence* that supports the notion that, in all walks of life - including software development - you really do get what you measure. I've seen it first hand. I bet we all have. Anybody who's worked in the public sector here in the UK - especially the National Health Service - will know exactly what I mean. Go in, admit it - you've gamed a metric, haven't you? The key idea behind the workshop is that your metric designs must evolve, and must be put to the test. It also strongly encourages balancing metrics across different - often competing - aspects of performance. We've all seen what happens when teams focus too much on deadlines, or on costs, at the expense of quality and long-term productivity, for example. I've never actually seen a team focus too much on quality, but I suppose it could happen ;-) So you might measure a team on how many lines of code they delivery every week, but if you place equal emphasis on the maintainability of that code and the level of defects in it, then they have to at least try to strike a balance between driving fast and driving safely. This is an oversimplified example, of course. The whole point is that, just as your first software design is likely to be wrong, your first metric designs will also have much room for improvement. That's why I strongly emphasis an iterative, evolutionary approach to designing and managing metrics. As well as an iterative and evolutionary approach to SPI that's feedback-driven. I should probably consider not calling it "Agile Software Process Improvement", for the same reason that people immediately think "he means 'UML'" when somebody says "model". How you apply metrics is equally important as how you design them, but the workshop covers that aspect, too :-) Hope that clears (or at leats, stirs) things up a little. I only have SPI data from 3 clients who have tried this approach, so I can't argue that it always works. But it seems to work. Performance really does skew in the directions specified by the metrics. And, if the metrics are evolved enough, performance really does improve. 3 out of 3 seems statistically significant enough for me to push forward and develop the techniques. That's the great thing about a metrics-driven approach - it measures itself. There's a more detailed description of the workshop at http://www.parlezuml.com/metrics/doyougetwhatyoumeasure.htm which has basic instructions on how to run your own metrics design workshops. Cheers Jason Gorman http://www.parlezuml.com * Just search for "You get what you measure" on Google.
From: H. S. Lahman on 11 Apr 2006 11:36 Responding to Jasongorman... > So you might measure a team on how many lines of code they delivery > every week, but if you place equal emphasis on the maintainability of > that code and the level of defects in it, then they have to at least > try to strike a balance between driving fast and driving safely. Alas, I have the same concern here that I had with the original post. This assumes that the developers have some direct stake in the metric results so that they will immediately adjust their behavior to the metric results (e.g., their salary review depends upon the metric). That's not the way a metrics program should work. Nor is it the way a properly process-oriented shop uses metrics. One should not "measure a team"; one measures processes. Metrics simply monitor the process the team uses and provide data for subsequent analysis. Changing the way things are done is a matter of systematic process improvement that has its own suite of activities (and validations to ensure the metrics aren't abused). Those process improvement activities are quite orthogonal to the metrics themselves and to what the developers are currently doing. That is, the developers continue with business as usual regardless of the metrics' results until some separate process improvement activity completes due diligence and changes things. While your personal experience may be with shops that do not have a good process orientation, that doesn't mean such shops should set the standard. IOW, I think you need to research the way process improvement is supposed to work. ************* There is nothing wrong with me that could not be cured by a capful of Drano. H. S. Lahman hsl(a)pathfindermda.com Pathfinder Solutions -- Put MDA to Work http://www.pathfindermda.com blog: http://pathfinderpeople.blogs.com/hslahman (888)OOA-PATH
From: jasongorman on 15 Apr 2006 04:10
H. S. Lahman wrote: > Responding to Jasongorman... > > > So you might measure a team on how many lines of code they delivery > > every week, but if you place equal emphasis on the maintainability of > > that code and the level of defects in it, then they have to at least > > try to strike a balance between driving fast and driving safely. > > Alas, I have the same concern here that I had with the original post. > This assumes that the developers have some direct stake in the metric > results so that they will immediately adjust their behavior to the > metric results (e.g., their salary review depends upon the metric). Your objection is based on an assumption you've made about how I think metrics programs should be applied. I have always maintained that linking pay directly to performance is unwise. But then again, what *should* we pay developers for? Delivery, or is that just crazy talk? (Answers on a postcard, please...) > That's not the way a metrics program should work. Nor is it the way a > properly process-oriented shop uses metrics. One should not "measure a > team"; one measures processes. Metrics simply monitor the process the > team uses and provide data for subsequent analysis. Processes don't have performance characteristics. Teams (or machines, or trained animals) performing processes do. In any process, the people executing it are always the biggest and most significant factor in performance. I strongly suspect that if you took 6 rewally strong developers and had them do, say RUP, and then took the same 6 really strong developers and had them do, say, eXtreme Programming, their performance might be more similar than a completely different team following exactly the same process as they did. Your tone suggests that teams execute processes in much the same way that Intel x86's execute programs. I must admit, I'm not a big fan of the reductionist view. The problem with process-orientation is that it tries to ignore the whole messy business of people and what they do when they get together in groups to get stuff done. Whether we like it or not, complex systems like software development teams - and herds of cats - are self-organising, and self-organising teams need clear and testable goals to self-organise towards. I find cats tend to self-organise towards cat food when they're hungry, for example. I've tried giving them a proper process to follow, but they must be one of the bad shops to which you refer, because they don't seem at all process-oriented. Of course, a true professional does exactly as he's told, doesn't he? Not like those naughty self-organising cats! What you describe is very in much in the Linear Management camp where one tries to impose a specific order (a specific process or architecture - or both) either from within - or, as it seems from your description - without the team. I am - you may have noticed - very firmly in the Nonlinear Management camp, where I advocate what amounts to process anarchy within very clear and testable boundaries. > Changing the way things are done is a matter of systematic process > improvement that has its own suite of activities (and validations to > ensure the metrics aren't abused). Those process improvement activities > are quite orthogonal to the metrics themselves and to what the > developers are currently doing. That is, the developers continue with > business as usual regardless of the metrics' results until some separate > process improvement activity completes due diligence and changes things. I've coined the term "Process Voodoo" for what you describe - http://parlezuml.com/blog/?postid=144 :-) > While your personal experience may be with shops that do not have a good > process orientation, that doesn't mean such shops should set the > standard. IOW, I think you need to research the way process improvement > is supposed to work. Thanks for the advice, but I neither seek nor recommend process-orientation. Far from it, in fact. Perhaps I'm the lone Druid at the Cardinals' Ball, but 10 years from now you'll all look back on this and say "y'know what, you really *can't* herd cats... what were we thinking?!" Jason Gorman http://www.parlezuml.com |