Do You Get What You Measure?
in [OOP]
|
From: Ilja Preu? on 21 Mar 2006 09:18 > As a different spin on AndyW's response, the stability refers to > process stability -- constructing software using the same basic > process in a consistent manner. IOW, all the developers are on the > same page. If the process is ad hoc where the developers do their > own thing, then basically the data forms a ball of points in a > Scatter Diagram because one is measuring apples & oranges. That depends on what we are measuring, doesn't it? For example, if we were measuring customer satisfaction, inconsistent coding styles shouldn't be a problem. That seems to suggest that there are at least two possible reactions to a metric being bastardized (?) by unstable processes: stabilizing the problematic part of the process, or finding a different metric. > Note that consistency is not equivalent to lack of creativity. The > process of software development is mostly about things that are > peripheral to the intellectual content of design. I'm not convinced. I'd think that the process can have a significant impact on creativity. > For example XP is > one of the most rigidly defined (repeatable) development processes to > come down the 'pike. It comes with a very *specific* set of practices, yes. I wouldn't say that it is exactly *rigid*, as you are supposed to adapt it to your situation, nor would I actually say that it is *defined* by the practices. And I wouldn't at all call it "repeatable", although I'm not sure I'm understanding that the same way you do. It might have a meaning in the process community I didn't quite grasp yet. > The process defines what you should do and when > to do it down to almost the code fragment level. "the Code is more what you'd call 'guidelines' than actual rules." - Barbossa > When you get into things like version control policies and refactoring > practices, the effects can be much more subtle. And if different > teams are using model-based vs. OOP-based design techniques some > metrics may have no meaning at all. Imaginable. I'm not sure, though, that this is a good argument for unification of the process. Perhaps it's a better argument for more appropriate metrics... > Bottom line: it doesn't matter very much which way one does > development but it is extremely important that there is only One Way > that everyone practices. That's because a fundamental thesis of > process improvement is that it is self-correcting and will converge > on the optimal process for the environment regardless of where one > starts out. But everyone needs to be doing the same thing as they > travel that path. I can see that a practice is much easier to assess when a team applies it consistently for some time. On the other hand, needs *are* different, not only between teams, but even between team members, and I don't think it's wise to ignore that for the sake of measurability. Measuring, after all, is not the only way to assess a process. Cheers, Ilja
From: Nick Malik [Microsoft] on 21 Mar 2006 10:53 Hi Ilja, "Ilja Preu?" <it(a)iljapreuss.de> wrote in message news:3audnT5KPpwDmb3Z4p2dnA(a)totallyobjects.com... > AndyW wrote: >> On Sun, 19 Mar 2006 07:54:07 +0100, "Ilja Preu?" <it(a)iljapreuss.de> >> wrote: >> >> >> In CMM terms, level 2 is the repeatable level - pretty much what I >> would suggest the majority of s/w teams are trying to achieve (based >> on my probably incorrect observation). > > Repeatable in which sense? > > I could imagine that for an Agile team, adaptability and reliability are > more important... > I think you misunderstand. Using Scrum or XP, there are very specific rules about what you do, and when, and who. Scrum has daily "stand up" meetings and monthly sprints. XP has pair programming and the planning game. These are rules that are not to be broken. By following them, you provide repeatability to the development PROCESS so that creativity can flow properly to solving the business problem. CMM is about having a stable and repeatable development process. >> Level 3 is probably the holy grail of teams that focus only on >> methodologies as being the savior of everything [this is where I have >> a poke at the agile and eXtreme folks :) ]. > > I don't get the point about Agility/XP here. > Agile methods rarely define what Levels 4 or 5 look like, while CMM clearly expects you to refine your system beyond stability, repeatability, and forecasting. >> Level four is the managed level. Its here that metrics really kick >> in. You have a nice stable process, what you are looking for is when >> it may become unstable (change). Also, we are looking to predict >> change rather than react to it. >> >> The example here is - "what is the volume of LOC that was produced >> each month for the last 6 months. Has it it varied this month - if >> so, by how much and what is the root cause reason for it. Did a >> developer have issues - what where they and how can we refine them out >> so they dont happen again". >> >> Its at this level when we are looking at historical metrics to see our >> past performance (our stable process) and current metrics to identify >> deviations to that (where change may occur), then predictive metrics >> to try and identify the risk of change occuring in the future. > > The above isn't an example of that last part - "predictive metrics", is > it? How does that work, and what kind of stability do we need to be able > to do it? > Level 3 is about a development process that is so predictable that you can plan and predict the outcome. Level 4 is about finding the things that are predictable, but not effective, and making them both predictable and effective. In other words, Level 2 means making it work, Level 3 is knowing it works, and Level 4 is being able to make it work better. Level 5 is changing the system to respond to outside factors. In other words, we got it to run and run well, now we need to change it because something else has changed (competition, internal company strategy, marketplace, etc). -- --- Nick Malik [Microsoft] MCSD, CFPS, Certified Scrummaster http://blogs.msdn.com/nickmalik Disclaimer: Opinions expressed in this forum are my own, and not representative of my employer. I do not answer questions on behalf of my employer. I'm just a programmer helping programmers. --
From: H. S. Lahman on 21 Mar 2006 13:21 Responding to Preu?... >>As a different spin on AndyW's response, the stability refers to >>process stability -- constructing software using the same basic >>process in a consistent manner. IOW, all the developers are on the >>same page. If the process is ad hoc where the developers do their >>own thing, then basically the data forms a ball of points in a >>Scatter Diagram because one is measuring apples & oranges. > > > That depends on what we are measuring, doesn't it? > > For example, if we were measuring customer satisfaction, inconsistent coding > styles shouldn't be a problem. Measuring customer satisfaction is actually a good example of the other point I was making that the goal of a good metric is not to change the way things are done (process improvement), but simply to track how one is doing (process monitoring). Let's say you measure ongoing customer satisfaction and it takes a dip. That's not good, but the metric doesn't tell you why. You have to collect and analyze data to determine what the root cause of the dip is and then correct it. While differing coding styles are, indeed, unlikely to have caused the dip, something did. The point here, though, is quite different. If the process is ad hoc, then one will tend to have high scatter in the metric because the developers are not doing things the same way from one application (or product or increment or whatever) to the next. IOW, the practices underlying one metric observation and the next will vary and the metric doesn't tell you which practices affected the observations. [Another example below] >>Note that consistency is not equivalent to lack of creativity. The >>process of software development is mostly about things that are >>peripheral to the intellectual content of design. > > > I'm not convinced. I'd think that the process can have a significant impact > on creativity. There may be a terminology disconnect here. I use 'methodology' to describe the intellectual practices that go into software design while I use 'process' to describe the supporting infrastructure like IID, which, once defined, is essentially done by rote. IOW, it require creativity to practice a methodology but it doesn't to follow a process. (Though it takes substantial creativity to design a process.) >> For example XP is >>one of the most rigidly defined (repeatable) development processes to >>come down the 'pike. > > > It comes with a very *specific* set of practices, yes. I wouldn't say that > it is exactly *rigid*, as you are supposed to adapt it to your situation, > nor would I actually say that it is *defined* by the practices. And I > wouldn't at all call it "repeatable", although I'm not sure I'm > understanding that the same way you do. It might have a meaning in the > process community I didn't quite grasp yet. Apostles like Martin have been on record several times over the years saying that if one does not do a practice or does the practices in a different order than XP prescribes, one is doing something other than XP. A team can add stuff to XP but they cannot change the out-of-the-box features. Note XP is primarily a process definition, rather than a methodology. Martin's Principles and Fowler's Refactoring constitute a de facto methodology for XP design. However, those practices are not defined by XP explicitly. Thus one is free to modify the way one contributes intellectual content, but not the surrounding process context. >>When you get into things like version control policies and refactoring >>practices, the effects can be much more subtle. And if different >>teams are using model-based vs. OOP-based design techniques some >>metrics may have no meaning at all. > > > Imaginable. I'm not sure, though, that this is a good argument for > unification of the process. Perhaps it's a better argument for more > appropriate metrics... The context here is ad hoc development where different people whose work is aggregated by the metric are doing things in different ways. Let's say you are measuring customer satisfaction in a shrink-wrap environment. Let's also assume the team doing model-based development is doing translation while the OOP-based group is doing elaboration. The translation software will usually have higher reliability because all of OOD/P is automated (i.e., the frequency of bugs inserted by the code generator is much lower than the frequency of bugs inserted in manual code writing). For shrink-wrap products the customer satisfaction metric only evaluates released products, regardless of how many IID steps there were to produce them. If the products are large, then there will tend to be step functions in the metric depending on which team's product was released most recently. (Defect discovery after release tends to be Poisson with m < 0.5.) If there is a large discrepancy between the reliability of each team's products, then the metric is going to hop up and down like a yo-yo and it really won't be very useful for describing the overall business' customer satisfaction. However, if one tracks customer satisfaction for each groups' products (i.e., the metric measures consistent processes), a useful pattern will emerge very rapidly. >>Bottom line: it doesn't matter very much which way one does >>development but it is extremely important that there is only One Way >>that everyone practices. That's because a fundamental thesis of >>process improvement is that it is self-correcting and will converge >>on the optimal process for the environment regardless of where one >>starts out. But everyone needs to be doing the same thing as they >>travel that path. > > > I can see that a practice is much easier to assess when a team applies it > consistently for some time. > > On the other hand, needs *are* different, not only between teams, but even > between team members, and I don't think it's wise to ignore that for the > sake of measurability. Measuring, after all, is not the only way to assess a > process. Metrics always have a context. NCLOC is a quite reasonable measure of product size. But to use it properly, especially for comparisons, one has to know the context of things like language, coding styles, type of software, etc. FWIW, in our shop we had a template that was used to document metrics. Before a QIT could institutionalize a metric the metric had to be documented and provided to the stakeholders. There were process sections on mechanics like data collection but the interesting ones vis a vis this point were one that described how to interpret the metric and another that described what its limitations were. Those sections were there to explicitly address the context issue. [As a personal favorite, a lot of the function points vs. LOC counting debate is really apples and oranges. Function points measure the size of the requirements while LOC measures the size of the software product. One can debate which one is more useful for something like schedule estimation but it is pointless to debate which measurement is inherently "better".] ************* There is nothing wrong with me that could not be cured by a capful of Drano. H. S. Lahman hsl(a)pathfindermda.com Pathfinder Solutions -- Put MDA to Work http://www.pathfindermda.com blog: http://pathfinderpeople.blogs.com/hslahman (888)OOA-PATH
From: Ilja Preu? on 23 Mar 2006 02:39 Nick Malik [Microsoft] wrote: >>> In CMM terms, level 2 is the repeatable level - pretty much what I >>> would suggest the majority of s/w teams are trying to achieve (based >>> on my probably incorrect observation). >> >> Repeatable in which sense? >> >> I could imagine that for an Agile team, adaptability and reliability >> are more important... >> > > I think you misunderstand. Using Scrum or XP, there are very > specific rules about what you do, and when, and who. Scrum has daily > "stand up" meetings and monthly sprints. XP has pair programming and > the planning game. These are rules that are not to be broken. That's a common misunderstanding - at least about XP: http://www.xprogramming.com/xpmag/jatNotRules.htm > By > following them, you provide repeatability to the development PROCESS > so that creativity can flow properly to solving the business problem. How does a repeatable process allow creativity to "flow"? Serious question. >>> Level 3 is probably the holy grail of teams that focus only on >>> methodologies as being the savior of everything [this is where I >>> have a poke at the agile and eXtreme folks :) ]. >> >> I don't get the point about Agility/XP here. >> > > Agile methods rarely define what Levels 4 or 5 look like, while CMM > clearly expects you to refine your system beyond stability, > repeatability, and forecasting. I'm not sure I would say that XP "expects you" to do anything - I like to think about it more as suggestions. On the other hand I'd say that the XP/Agile community clearly values and discusses those things that you group under CMM levels 4 and 5. > Level 3 is about a development process that is so predictable that > you can plan and predict the outcome. Level 4 is about finding the > things that are predictable, but not effective, and making them both > predictable and effective. In other words, Level 2 means making it > work, Level 3 is knowing it works, and Level 4 is being able to make > it work better. Mhh, what I've learned about tuning a process, I've learned from Agile community - iteration retrospectives and the like. Just found this on page 28 in "Extreme Programming Explained, 2nd ed.: "[...] the point of XP, which is excellence in software development through improvement. The cycle is to do the best you can today, striving for the awareness and understanding necessary to do better tomorrow." > Level 5 is changing the system to respond to outside factors. In > other words, we got it to run and run well, now we need to change it > because something else has changed (competition, internal company > strategy, marketplace, etc). I'm not sure why this has to be separate from level 4 - probably because those factors are traditionally outside the awareness of the development team. It's amazing how much difference close communication with "outside" stakeholders make. Of course you are right, though, that XP doesn't tell those stakeholders how to know about the outside changes they should monitor, should that be your point. Cheers, Ilja
From: Ilja Preu? on 23 Mar 2006 03:07
> The point here, though, is quite different. If the process is ad hoc, > then one will tend to have high scatter in the metric because the > developers are not doing things the same way from one application (or > product or increment or whatever) to the next. IOW, the practices > underlying one metric observation and the next will vary and the > metric doesn't tell you which practices affected the observations. > [Another example below] I see. I'm not convinced, though, that you can stabilize a process enough to have some metrics clearly pointing at the practice that needs improvement, that it typically is a single practice that is responsible, nor that it would be the best way to find out where to improve. I could even imagine that it would do more harm than good, even if it worked. But perhaps I'm taking all this too literally. The way I'm reading it, it sounds too mechanical and too susceptible to micro optimization to me, though. >>> Note that consistency is not equivalent to lack of creativity. The >>> process of software development is mostly about things that are >>> peripheral to the intellectual content of design. >> >> >> I'm not convinced. I'd think that the process can have a significant >> impact on creativity. > > There may be a terminology disconnect here. I use 'methodology' to > describe the intellectual practices that go into software design > while I use 'process' to describe the supporting infrastructure like > IID, which, once defined, is essentially done by rote. IOW, it > require creativity to practice a methodology but it doesn't to follow a > process. (Though > it takes substantial creativity to design a process.) The creativity is not in following the process, but in adapting it to the current situation. Blindly following a process is counter to Agility, the way I understand it. >>> For example XP is >>> one of the most rigidly defined (repeatable) development processes >>> to come down the 'pike. >> >> It comes with a very *specific* set of practices, yes. I wouldn't >> say that it is exactly *rigid*, as you are supposed to adapt it to >> your situation, nor would I actually say that it is *defined* by the >> practices. And I wouldn't at all call it "repeatable", although I'm >> not sure I'm understanding that the same way you do. It might have a >> meaning in the process community I didn't quite grasp yet. > > Apostles like Martin have been on record several times over the years > saying that if one does not do a practice or does the practices in a > different order than XP prescribes, one is doing something other than > XP. I think we already discussed this in the past, but I can't remember the outcome. Anyway, I don't hear Robert C. Martin say exactly that. And I hear Ron Jeffries say that he thinks XP isn't something you either do or not do. And Kent Beck writes in his second edition book (pg. 35): "Practices are situation depend. If the situation changes, you choose different practices to meet those conditions. Your values do not have to change in order to adapt a new situation. Some new principles may be called for when you change domains. [...] Applaying a practice is a choice. I think the practices make programming more effective. This is a collection of practices that work and work even better together. [...] Experiment with XP using these practices as your hypotheses. [...]" > A team can add stuff to XP but they cannot change the > out-of-the-box features. That's a comon misunderstanding, but simply not true. Cheers, Ilja |