Do You Get What You Measure?
in [OOP]
|
From: H. S. Lahman on 19 Mar 2006 10:08 Responding to Andy... > Although I've found often that many people instigate > the process change then implement metrics to try and see if its > working - which is rather pointless as you have nothing to compare it > with. That depends. Often the data that was analyzed to determine the improvement provides the "history" for the metric's future values. Also, most of the process improvement approaches require that any change be validated prior to actually institutionalizing it. That will usually require observations over time similar to the metric. IOW, the metric is often just a formalization of the data collection used during the process improvement. <aside> FWIW, we had a practice where we always collected metric data at a level of detail finer than actually needed for the metric itself (i.e., the metric aggregated data). The detailed data was ignored until we had a particular reason for analyzing it. The reason was to ensure that we would have detailed data in hand to analyze when a metric blew its SPC envelop. Quite often (though not always) that sped up the process improvement reaction time because we didn't need to take time to collect more data. IOW, the line between metrics and data collection can be blurry. [Of course that only works if the data collection is pretty non-invasive, so it works best if there are already an automation infrastructure and good data collection tools in place.] </aside> ************* There is nothing wrong with me that could not be cured by a capful of Drano. H. S. Lahman hsl(a)pathfindermda.com Pathfinder Solutions -- Put MDA to Work http://www.pathfindermda.com blog: http://pathfinderpeople.blogs.com/hslahman (888)OOA-PATH
From: H. S. Lahman on 19 Mar 2006 10:36 Responding to Preu?... >>Also, in general with metrics you are looking for change you can only >>find change if you have stability first (and this is what the CMM is >>about). So consistency is good as it means stability with your >>project. > > > This sounds interesting - could you please elaborate a little bit, perhaps > with an example? As a different spin on AndyW's response, the stability refers to process stability -- constructing software using the same basic process in a consistent manner. IOW, all the developers are on the same page. If the process is ad hoc where the developers do their own thing, then basically the data forms a ball of points in a Scatter Diagram because one is measuring apples & oranges. The CMM happens to use 'repeatability' as a buzzword but all the process frameworks emphasize consistency. [Note that consistency is not equivalent to lack of creativity. The process of software development is mostly about things that are peripheral to the intellectual content of design. For example XP is one of the most rigidly defined (repeatable) development processes to come down the 'pike. The process defines what you should do and when to do it down to almost the code fragment level. Yet it doesn't say anything at all about what classes, relationships, patterns, etc. one will actually abstract in the design. Nor does it define what technologies, strategies, standards, etc. one will implement with.] For example, let's say you decide you are going to measure code size with LOC for purposes of estimation. Assume some developers are demons for comments while others depend almost exclusively on naming conventions and idioms for documentation. Your metric will be distorted depending on which group wrote the most code in the metrics time interval. To a lesser extent it could be distorted in a similar fashion if you used NCLOC but half the developers used a C beautifier while the other half didn't. When you get into things like version control policies and refactoring practices, the effects can be much more subtle. And if different teams are using model-based vs. OOP-based design techniques some metrics may have no meaning at all. Bottom line: it doesn't matter very much which way one does development but it is extremely important that there is only One Way that everyone practices. That's because a fundamental thesis of process improvement is that it is self-correcting and will converge on the optimal process for the environment regardless of where one starts out. But everyone needs to be doing the same thing as they travel that path. ************* There is nothing wrong with me that could not be cured by a capful of Drano. H. S. Lahman hsl(a)pathfindermda.com Pathfinder Solutions -- Put MDA to Work http://www.pathfindermda.com blog: http://pathfinderpeople.blogs.com/hslahman (888)OOA-PATH
From: Laurent Bossavit on 19 Mar 2006 13:49 > Bottom line: it doesn't matter very much which way one does development > but it is extremely important that there is only One Way that everyone > practices. Assuming the process improvement benefits of standardizing on One Way (monoculture approach) largely outweigh the penalties. > That's because a fundamental thesis of process improvement > is that it is self-correcting and will converge on the optimal process > for the environment regardless of where one starts out. Assuming there is one optimal process, which equates to the assumption that there is one and only one "risk profile" for software projects. What grounds are there for either of the above two assumptions ? Laurent
From: Phlip on 19 Mar 2006 14:26 Laurent Bossavit wrote: > Assuming the process improvement benefits of standardizing on One Way > (monoculture approach) largely outweigh the penalties. I thought CMM[I] Level 5 specified that all teams were aware of each others' practices, not necessarily obeying them. >> That's because a fundamental thesis of process improvement >> is that it is self-correcting and will converge on the optimal process >> for the environment regardless of where one starts out. > > Assuming there is one optimal process, which equates to the assumption > that there is one and only one "risk profile" for software projects. And that's why not all teams converge on the same spot. (However, convergence requires, uh, iterations and high-quality feedback...;) -- Phlip http://www.greencheese.org/ZeekLand <-- NOT a blog!!!
From: AndyW on 19 Mar 2006 06:07
On Sun, 19 Mar 2006 19:26:59 GMT, "Phlip" <phlipcpp(a)yahoo.com> wrote: >Laurent Bossavit wrote: > >> Assuming the process improvement benefits of standardizing on One Way >> (monoculture approach) largely outweigh the penalties. > >I thought CMM[I] Level 5 specified that all teams were aware of each others' >practices, not necessarily obeying them. > No, thats really just level 3. At level 4 it starts becoming organisational, level 5 is purely the same process aross the organisation (at least, that how I have always viewed it). However, I think sometimes its woolly in its granularity. For example, one might specificy a single methodology across the org - which could be said to meet the criteria (I think this is what you meant) vs using a s/w tool to baseline a project (with all core documentation) and stating that this baseline is used when each new project is set up (each project then refines the process to reflect its subtle differences and customer specific legal nuances if any). I prefer the refining the project baseline approach because each project (especially for companies performing multiple simultaneous projects) typically starts working as a factory assembly line and as each project completes the post mortem review results are fed back into the original baseline to improve it. It does require a high performance team, which can be hard to create if the company has undergone some level of churn or is still small and reactive (I've found high performance teams occur when there is effective knowledge management process in place - rather than any specific methodology). In my opinion tho, refinement comes analyzing all techneques available in the industry at that time (and perhaps inventing some new ones). This is why I dont really hold with a single 'methodology' or 'technique' and why I dont espouse any specific technique such as agile/XP/Waterfall and I can sometimes be quite happy with waterfall in its iterative form. They are good to create a basline at the defined level and to provide ideas, but beyond that they start to always end up playing catchup or becoming restrictive. I used to know CMM level 5 as the Institutionalised level. That is, those techniques created at companies at this level usually become industry best practices (and get adopted by methods such as XP ). |