Organization Z’s Second Assessment: Summary of Achievements
A second assessment, conducted two years after the first, rated Organization Z’s maturity level at Level 3. The responses to the concerns raised by the first assessment already discussed in Section Organization Z’s Post-Assessment Efforts were judged successful. As a result, the organization had made project managers accountable for the software and process improvement components of their projects, the software process handbook they had distributed had successfully disseminated information about company-wide processes, and the organization’s strengthened inspection program had substantively increased the level of defects found before testing.
The results of these measures served not only to allow the organization to "pass" a Level 3 assessment but also to achieve difficult medium- and long-term business goals. Both the project managers and the organization president were given new visibility and control procedures, increasing their ability to understand and manage software development. Project personnel were able to make a more informed choice about software development processes. Projects were able to predict cost and schedule better; quality improved across the board. Finally, the organization needed to spend less time finding defects after testing.
As a result of these improvements, the organization started to see real improvement in efficiency, quality, and profits. By the time of the second assessment, at least 90% of the projects were delivering on time and on budget. Projects were finding 63% of their defects before test, producing marked decreases in development costs resulting in a 23% increase in productivity.
What part had assessments played in all this? The analytical power of the original assessment had demonstrated problems with the organization’s information and authority structures. Beforehand, the software managers were sure that their engineers were using the company’s software processes, yet during the assessment not one of the practitioners said they had properly understood them, much less used them. The assessment also pointed out that project managers had shirked a major part of their responsibility even though during interviews they said they thought they had their projects fully under control.
Meanwhile, it was the shock of the assessment that made not just the project managers but also the senior managers re-evaluate the organization’s management structure. Without that shock, it would never have been possible to reorganize the way one level of management reported to the next in such a dramatic and global way.
Organization Z’s Second Assessment: Remaining Issues
The second Level 3 assessment showed that Organization Z was now capable of implementing company-wide processes and measuring their effect. Now able to more reliably produce high-quality products, the organization stood on the verge of exploiting the potentials it had unleashed. Their organization-wide effort to measure how processes worked, for example, began to give them the data to institute quantitative goals for improving process and product levels.
At this stage, however, senior management did not yet have enough experience with the new data to set the kind of realistic quantitative goals that could drive new improvements. Setting such goals constitutes one of the capability characteristics of a Level 4 organization.
A second area of concern reported by the second assessment involved creating a program to prevent defects from occurring. As a result of the improvements that had led to Level 3, Organization Z could now find more defects early, giving it for the first time sufficient data and expertise to set up a defect prevention plan using causal analysis techniques. This is one of the characteristics of a Level 5 organization.
The second Level 3 assessment recommendations suggested that the organization combine the implementation of these Level 4 and Level 5 initiatives into one improvement plan, and senior management agreed.
Organization Z Second Assessment Findings
The general findings of the second assessment are presented in Table 12-6. See also Figure 12-2.
Table 12-6. General Findings of Organization Z’s Second CBA IPI Assessment, Conducted Two Years After the First One General Strengths
President and management staff continue to be actively involved.
Program managers and their teams have become better at managing and meeting software commitments.
A stronger process improvement culture exists now than at the last assessment.
A more widespread understanding exists that process improvement is not a waste of time.
Other disciplines have begun to see the improvements in software development as a model for themselves and to embrace process improvement.
Staff members have begun to recognize the value of the feedback from frequent health checks and assessments.
Early warning indicators are working.
Metrics have made it easier to recognize problems early.
Executive staff is now more focused on issues than individual blame.
Staff morale is higher, and personnel are more willing to "go the extra mile."
Training has improved and has been widely implemented.
Fagan Inspections have been successful enough in software engineering that other engineering disciplines are looking into their use.
Continuing Weaknesses
A continuous process improvement culture needs to be ingrained in the organization.
Personnel and management should be more willing to admit their own faults.
There should exist a more widespread awareness of the need for continuous process improvement.
Staff need to be more aware and more skilled in using measurement data.
Recommendations
Establish quantitative organization-wide process improvement goals with visible senior management sponsorship.
Keep raising the bar.
Establish a more structured integration of the individual process improvement initiatives across the organization directorates to ensure a single coordinated thrust.
Assign the best people.
Use software successes for leverage.
Continually monitor and adjust the program.
Continually monitor and adjust the program.
Institute causal analysis for defects.
Start implementing Level 5 process improvement technology
A Sample of Organization Z’s Proposals After a Second Assessment and What Difference They Made One and Two Years Later
Concern #1 (Creating Organization-Wide Process and Product Goals):
The assessment recommendations advised that the organization needed to establish quantitative (organization-wide) process and product improvement goals. These goals must be viewed as the president’s goals.
It also observed that, although the software organization had developed project defect detection goals and these metrics had been inserted into software development plans on pilot projects, management was not measuring progress in relation to these goals as a way of controlling the projects.
Initial Action Taken:
The president assigned the engineering division VP and his directors the job of developing quantitative goals for process and product improvement. They were, for example, to develop a plan to prescribe realistic numbers for increasing the organization’s productivity. Such a plan might specify that the organization was each year to increase quality by x%, decrease cost by y%, and increase the business flow by z%, all based on the measures made available by the tools instituted in the previous improvement cycle.
The president also required project managers to start using already devised software quality goals.
Comment:
Managers in Organization Z tried to devise a set of general goals, but they were unsuccessful because they could not come up with a set of goals sufficiently clear to make sense to the entire organization.
Companies don’t always manage to implement all of their improvement plans even when they make honest attempts to do so, and this is a good example of why improvements are sometimes delayed. The managers assigned to develop Organization Z’s goals were middle managers and had only a partial view of the whole organization. They were too focused on details, and the goals they created were too complex. Every time they presented these goals to the president, he would send them back to the drawing board.
Nor was the organization’s plan to utilize software quality goals successful because project managers had not been given training in the use of such metrics for planning and tracking.
One Year Later:
A health check carried out a year later highlighted the continuing lack of organizational goals and confirmed that existing software quality goals were not being used.
The health check recommended that the new goals, for example, should require x% productivity improvement for each program and should involve a statement of what percent of that figure was to be allocated to systems, software, hardware, production functions, and so on.
Concerning the software organization, the health check added that it needed to re-examine its progress in achieving stated goals and to establish whether the goals’ assumptions and expectations were valid, and if not, to adjust them accordingly. The health check also recommended that existing software goals that required an x% improvement in productivity might be required to specify which part of this would come from technology, which part from reuse, which part from inspections, and so on.
Finally, the health check recommended that project managers should present to the next assessment team an account of how they had used Levels 4/5 quantitative management processes and a plan for achieving organization-wide goals.
The health check helped focus the entire organization on the notion of process improvement. It encouraged the president, for example, to think through a set of goals on his own. (The ones he came up with were clear and encompassing, and the organization rallied around them.) The health check also encouraged the president to motivate his project managers to start deploying quantitative measures in their monthly reviews.
Actions Taken Immediately After the Health Check:
After the health check, the president himself sat down with a small group from the engineering division and on a whiteboard sketched out four organizational goals that made sense to him. His guiding rules were to keep them simple, clear, doable, and measurable.
The president identified four organization-wide goals concerning quality, productivity, customer satisfaction, and business growth and required that they flow down to each of the vice presidents (engineering, programs, contracts, procurement, human resources, etc.). The vice presidents would then flow the goals down to the director level, and the directors down to the rest of the organization. These goals formed the basis of Organization Z’s improvement initiative.
Two Years Later:
A third CBA IPI assessment, conducted two years after the second, reported the following:
"Clarity of strategic direction, which is the organization’s goal, is greatly improved. Whole company more focused on goals."
Concern #2: Setting Up a Defect-Prevention Program
A second area of concern from the second assessment had to do with creating a software program for the causal analysis of defects (defect prevention).
Initial Actions Taken:
- The president assigned the software director to create an infrastructure for quantitative management and defect prevention and to require that appropriate personnel focus on processes, procedures, metrics, and training for causal analysis.
- The software director assigned a causal analysis manager to use data from a first round of Fagan Inspections to design a plan for quantitatively managing defect injection rates on the projects. Control limits were established, and a predictive model was created.
One Year Later:
A health check conducted one year later indicated the following improvements:
- Defect prevention plans are in place on the pilot projects and are being used to manage the defect prevention activities.
- Corrective action plans are being generated and are being tracked to closure.
- Measurements on the effort involved in the corrective action process are captured using a standard method.
- Software project leads not involved in the pilots already are recognizing the value of the defect prevention process.
The health check also indicated some continuing weaknesses:
- Changes identified in the corrective action plan have not yet been flowed into the software development plan or, even further, into the software process handbook.
- Early involvement of the program managers does not appear to be standardized.
Two Years Later:
A third CBA IPI assessment, conducted two years after the second, showed that:
"Most programs hold causal analysis meetings monthly, categorize root causes, and track corrective action proposals to closure."
"Software practitioners are kept informed of the results of causal analysis activities. Buy-in seems to have occurred."