Assessment teams use the same decision-making process for ratings as they did in previous consolidationsthey do it by consensus. However, although the final consensus should represent an easy and logical last deduction from the numerous moments of consensus that have preceded it, occasionally this is not the case, for reasons that are infrequently discussed but nevertheless important to remember.
Coming to Consensus About Rating KPAs/PAs
One of the most difficult moments in an assessment is when the assessment team must come to consensus about whether the organization has satisfied the goals of assessed KPAs/PAs. The "satisfied/not satisfied" rating of each KPA/PA directly impacts the maturity level rating of the organization. If a goal is found to be unsatisfied, the corresponding KPA/PA also must be judged not satisfied and therefore also the maturity level on which the KPA/PA resides. (If the organization chooses to use the continuous model, individual PA capability level ratings are compiled, but an organizational maturity level rating is not assigned.)
Even if the assessment team agrees about the requirements of the model and has previously come to consensus about observation statements indicating weaknesses, a great deal of heat can be generated when rating begins.
In general, if the organization’s practices for each KPA/PA are either all strong or all weak, there isn’t much room for controversy. However, when there is a mix of strong and weak evidence for a number of practices, the team’s task becomes truly vexing. Finding a weakness does not automatically mean that a given goal is "unsatisfied." The team must use judgment as to whether the weakness is significantthat is, whether it impacts the implementation of the goal.
One kind of problem at this stage of an assessment involves the fact that both CMM/CMMI models may seem ambiguous in their description of some required practices to team members who are not really familiar with the logic of the CMM/CMMI. During the rating session, these apparent ambiguities can provoke heated arguments. Teams often quarrel, to give only three examples, about what is meant by (1) "objective" SQA of the process and product, (2) product size, and (3) the need in large organizations to involve not only software managers but also project managers in the assessment process and in the improvement program.
Arguments have also traditionally arisen concerning a more general ambiguity about how completely practices must be implemented to satisfy a goal. (Sometimes even if a practice was not implemented at all, it was still possible to satisfy the goal if the team thought the gap was not significant.) The SCAMPI method has attempted to address this larger problem by requiring that the team come to consensus in its day-to-day consolidation on whether a practice is fully, largely, partially, or not implemented. If a practice is partially or not implemented, the corresponding goal must be rated not satisfied.
On an assessment of Organization V, after each day’s interviewing, one internal team member argued with the team about every weakness the team reviewed. These daily consolidation meetings lasted well into the evenings. The other team members and the Lead Assessor would repeatedly remind the skeptic that determination of weakness was always based on what had been said in the interviews. The internal team member was asked if his memory of the interviews was different from the others, and he would say no, but he would add that he didn’t agree with the team’s characterization of what the interviewees said. He was asked if there were additional people he thought the team should interview or if some people should be interviewed again. Every night after long battles the team would finally come to consensus.
The skeptic was of course afraid that any weakness the team recorded would result in a lower rating for his organization. When it came time for the final ratings, however, the situation curiously reversed itself. The internal team member argued that the weaknesses that had been agreed upon meant that the organization could not satisfy the pertinent KPAs. The other team members, however, now equivocated. All the issues were argued out again. The organization achieved Maturity Level 3, but only after universal exhaustion.
One of the happier endings to a ratings quarrel involved Organization F. In the middle of an assessment, the management of Organization F thought it was performing many Level 3 activities and was certainly a solid Level 2 organization. However, as the assessment progressed, it became clear that many of its Level 2 processes exhibited serious weaknesses. The team dutifully rated the full set of Level 3 KPAs, but all proved to be unsatisfied. Tempers flared when the team started to look at Level 2 KPAs, and the Lead Assessor suggested that the team break for the day and reconvene the next morning. After a night’s sleep, the organization’s SQA team member acknowledged that the organization was not performing the Level 2 activities described in the model and acknowledged that it was important for the organization’s senior management to understand this so that they could encourage real improvement. Because of his remarks, the team took another hard look at the evidence and was finally able to reach consensus. Because the team undertook its task with a new sense of seriousness, the remaining Level 2 KPAs were assessed dispassionately, and the organization was rated "Level 1." All this took place two hours before the final briefing. The assessment was completed, and the organization did improve afterwards, but the team came very close to not reaching consensus.
How Lead Assessors can Negotiate Instances of Intransigence as Ratings Are Consolidated and Assigned
At the crunch time of any assessment, there is almost always a tendency for team members from the organization to resist ratings of weakness and for the team as a whole to feel sympathy for an organization that is about to fall short. However, the Lead Assessor, who is charged with keeping in mind the long-term benefit of the organization, has a duty to counter these reactions.
An experienced Lead Assessor should negotiate problems of ambiguity by patiently pointing to places in the models that clarify these issues.
When ratings based on a clear pattern of evidence are resisted, it is best first to go back to the facts that have been collected and ask if more data needs to be collected, and if any observation that had been agreed to should be changed (and why). This will focus the ratings discussions on the evidence that has been consolidated since the beginning of the assessment.
In cases where negative consequences to the organization resulting from an honest assessment are present, it may be useful to talk about those consequences and if possible try to mitigate them. (If an internal team member is worried about losing his job, the Lead Assessor can go and speak to the president or managing director about the situation. In most cases these fears are unfounded.) In cases where fears exist that the whole organization would suffer, it is sometimes possible to turn the assessment into a "health check" and schedule another assessment in the future when the organization’s deficiencies have been corrected.
John, the process improvement manager and a VP in Company D, scheduled an assessment in the last quarter of the year. The year before, a health check had been conducted, which showed significant weaknesses at Levels 2 and 3. Although not much process improvement work had been conducted, John felt it was important for his organization to have an official assessment rating, even though a poor rating would negatively affect his company’s bid for an important contract. The assessment was begun, and eventually it became clear that the organization would be rated at Level 1. Because the rating could have prevented the organization from winning its contract (a contract that corporate headquarters had spent a substantial amount of money bidding on), the Lead Assessor checked with both the president of the organization and the president of the corporation. They were both grateful for the consultation, and both agreed that the assessment should be turned into a health check. The bid for the contract remained unaffected, and the corporation’s commitment to process improvement was sustained.
Sometimes teams try to argue that if an organization is given the benefit of the doubt, it will respond to this encouragement by an eagerness to improve. However, the reverse is often the case, and the Lead Assessor should communicate this to the team. Telling an organization that it is better than it really is can make it complacent, causing it to postpone or cancel improvement activities and risk future decline. (The organization may even take the team’s "generosity" as a sign that process improvement is subjective and without rigorjust another empty program that will fade away like all the others.)
This rule holds even when the rating only concerns one KPA.
Organization E had followed a course of process improvement for over three years and had nearly finished its first assessment. As in many immature companies, however, some of its managers felt that they only needed to improve in a few areas, and so they had not taken the program seriously. During the ratings stage of the assessment, the assessment team agreed that the organization had only satisfied one Level 2 PA (requirements management) and had failed to satisfy the others. Then on the last day of interviewing, it became clear that there were real problems even in requirements management. When it came time to finalize the maturity ratings, some team members felt that the organization would be demoralized if they did not at least satisfy one PA. One team member identified strongly with the organization and suggested how bad any of them would feel if their organization was judged deficient in every area. After many hours of discussion, though, the team stuck to the facts and finally agreed that no Level 2 PA had been satisfied. No one felt good about the decision, but about two months later, the Lead Assessor had the chance to speak to some of the organization’s developers, who told him that their management requirements procedures were even worse than they looked, and that they would have considered the assessment discredited if the requirements management PA had been marked "satisfied." An honest rating in this case was the beginning not of discouragement but rather of heightened morale in the organization.