During the middle stages of an assessment, it is easy for an assessment team to regard the process of consolidation as a mechanical exercise that can be put off until later. If it is already late in the day, the team may wonder why consolidation has to be done now. But too little early discussion almost always leads to major disagreements later on.
The same difficulties arise when, during daily consolidation meetings, the team as a whole defers to mini-team members instead of engaging with the adequacy of the evidence being presented to them.
If a Lead Assessor allows this pattern to continue, at the end of the assessment when ratings must be assigned, the shallowness of early moments of consensus will become apparent, and the team may be left with real disagreements about important issues. Problems circumvented by earlier shortcuts have a way of reappearing. Without real participation and discussion in the daily consolidation sessions, ratings meetings can disintegrate into drag-out quarrels.
Organization F asked the seven members of its assessment team to look at all Level 2 and Level 3 PAs. The organization had been involved in process improvement for at least five years and was confident not only that they would be rated at Level 2 but also that they were well on the way to reaching Level 3. The Lead Assessor divided the team into two mini-teams. One team was responsible for project planning, project tracking and oversight, and integrated software management. The other team was responsible for software quality assurance, software product assurance, and peer reviews. Each team asked detailed questions and at the end of the interviewing period conscientiously filled out the PA sheets. For the first several nights of team consensus meetings, one of these mini-team members led discussions about whether PAs had been satisfied. The part of the team not involved in that area was glad to nod in agreement on each point because they did not want to contest their colleagues’ authority and because they wanted to get home as soon as possible. However, as the assessment went on, it became evident that the organization was not performing some Level 2 activities and that many team members had not really thought about any PAs other than the ones for which their own mini-teams were responsible. When the team as a whole was required to come to consensus about a rating based on the full range of PAs, a very acrimonious disagreement ensued, and the team encountered many more (and more unpleasant) late hours than they had avoided.
One obvious way to avoid this kind of assessment-busting situation is not to use mini-teams. If the whole team works on all KPAs/PAs, one cause for late disagreements can be entirely eliminated. But this solution only works with small teams of four or five people.
Another common but avoidable problem: In the early stages of an assessment when filling out the data sheets for each KPA/PA, many team members will note strengths but not weaknesses. They assume that whatever weaknesses they have noticed can be traced to information they have not yet encountered, rather than real holes in the organization’s processes. Thus they do not flag the holes in the information, even after they have been instructed to note all potential weaknesses. Among other things, this habit means that follow-up activities must be piled up at a very late stage of the assessment. Even more seriously, it allows acrimonious debates about satisfying KPAs/PAs to take place by displacing some information and giving a temporary advantage in consensus discussions to team members who want to make the data look more positive than it is.