Experienced assessors understand that "strength" and "weakness" statements recorded in initial notes do not always translate into judgments that a given KPA has been satisfied, and they recognize that retaining a great many irrelevant statements of "strength" or "weakness" will only prolong and confuse ratings decisions later on. The problem is actually more common with statements of "strength." Inexperienced team members do not always understand what data is relevant to a judgment about whether a practice is being performed. They are also naturally eager to give the organization the benefit of the doubt. However, if too many insignificant "strength" statements are recorded and retained during the assessment, it is possible to misread them at the end of the assessment as proof that the organization is actually performing a practice when in fact it is only performing a portion (sometimes a tiny portion) of it. It is therefore essential from the beginning of consolidation to try to winnow and summarize information according to the relevant categories of final judgment, rather than including and retaining every little positive (or negative) detail.
On the other hand, it is also possible for inexperienced assessors to place too much emphasis during the early stages of consolidation on the full extent to which the organization has implemented a practice. Such assessors want to make fine distinctions and quantify success, but they misunderstand how process maturity models work. The models ask only whether the organization has implemented process improvement goals, not how well individual practices have been implemented. Because a practice has not been implemented well does not necessarily mean the organization will not satisfy that KPA/PA because satisfaction depends on goals, not practicesends, not meansand there may be other practices the organization is performing well that will satisfy the same goal. It is one of the prevailing rules of process improvement models to encourage organizations to find their own way to satisfy goals, not mindlessly to tie them down to the letter of the law. Many inexperienced assessors have not realized this, however, and want to hold organizations accountable to the way they think a practice should be performed (often the way it has been performed in their own organizations).
A related problem arises when inexperienced assessors see that the organization has shown no evidence of performing a particular activity. Especially if the assessors come from inside the organization, they begin to worry that the goal will not be satisfied, even though the practices needed to satisfy goals can involve a variety of activities. It is very useful for a team to understand exactly what it means for a practice to be performed. Noting that no evidence seems to exist for a certain activity can actually have a positive effect. At this point, the Lead Assessor can go to the organization site coordinator and ask him or her to assemble other people or documentation for review. It often turns out that the organization is actually satisfying the model’s practices and goals via activities that no one remembered while being interviewed. Only after all the data has been collected can the team decide whether a practice is being performed that will satisfy the overall intent of the KPA/PA goal.