Assessments culminate with the rating of individual KPA/PAs and the establishment of a global maturity level rating for the organization, which is the end of a long process. This book has concentrated throughout on staged process improvement models, which result in a single maturity level rating for the organization, and that emphasis is maintained here. However, in what follows, an additional section has been provided to explain how the process of establishing a capability level rating works using the CMMI continuous representation model.
Final Consolidation
After the draft findings meeting and any needed additional interviews, it is time for the assessment’s final consolidation. This is usually done the day (or the night) before the last day of the assessment and the final findings presentation.
During final consolidation, the assessment team completes the KPA/PA worksheets for the last time and, before any rating takes place, determines once again if there is sufficient data to clearly describe all the processes of the organization within the assessment scope. If there is still not enough coverage for a KPA/PA, that KPA/PA will be clearly marked "Not Rated." This, however, will impact the maturity level rating because all applicable KPAs/PAs need to be rated for any maturity level to be achieved.
Rating: The Process
As with the previous stages of consolidation, consensus is required for all rating decisions.
Goals and KPAs/PAs in the CMM/CMMI are characterized as satisfied or unsatisfied based on the implementation of the model practices by the organization.
Rating is slightly different from the preparation of draft findings because it involves the rigorous translation of the organization’s strengths and weaknesses into the analytical categories of the model. For this reason, the Lead Assessor, who will be more familiar with the model than the rest of the team, may have to play a more prominent role in this stage of consolidation than in previous ones.
Rating judgments made by the assessment team depend on (1) the quality of the data available to them, (2) their ability to reliably judge the implementation and institutionalization of practices in the organization, and (3) their ability to correlate these practices with the model (CMM/CMMI).
Rating is performed for each goal for each KPA/PA within the scope of the assessment. Supplying an organizational maturity level rating is optional in both a CBA IPI and SCAMPI. The organization’s sponsor must determine prior to onsite activities if he or she wants a maturity level to be recorded. Ratings are reported to the organization during the final findings presentation.
Judgments made concerning ratings at each level within the CMM/CMMI structure build on judgments made at finer levels of detail within the structure:
The rating of a goal for a particular KPA/PA is based on examination of each relevant practice and the extent of its implementation and institutionalization, after sufficiency of coverage has been achieved.
The rating of a KPA/PA is determined when each of the KPA’s/PA’s goals have been rated.
A maturity level rating can be determined when all KPAs/PAs within that level and each level below have been rated.
Prerequisites for Rating
Most of the groundwork for ratings usually has been accomplished during previous phases of the consolidation process. In particular:
The team should already have a good understanding of how the organization has implemented each relevant practice within the assessment scope because data has been gathered for each practice since the beginning of the assessment and has been consolidated on the team’s worksheets.
From the beginning of consolidation, when a weakness has been identified with regard to model-related activities, the team will have explored whether the organization has managed to mitigate this weakness with an acceptable alternative practice.
The assessment team reconfirms (see Section 10.3.1) that the observations used as the basis for ratings are:
Accurate
Valid
Consistent
Thorough (i.e., they fully cover the area being investigated)
The assertion that observations are "consistent" implies that none of the observations conflict with other observations and that no identified weakness has been mitigated by any acceptable alternative practices.
The fact that the observations meet the "coverage" criteria for the area of investigation means that they address each of the practices of the model in enough depth to determine the extent of their implementation.
If no weaknesses are found in a set of KPA/PA practices that are based on accurate, valid, consistent, and thorough criteria, then the team determines that that KPA/PA has been satisfied.
Final Rating Process
After all the data consolidation activities are finalized, the final rating process begins:
All KPAs/PAs that are outside of the organization’s scope of work are designated "not applicable."
All KPAs/PAs that are outside of the assessment scope or for which insufficient data was collected are designated "not rated."
For the remaining KPAs/PAs, each goal is examined for each KPA/PA and rated as "satisfied" or "unsatisfied."
For a KPA/PA to be satisfied, each of its goals must be satisfied. Otherwise, the KPA/PA is rated "unsatisfied."
If all KPAs/PAs within and below the maturity levels under consideration are rated, the maturity level can be determined.
Rating Goals
An organization’s process relative to a particular goal associated with a KPA/PA must have one of the following two ratings:
Satisfied Each of the organization’s practices associated with the goal is implemented and institutionalized either as suggested by the model or with acceptable alternative practices. The aggregate of the weaknesses, if any, does not have significant negative impact on the goal being achieved.
For SCAMPI, the definition of a goal rated satisfied is more precise:
All associated practices are rated either largely implemented (LI) or fully implemented (FI).
The aggregation of the weaknesses does not have a significant impact on achieving the goal [Members of the AMIT 01].
Unsatisfied One or more of the practices associated with the goal are not adequately implemented or institutionalized either as suggested by the model or with adequate alternative practices, and the aggregate of the weaknesses associated with the goal has a significant negative impact on the goal being satisfied [Dunaway 01c].
All goals are considered to be applicable if the KPA/PA is applicable. If a KPA/PA is designated "not applicable," then none of the goals are deemed applicable. If a KPA/PA has been designated as "not rated," none of its goals are rated.
The following rules are followed when rating a particular KPA/PA goal.
The goal must be satisfied if the findings:
Indicate that the organization effectively implements all model practices related to the goal.
or
Indicate that the organization implements alternatives to the model practices, which satisfy the goal.
Identify related weaknesses that when considered in aggregate do not have a significant negative impact on the associated KPA/PA goal.
The goal cannot be satisfied unless the findings:
Identify valid strengths in the organization’s implemented practices.
Indicate that the process is institutionalized and has been in place for a satisfactory period of time relative to the development life cycle in use. (The rule of thumb sometimes used is a minimum of six months; however, a satisfactory length of time depends on the specific process and its duration.)
The goal must be unsatisfied if the findings identify related weaknesses in the organization’s implemented and institutionalized practices that when considered in aggregate have a significant negative impact on the goal.
The goal cannot be rated if the organization is unable to perform the practices or if the practices have only been performed outside of the assessed organization and therefore outside of the assessed organization’s control.
The goal cannot be rated unless the related data are sufficient to determine the extent to which the associated model practices are implemented and institutionalized.
In other words, the assessment team assumes that a particular KPA/PA goal is satisfied unless associated weaknesses, considered in aggregate, have a significant negative impact on the related KPA/PA goal. In the latter case, a rating of unsatisfied is given [Dunaway 01c] [Members of the AMIT 01].
Rating of Process Areas/Key Process Areas
When all goals for a KPA/PA have been rated either "satisfied" or "unsatisfied," the KPA/PA can be rated as follows:
A KPA/PA is rated "satisfied" if and only if each of the goals is rated "satisfied."
A KPA/PA is rated "unsatisfied" if one or more of its goals are rated "unsatisfied."
A rating of "partially satisfied" can be indicated for a KPA/PA that has at least one of its goals "satisfied." However, this counts the same as "unsatisfied" when considering maturity level rating.
If a KPA/PA is outside the organization’s scope of work, the KPA/PA is designated "not applicable"a designation that will not impact the maturity level rating. If a KPA/PA is outside the assessment scope, or if sufficient data were not available to determine goal satisfaction, the KPA/PA is designated "not rated," and the KPA/PA must be considered "unsatisfied" for the maturity level rating.
Maturity Level Ratings
An organizational or global maturity level rating is not required for an official CBA IPI or SCAMPI. Some organizations using the CMM/CMMI only want to focus on certain KPAs/PAs and do not want to assess all KPAs/PAs within a maturity level. The results of this kind of an assessment can be just as useful in furthering an organization’s process improvement program as the results of maturity level assessment.
For organizations that choose to have the assessment team provide an official maturity level rating, the scope of the assessment must include all the KPAs/PAs necessary for the maximum maturity level included in the investigation.
The highest level that is examined will determine the organization’s highest possible rated maturity level. For example, the fact that not all Level 4 PAs/KPAs have been rated forces the highest possible rating to be Level 3, even if some Level 4 PAs/KPAs have been rated as satisfied.
Ratings for each goal for each KPA/PA are recorded on a KPA/PA profile worksheet that shows the composite KPA/PA ratings that are to be presented at the final findings presentation.
In addition, if the organization desires to receive individual KPA/PA profiles for each maturity level, the team prepares those as well, alongside the composite KPA/PA profile that shows all maturity levels.
Results of two assessments cannot be combined to produce a maturity level rating. Although it may be useful for an organization to conduct a partial assessment to monitor progress, combining results from assessments performed at different times has a high risk of being misleading .