General Rules
The process of transforming notes into observations is a complicated one, about which teams may need guidance. Ultimately, the team’s observations are systematized into observations that are specifically keyed to practices within a KPA/PA. The following explains how notes should be transformed. Helpful rules for creating observations include (1) use simple complete sentences, (2) make observations as relevant to the categories of the assessment (or "finding-like") as possible to avoid rework later, and (3) avoid attributing statements to any individuals.
Observations are usually statements created by mini-teams that categorize information either heard or seen in data-gathering sessions. When an observation is presented to the team, discussions usually occur to make any modifications needed to satisfy the team members. Accuracy of each observation needs to be agreed to by the entire assessment team.
Each observation should be discussed in regard to its accuracywhether it is based on information provided, whether it is relevant to a component of the model or otherwise has a significant impact to the organization’s process capability, and whether it has some significance in relation to a strength or weakness of the organization or to a practice of the model.
After an observation is deemed accurate, it is the team’s responsibility to verify that the observation is validthat is, that the accurate observation is corroborated by at least two independent sources and is consistent with other observations.
In addition, for each practice in the model being investigated, the team must agree that the data collected sufficiently cover the practice to (1) understand the extent of implementation of the practice, (2) represent the organizational scope of the assessment, and (3) represent the life-cycle phases in use within the assessment scope.
Sufficiency of coverage relates to a particular practice or component of the model and may be satisfied by a composite made up of multiple observations relating to the practice.
The overriding concern during consolidation is obtaining sufficient objective information to cover all the practices of every KPA/PA included in the assessment. Sufficient data must be collected for each practice within the assessment scope to cover the organization being assessed and the development life cycles used by the organization.
Observations should be accurate, valid, and sufficiently thorough to provide for final rating decisions (see Summing Up and Presenting Results). Until sufficient coverage is obtained for each practice within the assessment scope, data collection activities must continue.
For each KPA/PA, the individual mini-teams will:
- Review their notes for the assigned KPAs/PAs.
- Write accurate observations and transfer them onto the individual KPA/PA worksheets.
- Update existing observation(s) to include additional sources, improve accuracy, or eliminate redundancies and inconsistencies.
- Update the needed information section.
For each KPA/PA, the team as a whole:
Reviews notes relative to given KPAs/PAs. (Generally KPA/PA mini-teams present new or updated observations.)
Ensures that observations are accurate.[1] If not, they must be discussed and either updated or removed.
[1] Accuracy is required for each observation according to the following criteria:
Worded appropriately Clear, no absolutes, expressed in terms of the organization being assessed, and phrased without attribution to a particular source. Observations must be worded clearly and phrased in organization terms that can be understood by both the assessment team members and the organization. Use of absolute statements should be avoided because the sampling that is performed during the assessment is never complete enough to substantiate such statements. Also, observations should also be phrased so that the source of the observation is not recognizable and confidentiality commitments are not compromised.
Based on specific information Based on information that has been heard or seen by the assessment team. The absence of information is itself a piece of data but must be validated by other observations.
Relevant Can be categorized in terms of the CMM/CMMI or has significant impact on the organization’s process capability. Observations must be relevant. If an observation cannot be associated with a particular practice, it cannot be recorded as an observation relative to a KPA/PA. Such an observation can, however, be recorded as a non-model observation if the assessment team believes it has a significant impact on the organization’s process capability. When some of the practices performed in a KPA/PA are confirmed, but others are not, the observation must include more detailed information.
Significant Can be classified as evidence of strength, weakness, or acceptable alternative practice. This can apply to both model-related observations and non-model-related observations.
Not redundant An accumulated record should not contain redundant observations. When KPA/PA mini-teams organize observations into a single set for each KPA/PA, redundancy is eliminated by combining similar observations, rewriting overlapping observations, and deleting duplications. All of the sources upon which the new observation is based must be identified .
Ensures that observations are valid[2] (i.e., accurate, corroborated by multiple sources, and consistent). If an observation is not accepted as valid, the team may rewrite the observation and/or decide that more information must be collected.
Validity of observations The assessment team must judge each observation to be valid by consensus as each mini-team presents the observations, one at a time for each KPA/PA in the scope of the assessment. Each observation will be determined to be valid according to the following criteria:
- Accurate using criteria discussed in the preceding sections
- Corroborated
- Consistent with other validated observations
Corroboration An observation is considered to be fully corroborated if the minimum rules of corroboration are satisfied:
The observation is based on data from at least two independent sources, e.g., two separate people or a person and a document.
The observation is based on data obtained during at least two different data-gathering sessions.
At least one of the two data sources indicates work actually being done; e.g., a person performing the work or an implementation-level document [for a CBA IPI].
The intent of requiring corroboration that "work actually being done" is to ensure that a practice is really in place and not just documented. Examples of work being done would be meeting minutes, status reports, action item tracking reports, or someone saying that they enact a practice.
Although questionnaire responses are considered valid data sources, a questionnaire response is the weakest source of data due to possible ambiguity or misunderstanding of the respondent. Interviews must be adequately incorporated into the data collection process. A questionnaire response cannot corroborate a document without getting the information from an interviewee to ensure that the questionnaire response was understood.
Consistent A set of validated observations is consistent if it has the following characteristics:
It does not include weaknesses for CMM/CMMI-related practices that directly conflict with evidence of acceptable alternatives.
It does not include weaknesses that directly conflict with evidence of strengths.
A direct conflict between two observations requires that they both address the same aspect of a practice or set of practices and apply to the same part of the organization. It is possible for some aspect of a practice of be strong while another is weak. In such cases, two observations should be made: one observation indicating that some projects are weak in a particular area, and one observation indicating that some are strong in the same area.
Ensures that there is sufficiency[3] of data for each CMM/CMMI practice within the assessment scope. If there is not sufficient data, the team decides what additional information is needed and which interview or document review should be arranged to obtain the information.
[3] Sufficiency of dataSufficient data must be collected for each practice during an assessment to cover:
- The organization being assessed.
- The development life cycle(s) in use in the assessed organization.
- The model components within the assessment scope.
Coverage of the organization is sought by choosing representative projects and participants who are interviewed during the assessment. Coverage of the development life cycle is sought by the cross-section within the organization of projects and groups of people representing the project activities. The data-gathering techniques employed in the CBA IPI method are intended to ensure that the same topics are covered with representatives of the different projects selected. Coverage of the organization and development life cycle must be considered during the consolidation of information to ensure that this coverage has been achieved.