The Lead Assessor and the assessment team should now begin to review documents. Document review constitutes one of the principal efforts undertaken during the assessment, and the techniques and criteria used by the team will be considered in depth in The Kick-Off Meeting and Other Presentations, "Onsite Activities, Part 1 (which is the first chapter devoted to significant onsite assessment activities). For present purposes, it is important to emphasize how the Lead Assessor and the team cooperate with the organization to assemble the documents to be reviewed, whether they are reviewed before or during the onsite assessment proper.
By the time the team has been assembled, the organization needs to have collected a "library" of documents to be used as evidence of the model practices it believes it has implemented. The Lead Assessor will have specified a particular format or level of detail needed for these documents, which are particularly important for a SCAMPI assessment, where documents carry more weight than they do for a CBA IPI.
The "library" of documents that the organization supplies may take the form of hard copy or electronic copy. The organization does not need to make copies of the documents but can loan them to the team. The organization should remember that the documents will continue to remain available when the assessment team is not using them. (The documents will remain in the team workspace until after the assessment is completed.)
Company confidential or proprietary information requires care. In general, all team members not from the assessed organization need to complete non-disclosure agreements prior to the assessment. However, at least one person on the team must be able to review the classified information.
Someone from the organization (sometimes the organization site coordinator) is designated the team’s "librarian." This person is responsible for compiling not only the documents but also a master document index, which comprises the assessment’s initial document inventoryits initial data set. The index contains a list of documents and also refers to related documents and to personnel who can provide relevant information. (Some organizations experienced in process improvement may already have collected this type of data to track their own progress.) A document index refers to all development and management activities both from the central organization and from each project to be assessed. The Lead Assessor may wish to verify the document inventory in advance, prior to the assessment team’s arrival.
The SCAMPI Instruments Used During an Assessment
Although SCAMPI does not require the SEI questionnaire, it does allow and encourage any instruments (e.g., surveys, questionnaires, or an objective evidence database) that help the organization characterize their process and support showing how the organization has put CMMI model practices in place. There is no need to provide the same information in two or more formats; one will do.
The team’s librarian is responsible for explaining how the documents (hard copies, soft copies, or web hyperlinks) have been indexed and arranged. The librarian briefs the team about the organization’s library management procedures during the initial team meeting.
The documents typically compiled and reviewed by the assessment team include:
- Organization policies mentioned in the model used
- High-level standard procedures in use across the organization
- Copies of project-level procedures
- Project development plans
- Samples of standard management reports
- Descriptions of the life cycle(s) used
- Descriptions of the process architecture
- Copies of defect records
- Training material
- Checklists
As Kick-Off Meeting and Other Presentations will elaborate, the team review process is intensive. When documents are reviewed, team members flag critical sections for future reference. (Team members who are part of the organization are extremely useful to answer other team members’ questions about the documents.) Before the assessment is finished, the team will need to verify objective evidence for "each instance of each practice" within the scope of the assessment.
It is desirable for the team to compile a complete set of Process Implementation Indicator (PII) database worksheets, one for each PA/KPA, before the onsite assessment begins. If the document database is incomplete, the onsite assessment will take more time.
The following illustrates a sample PII worksheet page of a kind typically filled out by the assessment team. The page in this case refers to part of the requirements worksheets, one for management PA.
During the pre-onsite phase of the assessment, the assessment team begins its document worksheets, one for review. First, individual team members or mini-teams identify the information needed to verify each KPA/PA practice. Documents are then assigned and selected for review. The Lead Assessor may make specific review assignments or may let experienced team members select their own documents for review.
Whether collected via questionnaires, document review, presentations, or interviews, the data used for an assessment is organized in categories related to the practices of the model. For every practice and for every instance of each practice, objective evidence must be found to determine the extent to which the practice is implemented. The kinds of documents that may show an assessment team that a CMMI practice may be implemented are:
- Direct Artifacts These represent the primary tangible output of a practice. They are typically listed as work products in the CMMI. One or more direct artifacts will be necessary to verify the implementation of model practices.
- Indirect Artifacts These represent artifacts that are a consequence of performing the practice, but not necessarily a consequence of the purpose for which it is performed. These are typically meeting minutes, review results, or written communication of status.
- Affirmations These are oral or written statements confirming the implementation of a practice. They are typically validated using interviews, questionnaires, and so on.
Documents are reviewed not only for verification but also for the purposes of establishing context and formulating additional areas to be probed. (See also Chapter 8.) Team members take notes as in any other data-gathering session. "Observations" (conclusions about each practice) are generated from these notes. Follow-up actions are recorded first by identifying additional needed information and then by adding notes to the worksheets to remind the team to request additional information.
Regarding the importance of documentation, a major difference in focus exists between a CBA IPI and a SCAMPI assessment. In an effort to reduce the time required to do an assessment, SCAMPI takes a "verification" rather than a "discovery" approach. In a SCAMPI, the responsibility for collecting and presenting relevant documentation to the assessment team is placed with the organization, and the team verifies the documentation’s existence. A CBA IPI assessment team reviews the documentation itself to substantiate its relevance to the reference model. When an organization has assessment experience, it does not need much assistance from the Lead Assessor or the assessment team in its collection of documentation. Inexperienced organizations, it should be noted, may have difficulty with the SCAMPI method, which requires an organization to collect and annotate documentation before the Lead Assessor and the team appears. Inexperienced organizations may find this activity extremely time consuming.
Experienced organizations find that in a Level 5 assessment, this activity requires a minimum of two person months to collect all necessary practice and PA documentation.
At the end of each document review session, the team reviews significant items found and notes any additional information that needs to be collected. Any objective evidence that is not identified in advance of the onsite assessment must be sought during onsite interviews and presentations. After the document review and the questionnaire review, the Lead Assessor may elect to conduct a consolidation session.
Analysis of the initial data set provides critical information for the overall planning of the assessment and helps form the basis for the detailed data collection plan that should be developed before the onsite assessment (data collection) begins. The analysis of initial objective evidence at this stage is focused primarily on the adequacy and completeness of information and the implications for future data collection. The results of this analysis will be the primary basis for determining the extent to which the assessment will be one of verification or discovery.
The assessment team analysis of how much more data it needs to look for may affect the onsite assessment schedule. After the organization gives its initial data set to the Lead Assessor, he or she must decide if there is likely to be insufficient direct artifacts to go ahead with a timely assessment and whether the organization is likely to satisfy the process areas of the assessment. (If documentation does not exist for each practice, the rating of the goals for each PA is affected.) If more than 20% of the objective evidence is missing at the time of team training, the organization might want to rescale or replan the assessment.
It probably goes without saying that in no case should an organization create a misleading document specifically for the purposes of an assessment. This is the equivalent of fudging the data, and it is a complete waste of time and money. Documents will be cross-indexed with interviews and other information, and bogus documents will quickly be revealed.