As discussed in Assessment Team Training, "Planning and Preparing for an Assessment, Part 4: Assessment Team Training and Post-Training Activities," documents both provide real information and establish expectations for interviews and other data collection techniques.
A document is any lasting representation of information that is available to the people in the organization doing the work (development and management). Documents include paper (or electronic) copies of policies and procedures, code libraries, electronic records, visual media (e.g., slides, poster, and training materials), graphs, charts (e.g., pert charts), working notes, and so on. A document is viewed as external "memory" for the people in the organization. Documents are used as tools for communicating complex issues over time. Documents are not restricted to formally published policies and proceduresthey include any lasting representation of information that is available to the people doing the development and management .
Documents provide objective evidence of the processes used. A fundamental assumption of the CMM/CMMI models is that a process must be documented to be effective: "An effective process can be characterized as practiced, documented, enforced, trained, measured, and able to improve" .
Document reviews during pre-onsite activities help establish contexts and reveal areas that need to be probed. During the assessment, emphasis shifts to data confirmation and answering specific questions.
The level of formality in the documentation reviewed varies with the size and culture of the organization and projects. Documenting your work processes help to ensure that they are consistently followed. However, it should always be remembered that the processes described by documents do not necessarily correspond to practices that are consistently followed. Documents are "necessary" but not "sufficient" to substantiate the existence of working processes .
An assessment team reviews documents to learn the processes that the organization uses and the organization’s strengths and weaknesses relative to the reference model. Document review corroborates data from other data sources.
An assessment team examines product-related documents not to ascertain whether documents are well written but rather to check whether documentation exists, is used, and is systematically updated after requirement changes.
Three conceptual "levels" of documentation are reviewed:
- Organization-level documents define policies and practices that developers associated with all development activities should know, understand, and use.
- Project-level documents describe procedures that implement organizational policies and contain the engineering activities related to a particular development effort.
- Implementation-level (or track record) documents have to do with the day-to-day processes associated with a particular engineering activity and are used to measure and track the work performed. They provide evidence of processes being implemented .
Most document review efforts concern implementation-level documents because they correspond most closely to work actually being done.
Team members from the organization can point external team members to specific documentation associated with related topics based on their knowledge of the organization’s working environment.
Systematic document searches involve tracing particular "threads" through a related set of documents. Threads often follow the sequential relationship between different work products in the development life cycle. For example, a reviewer might trace a requirements change as it is implemented in the cycle from requirements management to requirements analysis, design, and code and test activities. Alternatively, the reviewer might trace organization-level policies to project plans, operating instructions, and track record documents .
The assessment team checks that the organization’s processes are consistent and complete. Gaps and inconsistencies represent activities that are not well defined or controlled by management. These gaps are identified, and they become the subjects of further questions.
When assessing Organization Q, a team member reviewed a project plan and tried to follow the thread for the procedures that the project used to develop size estimates for their products. When these were examined, it turned out that there were at least four different recommended ways to develop the size of the work product. At the organization level, this disparity would not have been inappropriate. But the organization-level document should give guidance as to which method works best for different types of projects. In this particular case, no such guidance existedthe project document did not state which method was used. This was identified as a possible problem and was noted as information needed. When the project manager and software managers were interviewed, they couldn’t explain what method was used or why. It turned out that although the process improvement manager had given them copies of the organization template for product size estimating, no project had actually estimated the size of the product. They had only estimated the resources needed and the schedule and could not explain how the size of the product related to resources or the schedule.
Interviewees are asked to provide examples of their "day-to-day" documents and work products. It is preferable for the documents to be given to the organization site coordinator during the pre-onsite activities when there is time to review them. However, a document may be brought to the interview.
When an interviewee is discussing a particular activity, the team will ask questions such as:
- How is that activity tracked?
- What records are kept?
- What measurements are taken?
These questions will often cause the interviewee to refer to a document. If the interviewee has the document with him, it is useful for him to take a few minutes and show the team where the information is located in the document. If the document is not at hand, the team needs to ask the interviewee to bring the document to the organization site coordinator.
The assessment team should establish what needs to be examined as specifically as possible. The team must, however, be sensitive to possible differences in terminology. (The reason the information has not been found may be because the team has been asking for the wrong thing.) Internal team members can usually help clarify such misunderstandings.
Reviewing Organization-Level Documents
Organization-level policies and procedures establish the development environment for all the organization’s project activities. These documents define those process guidelines that all projects are expected to know, understand, and use. The richness of the documents demonstrates the degree to which the organization supports the project’s development and maintenance of products. The documents define a set of organization standard processes that have been proven to work on past projects. Review of organization-level documents is usually done early in the assessment; however, they can be referred to at any time during the assessment.
Examples of organization-level documents are:
The assessment team looks for the following kinds of information in organization-level documents:
- Planned and required management reports and reviews
- Tailoring guidance
- How exceptions are handled and documented
- Who is assigned responsibility in the document
- Whether the documents have a history of revision
- Sponsorship for the process activities
- Reporting channels for exceptions
- Defined roles for configuration management, quality assurance, and subcontract management
- Organizational training requirements
- Particular development activities that are required (e.g., peer reviews)
- Organization procedures for estimating size and cost
- Status reporting practices required on all projects
- Metrics required for all projects
- Risk identification and mitigation plans for a project
- Tailoring guidelines and waiver procedures related to organization standards
- Training plans for the organization
- Policies, procedures, and standards for engineering activities such as requirements, design, programming, and testing
- Policies, procedures, and standards for engineering support activities such as configuration management, quality assurance, etc.
Reviewing Project-Level Documents
The next level of process documents reviewed by the assessment team includes those that define the project development. These documents relate to activities needed to coordinate and integrate a particular development effort.
Ideally project-level documents should trace to the organization-level documents. Project-level procedures should be consistent with the organization’s standard set of processes. However, in less mature organizations, organization-level documentation may not be available. Therefore, the burden of defining all processes and procedures falls on the individual projects. In these kinds of organizations, the assessment team should look to the project documents for these (engineering and management) procedures and processes.
A project’s documents define the detailed processes that are used to manage, coordinate, and integrate all required engineering activities. This level of documentation provides structure to the development by:
- Translating high-level organizational policies and procedures into detailed procedures, plans, and guidelines.
- Defining specific project roles and responsibilities.
- Allocating project personnel and other resources.
- Defining detailed procedures and operating instructions to supplement and enhance organization-level procedures.
- Specifying and planning for required project training.
- Tracking adherence.
- Defining measurements used to manage project activities .
The assessment team reviews project-level documents to establish context, to determine areas to be probed, to confirm interview data, and to answer specific questions about processes in use. Review of project-level documents begins before the onsite visit and continues throughout the assessment.
The information a team looks for in project-level documents includes the:
- Differences between the processes used in different projects
- Planned/required management reports and reviews
- Tailoring guidance
- How exceptions are to be handled, are handled, and documented
- How changes to the schedule or requirements are handled over the life of the project
- Who is assigned responsibilities by the document
- Defined project-level training requirements
- Resources planned or allocated by the document
- Sufficient detail in the processes to guide actual work practice
- How the tasks described are tracked, measured, and verified
- How exceptions are reported
- How product size is estimated, measured, and revised
Project-level documents may include (but are not limited to):
- Systems engineering plan
- Software development plan
- Quality assurance plan
- Configuration management plan
- Schedules
- Detailed procedures and work instructions
- Project notebooks that define how the project collectively understands and integrates the engineering processes
The assessment team will consider how project documents are related to organization documents. The team may also check if and how organization processes are tailored, if required project documents exist as dictated by organizational documents, and if exceptions or waivers are handled and documented in accordance with organization guidance. The assessment team will also look at the relationship between project-level documents and implementation-level documents. For example, they will examine the tracking methods that are defined and used, the measurements that have been defined and are used, and whether standard report formats are defined and used.
Reviewing Implementation-Level Documents
The third type of documents that the assessment team reviews are implementation-level documents, such as status reports, minutes, schedules, and so on. (In SCAMPI, these are known as indirect artifacts.) These documents provide an audit trail or "track record" of all processes used on the implementation.
The purpose, format, and content of implementation-level documents are traceable to organization- and project-level procedures and standards. Implementation-level documents capture information that identifies work activities. They should be easy to use and should capture actual information about work accomplished.
The primary purposes for the assessment team reviewing implementation-level documents are to indicate areas that need to be probed, to confirm interview data, and to provide evidence of actual work being done.
The information the team looks for in implementation-level documents includes:
- Evidence of actual practices used
- Record of resources used
- Data on process improvement efforts
- Who uses the information in the document and how
- Who reviews the work products
- What measurements are taken and how they are used
- Consistency of work products with project-level plans, procedures, standards, etc.
- How regularly reporting, reviews, and verification activities are conducted
- How deviations from planned work are documented and handled
Implementation-level documents include:
- Meeting minutes (e.g., from project management meetings, configuration control board meetings, etc.)
- Project status reports and schedules
- Change request forms
- Test records
- Training records
- Software development folders
- Historical data derived from past schedules and status reports
- Analyses of resource trends, cost trends, size trends
- Work products such as requirements specifications, high-level and detailed design documents, etc.
The assessment team will consider how the implementation-level documents track to the project-level documents.
How the Assessment Team Prioritizes Documents for Review
An assessment does not allocate enough time for the team to examine every pertinent document. However, a large sample should be reviewed to ensure coverage according to the assessment corroboration rules.
The organization can expect the team to spot-check documents and work products mentioned during the interviews and given to the team. For example, there might be a number of problem reports or change reports to choose from. Typically the team will choose a relevant few for review.
The assessment team asks for independent corroboration of data. This may take the form of direct or indirect artifacts as well as interviews. For example, if a project manager says that all programmers use a particular coding standard, the team might ask a programmer to show them the standard he or she uses (along with some code modules produced using the standard). This allows the team independently to confirm the manager’s statement and also to examine how well the standard is followed.
Reviewing Documents for General Context
Documents may provide valuable information not related to a particular KPA/PA. The organization will benefit if a team uses this information as a basis to ask further questions. Examples include:
- Creation dates on procedures might indicate that a process is new. This can lead to inquiries about how the information was made available to the people using the procedure and how many people have actually used the new procedure. A new procedure with a very recent creation date could indicate that no one has ever used it and that it was created for the assessment.
- A revision date might indicate that a given process had not been updated for several years. The team would look for signs that the process is still in use and that work is really being done this way.
- The authorization or approval "signature" indicates the level of sponsorship for a process. If, for example, the Software Engineering Process Group (SEPG) manager signs an organization policy, questions would be asked about how important people in the organization regard the policy. Normally you would expect to have an executive sign an organization policy.
- The size and format of a document has a direct bearing on its usability. If a document were extremely large, the team would look to see if people actually knew what the document said and how it was used.
- Cross-references to other documents provide critical links to KPA/PA-related information. (An organization is not expected to keep its process documentation in KPA/PA order, even if it would make assessments easier.)
- For electronic documents, a listing of who is allowed access and what kind of access (create/read/modify) will indicate who controls work product configuration.
- Procedures for version control and archiving indicate the stability of information .
When assessing Organization M, the assessment team noticed that all of its policy and procedure documents were dated a week or two prior to the assessment and that there was a consistent list of signatures on the documents. When they examined this further, it turned out that a small committee had been put together in the month before the assessment to produce all of the company’s policies. The documents, needless to say, had little to do with actual practices. Instead they provided strong evidence that creating policies had meant little to the organization.
Library Management
As discussed in Assessment Team Training and Post-Training Activities, documents for the assessment are maintained in a "library." The organization may supply documents in a hard copy or electronic copy. The organization does not need to make copies of the documents but can loan them to the team. The organization may use them any time the assessment team is not using them, but the documents still remain in the team workspace until after the assessment is completed. When documents are reviewed, the team members flag critical sections for future reference. Documents are usually managed with reference to the KPA/PA worksheets. The team may also maintain separate document reference sheets.
The organization should provide the assessment team efficient, controlled access to all documents while onsite. The assessment team’s library management procedures must be user-friendly if access to documents takes too much time, it will interfere with critical onsite activities.
The major library management tasks are described in the following. All tasks except for the first and last are repeated throughout the assessment.
The librarian verifies the initial document inventory by:
- Checking that the documents requested prior to the site visit have been provided
- Verifying or preparing the initial data sheets
- For an assessment team member to request a document during an interview, the team member may ask for it. If the document is immediately available, the document will be added to the inventory. If the document is not available, the librarian will note the request and follow up with the appropriate person to get the document.
- Outside of interviews, the team member will note the document title, session, source, and the team member’s name, and ask the librarian to get the document.
The librarian or Lead Assessor:
- Coordinates the requests with the organization site coordinator
- Adds the document to the inventory by assigning a number and entering the number on the data sheets next to the title
- Arranges for the organization site coordinator to return all of the documents to the owners at the end of the assessment, or during the assessment if requested by site personnel
The team can request documents during interviews, when the team is reviewing other documents, or during team consolidation periods. In SCAMPI, the document collection plan is updated daily. A data collection report should be available to the team at the start of each day. The librarian (or Lead Assessor) needs to coordinate requests with the organization site coordinator.
Company confidential or proprietary information may require additional care. In general, all team members who are not from the assessed organization need to complete non-disclosure agreements prior to the assessment. There must be at least one person on the team who is able to review the classified information.