An assessment wrap-up is an informal meeting between the members of the assessment team devoted to discussing lessons learned, completing any required SEI feedback forms, and talking about ways to improve before the next assessment.
Three major goals are associated with an assessment wrap-up:
- To complete the transmission of assessment results to the assessed organization, prepare a context for assessment follow-on activities, and assign responsibility for writing the final report, if one was required by the sponsor.
- To ensure that the assessment team members have an accurate understanding of their roles in any post-assessment activities, such as final reports, action planning, and so on.
- To provide feedback to the SEI. This feedback informs the SEI about the results of the assessment and helps the SEI to improve the assessment method. The achievement level of the organization assessed also enters the SEI’s database and provides additional information about the state of the practice in process improvement around the world.
Organization Wrap-Up
Because members of the assessment team are also members of the organization assessed, the wrap-up meeting impacts the way the assessment results are absorbed by the organization. The team should make sure that any issues already raised by the assessment’s sponsor about the assessment findings are addressed, and that there is a common understanding of the manner in which any remaining commitments to the organization will be fulfilled. (The latter includes further guidance to the sponsor about completion of the assessment and responsibility for improvement activities.)
A confidence report is communicated to the sponsor in the form of a verbal or written statement that identifies those elements of the assessment that the Lead Assessor believes should produce exceptionally high or low confidence in the assessment results.
The Lead Assessor identifies (if necessary) individuals responsible for fulfilling remaining obligations and sets a schedule for work on a final report if the organization so wishes. If a final report is requested, it should be submitted within 60 days of the end of the assessment (refer to "Planning and Preparing for an Assessment, Part 1: Senior Management Responsibilities"). Often, the final report is forgone in favor of an action plan to be created by members of the assessed organization. This plan serves as a bridge between the assessment and follow-on improvement activities.
Finally, the Lead Assessor hands all detailed assessment data (minus identification of sources) to a responsible member of the organization (usually an organization assessment team member designated by the sponsor). Confidentiality is to be maintained indefinitely. Team notes are shredded. Documents are returned to their sources by the organization site coordinator.
Assessment Data Sent to the SEI Appraisal Database
The Lead Assessor is required to make sure that a statement of the assessment data as recorded in appropriate forms is returned to the SEI within 30 days of the conclusion of the assessment. These statements include:
A copy of the final findings presentation, including the KPA/PA profile and the maturity rating.
A PAIS form (Record of Entry to Process Appraisal Information System), which describes the organization to the SEI and registers the assessment data with the SEI.
For a SCAMPI: An Appraisal Disclosure Statement (ADS) is required. The ADS includes information about the following:
A description of the organization assessed
Assessment dates
Assessment method used
Reference model used
Process areas within the assessment scope
Key participants (sponsor, team leader, team members, site coordinator)
Ratings (process areas, maturity level)
Evaluation by the Lead Assessor of whether the assessment was properly conducted and of the accuracy of the results.
The SEI keeps all assessment data confidential. It does not disclose the name of an organization or project that has returned data to the SEI. The SEI uses assessment information to prepare reports on the worldwide state of the practice, and these reports are posted as the Maturity Profile on the SEI web site, www.sei.cmu.edu, and updated twice annually [SEMA 03].
Feedback Relating to the Effectiveness of the Assessment Method
The SEI requires several feedback forms be sent to back to them after the completion of the assessment.
A Team Member Feedback Form, which addresses questions such as:
What model (CMM/CMMI) training was provided prior to the assessment?
How would you rate your model knowledge?
Did your experience meet the team member selection criteria?
Did the Lead Assessor provide assessment team training for you? Describe.
How well did the training provide for your role as a team member?
Were the interviewees forthcoming with information? Describe.
How would you rate the performance of the assessment team?
How would you rate the performance of the Lead Assessor?
Additional questions are asked regarding the level of difficulty of model understanding, time management, and execution of onsite activities.
How would you improve the assessment method?
A Lead Assessor Feedback Form. This form addresses questions such as:
Do you believe that the assessment findings accurately describe the state of the development process in the assessed organization?
Were the interviewees forthcoming and honest in providing information during the assessment?
Were the assessment team members objective in performing their responsibilities?
How would you rate the performance of the assessment team?
How confident are you that the organization will support the implementation of improvements based on the assessment findings?
Additional questions are asked regarding the model understanding, time management, and execution of onsite activities.
What performance improvements would you like to see in future assessment teams?
A Sponsor Feedback Form. This form addresses questions such as:
How important are these issues in your organization; i.e., defects found after release, schedule and planning more effectively, estimate resources more accurately, and satisfy corporate requirements for process improvement?
How important were the objectives of the assessment; e.g., to initiate the process improvement program, achieve a maturity level, and/or monitor process improvement progress? Were these objectives achieved?
Do you think that the assessment findings accurately reflect the state of the software process in the organization?
How well did the Lead Assessor set your expectations for the assessment?
How would you rate the performance of the assessment team for this assessment?
How would you rate the performance of the Lead Assessor for this assessment?
How much involvement did you have in determining the assessment scope, both the organization scope and the reference model scope?
How well do you expect the assessment findings to provide guidance for planning follow-on process improvement activities?
How confident are you that you will be able to actively support the implementation of improvements based on assessment findings?
What improvements to the assessment would you suggest?
For a CBA IPI: A Lead Assessor Requirements Checklist. This form can be used early in the assessment by the Lead Assessor as a planning tool. It consists of a checklist to aid the Lead Assessor in ensuring that each of the required activities of a CBA IPI have been satisfactorily conducted [Dunaway 01b].
Confidence Report Template for CBA IPI
For a CBA IPI, the Lead Assessor must (either verbally or in writing) provide a confidence report to the assessment sponsor to report any issues that might affect confidence in the accuracy and completeness of the assessment results. The confidence report is delivered during the assessment so that remedial action may be taken if needed and if possible. The confidence report must address the kind of risks that could produce a low confidence in the accuracy and completeness of the assessment results. Examples of confidence factors that need to be considered are:
Composition of the assessment team.
Members’ qualifications, e.g., team members need sufficient model and method training. If most team members are inexperienced in assessments, the use of the CMM model, or the organization’s products, this may be problematic for the team’s effectiveness.
Size of the team, e.g., a requirement of four team members must be met. A team larger than 10 is problematic and can affect time management and the ability to reach consensus.
Potential biases of any team members, particularly if they have a vested interest in the assessment results.
Organizational scope of the assessment.
Cohesiveness of the appraised entity in terms of its management structure, geographic location, and product lines.
The extent to which the participants are forthcoming concerning any shortcomings in the organization processes.
The number of projects and the extent to which they are representative of the appraised entity; e.g., a significant project might be omitted due to some process insufficiencies.
The number of people and the extent to which they are representative of the appraised entity.
The extent to which data collected covers the selected projects and is corroborated by the selected participants.
Use of data collection techniques planned and achieved.
Combination of data collection techniques used; i.e., instruments, documents, interviews, and presentations.
Extent to which the data collected is corroborated.
Validation of findings.
Rules of corroboration used in validating findings.
Validation of findings by the appraised entity; e.g., draft finding presentations.
Time spent (the amount of time spent per unit appraised; e.g., process area and organizational unit).
Adherence to the assessment plan.
Assessment Acknowledgement LetterTemplate
Lead Assessors are frequently asked by the assessed organization or an associated contracting agency to acknowledge that an organization has performed a CBA IPI or SCAMPI assessment. Because the SEI does not certify or acknowledge received assessment results, a Lead Assessor may choose to send a letter such as the following:
[Date]
[Sponsor Name and Address]
Dear [Sponsor's Name]:
Thank you for permitting me to participate in your recent [specify CBA IPI or SCAMPI], which concluded with the Final Findings Briefing on [last day of onsite period]. As the Lead Assessor, I want to commend each of the assessment team members for their dedication and for the fine job that they did. [Name of site coordinator], as site coordinator, made our jobs easier with [his/her] smooth handling of the assessment planning and logistics. It was a pleasure to work with this intelligent, cooperative group of people.
[Name of assessed organization] has worked hard on the process improvement initiative. Your own strong management support has been a key element in your accomplishments. I share your pride in the assessment team’s findings that indicate [Name of assessed organization] has achieved a Maturity Level [X] by satisfying all of the (key) process areas at Level [x,y,z] in the [CMM or CMMI]. The assessment also acknowledges, as indicated in the Final Findings Briefing, that [additional PAs/KPAs rated] have also been satisfied.
I look forward to further work with [Name of assessed organization] in your continued efforts to achieve process improvement.
Best regards,
[Name of Lead Assessor]
SEI-authorized Lead Assessor
Copies to: [Assessment Team Members]