Onsite activities include the following:
Opening meeting
Other presentations
Interviews
Recurrent team meetings for the consolidation of information, including document reviews
The onsite phase of the assessment prepares for phase three of the assessment: the preparation of draft and final findings, and the staging of draft and final findings presentations.
The Kick-Off Meeting and Other Presentations
A kick-off or opening meeting always begins an onsite assessment. The assessment sponsor explains to his (or her) organization (at a minimum, all the people who will be interviewed) why he has authorized the assessment and what he expects from his people. He must make it absolutely clear that he fully supports the assessment and the subsequent improvement efforts. He must make sure that the assessment is one of the organization’s top priorities for the week and insist that people who are scheduled to be interviewed be available at the scheduled time. At the kick-off meeting, the sponsor should encourage the participants to be open and forthcoming and explain why honesty is important. He should also explain that all assessment information is confidential, that no reports will be made about who says what, and that no one will be penalized for what they say. The Lead Assessor will introduce the team and explain what will happen during the assessmentwhat kind of questions will be asked, what material should be brought to the interview (and what need not be brought), and so on.
Other initial presentations made by the organization to the assessment team are also useful. Some examples include 30- to 60-minute briefings by project managers to explain the nature of their projects, demonstrations by project teams of what they produce, and briefings by department managers about how they run their departments or how they interact with other departments. Executives also might want to brief the team about their current perceptions of the organization’s strengths and weaknesses and its future plans.
All of these presentations help set the stage for the assessment team and help managers and developers put on record what they regard as important.
Collecting and Managing Data
Four primary sources of data are collected during an assessment:
- Questionnaires Usually administered prior to the onsite period to a small number of key individuals.
- Documents Documentation at the organizational level, project level, and implementation level is collected starting before the assessment begins and is added to and consolidated during the entire course of the assessment.
- Interviews Project, middle, and senior managers and practitioners (sometimes called functional area representatives, or FARs) are interviewed during the onsite period of the assessment.
- Presentations Project presentations given to acquaint the assessment team with the organizational work; draft finding presentations that validate the information heard and seen by the assessment team.
The assessment team makes notes based on each kind of data. The information from each source establishes an organization context, reveals areas that need probing, corroborates other data, provides information relative to appropriate practices in the model, and answers specific questions about the processes in use by the organization.
In an ongoing process of review, the assessment team consolidates its notes onto worksheets organized by specific practices in each of the model’s KPAs/PAs. The team discusses each observation created and arrives at consensus about the validity of the information.
Interviews
Interview questions are meant to help the team connect the dots in regard to how the organization actually implements its engineering and management processes. (People cannot describe a process in detail if they do not use it.) They also help identify areas that people believe can and should be improved in the organization, explain how work is performed, and aid the team in understanding relationships between organization- and project-level processes.
Interviews educate by explaining the reason behind processes that practitioners may only know about but not actually use.
Interviews are the most important component of an assessment, both for the purposes of analysis and cultural transformation. Employees should be encouraged to tell the team anything and everythingboth technical and culturalabout the organization. Interviewees are not to take notes in deference to the confidentiality requested for other interview participants. The assessment team takes copious notes so that subsequent discussions of what was said and heard can be recalled accurately.
During interview sessions, it should seem to all concerned that an interesting and useful discussion is taking place. Employees should be able to go away from the interview feeling that they have learned something new and useful.
Consolidating Data from Interviews and Other Sources
An assessment team consolidates data at the end of each day’s interviewing. This activity consumes by conservative estimate approximately one-third of the onsite period. (Any way that teams can automate this process while retaining the accuracy and team consensus is encouraged.) The assessment team members are dealing with a large amount of information from many different sources. Their job is to synthesize this information into precise statements that can be agreed to by the assessment team as a whole.
The three primary objectives of consolidation are to summarize the information obtained during the interviewing and document reviews and consolidate it into a manageable summary, to determine whether there is enough data for the team to make a rating judgment, and to determine if further revisions are needed to the schedule to get more information.
"Observations" are created that represent the assessment team members’ understanding of information either seen or heard during the data collection activities. An observation is an assertion about a practice implementation (or lack thereof) developed by team members and agreed upon by the entire team using the consensus process. An observation addresses some aspect of the extent to which a practice is or is not implemented. An observation may be unrelated to the reference model, but it identifies an issue that has a significant impact on process capability or organizational maturity.
The assessment team is responsible for collecting data for each practice for each process area within the assessment scope. Observations must be corroborated by multiple independent sources. The set of observations must, in the judgment of the assessment team, provide sufficient coverage of the extent of implementation of each of the practices, be considered representative of the organization being assessed, and represent the life-cycle phases in use within the organization.
Observations do not record any project name or individual’s name in order to preserve confidentiality, which is crucial to the assessment method.