How Far in Advance to Begin Formulating the Assessment Plan
An assessment constitutes something of an ordeal. How could it not when it involves an outside analysis of one’s own work and an intrusion that disrupts normal business for anywhere from three weeks to three months? Because an assessment involves coordinating many people (for Level 3 assessments, as many as 75 people in the organization) and considerable resources, serious planning is an essential step.
On average, the organization should assume that it takes a minimum of three months from the time an assessment is scheduled to prepare and execute a workable plan so that the on-site assessment can be efficiently staged.
How (and by Whom) the Assessment Plan Is Written
In the first phase of assessment planning, the Lead Assessor studies the organization’s key strategic objectives and advises the assessment sponsor about the parameters of the assessment.
The Lead Assessor works first with the senior executive and then with the assessment sponsor to define the boundaries for the assessment plan. The Lead Assessor and senior management must have a clear understanding of the assessment context, its goals, and its objectives. They must agree on the scope, outputs, and any constraints on the assessment process (cost, schedule, and participants), and the Lead Assessor must be given written authorization to proceed with the assessment.
When all this is agreed upon, the Lead Assessor gives the organization site coordinator the general contents (template) of the assessment plan. The organization site coordinator fills in the details of the plan. The Lead Assessor reviews and revises as necessary.
General Features
The assessment plan is an organization’s map through an assessment. It is also an important way for the organization and Lead Assessor to document mutually agreed upon assumptions, establish and maintain sponsorship, track and report the performance of the assessment process, and reinforce commitments at key points in the assessment process.
The plan is continuously refined. Typically scope-related decisions, a planned list of assessment outputs, and an evaluation of resources are finalized early on, along with risk and risk-mitigation plans. Cost, schedule, and logistics are finalized later. By using the organization information found in the Organizational and Project Questionnaires and the Project Selection Matrix (see Creating an Assessment Team), the Lead Assessor can refine the scope of the assessment.
After their initial discussions, the Lead Assessor and the assessment sponsor meet to nail down the scope of the assessmentthat is, which parts of the organization are to be assessed, what maturity level of the organization is to be assessed, and the characteristics and mix of people who will make up the assessment team.
At its most basic, an assessment plan initially reflects: 1) which portion of the selected model will be investigated, and 2) the bounds of the organization for which the results can be considered valid (e.g., a project, a product line, an operating division, a business unit, or an entire global enterprise), and the rationale for selecting these projects. After these are determined, the assessment plan then proceeds to refine the scope of the assessment by prescribing more specific goalsprincipally the specific process areas (PAs) to be assessed.
The characteristics of an assessment team must be agreed upon (how many in all, how many from inside and how many from outside the organization, at what ranks, with what specialties, etc.), and assessment team members must be appointed and have their names and qualifications recorded in the plan. A schedule for when the assessment team must meet for training, presentations, review of questionnaires, data management, and so on also must be constructed.
The plan prescribes that prior to the beginning of the assessment, the Lead Assessor and all members of the assessment team will sign a confidentiality agreement. (It is also good policy to have the assessment sponsor sign the agreement.)
The plan must provide a list and rationale for all those to be interviewed (interviewing too few people, or not interviewing the right kind of people, may lead to major problems) and must provide a schedule for the interviews that are to be conducted. (A list should also be made of who else should be interviewed if people named on the first list should have to travel. The plan should anticipate the risks these decisions are likely to incur.)
Decisions must be made about which data collection, analysis, and validation approaches will be utilized, including supporting work aids and tools. This is an issue for the assessment team, whose members must decide what mechanisms will be used to control the data being collected. Which parts will be computerized and which parts will be manual? What combination of questionnaires, document reviews, interviews, and presentations will be used? What rules will be used to corroborate the findings, and how will the team validate the findings with the organization?
Besides indicating who should be involved in the assessment and at what stage, an assessment plan must specify when the major milestones for the assessment’s completion are due and what outputs from the assessment are needed by the organization to be able to plan further improvements.
The assessment sponsor and the Lead Assessor must approve the final assessment plan. Having the entire assessment team sign the plan is an even better arrangement because it helps team members buy into an assessment. Formal approval of the plan demonstrates executive and team commitment and ensures a mutual understanding of the assessment goals, constraints, outputs, responsibilities, and confidentiality.
The Assessment Plan: Identifying Confidence Factors and Risks in the Coming Assessment
One part of the assessment plan not yet discussed has to do with identifying risk and confidence factors beforehand for the coming assessment.
Defining confidence is a way of stating whether the assessment’s characterization of its representative projects can predict the capabilities of similar projects throughout the organization.
A rigorous assessment correlates with a high confidence factor. The quality of the assessment can be affected by:
- The adequacy of the composition of the assessment team, taking into consideration:
- Qualifications of individual team members
- Team size
- Potential biases of organization site team member(s)
The model coverage planned and achieved:
- The extent to which the data maps to the (key) process areas within the scope of the investigation
- The extent to which the data collected covers the organization’s defined life cycle activities
The organization coverage planned and achieved:
- The number of projects and the extent to which they are representative of the organization being assessed
- The cohesiveness of the appraised entity in terms of its management structure, geographic location, and product line
- The extent to which the participants are forthcoming concerning shortcomings in the organization processes
- The number of people interviewed and the extent to which they are representative of the organization being assessed
- The extent to which data and observations are corroborated by the selected participants
Use of data collection techniques planned and achieved:
- The combination of techniques used to obtain objective evidence (instruments, document review, interviews, presentations)
- The extent to which the data collected is corroborated through use of the selected techniques
Validation of findings:
- The rules of corroboration used in validating findings
- Validation of findings by the organization assessed
- Time spent (the amount of time spent per assessment)
Adherence to the assessment plan
The assessment plan should identify circumstances that could diminish confidence in the accuracy and completeness of the assessment results and should include prescriptions for mitigation in case risk factors begin to threaten the quality of the assessment. The assessment sponsor should be kept abreast of risk factors throughout the assessment. At the end of the assessment (either verbally or in the Final Report), an assessment confidence statement must be provided to the assessment sponsor.
While the Details of the Assessment Plan Are Being Finalized: A List of Interim Tasks
While the plan is being written and before its completion, the following tasks will be accomplished:
- Analyzing the project questionnaires
- Reviewing the organization site information packet
- Refining the scope of the assessment if necessary within the constraints established during the beginning of the assessment
- Identifying the goals for how projects and interviewees will be selected
- Selecting the projects and interviewees for the assessment
- Preparing a detailed assessment schedule
- Selecting assessment team members
- Identifying the list of documents for document review and coordination (objective evidence)
- Identifying logistics requirements, which included booking rooms, transportation, accommodations, and managers
- Preparing assessment participants’ briefing and opening briefing
The Assessment Plan: Managing Resources and Estimating Costs
Resource management is typically not included in the assessment plan document. A separate, more confidential plan must be constructed to provide estimates of the effort and time to be invested by the organization and the assessment team in preparation, including pre-onsite data collection and analysis and the on-site assessment. This plan is formulated by the assessment sponsor with the help of the organization site coordinator.
While planning, the organization develops and includes a top-level cost breakdown and schedule. General activities include estimating:
- The duration of key events as a basis for deriving a comprehensive schedule.
- The hours likely to be required of the people participating in the assessment.
- The costs associated with using facilities and equipment (as appropriate).
- The costs for incidentals (e.g., travel, lodging, meals) as appropriate.