In the past, software process assessments always began with the distribution of a maturity questionnaire. Recognizing their deficiencies, today’s assessment methodologies use them not as the basis of the assessment but rather primarily to identify issues to be explored further during the onsite period and to be corroborated through document review and interviews. (In SCAMPI, questionnaires are optional. If a SCAMPI questionnaire is to be used, that fact needs to be identified in the SCAMPI Data Collection Plan.)
The CBA IPI maturity questionnaire is required to be distributed to participants before the start of an assessment . As it includes a glossary of terms and KPA descriptions, the questionnaire partly serves as an aide to CMM terminology. It covers all 18 KPAs of the CMM, addressing each KPA goal in the CMM but not all of the key practices. Limited to six to eight questions per KPA, the questionnaire can usually be completed in one hour.
The use of questionnaires to gather written information from members of the organization represents a relatively low-cost way to collect data when done well. Questionnaires typically require yes or no answers. (Respondents are given an opportunity to comment if they wish.)
Questionnaire data, however, may be problematic. Because the questionnaire has specific CMM language in it, those who fill out the questionnaires sometimes don’t understand the questions. The questionnaire therefore needs to be administered by someone familiar with the model in case explanations are needed.
When the team evaluates questionnaire responses about whether a particular process is or is not present, a determination needs to be made as to whether the respondent has understood what he has been asked. The team should also consider whether the respondent has answered yes to something merely because he thinks that a loyal response is better than a truthful one.
It is the task of the organization site coordinator to summarize the responses to a given question on the questionnaire. Such summaries give the team a quick overview of what respondents think are the company’s strengths and weaknesses, but they should be treated by the team with some skepticism because of the difficulties just described. Answers to questionnaires are most useful in preparing interview questions about specific technical areas.
The widespread distribution of questionnaires may be more trouble than it is worth. It is recommended that the questionnaire be filled out by four to ten peopleideally by each participating project manager. In addition, key technical personnel or functional managers for the organization may be asked to fill out the questionnaire as well.
Whenever possible, the organization site coordinator should collect the documents mentioned in questionnaire responses so that the team can review them early in the process.