The senior executive should start with a sense of the business goals to be addressed by the assessment and its follow-on process improvement plan. What comes out of an assessment will have a direct bearing on the organization’s system/software management, development, and maintenance activities by reducing the cost of developing and maintaining software product; improving quality, cost, and time to market of software products; and by improving the timeliness and reliability of software development services.
Organizational Scope (Choosing a Division or Cross-Section of the Company to Be Assessed)
Assessments look at the managerial and technical capabilities of companies by examining the way the whole interacts with a chosen sample of divisions or parts of the company. The sum of these parts is called the "organization being assessed." It is necessary to choose as "the organization being assessed" parts of the company that are both representative and critical to the company’s success. This is done by selecting a sample of projects that represent the typical type of work being done in the organization.
Choosing the projects to comprise the "organization being assessed" is of course a big decision. The obvious factors that have to be considered are management structure, geographic location (if the organization is in more than one physical location), and product lines being developed.
It is also important to identify the part of the organization that is at the center of the process improvement initiative. This usually means a group of managers who are genuinely interested in allocating resources to process improvement (and to an assessment).
In identifying the assessment scope, both the breadth and depth of the organization should be considered.
The depth of the organization has to do with a selection of suitable size (usually four or five specific projects) in the organization’s business base. The breadth of the organization refers to a cross-section of practitioners who work across the organization.
Factors that are eventually considered in setting the organizational scope of the assessment include:
- The feasibility of including multiple geographic locations (generally no more than two locations are recommended).
- The feasibility of including multiple divisions/business units (Are the units committed to the development of common procedures?).
- The feasibility of including multiple product lines or process focuses (e.g., do they have equal impacts on the organization’s business goals?).
- Exclusion of outlier projects (e.g., projects that use less-mature processes and where improvements will have little cost benefit to the organization).
However, the company senior executive does not need to define all the details of this selection beforehand. They will be finalized in an assessment plan to be prepared by the organization site coordinator along with the Lead Assessor and then approved by the senior executive. The assessment plan will identify the full organizational scope of the assessment, the rationale for establishing the scope, and the risks associated with addressing it during the assessment process.
The Special Problem of Matrix Organizations
CMM/CMMI assessments are designed to evaluate the management structure of an entire organization as it is embedded in given projects of the organization. Special care needs to be taken, however, when an organization operates through a matrix structure: That is, when all of engineering (including software engineering) is in one division, program management is in another, contracts are yet in another, and so on. In such cases, it makes no sense to define the organization as being the software engineering directorate only. An assessment that looks only at this one area does not address the responsible structure (the president, managing director, and executive staff) or the real authority on the projectsthe individual project managers and program managers. With both the CMM and CMMI, it is clear that project management and senior management have specific roles to play, and an assessment cannot ignore them and pretend to offer reliable evidence about a company’s strengths and weaknesses.
On one assessment, Company A wanted to determine if it was at Level 3. The company was organized in a matrix. On previous assessments, they had always reviewed only the software division as the "organization." When the assessment was planned, the Lead Assessor explained to the president that the entire company needed to participate, with the president being the sponsor because he was the point where the authority for all groups merged. This allowed project managers, software engineers, executive VPs, and so on to participate. What was found in this assessment was that the software engineering directorate had been isolated, not only in previous assessments but also on projects. Although software engineering processes were theoretically in place, they were not actually functioning properly. The project managers were not involved; they knew little about the software portion of the project. When something failed, it was believed to be only the software engineers’ fault, not the project managers’. As a result of this assessment, the president saw the problem and directed that at every monthly program meeting, the project manager was to report to him on the status of the entire project and specifically on the status of the software. Within six months, the company’s quality culture had improved radically.
Companies who regard assessments as tests to be passed often prefer to consider one small and successful project as the organization. By having a single project assessed, the companies hope to demonstrate that they are better than they really are. But the culture of an organization is pervasive. Although some projects may be better than others, the same management or mismanagement decisions tend to drive the project managers and their project teams, and this culture can only be identified by sampling a broad range of projects.
Another problem with assessing a small group is that it is extremely difficult to preserve confidentiality during and after the assessment.
An example of a division that inappropriately tried to have one project represent the "organization" involved Company B, a very large company. The division at issue in Company B was made up of about 200 people, but it only wanted to have one pilot project (about 10 people) assessed. The Lead Assessor had to explain to the managing director of the division that this was not in the company’s best interest because one project can easily give a distorted view of a company or organization and also because it is nearly impossible to maintain confidentiality (one of the primary elements of an improvement effort) when only a few people are involved. The division, which was certain that the pilot project was superior to its other projects, disagreed for obvious reasons. They managed to get their way, and the assessment was almost undermined by confidentiality issues. Ironically, though, the methodology proved resilient enough that the distorting effect of good individuals did not entirely decide the rating. Even the pilot project suffered from institutional problems that affected the other projects to a greater degree, and the division, though it learned more than it wanted to about its procedures, did not "pass the test."
The Special Problem of Small Organizations
The question is frequently asked whether it is feasible to assess small organizations. Data collected by the SEI from Lead Assessors have reported that 22% of the organizations assessed have 20 to 50 software developers [Dunaway 01a]. These types of organizations can be assessed with relatively little difficulty.
For organizations with fewer than 20 software developers, however, the reference model is not ideal. The CMM and the CMMI focus on project-level management and development practices at Maturity Level 2. At Maturity Level 3, they ask organizations to consolidate their best practices from individual projects into an organization-wide set of practices. These issues have reduced relevance when there are no more than 20 people in an organization. (Having said this, it should also be said that a number of organizations with fewer than 20 software developers have reported that conducting CMM or CMMI assessments proved of real instructional benefit to them.)