More frequently than not, an organization being assessed is under considerable explicit or implicit pressure to do well. (A contract may be riding on the rating, or people may fear that the jobs of relevant managers are on the line.) Such circumstances make it tempting to "game" the assessment to get a higher rating than what the organization’s real capability warrants. A company’s long-term interests, though, are best served by facing whatever problems exist. (If no problems existed, there would be no reason for an assessment.) The company’s real, long-term interests can only be satisfied by honesty and openness in the assessment process. Achieving a culture of openness is in fact the first big step in improving the organization’s future possibilities. By contrast, a culture of secrecy and "gaming" forecloses the possibility of real improvement and can turn an assessment into a counter-productive waste of money.
Despite these facts, immature organizations (which by definition cannot see beyond short-term issues) will try to manage an assessment so that it produces a positive outcome, so it is crucial for everyone who has the organization’s survival at heart to anticipate the kinds of interference that can be exerted. For example:
- Preparing interviewees in mock interviews to talk about what should happen rather than what does happen.
- Grilling interviewees after their interviews to discover what they said so that damaging statements can be explained away.
- Trying to intimidate the assessment team by suggesting there is something lacking in the assessment process or by insisting that the organization is doing more than is being reported in the interviews.
- Encouraging the organization’s representatives on the assessment team to discount negative comments made during interviews.
- Defending the organization’s maturity in spite of the evidence.
No one likes to admit thinking about such thingsmuch less doing thembut such maneuvers are nearly universal in immature organizations. It is important to anticipate them and understand why they are counter-productive.
The Abuse of Mock Interviews
Especially during a first assessment, immature organizations attempt to spin things positively by staging mock interviews during which "the right answers" are rehearsed. This, however, undermines the way assessments motivate improvementby providing a forum for the free discussion of how people are doing their jobs, a process that can and should lead to an understanding by all concerned of the overall strengths and weaknesses of the work process.
A good assessment team will get interviewees to think about how they do their jobs and sometimes provoke people to understand why they do what they do for the first time. The point of interviews is not to rehearse the company’s written processes and procedures but rather to give people in an organization a good idea of which procedures are really used and how well actual practices work or don’t work. Being honest sometimes can mean a lower rating in the short term, but it provides the only sure basis to maintain long-term improvement, often by pointing out potentially useful processes that people are not usingbecause they mistrust them, because the processes need to be improved, or because the personnel need to be trained or retrained in the processes.
Assessment teams themselves can make things worse by rigidly adhering to a set of scripted questions, which can to an immature organization seem like an invitation to prepare a set of scripted answers. Team members should regard interview scripts only as jumping-off points for actual interviews. The best interviews are those that become a conversation between the assessment team and the people being interviewed.
During the first assessment of Organization X, a relatively inexperienced assessment team asked entirely scripted questions during the first three days of interviewing and seemed to get a promising picture of the organization. However, senior team members came to sense the absence of concrete examples of how the organization really worked. The team therefore decided not to follow the announced script during the next day’s interviews. It quickly emerged that earlier responses had described practices that the projects had never really implemented. Later the team learned that each person interviewed had spent at least 50 hours practicing with a consultant the organization had hired. (There were at least 50 people interviewed.) The organization was trying to achieve a Level 2 rating without actually implementing Level 2 practices. But the attempt at "gaming" the assessment was itself a sure indicator of an immature organization. The organization could have better spent the 2,500 wasted consultation hours in almost any other way. Its real desire was not to improve but ratherdesperatelynot to change.
Violating Confidentiality
An organization’s intentional or unintentional interference can go beyond preparing interviewees. In immature organizations, participants are sometimes instructed to answer questions in a narrow way and not to volunteer information. Managers in such organizations may even call in participants after an interview and ask them what transpired. Individuals are put in an impossible position if they know the boss is going to ask them what they said, and from that point on, all openness ceases, and the assessment becomes a waste of time. Process improvement activities are designed to fix the systemnot to blame the people. This is why the SEI made confidentiality and non-attribution one of the most important rules of an assessment.
Intimidation
Organizations also interfere in more obvious ways, including trying to intimidate the assessment team.
For instance, Company Y’s managing director (who questioned his employees after every interview) asked to see the Lead Assessor halfway through the assessment. He told the Lead Assessor that people were complaining about not getting a fair interview. He had been told, he said, of employees who had wanted to give the team more information but were cut off. The Lead Assessor responded that he would check into it but that there was no immediate need to worry because interviews were often truncated if the territory had been covered at other sessions. However, the manager hardly listened, and he grumbled that the assessment was being improperly conducted. It turned out that the managing director knew the organization was weak in crucial areas and that his questions were aimed at preparing the ground to argue that failure to be assessed at Level 2 was the assessment team’s fault and not the company’s.
Deliberate Obstruction
Organizations can also try to interfere through their representatives on the assessment team. The organization’s team members may consistently discount any problems raised in interviews or may violently disagree with the rest of the team during the consolidation period. Even in less extreme cases, it is natural for internal assessment team members to become protective. Sometimes a team member will not realize what he is doing until the team leader points it out to him, and only then will he adjust his behavior. If the situation does not improve, the team leader must ask the team member to withdraw from the assessment. Although this can be painful at the time (especially if it is a small team), retaining a recalcitrant team member is good neither for the team nor for the long-term prospects of the organization.
A team member from Organization Q was concerned that if the organization did not achieve a certain maturity level, both the managing director and the SEPG manager would lose their jobs. The team member agreed with each finding during the course of the assessment, but on the day that the team was to consolidate its findings, he realized that the organization was not going to achieve its desired maturity level. He told the team that under no circumstances would he give a maturity level less than what the organization expected. The Lead Assessor had to go to the managing director, who was forced to remind the team member that his obligation was to be objective.
Such cases smack in ordinary language of "cheating," but organizations that permit them only care about the maturity rating they will receive. Such interference clearly indicates that the organization’s personnel are convinced they have no need to improve.
Assessments are designed to provide the kind of shock that requires a company to look at itself honestly. After that shock, it becomes much easier to change a technical cultureto keep what works and improve what doesn’t.
How an Experienced Lead Assessor Should Handle Sensitive Situations
To have any possibility of resolving the problems just described, the Lead Assessor must be perceived as someone who will respect all parties, listen carefully, tell the truth, and uphold the assessment’s ground rules. This requires time and leads to the development of working relationships. Lead Assessors can demonstrate trustworthy behavior first and foremost by keeping confidences even on trivial issues. He or she should practice reflective listening: rephrasing what was said in the listener’s own words, sharing one’s own experience with similar concerns, and offering observations about the situation that demonstrate interest and compassion.
During assessment team training, if some team members obviously have a hidden agenda that will result in inaccurate assessment results, the Lead Assessor may have to speak with the sponsor and have the team member replaced with another person who can be more objective.