Assessment Center Technology and Techniques
Whenever a large organization undergoes a change in executive leadership, an excellent opportunity presents itself to take stock of existing human resources as a basis from which to understand and maximize the operations of the organization. Precisely such a situation arose when Minister Luis R. Villafuerte was called upon to head the Ministry of Trade in July of 1979 and to reorganize this Ministry for maximum productivity. As an initial step for assessing his human resources leading to reorganization, Minister Villafuerte contracted the services of Development Dimensions (ASEAN) International (DDI) to run assessment centers for division chiefs of all bureaus and agencies within the Ministry and for all commercial attaches throughout the world. It is the purpose of this brief presentation to summarize the Ministry experience in the use of the assessment center technology.
I
The methodology of employing the assessment center technology is one that is already familiar to those who have used this and those who have used the simplification of this technology in the form of targeted selection system. A quick summary of the process might nevertheless be helpful.
Step One: Identify the Target Audience. It is obviously important to start with a very clear idea of the clientele which is to be assessed. It is equally important to note that the target audience here is not so much a group of people as it is a group of functions or equivalent functions at a clearly identified target managerial level. The Minister had identified division chiefs and commercial attaches as the two critical groups upon which he would rely and for which he needed immediate assessments.
Step Two: Do a Needs Analysis: This analysis, also called the "what counts" study, is the crystallization of which specific sets of skills or dimensions are most critical for successful and acceptable performance in that type of job and at that job level for which the program is designed. The task of isolating and defining key elements of success in terms of skill dimensions is an entire process by itself and requires inputs from several sources and the use of several tools. A random sample of incumbents of the given target level is chosen and selected incumbents are interviewed about what they think contributes most to their effectiveness. Validating interviews likewise take place with their subordinates and with their superiors one and two levels above them. The chief executive is himself asked to paint a picture of his ideal performer at the job. To acquire as thorough a grasp of the job requirements as possible, questionnaires, job description content analysis, and other techniques are also used.
Step Three: Choose and Define the Critical Skill Dimensions. Following the needs analysis process, the inputs are methodically categorized, ranked, tabulated, prioritized, and evaluated until a relatively clear list of which dimensions matter most emerges from the study. Such a list might include, but would not be limited to some of the following, for example:
Problem identification Delegation
Information gathering Initiative
Methodological research Oral communication
Problem analysis Written communication
Risk diagnosis Motivation
Planning Public Relations
Creativity Technical natural systems
Decision making (judgement) expertise
Organizing
Leadership
Sensitivity
The needs analysis for a given job level skill isolate which ones are to be included among the critical list for evaluation. The next step is to arrive at a clearly expressed consensus definition for each dimension, not only for the sake of a working common understanding among the evaluating team, but also that the concepts describing each dimension do not overlap one with the other. It is precisely the objective of the assessment center to avoid the "halo effect" of too hastily judging as good (or as poor) the less obvious skills of a manager on the basis of his good (or poor) performance in the more obvious skills; therefore, skill definitions must be discrete as much as possible, to enable assessors to make such precise statements as, say, "He is good in analysis, but poor in judgement," or "He is good in initiative but poor in creativity."
In the Ministry of Trade experience, the needs analysis methodology produced two separate lists of skill dimensions, one list of 13 dimensions for division chiefs and another list of 11 dimensions for commercial attaches.
Step Four: Design Exercises and Measures of Evaluation. The next step, and perhaps the most difficult one, is that of designing the yardsticks with which to measure skill levels of given individuals for each of the dimensions in the finalized list. Fortunately, considerable work has taken place in this area, and it is here that the research and previous efforts of human resources evaluation and development specialists can be most helpful. Individuals with adequate professional training are able to look at the list of dimensions in particular and the job in general and from these design a series of structured experiences, or exercises, which can evoke observable and measurable manifestations of these dimensions. These exercises, many of them simulation based, are so designed in such a way that when an individual goes through them he would apply to these experiences the same approaches and skill levels that he would at the actual job, thus drawing a portrait of his managerial style according to the dimensions earlier identified.
Typically, a group of three to seven such structured experiences would be included in a design, since it would be impossible for any single exercise to adequately tap into all skill dimensions. A typical series of exercises might include:
A role-play interview simulation
A group "lump sum" type game
An in-depth interview structured by the assessor
An analytical case study/exercise
An in-basket exercise of correspondence for processing
A group creativity exercise
As earlier indicated, not all exercises tap into all dimensions; for example, only those exercises with written components can be used to judge an individual's written communication skills. Any single exercise will, however, probably tap into between three and eight dimensions, so that when the assessment clinic is run and all the exercises are conducted, a grid will show that for every skill dimension that is to be measured there will be at least two or three exercises that can provide behavioral evidence on the level of that skill. For example, the skill dimension of analysis will show itself in the in-basket exercise, in the analytical case study, and in the written report exercise; the leadership skill will show in the lump-sum game, in the creativity exercise, and in the interview simulations; and so on.
The important aspect of the design of the exercise is that measures for evaluating specific skills must be preliminarily built in. For example, it must be decided how many skills are going to be measured by the in-basket exercise, and which ones they are. And then for each skill, ways of measuring this must be built into the design. If, in the example of the in-basket exercise, the skill dimension of delegation of authority were to be measured, then several items of correspondence in the in-basket should be geared as items properly to be delegated in the simulation. This is done so that by analyzing the participant's responses to the items in the exercise, assessors can determine whether be delegates to the appropriate or to the inappropriate persons in the organization. The same would have to be said for the performance measures of the other skills (such as written communication, analysis, sensitivity, etc.) to be tapped by the in-basket exercise.
For instances, the completed basket of exercises will include a combination of standardized exercise of universal applicability, modified exercises with reference to natural systems settings, and brand new exercises designed specifically for the diagnostic clinic to be run. In any case, the use of outside expertise of consultancy generally accompanies this step.
Based on the two lists of dimensions drawn up, DDI in cooperation with DDI Pittsburg, designed, modified or created exercises for two types of assessment centers. The assessment center for division chiefs included an in-basket exercise of correspondence for processing, a group meeting type exercise and analysis exercise and a supervisory interview role play, together with an in-depth interview. The commercial attaches' assessment center included an analysis exercise with an oral presentation component, a marketing exercise, and an in-depth interview.
Step Four: Train Assessors and Calibrate the Criteria. Once the materials for the diagnostic clinic are completed, the next step would be to identify a group of assessors and prepare them thoroughly for their task. A number of alternatives exist in the choice of which individuals to choose to consist the team. Most corporations prefer to use their own line managers, rather than external consultants or even personnel staffs of their own corporation. Generally these line managers are from two levels above the participant group, and are therefore quite senior, especially when the assessment clinic is being conducted to identify individuals for promotion. Government bureaucracies, on the other hand, tend to favor an assessing team with a combination of external consultants and internal officials, or even an entirely external team, so project the image of objectivity and to shield the clinic from possible political influences. In any case, whether the assessors are internal or external, or a combination, and whether they are professional human resource specialists or not, an orientation and training session is an absolute necessity.
The objectives for such a training session are several. First, the purpose, nationale, and philosophy of the assessment center must be clearly understood by all. It must be clear, for example, that their task is not to be amateur psychologists trying to read motivations and personality into the participants' performance in the exercises, but merely to be recorders of outside observable, measurable behavior which can be reported during the assessor discussions and supported with specific evidence.
Second, the definitions of the skill dimensions must be accepted and understood by the assessors. This is done, not only by discussions if the skills themselves, but also experientially by identifying their manifestations in the exercises.
Third, everyone should have a working familiarity with the actual exercises and the measures of assessment built into them. This is of course discussing the procedure of gathering evidences via extensive notetaking in pre-designed forms for specific dimensions.
Fourth, and very important, the fine tuning of the calibration for the particular individuals of the particular diagnostic clinic must take place. As earlier mentioned, a some exercises will be of rather general applicability, and even when designers of these build in measures for specific skills, the assessors must decide and define what is acceptable for their particular group.
Thus for example, when evaluating for a mid-level manager target level, three errors out of a possible eight in the delegation skill in the in-basket exercise may still be acceptable, but when evaluating senior level officials (should delegation be on their critical list and should the same in-basket exercise be used) three errors out of eight may not be acceptable. This type of calibration is determined, not by the designers of the exercise, but by the assessors themselves, since they generally have a better feel for what is acceptable at the target job level in question.
If the materials are properly packaged and the assessors correctly chosen, the entire training session should not last more than three days. The first day would typically be spent discussing the philosophy, and running mini-exercises on recording observable behavior. The second and third days would be taken up with running the exercises and discussing the definitions of the dimensions and reaching consensus on the calibrated acceptable levels of each skill dimension in each exercise.
Step Six: Run the Assessment Center . The actual assessment center should ideally follow immediately after the assessor's training session. In fact, it is commonly held at the same live-in venue, so that the assessors do not have to be pulled out of work two separate times.
A typical schedule would process participants at an assessment center in groups of six. The assessing team, consisting of one administrator and three assessors, would stay with this group of six and run them through the exercises by two assessors, although only one would assume prime responsibility for note taking. If the administrator handles the scheduling correctly (simultaneous sessions in different cubicles could take place on different exercises), then the assessing team should have evidences from each participant for each skill from at least two exercises as recorded in notes by at least two assessors.
Where more than six participants are to be assessed, the assessment center can handle as many sub-groups of six as there are additional administrator and three assessors. In all cases, experience has shown that because of the tightness of the schedules and the confidentially of the materials, it is almost necessary that this be an uninterrupted experience at a live-in workshop site.
Step Seven: Have Assessors Recast Notes, Conduct Discussions, and Finalize Write-ups. After the assessment center proper, participants go home but assessors stay on. Two extra days are needed by them to recast their copious notes into the frameworks and formats that they will bring to the assessors' discussions. During these two days they stay in their rooms and assemble the evidences per exercise per candidate for those assigned to him, and arrive at a preliminary individual decision on a rating of much less that acceptable, less than acceptable, acceptable, more than acceptable, superior, or undecided on the basis of insufficient evidence gathered during the clinic. The administrator goes around among his three assessors to supervise the notes recasting and the preliminary judgements.
Starting the third day, and probably lasting until the fifth day, the three assessors and the administrator get together for discussions of their findings. Each participant is taken up separately, and evidences for each skill are brought out and discussed individually, with preliminary individual ratings brought out. Each participant will probably require about three hours worth of discussion. As each skill is covered and all the evidences are reported, individual ratings (there would be at least two per skill) will be defended or be changed after extensive discussions and even arguments until the group achieves consensus on a group rating for that skill. Voting or averaging is not permitted as an easy substitute for consensus on the basis of evidence and agreed upon calibration of criteria.
At the end of the discussions, the administrator is usually charged with the task of writing up the consensus results for each participant. A typical summary write-ups is not three pages long, and merely identifies (after a standard introduction on the nature of the assessment center) the individual's skills in which he is more than acceptable of outstanding, the skills in which he needs further training to become acceptable, and the skill where he is acceptable. A typical item is this write-up which reads as follows:
In the following skills, the participants was rated as the assessors as more than acceptable:
Sensitivity. The participant showed evidence in a number of exercises of an ability to adjust appropriately to the feeling of others, and to maintain his effectiveness in delicate emotional settings. In the lump sum game, for example, he called for opinion of the quietest member of the group on at least three occasions. His interview simulation was started with questions about the well-being of the interviewee and his family.
During the in-depth interview, he reported on improving his division's morale by introducing rewards in productivity as one of this three greatest achievements last year.
During the in-depth interview, he reported on improving his division's morale by introducing rewards in productivity as one of this three greatest achievements last year.
Analysis. In several exercises, but particularly in the exercise on the analysis of the waterway project, the candidate manifested more than adequate skills at relating pertinent information, establishing casual links, and arriving at a thorough grasp of the factors that most affected the project.
He showed.
In preparing this write-up, of course, the administrator has to be both guided and limited by the group consensus.
II
The results of the assessment center within the Ministry of Trade were prepared in duplicate, one for the participant assessed and on for the Minister himself. It soon became obvious that this written outputs for each participant were useful for a wide variety of purposes.
First and most importantly, the results were key inputs together with such other inputs as actual performance analysis, seniority, and civil service eligibility for identifying the most appropriate individuals to head created or modified units within the reorganization. For commercial attaches, information was also useful in the consideration of the next round of transfers and cross posting that were to take place. It become apparent for example, that attaches whose strengths were marketing and entrepreneurship should be posted in commercial capitals such as New York and Frankfurt, whereas attaches whose strengths were in analysis, research and negotiations were more appropriate in political and negotiating capitals such as Washington, Bonn and Geneva.
The results were also useful for identifying training needs of individuals or sub-groups. It is easy to tabulate and identify which division chiefs are weak in planning and organizing, in order to provide them a training module or to give them a self paced instructional program. This is also useful for attaches or that they can either band in small groups or undertake individual training programs for their specific weaknesses, and thus avoid the expensive and often wasteful wholesale recalling of attaches worldwide to some central training center on the false assumption that everyone needs all aspects of a general training program.
In looking for individuals for promotion or selection, again the results of the assessment center are useful. The assumption is that the best person for a particular job must be found. Traditional test have not always produced reliable results; interviewing has been hit-or-miss depending on the skill of the interviewer or the sincerity of the interviewee; past performance records reflect success at a different job whose skill requirements are probably quite different from those that new job requires.
The assessment center is thus used to put the candidate into a series of structured experiences that simulates similar settings to the new job. In this manner the choice is made on the basis of results that have a far greater degree of predictable validity.
III
In general therefore, the assessment center results have been useful, not only to the Minister in the course of exercising his human resources executive responsibilities but also to the individual division chiefs and commercial attaches who have been given an objectives and accurate picture and their strengths and weaknesses as basis for self improvement and maximizing their productivity.
