Bob Clifford explains how to evaluate, improve and value your selection process to ensure your decisions are accurate and objective.

In a previous article, I discussed that it is only through conducting studies in individual organisations that the effectiveness of a particular selection system can be established.

The benefits of conducting a straightforward evaluation of your selection process include pinpointing the parts of the process which best predict job performance. Other parts can be adapted or removed. Also, the value of your selection process ('return on selection investment') can be measured, providing you with the hard facts you need to influence the business.

A structured selection process not only adds objectivity, and therefore accuracy, to selection decisions, but also makes evaluations an uncomplicated matter. For example, person specifications form the basis of selection. Assessment scores against these criteria form the basis not only of selection decisions, but also the evaluation exercise.

Step-by-step approach

1. Choose a role with at least 50 members of staff. Less than this number and the analyses are less robust.

2. Collect assessment data. In other words, collate scores from the original selection process in a spreadsheet, with one candidate per row and scores from different assessment methods in columns. For example, column A may have a score for competency one assessed during the interview; column B may have a score for competency two assessed during the interview; and other columns may have scores from other competencies assessed during the interview; scores for competencies assessed using other assessment methods; and scores from different scales from psychometric assessments

3. Collect performance data. This can be as simple as line manager ratings, using a single score on each individual's job performance. A scoring/rating guide – often called a 'behaviourally anchored rating scale' (BARS) – should be provided to ensure maximum objectivity. Performance data becomes another column in your spreadsheet.

4. Conduct statistical analyses:

• Calculate the mean score on each component of the process. Quality psychometric tests will have comparison groups (norms), so it is possible to compare the typical profile of your cohort with a relevant sample.
• Calculate the correlations between each component of each selection method and performance data. This will give an initial 'relationship' between the assessment score and job performance. This can be done manually, but there are packages which will calculate correlations at the push of a button, such as Microsoft Excel.

5. Calculate the return on selection investment. To illustrate this step, I have assumed that some type of psychometric assessment has been added to an existing selection process, which has improved selection accuracy. Figure 2 (below) is a worked example, based upon a number of assumptions:

• The value to the business of an excellent performer is 40 per cent more than an average one (it is generally accepted to be between 40 and 70 per cent).
• Salary = £40,000 pa.
• Incremental rise in predictive validity of 0.20. This is the crucial part: predictive validity is simply how well an assessment method predicts behaviour. It is the correlation between a score on a test (or interview or any other assessment method) and job performance, which is normally just a manager's rating (see step four). In this case, analysis has shown that the new selection process – with the testing stage added in – has increased predictive validity by 0.20.
• There is one post with five candidates left.
• The tests cost £50 per candidate.
• Estimated tenure of recruit is five years.
• The 'benefit/cost' ratio is 64 (in other words the return is 64 times the investment, which can be stated as a percentage return on investment of 6400 per cent).

Figure 2 (click image to view larger version)

6. Interpret the findings in light of the organisational context and report on them. For example, if the study shows that the best customer service advisers are more task-oriented than people-oriented, this may make sense if it is a high-volume, process-driven environment.

7. Use the findings to improve the selection process and to develop existing staff. For example, if certain personality traits are shown to predict the best performers, it is those that should be focused on during feedback interviews.

What results and benefits can I expect?

Simple studies that evaluate the effectiveness of existing (or new) selection processes for particular roles in particular organisations can be one of the most valuable, and straightforward, exercises an organisation can undertake. These 'local validation studies' can statistically demonstrate whether high scorers on the overall selection process, or just a particular part of it, predict the best performers.

This information can certainly be very useful for verifying, adapting and improving selection processes; but also can be equally beneficial for developing staff.

Interventions can be based upon the knowledge that high performers behave in a certain way, and areas for development can be addressed in the sure knowledge that they are correctly targeted. This dramatically increases the likelihood of a successful outcome.

It is for this reason that the web application behind my own business has been designed to enable us to work with organisations very easily to demonstrate which profiles are statistically most predictive of good job performance in different roles.

Respondents and line managers are set up online in seconds and invited by automated email to respectively complete a short questionnaire and rate respondents in a number of key job performance areas. Data is downloaded and analysis and reporting is then a straightforward matter.

Bob Clifford is managing director of the Plumbline Test Company. He is a psychometrician, with expertise in designing and validating psychometric instruments.