Understanding your organisation ensures your actions make the most effective and relevant impact. Whether it’s a major change project or an introduction to a new benefits packages, your efforts can only be directed and harnessed efficiently if you know what’s going on outside your HR bubble, and one way businesses do this is by using surveys – but how much can we really rely on surveys?

In essence, we ask specific questions to all staff so that the responses are tallied in a clear and consistent way. We analyse the results so to create some sort of dataset that we can use as a business case to senior management, providing them the cold, hard MI to back up our decisions.

People are prone to alter their normal behaviour in an environment where their behaviour is being studied, like in a survey. 

Is this too clinical and quantitative, however? We are using data collecting methods that rely on providing facts – information that cannot be questioned and can easily be analysed quantitatively. We’re asking this of people, the strange, quirky and emotional animals that we are, each with irrational thoughts, unpredictable behaviours and (sometimes) unfounded perceptions.

These factors do not, and cannot provide irrefutable evidence to form most emotion-derived decisions; they provide us with anecdotal evidence and suggested ideas to focus our efforts, but as the answers require emotive responses, we need to be offering better evidence-gathering solutions that take emotions into account.

We overlook ‘response biases’ in survey results; biases that influence how people respond to surveys that jeopardise the data, whether deliberately or not, ultimately questioning the integrity of the evidence. Before you conduct your next survey, consider these factors that may affect its integrity.

Not everyone responds and therefore it cannot tell the whole story

How can we get a clear picture of what the collective workforce think and feel if a portion doesn’t respond? Any less than a 100% response rate means it isn’t the collective workforce but the opinions of a non-select and random sample. This fraction is not pre-identified as being a core and reflective representation of the workforce, and therefore cannot be assumed that the thoughts of a few reflect those of the entire workforce.

This isn’t to say these should be disregarded – any feedback is insight for our work – but the weight we apply to these responses should be proportionate to the actual response rate. There’s also a risk of already underrepresented staff not responding (‘undercoverage bias’) if they are within the non-responding group.

People have moods

It is a psychological and physiological fact that people have good and bad moods. Moods make us act in ways that severely alter the way we think, how we perceive the outside world, and how we treat those around us. The smallest of triggers can set these moods off and in turn could jeopardise the integrity of the survey if they choose to respond in this mood.

Without predicting these moods – most are short-lived – or understanding the level of emotional self-awareness each employee has to cope with them, we cannot determine the best time that most accurately records long-term, undistorted opinion.

People have a score to settle

People may have an unjustified point to make if they’re disgruntled. They may do this by providing hyperbolic answers (known as ‘extreme responding’), answers that are the extreme reflection of what they feel, whether or not they feel it in the extreme.

We cannot deny that feelings and perceptions can affect how we experience reality and our objectivity. Couple this with the lack of understanding of certain terminology, and HR risk misreading a situation.

For example, when an employee is performing poorly and, despite the manager’s best efforts to support them, they continue to lack the required performance and it results in a written warning, the employee will want their feelings known through the survey. Their response to what is essentially a fair and transparent management process could be one of vengeance and spite, and by marking poorly on all answers relating to management, or even the company in general, they feel a sense of justice by showing everyone how they currently perceive the organisation.

Are we to disregard their response because of this? Absolutely not, but nevertheless we cannot deny the results have lost integrity.

People experience demand characteristics

People are prone to alter their normal behaviour in an environment where their behaviour is being studied, like in a survey. This is called ‘demand characteristics’ which influence the results of surveys, as participants consciously choose answers they feel are expected of them. Or they may even choose to consciously provide unexpected answers, just to scupper the results, a common occurrence in research practices.

There may be many reasons why they do this (or none whatsoever) but it alters the results and thus jeopardises the survey’s integrity. It could be that they are particularly biased against change; if they make an assumption based on first impressions of a new annual leave booking system and they dislike it, they may make a more concerted effort to find faults with it to confirm their own biases, and reflect this in a user questionnaire.

People aren’t objective

We cannot deny that feelings and perceptions can affect how we experience reality and our objectivity. Couple this with the lack of understanding of certain terminology, and HR risk misreading a situation.

Signs of harassment and discrimination are extremely important for us to identify and immediately address, as no one should be subject to these in the workplace. What ‘discrimination’ is to HR may be different to a comparative layperson, however. For example, if an employee is denied enrolment to particular training course because it’s not in line with their immediate or future roles, they may feel they have been discriminated against.

Although this perceived ill treatment is applied to the rest of their team for the same reason, and therefore nothing to do with any protected characteristic the individual may or may not have, they could use the term ‘discrimination’ superficially in a survey without understanding what it means in legal terms.

I have seen this happen a number of times, as many peers have too. Still, follow up every accusation where possible (for example if you know who raised it or the area within which it was raised) as any form of bullying, harassment or discrimination cannot be tolerated and therefore any accusation cannot be disregarded.

[cm_form form_id=’cm_65a14c3f5da64′]

The unpredictable nature of human behaviour is just one of the many reasons our profession is so interesting, and we really should embrace who we all are as individuals. We risk losing this if we are to understand our people quantitatively, rather than through observation or real engagement.

Feedback surveys are a quick and convenient way to gather information but we need to be aware of the risks and the things we miss out on in exchange of this convenience. And even without biases, we can only rely on the responses if our organisation already offers an open and no-blame culture – without this, how much are employees not telling us?

If you do decide to go ahead with them, make sure these considerations are accounted for by offering more opportunities to expand on answers (‘explain why you feel discriminated against’), asking much more specific questions that are based on factual incidents, not immeasurable emotion (‘how often has your manager denied you training?’) and follow surveys up with more observational evidence (focus groups, interviews, group activities and real-life shadowing).

Interested in this topic? Read Five communication tips to give engagement surveys a happy end.