How interviewers score behavioural answers

Two interviewers can hear the same story and write very different notes: one “good communicator, strong 4”, the other the three things the candidate actually did, scored 2. A structured interview exists to make the second kind of note the rule, because the decision is taken later, from what was written, sometimes by people who never heard the answer.

Below are the path from an answer to a decision, how an anchored rating scale makes a score mean the same to every interviewer, one story scored sentence by sentence on a five-part practice rubric, what raises and lowers a rating, and why polish counts for less than evidence. At the end you score three short answers yourself and compare your scores with the reference.

An answer becomes notes, then a rating, then a decision

Take a candidate who has just told a story about a disagreement over a release date. Nobody records the conversation; what survives it is what the interviewer wrote down. Step through what happens to the answer next:

Every stage in that path is a choice an employer makes. A survey of interview research by Campion, Palmer and Campion identified fifteen components of structure, some about the content of the interview and some about how answers are evaluated, and the stages above are several of them at work. Three consequences follow for you:

• Only what is written reaches the decision. Meta’s interview guide says interviewers can assess you only on “what you show them during your interview”. A strength you have but did not show is, for the panel, a gap.
• Each interviewer scores alone. An interviewer cannot borrow a result you gave someone else, so every round needs the whole story.
• The debriefdebrief: The meeting after a set of interviews in which the interviewers read their feedback together and talk through the differences between their ratings before a decision is made. compares notes. If one interviewer writes down that your fix halved the failures and another writes three quarters, the notes stop agreeing the moment they are read together.

Anchored scales make a score mean the same to every interviewer

Without a scale, “3 out of 5” means whatever the interviewer thinks a 3 is. A behaviourally anchored rating scalebehaviourally anchored rating scale: A rating scale whose points are tied to descriptions or examples of behaviour, called anchors, so that the same evidence gets the same score from every interviewer. fixes that by tying each point to a description of behaviour, called its anchor. The US Office of Personnel Management (OPM) publishes an example. Its competency is interpersonal skills, its lead question is “Describe a situation in which you dealt with individuals who were difficult, hostile, or distressed”, and its probes ask who was involved, what specific actions the candidate took and what the outcome was. Here is its rating scale, redrawn:

Read the benchmark column: the anchors rise with how hard the interpersonal situation was, not with how well the candidate talks about it. Cooperative, courteous behaviour earns a 1 when nothing was at stake and a 5 when the meeting was emotionally charged. That has a direct consequence for you: the story you choose sets the highest score it can reach, before you say a word of it.

Google’s guide to structured interviewing says researchers call such scales behaviourally anchored rating scales, that Google calls them rubrics, and that Google often describes, with examples, the content of a poor, a borderline, a solid and an outstanding answer. Employers keep their own anchors to themselves, so this track practises with a rubricrubric: A written scoring guide that says, for each thing being assessed, what an answer at each score contains. Google uses the word for its anchored rating scales. of its own, written by MySmartCoPilot, which describes no employer’s internal rubric. It scores five parts of a story from 1 to 4, and a 1 means the answer left no evidence for that part:

One story, scored sentence by sentence

Rohan, an invented mobile developer, answers “Tell me about a time you disagreed with a decision at work” like this:

“We had a release ready for a big sale weekend, but the new checkout was crashing on some older Android phones. The product manager wanted to ship anyway because the sale started on Friday, and I disagreed. We looked into it and decided to switch the new checkout off for those phones. I wrote the switch, and our QA lead tested it. We shipped on Friday and the sale went fine. I think it went well because I stood my ground.”

Step through how an interviewer reads it, one part of the rubric at a time. Each step quotes the sentence that decides the score:

The scores read 3, 3, 3, 2 and 2: 13 of 20, a decent story whose ending gives the interviewer almost nothing. Three sentences change that, each one adding evidence rather than polish:

• “I pulled the order data: those phones had placed about 9 % of orders on the last sale weekend, so I proposed shipping on time with the old checkout for them.” Actions rise from 3 to 4: a decision of his own, with its reason.
• “Over the weekend there were no checkout crashes on those phones, and they placed about 8 % of the sale’s orders through the fallback I wrote.” Result rises from 2 to 4: measured, with his part in it clear.
• “Since then I bring the data before I argue about a release, as I did again a few months later for a change to payments.” Reflection rises from 2 to 4: a habit he changed and has used again.

With them the story scores 18 of 20. The situation and the difficulty stay at 3, because a disagreement with one product manager over one release is a real obstacle but not a wide one; no sentence can raise those two scores without a harder story.

What raises a rating, and what lowers it

Every rubric in this lesson rewards the same kinds of evidence and marks down the same gaps:

Two of these rows come straight from employers’ guides. Google’s interview tips say its behavioural questions look for specific examples rather than “I always do X”. Meta’s guide says that if you go into a theoretical tangent, the interviewer will redirect you to a concrete example. The same guide asks candidates to be honest, since not every project succeeds, and says that being transparent about such a project won’t be counted against you, while what you learned from it is valued.

A part with no evidence is not filled in with a guess. If your story never says how it ended, the interviewer cannot assume it ended well: the notes say nothing about the result, and the result scores 1.

Polish counts for less than evidence

Candidates often worry most about how they come across: their clothes, their confidence, the small talk at the start. Those things do move ratings, and research finds they move them most where interviews have least structure. A meta-analysis by Barrick, Shaffer and DeGrassi looked at three self-presentation tactics, appearance, impression managementimpression management: What a candidate does to be seen favourably, such as self-promotion or flattery, as distinct from the job-related evidence in an answer. and verbal and nonverbal behaviour, and found that they influence interviewer ratings, that unstructured interviews are particularly affected, and that what an interviewer sees in the interview may not be what they get on the job.

First impressions also fade as a structured interview goes on. Swider, Barrick and Harris studied 163 mock interviews that began with a short rapport-building conversation. The impressions formed in those minutes were related more strongly to the ratings of the early answers than to those of later ones, and a good part of what drove them was job-related content, not charm alone:

So start well, with a greeting, a calm first minute and your first story ready, and then let evidence carry the rest. In a structured interview the notes decide, and the notes are made of what you did.

Practise: score three answers yourself

Score each answer on the five parts of the practice rubric, from 1 to 4, before you read on. All three answer the same prompt, “Tell me about a time you had to learn something new quickly”, and all three speakers are invented:

• A, Meera: “Our team moved its alerts to a new monitoring tool, and I had a week to learn it before my first on-call shift with it. I read its documentation each morning, rebuilt two of our old alerts in it, and asked the engineer who had set it up to check them. On my shift, one of my alerts caught a disk filling up, and I cleared it before users noticed anything.”
• B, Kunal: “I’m honestly a fast learner. Whenever there’s a new technology, I dive straight in, watch tutorials, and I’m productive within days. At my last job we adopted Kubernetes and I picked it up really quickly, and the migration was a big success. My manager was very happy with how it went.”
• C, Ananya: “In my final-year project, our team chose Flutter for the app, which none of us knew, with six weeks to the demo. I spent the first week building a small to-do app and wrote the others a page on the three things that had tripped me up. Two teammates used the page, and we finished all the app’s screens by week four. The demo worked, but the app was slow on older phones because I never tested on one. Now I test on the oldest phone I can borrow from the first week.”

Then press Edit, put your scores in MY_SCORES and run the program. It compares each score with the reference and quotes the evidence wherever you are two points or more away. As it stands it holds one learner’s first attempt:

// Calibration: score the lesson's three transcripts on the five-part practice rubric, then compare your scores with
// the reference scores and the evidence behind each. Replace MY_SCORES with your own (1 to 4 for every part), and run
// it again. A score within one point of the reference counts as calibrated.
//
// The prompt of all three: "Tell me about a time you had to learn something new quickly." Meera, Kunal and Ananya
// are invented, and so are their answers.

const PARTS = ['situation', 'actions', 'difficulty', 'result', 'reflection'];

// The reference score of every part, with the words of the transcript that decide it.
const REFERENCE = {
  'A (Meera)': [
    [3, 'one real situation, clear: a week to learn the tool before her first on-call shift with it'],
    [3, 'her own steps in order: read the documentation, rebuilt two alerts, asked for a check'],
    [2, 'routine learning with a week to do it, and little at stake'],
    [3, 'a visible outcome: her alert caught a disk filling up before users noticed'],
    [1, 'no reflection at all'],
  ],
  'B (Kunal)': [
    [2, 'named but vague: "we adopted Kubernetes", with no time, team or goal'],
    [1, 'no action of his own in that situation: diving in and watching tutorials is a habit'],
    [1, 'no obstacle and nothing at stake is described'],
    [2, 'an outcome without evidence: "a big success", "my manager was very happy"'],
    [1, 'no reflection at all'],
  ],
  'C (Ananya)': [
    [4, 'clear, with what was at stake: nobody knew Flutter and the demo was six weeks away'],
    [3, 'her own steps: a small practice app first, then a page for the others'],
    [3, 'a real obstacle for the whole team, under a deadline'],
    [3, 'an outcome with numbers: two teammates used the page, all screens done by week four'],
    [4, 'a changed habit she keeps: she now tests on the oldest phone she can borrow'],
  ],
};

// Your scores, in the order of PARTS. These are one learner's first attempt.
const MY_SCORES = {
  'A (Meera)': [3, 3, 2, 3, 2],
  'B (Kunal)': [3, 3, 2, 3, 2],
  'C (Ananya)': [3, 3, 3, 2, 4],
};

let within = 0;
let total = 0;
const allAbove = [];
for (const [name, reference] of Object.entries(REFERENCE)) {
  const mine = MY_SCORES[name];
  console.log(`Transcript ${name}`);
  let mySum = 0;
  let refSum = 0;
  let above = 0;
  PARTS.forEach((part, i) => {
    const [ref, evidence] = reference[i];
    const diff = mine[i] - ref;
    mySum += mine[i];
    refSum += ref;
    total += 1;
    if (Math.abs(diff) <= 1) within += 1;
    if (diff > 0) above += 1;
    const verdict = diff === 0 ? 'same' : `${Math.abs(diff)} ${diff > 0 ? 'above' : 'below'}`;
    const line = `  ${part.padEnd(11)} you ${mine[i]}  reference ${ref}  ${verdict}`;
    console.log(Math.abs(diff) >= 2 ? `${line}  <- ${evidence}` : line);
  });
  console.log(`  total: you ${mySum} of 20, reference ${refSum} of 20`);
  if (above === PARTS.length) allAbove.push(name);
  console.log('');
}
console.log(`${within} of ${total} scores within one point of the reference`);
for (const name of allAbove) {
  console.log(`Every part of ${name} scored above the reference: check whether fluency, not evidence, set those scores`);
}

Transcript A (Meera)
  situation   you 3  reference 3  same
  actions     you 3  reference 3  same
  difficulty  you 2  reference 2  same
  result      you 3  reference 3  same
  reflection  you 2  reference 1  1 above
  total: you 13 of 20, reference 12 of 20

Transcript B (Kunal)
  situation   you 3  reference 2  1 above
  actions     you 3  reference 1  2 above  <- no action of his own in that situation: diving in and watching tutorials is a habit
  difficulty  you 2  reference 1  1 above
  result      you 3  reference 2  1 above
  reflection  you 2  reference 1  1 above
  total: you 13 of 20, reference 7 of 20

Transcript C (Ananya)
  situation   you 3  reference 4  1 below
  actions     you 3  reference 3  same
  difficulty  you 3  reference 3  same
  result      you 2  reference 3  1 below
  reflection  you 4  reference 4  same
  total: you 15 of 20, reference 17 of 20

14 of 15 scores within one point of the reference
Every part of B (Kunal) scored above the reference: check whether fluency, not evidence, set those scores

That first attempt was within one point on 14 of the 15 scores, yet it scored every part of Kunal’s answer above the reference, 13 against 7. His answer is the most fluent of the three and has the least in it: no action of his own, no obstacle and no evidence. If you scored him high as well, read his answer again and ask of each sentence what an interviewer could write down.

Then test yourself on the whole lesson. The questions come one at a time, the weakest idea first:

Interview questions

What is a behaviourally anchored rating scale, and why do interviewers use one? A warm-up for every level.

It is a rating scale whose points are tied to descriptions or examples of behaviour, called anchors, so that a score means the same to every interviewer. Instead of “rate communication from 1 to 5”, each point says what an answer at that level contains. In a public example from the US Office of Personnel Management, the anchors rise with how much tension the candidate handled, from little or none at 1 to a high degree at 5. Interviewers use such scales because they make ratings consistent across interviewers and candidates, which is part of what makes a structured interview fair and predictive; Google calls them rubrics. For you, the point is that a score comes from evidence matched to an anchor, so a harder situation, told with specifics, can reach a higher one.

Then answer these two aloud, one with a story of your own and one as the person who runs the interviews, before you look at their first steps:

Key takeaways

• An answer becomes notes, a rating against anchors, feedback written alone and a debrief, and the decision rests on what was written.
• An anchored rating scale ties each score to a description of behaviour, and in OPM’s example the anchors rise with how hard the situation was, so the story you choose sets its ceiling.
• On the practice rubric a story scores five parts, and the cheapest gains are usually a measured result and a reflection that changed what you do.
• Specific situations, your own decisions, difficulty, evidence and reflection raise a rating; habits, “we”, theory, blame and missing endings lower it.
• Polish moves ratings most in unstructured interviews, and first impressions fade after the early answers, so start well and let evidence carry the rest.