A Verge reporter sat down for a job interview recently and found themselves talking to a bot. No human on the other end. Just a conversational AI asking structured questions, analyzing responses, and making a determination about whether to advance the candidate. The experience felt uncanny, a little dehumanizing, and it generated predictable outrage in the Hacker News thread that followed.
But the discomfort most people feel about AI interviews tends to focus on the wrong thing. The problem isn’t primarily that it’s weird to talk to a bot. The problem is deeper and more structural: these systems are almost universally trained on historical hiring data, and that means every model encoding “what a good candidate looks like” is actually encoding whatever patterns the employer’s past hiring managers happened to prefer.
How These Systems Actually Work
There are a few distinct categories of AI interview tooling, and they operate differently.
The oldest pattern is asynchronous video, best represented by HireVue. A candidate receives a link, opens a browser tab, and records answers to pre-set questions with no one watching. The system then analyzes the recording: speech rate, vocabulary, use of filler words, semantic content extracted from the transcription, and historically, facial micro-expressions. HireVue dropped the facial analysis component in 2021 after years of pressure from researchers and the Electronic Privacy Information Center, which had filed an FTC complaint in 2019. The facial analysis was premature at best; the scientific basis for inferring job performance from micro-expressions remains thin.
The second pattern is conversational, exemplified by Paradox’s Olivia. This is a chat-based system that interviews candidates over SMS, WhatsApp, or a web interface. It asks structured screening questions and scores responses against pre-defined criteria. It’s used heavily in high-volume hiring: retail, food service, logistics. McDonald’s was a notable customer. The experience is closer to filling out a form than having a conversation, but the system makes real screening decisions and can advance or reject candidates autonomously.
The third pattern, which is newer and less controversial, is live augmentation: tools like Metaview that join human-conducted interviews as silent observers, generating transcripts, structured notes, and scorecards. The interviewer is still a human; the AI is handling documentation.
The current wave, which the Verge article appears to encounter, is the agentic variant. LLM-based systems conducting dynamic multi-turn spoken or text interviews, capable of following up on answers and adapting questions. These are more flexible than the scripted asynchronous model. They are also less predictable, which creates new problems.
The Training Data Problem
Across all of these approaches, the scoring mechanism matters more than the interaction format. Most platforms use some combination of keyword matching, semantic similarity against “ideal answer” corpora, and ML models trained on the employer’s historical hiring outcomes.
That last component is where things go wrong structurally. If you train a model to predict “which candidates will this company hire,” and you train it on a company’s historical hiring data, you are training it to replicate whatever patterns the human hiring managers followed. If those managers hired disproportionately from certain universities, certain communication styles, or certain demographic backgrounds, the model learns to prefer those patterns too. It doesn’t know why the pattern existed; it just knows the pattern exists.
This isn’t a hypothetical. The EEOC’s 2023 guidance on AI hiring tools explicitly warns that AI screening systems can violate Title VII if they produce disparate impact on protected classes, and that employers remain liable even when using third-party tools. The guidance followed documented cases where automated systems produced measurable disparate outcomes by race, gender, and age.
NLP-based scoring has a specific and well-documented variant of this problem: performance gaps across accents and dialects. Models trained predominantly on Mainstream American English score speakers of other English varieties lower on “communication” metrics. Non-native speakers and speakers of African American Vernacular English have both been shown to score lower on automated language assessments even when the substantive content of their answers is equivalent. The system doesn’t distinguish between dialect and competence; it just measures distance from the training distribution.
Neurodivergent candidates face a related set of issues. A candidate with autism may not maintain eye contact with a camera in a way that resembles neurotypical interview behavior. A candidate with ADHD may answer questions in a non-linear way that confuses an NLP system looking for structured responses. These are differences in presentation, not in job capability, but they consistently produce lower scores in systems trained on neurotypical behavioral norms.
The Validity Question
Above all of the bias concerns sits a more fundamental question: do any of these metrics actually predict job performance?
The answer, based on the available research, is: barely, and far less than proponents claim. Schmidt and Hunter’s 1998 meta-analysis remains one of the most comprehensive assessments of hiring method validity. Structured interviews performed reasonably well (validity coefficient around 0.51). Unstructured interviews performed poorly (around 0.38). Work samples performed best (around 0.54). AI-scored video interviews are effectively a proxy for unstructured interviews filtered through a language model; they don’t obviously inherit the validity of structured interviews just because the questions are standardized.
The more troubling dynamic is the one HireVue’s Unilever case illustrates. Unilever reported a 75% reduction in time-to-hire using AI interview screening. That’s a real operational improvement. But time-to-hire and quality-of-hire are different metrics. A system that is fast and consistent can still be consistently wrong. If the hiring outcome it was trained to predict is “did a human hiring manager at this company advance this candidate,” and human hiring managers were biased, the system has learned to rapidly replicate biased decisions. Speed is not the same as validity.
Regulation Is Moving, But Slowly
A few jurisdictions have started to address this directly. Illinois passed the AI Video Interview Act in 2020, the first US law specifically targeting AI interview tools. It requires employers to notify candidates, explain what the AI analyzes, collect demographic data, and delete videos on request. New York City’s Local Law 144, effective 2023, requires annual independent bias audits of automated employment decision tools. The EU AI Act classifies employment AI systems as high-risk, requiring conformity assessments and human oversight.
These are meaningful steps. They are also insufficient. Bias audit requirements under NYC Local Law 144 have been criticized for allowing employers to define the audit methodology, which produces obvious incentive problems. The Illinois law’s notification requirement is useful but doesn’t address whether the tool is valid or whether candidates can meaningfully consent to an evaluation method they don’t control.
The Gaming Problem
There’s a final structural issue that doesn’t get enough attention: these systems are being gamed at scale.
A whole industry of AI interview coaching has emerged, including tools like Interview Kickstart, Final Round AI, and various LLM-based practice platforms, specifically designed to help candidates optimize their responses for AI scoring systems. Some of these tools run as browser overlays during live AI interviews, feeding candidates real-time response suggestions based on the question being asked.
This creates a measurement collapse. If the AI interview is supposed to evaluate the candidate, but the candidate is using a separate AI to generate the answers, then the AI interview is no longer measuring the candidate at all. It’s measuring how well the coaching AI approximates the hiring AI’s preferences. The signal degrades entirely.
This is the same failure mode that plagued ATS keyword optimization years ago, but operating at a higher fidelity. The form changes; the underlying problem, that automated filtering creates an arms race between screening systems and candidates trying to pass screening systems, remains.
What This Means
The Verge article captures something real about how strange and alienating it feels to be evaluated by a machine with no human context. But the strangeness is almost a distraction. The more pressing concern is that these systems are deployed at serious scale, often with minimal transparency, and their validity for the task they claim to accomplish is weak. Training on historical hiring data encodes historical bias. NLP scoring disadvantages non-mainstream speakers. Speed-to-hire metrics obscure whether quality-to-hire is actually improving.
For developers building tooling in this space, the EU AI Act’s framing is worth taking seriously: employment AI systems are high-risk by default, and the burden of demonstrating that they are not causing disparate harm should sit with the employer deploying them, not with the candidate trying to navigate them.