How voice deepfake detection works, and its limits

"Is this voice real?" sounds like a yes-or-no question, but it rarely has a simple answer. This guide explains in plain language how AI-generated voice detection works, why some recordings are much harder to judge than others, and how to read a detector's result without over-trusting it.

We build SpotACloneso we deal with these trade-offs daily. Everything below applies to voice deepfake detection in general, not just our tool.

What a voice detector is actually looking for

Modern voice cloning works by generating speech from a model of someone's voice. The output can sound very convincing to a listener. But generating audio is not the same as recording a human throat, mouth and room, and the process tends to leave traces. Detectors look for these traces in three broad ways.

1. Spectral and phase cues

Any audio can be broken down into its frequencies over time, a picture called a spectrogram. Real speech has a messy, physical richness: breath noise, small irregularities in the voice, and the way a real room colours the sound. Synthetic speech can show patterns that are too regular, odd energy in certain frequency bands, or traces left by the component that turns the model's output into a waveform.

Phase describes how sound waves line up over time. People can barely hear it, but some generators handle it in ways that differ from natural recordings, and detectors can learn to spot that.

2. Learned features from self-supervised speech models

Many current detectors start from large speech models that were first trained on huge amounts of unlabelled audio to learn the general structure of speech. A well-known example of this approach is wav2vec 2.0. The detector is then fine-tuned on labelled real and fake examples. These learned features capture patterns that are hard to describe in words but useful for telling genuine speech from synthesis.

3. Prosody: rhythm, stress and intonation

Prosody is the music of speech: where pauses fall, how pitch rises and falls, how stress shifts with emotion. Cloned voices sometimes get the tone of a voice right but the rhythm slightly wrong, especially in long sentences or emotional speech. Prosody is a weaker signal on its own, but it adds evidence.

In practice, a detector combines these cues into a score. That score is a probability-like estimate, not a fingerprint match.

Why phone calls and voice notes are harder

Most scam audio doesn't arrive as a clean studio file. It comes through phone networks and messaging apps, and every step removes information.

• Narrowband phone audio. Many phone calls carry only a limited range of frequencies, often described as 8 kHz telephony. The high-frequency detail where some synthesis traces live is simply cut off.
• Compression codecs. Calls and voice notes are compressed to save bandwidth. WhatsApp voice notes, for example, typically arrive as Opus-encoded .opus or .ogg files. Compression is built to keep speech understandable, not to keep forensic detail, so it can blur the traces a detector looks for. It can also add artefacts of its own that resemble synthesis.
• Background noise. Traffic, fans, TV and crowd noise mask fine detail and can push a detector toward the wrong answer.
• Short clips. A three-second "hello, it's me" gives a model very little to work with. Longer, clearer speech gives more evidence.
• Re-recording. Playing a voice note on one phone and recording it on another adds a room, a speaker and a microphone, which changes the signal again.

The research community treats this as a core problem. The ASVspoof 2021 challenge added channel and compression variability to its tasks, and the authors note that this makes detection harder.

Why new generators can slip past detectors

A detector learns from examples. If a new voice-cloning system leaves traces it hasn't seen, it may miss them. This is a moving target. Generators improve, detectors retrain, and generators change again.

Public research shows how big this gap can be. A 2022 study, "Does Audio Deepfake Detection Generalize?"collected real-world deepfake and genuine recordings and found that detectors from earlier research performed poorly on them compared with benchmark results. The fifth edition of the challenge series, ASVspoof 5added adversarial attacks for the first time. These are fakes deliberately crafted to fool detectors.

So treat any claim that a detector catches every deepfake with suspicion.

False positives, false negatives, and why "Unsure" matters

Every detector makes two kinds of mistakes:

• False positive: a real person's voice is flagged as AI. This can cause needless panic or unfairly cast doubt on someone.
• False negative: a cloned voice is passed as real. This can give false comfort at exactly the wrong moment.

Lowering one kind of error usually raises the other. A tool that always answers a firm yes or no hides this trade-off. On a noisy, compressed, two-second clip, a forced answer is often closer to a guess.

That is why SpotAClone has three outcomes: Likely AILikely real and Unsure. Unsure is not a failure. It's the tool honestly telling you the audio doesn't support a confident call, so you should rely on other checks.

How to read any result responsibly

• Likely AI: treat the request as suspicious. Don't send money or share codes. Verify through a separate channel.
• Likely real: the voice shows no strong signs of synthesis. This doesn't prove who is speaking, and a real human voice can still be a scammer.
• Unsure: the audio isn't good enough for a confident call. Verify as you would for Likely AI.

Whatever the result, the safest checks don't involve audio analysis at all. Call back on a number you already know, use a family code word, and never share an OTP or PIN. The US FTC gives the same advice: don't trust the voice, verify the story.

So, is an AI voice detector legit?

A detector is a useful second opinion when it is honest about uncertainty and you use it as decision support, not proof. Look for tools that admit their limits, show an uncertain outcome, and tell you to verify independently. Be wary of any tool that promises certainty.

For practical next steps, see how to tell if a voice note is AI and the AI voice cloning family emergency scam.

Sources

• ASVspoof 2021: accelerating progress in spoofed and deepfake speech detection (arXiv)
• ASVspoof 5: Crowdsourced Speech Data, Deepfakes, and Adversarial Attacks at Scale (arXiv)
• Does Audio Deepfake Detection Generalize? (arXiv)
• wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations (arXiv)
• US FTC consumer alert: Scammers use AI to enhance their family emergency schemes