Methods and evaluation

How multimodal AI detection is evaluated

This page explains useful signal families, evaluation metrics, and public reference datasets. It does not publish a universal accuracy claim because performance changes by model, sample, language, transformation, and threshold.

Text

Image

Code

Voice and audio

Video

Signal families by modality

These are common signal categories for interpreting detector output and designing representative evaluations. No individual signal proves authorship.

Generated text

Signals to examine

Text evaluation can combine statistical writing signals, classifier output, and robustness checks across editing and paraphrasing.

Evaluation checks

Sentence variationStylometric patternsModel-family shift testsParaphrase robustness

ReportAUROC, FPR, FNR, calibration

Useful evidenceSentence-level indicators and confidence context.

Reference samplesPublic corpora and held-out samples

Generated code

Signals to examine

Code evaluation should separate common boilerplate from generator-associated patterns and test how results change after human edits.

Evaluation checks

Structure and identifier patternsComment and documentation signalsLanguage-specific baselinesHuman-edit robustness

ReportAUROC, FPR, FNR, calibration

Useful evidenceFunction-level indicators with repository context.

Reference samplesVersioned human and AI code samples

Voice cloning and synthetic speech

Signals to examine

Audio evaluation can examine spectral, timing, prosody, and channel signals across clean, compressed, and re-recorded samples.

Evaluation checks

Codec robustnessNoise and replay testsSpeaker consistencyClip-length analysis

ReportEER, FPR, FNR, calibration

Useful evidenceTimestamped spectral and prosody indicators.

Reference samplesPublic anti-spoof datasets and held-out audio

Generated music

Signals to examine

Music evaluation can compare spectral repetition, timbre, structure, and metadata while accounting for mastering and distribution changes.

Evaluation checks

Spectral consistencyStructure and repetitionCodec robustnessSource metadata review

ReportAUROC, FPR, FNR, calibration

Useful evidenceSegment-level indicators with source context.

Reference samplesVersioned human and generated music samples

Deepfake and generated video

Signals to examine

Video evaluation should test frame-level, temporal, face, and audio-video consistency across resolution and recompression changes.

Evaluation checks

Temporal consistencyFace-region reviewAudio-video alignmentRecompression robustness

ReportClip AUC, frame AUC, FPR, FNR

Useful evidenceFrame and timestamp-level indicators.

Reference samplesPublic deepfake datasets and held-out video

Public datasets relevant to evaluation

Dataset	Domain	Notes	Metric(s)
HC3 / HC3+	Text	High-quality ChatGPT vs Human	AUROC, FPR@TPR
RAID	Text	Adversarial attacks & domains	Robustness Score
AIGCodeSet	Code	Python generation tasks	AUROC
ASVspoof 2019/21	Audio	Logical/Physical Access	EER, min t-DCF
ADD 2022	Audio	Audio Deepfake Detection	EER
DFDC	Video	100k+ clips, Facebook backed	Video-AUC
FaceForensics++	Video	Diverse manipulation methods	Frame-AUC
Celeb-DF	Video	High-quality Deepfakes	AUC

These datasets are public references for designing evaluations. Listing a dataset does not claim a published ZeroTrue score on it.

Evaluation Metrics

Text

AUROC

Across thresholds

Code

AUROC

By language

Audio

EER

Error tradeoff

Video

Clip AUC

Frame and clip

Operational Bands

Illustrative labels for reading confidence with policy context.

Lower confidence

Manual review

Higher confidence

Selected References

Evaluation priorities

Coverage

Generator and domain shift

Test held-out generator families, languages, media sources, and content domains.

Robustness

Transformation resistance

Measure behavior after compression, cropping, paraphrasing, editing, replay, and re-recording.

Quality

Calibration and error analysis

Publish false positives, false negatives, confidence calibration, model version, and test date.