REFRAMED: Towards Realistic Audio Description Generation for Movies

This paper introduces Re frame d, a new dataset and benchmark for realistic Audio Description (AD) generation for movies. AD is defined as a verbal narration of key visual information necessary for understanding a video, timed to fit into natural gaps in dialogue so it minimally interrupts the original soundtrack. The authors note that both American and British legislation will soon mandate AD at scale: Title II of the Americans with Disabilities Act will require public entities to provide AD for pre-recorded video, while the UK's Media Act sets a streaming quota of 10% by 2030, creating an urgent need for automating AD in a way that addresses real-world requirements.

The paper identifies that existing approaches formulate AD generation in an artificial setting where both the content and timing of descriptions are pre-specified, reducing the task to clip-level captioning. They further rely on noisy transcription and alignment pipelines, and lack the rich parallel data required for modeling narrative context.

The authors make four main contributions:

1. A realistic task formulation for AD generation where a model must decide when and what to describe, together with evaluation metrics that exploit multiple references.

2. Re frame d dataset: 2,023 video excerpts (3,302 scenes) from 206 movies with dual AD versions (US and UK), dialogue subtitles, SDH, screenplays, and video.

3. The Re frame d challenge set: 10 fully manually annotated movies with 2–3 professionally transcribed AD versions each, scene boundaries, AD-to-scene labels, screenplay alignment, and professional subtitles.

4. A suite of benchmarks for ASR quality, temporal alignment, screenplay alignment, scene segmentation, AD splitting, and AD generation on the above data.

The paper reformulates AD generation as a joint decision problem over when and what to describe. Given a video and gaps in dialogue, a model must determine where descriptions are needed and what to say. This matches real-world requirements, unlike prior work which assumes AD placement is given and only generates the description sentence.

The dataset is built from movie excerpts licensed by Fandango, with raw videos without watermarking available publicly on rottentomatoes.com. Movies were selected with at least five movie excerpt videos and listings on audiovault.net for both American and British AD versions. Animated movies are excluded as they represent a substantially different visual domain.

For each movie excerpt, the dataset provides:

• Professional English dialogue-only and SDH subtitles from opensubtitles.org

• Aligned screenplays for 85 movies from MovieSum

• Both American and British AD versions

Gold-standard Transcription: A film post-production company performed fully manual AD transcription and timecoding to the nearest frame (approximately 0.04 seconds) for evaluation data. This new resource is called MAD-v3-eval.

AD Transcription and Diarization: The paper evaluates multiple ASR systems and finds existing AD corpora are extremely noisy (WhisperX achieves WER=12.8; MAD-v2-eval achieves 22.1). A Speechmatics-based system with character-name adaptation achieves WER=2.9 and is used for training data.

Temporal Alignment: Cross-correlation is used to align scene videos to full-length AD tracks, accounting for frame-rate differences between PAL and NTSC formats. This achieves 100% alignment accuracy to within one frame for scenes of length 32 seconds or greater.

AD Splitting and Scene Segmentation: A RoBERTa-based model segments AD sentences into description elements (F1=64.8 vs. comma-splitting baseline F1=36.9). A classifier identifies scene boundaries (F1=59.2).

Screenplay Alignment: Vecalign is applied to align screenplay scenes to movie scenes (F1=80.9 vs. text-based dialogue-only baseline F1=63.8).

The paper proposes three categories of evaluation metrics:

This evaluates performance at the level of dialogue gaps, directly reflecting the main constraint that applies to AD placement. Gaps of at least 1 second in dialogue subtitles are collected, and references/generations are assigned to gaps that fully contain them. CIDEr and METEOR scores are computed within each gap.

SODA-M and SODA-T are adapted from SODA metrics. SODA-M is the resulting F1 measure over METEOR scores from an optimal monotonic alignment. SODA-T is the proportion of matched pairs with timespan midpoints differing by less than τ = 10 seconds.

QEval measures answer correctness (the proportion of questions with system-predicted answer matching the gold-standard answer) and QEval-T also enforces temporal alignment. Questions are generated from reference AD and filtered aggressively to ensure they depend on specific narrative content.

Despite the subjectivity of AD, inter-human agreement is high. A QEval score of 69.6% shows professional describers agree well on which elements of a movie are narratively salient, and QEval-T of 61.2% confirms they also describe them at similar times.

The LLM evaluator answers 34.1% of questions correctly from random descriptions (QEval; chance-level=20%), demonstrating parametric knowledge. However, random performance drops to 2.3% once temporal grounding is required (QEval-T).

Shot-by-Shot and DistinctAD outperform the random baseline, with the largest gains on CIDEr (DistinctAD: 15.6 and Shot-by-Shot: 16.3, vs. 1.3). SODA-T is trivially high, as expected, since these models are provided with gold-standard AD placements.

With full-movie input, LLMs achieve significantly lower scores across all dialogue gap and aligned metrics compared to the pre-existing models. LLMs achieve the best results with chunked input (ten minutes of context), significantly outperforming the full-movie approach across all metrics. The paper concludes current LLMs cannot yet leverage full-movie narrative context, which is a requirement for realistic AD generation.

The paper identifies three coupled challenges that current models struggle with:

1. Content selection: identifying narratively salient characters, objects, actions, and visual details across long contexts

2. Temporal placement: finding suitable gaps within dialogue or salient audio

3. AD realization: producing descriptions that are informative yet match the available narration time

The paper notes that AD must faithfully describe all key visual elements, including potentially sensitive topics such as sexual content, abuse and violence. During evaluation, one full movie (Charlie St. Cloud (2010)), two 10 minute chunks within the same movie, and two movie excerpts from Speak No Evil (2024) triggered non-configurable safety guardrails in Gemini. All evaluation data transcription was fully manual human transcription with the post-production company paid their full rate for professional feature-movie subtitling.

The paper introduces a new formulation of AD generation that reflects real-world requirements: a model must decide both when and what to describe. The Re frame d dataset provides 2,023 movie excerpts paired with multiple AD transcripts, dialogue subtitles, SDH, and, for 85 movies, aligned screenplays, together with a fully annotated benchmark of 10 full movies. The evaluation shows that Prior AD systems and zero-shot LLMs outperform a random baseline but remain far below human performance.

Based on the paper, here are the specific improvements I can make to AI systems:

• Improvement: Build a model that simultaneously decides whether to describe a visual element and when to insert it, rather than treating AD as clip-level captioning with pre-specified placements.

• What it can do: Given a movie video and dialogue subtitles, it identifies dialogue gaps ≥1 second, selects narratively salient visual events (characters, actions, objects) within those gaps, and generates descriptions that fit the available time—matching real-world AD constraints.

• Improvement: Train on full-movie or ≥10-minute chunked inputs with dual AD references (US/UK) and aligned screenplays to learn narrative consistency (e.g., character identities, plot-relevant objects) across scenes.

• What it can do: Generate AD that references earlier plot points (e.g., the same knife from the kitchen) and avoids contradictions, unlike current LLMs that degrade with full-movie input.

• Improvement: Incorporate the paper’s alignment-based metrics (SODA-T, QEval-T) into training loss, penalizing descriptions whose midpoints deviate >10 seconds from gold-standard AD times.

• What it can do: Produce descriptions that land precisely within dialogue gaps, minimizing overlap with speech or salient audio—critical for accessibility compliance.

• Improvement: Use the Speechmatics-based pipeline with character-name adaptation (WER=2.9 vs. WhisperX’s 12.8) as a pretraining step for AD-specific speech recognition.

• What it can do: Automatically transcribe and timecode new movie AD tracks with near-human accuracy, enabling scalable dataset expansion and real-time AD generation for live broadcasts.

• Improvement: Leverage the Vecalign-based screenplay alignment (F1=80.9) to condition AD generation on script context, identifying which visual elements are scripted vs. improvised.

• What it can do: Prioritize describing unscripted visual actions (e.g., a character’s subtle glance) that screenplays omit, while skipping redundant descriptions of scripted dialogue-adjacent actions.

• Improvement: Train using the dual US/UK AD versions and multiple professional transcriptions (MAD-v3-eval) as diverse references, optimizing for CIDEr/METEOR across all references rather than a single gold standard.

• What it can do: Generate AD that captures the most narratively salient elements agreed upon by multiple describers, reducing subjectivity bias and improving robustness across movie genres.

• Improvement: Develop a specialized AD module that can handle sensitive content (e.g., violence, sexual abuse) without triggering non-configurable safety filters, using the paper’s noted failures as training data.

• What it can do: Faithfully describe all key visual elements in challenging scenes (e.g., he raises a gun to her head) while maintaining ethical compliance, avoiding omissions that would mislead blind viewers.

• Improvement: Train a model to estimate available narration time per dialogue gap (from subtitle timestamps) and adjust description verbosity accordingly, using the paper’s AD splitting model (F1=64.8) as a post-processing step.

• What it can do: Generate concise descriptions for 1-second gaps (She nods) and richer ones for 5-second gaps (He picks up the letter, reads it, then crumples it in anger), matching professional pacing.

• Improvement: Use the RoBERTa-based scene boundary classifier (F1=59.2) to segment long videos into narratively coherent units, then generate AD per scene with context from previous scenes.

• What it can do: Maintain narrative flow across cuts and scene changes, avoiding abrupt topic shifts in AD that confuse viewers.

• Improvement: Train a model to predict QEval scores (human agreement=69.6%) and output confidence scores for each generated description, flagging low-confidence outputs for human review.

• What it can do: In production, prioritize human oversight for descriptions where the model is uncertain about salience or timing, improving overall reliability for accessibility services.

Audio Description (AD) is a verbal narration of key visual content in videos, enabling access for visually impaired audiences. Unlike standard video captioning, AD is a structured editorial task: descriptions must be inserted into gaps in dialogue and must convey only what is needed to understand the narrative being told. However, existing approaches formulate AD generation in an artificial setting where both the content and timing of descriptions are pre-specified, reducing the task to clip-level captioning. They further rely on noisy transcription and alignment pipelines, and lack the rich parallel data required for modeling narrative context. We introduce a new formulation of AD generation in which models must jointly decide what to describe and when to do it. To support this, we present REFRAMED, a high-quality dataset of 2,023 videos that span 3,302 scenes from 206 movies, with professional AD transcripts (both American and British versions), professional subtitles and aligned screenplays. We also provide a manually curated challenge set that pairs full movies with multiple AD references, together with evaluation protocols that leverage dialogue gaps and multi-reference comparisons. Experiments with state-of-the-art AD systems and multimodal LLMs show that they outperform trivial baselines but fall far short of expert human performance. Our dataset and benchmark establish a new foundation for research on video understanding.