ViTeX-Bench: Benchmarking High-Fidelity Video Scene Text Editing

Video scene text editing remains underdeveloped, particularly for precise local edits that must preserve original scene dynamics, and this work introduces ViTeX-Bench to systematically study these trade-offs. The benchmark suite comprises the ViTeX-Dataset and a three-axis evaluation protocol designed to measure text correctness, visual and temporal quality, and edit locality in video scene text editing.

How it works

The core of the methodology involves creating a comprehensive dataset and a rigorous evaluation protocol. The ViTeX-Dataset contains 387 real-world 720p videos with per-frame text-region masks and source–target string instructions. This dataset is split into a training set of 230 paired edits for training and a frozen evaluation split of 157 videos without edited versions. The construction pipeline involves four assets: the dilated text-region mask M, a source–target string pair (ssrc, stgt), a clean background video Vclean produced by removal-1.3B, and a first-frame target-text patch p new 1. Two strategies are used for training data: Strategy A (alpha composition) for static videos and Strategy B (PISCO inserter) for dynamic videos.

Evaluation Protocol

The protocol evaluates outputs across three primary axes using 13 metrics, with one primary metric per axis and a Pareto comparison to interpret trade-offs. The three axes are: text correctness, visual and temporal quality, and edit locality. Text correctness is probed using an OCR recognizer (PP-OCRv5) on source-detectable frames to calculate SeqAcc (exact substring match), CharAcc (partial credit for near-correct renderings), and TTS (temporal text stability). Visual quality uses two spatial scopes: full output frame MUSIQ and a text-crop region MUSIQ, measuring FlickerS, WarpS, and MUSIQS. Edit locality is measured by constructing a locality-only prediction ˆf loc t that retains predicted pixels outside the mask while substituting source pixels inside, quantified by metrics like PSNR/SSIM (higher is better) and LPIPS/DreamSim (lower is better).

Baselines and Results

Eight baselines from four editing families are compared across these axes. The results expose distinct failure modes: per-frame editors achieve the highest character accuracy, the reference editor has low temporal error, and bounding-box-local methods preserve the surrounding scene particularly well. For instance, ViTeX-Edit-14B achieves CharAcc 0.688, the highest mean among evaluated video-native editors. In terms of visual quality, FLUX-Text shows high correctness but large text-region residuals (Flickerc = 14.81). Edit locality is assessed by metrics like DreamSimloc, where ViTeX-Edit-14B achieves a low score of 0.024 on the evaluation split.

Calibration and Robustness Analyses

To interpret the scores, calibration and robustness analyses are performed. OCR accuracy on detectable source frames is 0.851 for exact match and 0.966 for CharAcc, providing context for recognition errors without rescaling benchmark scores. Human evaluation confirms these findings, with text-crop Warp aligning more closely with temporal ratings than full-frame Warp (ρ = −0.40 vs. −0.20), supporting its selection as the temporal primary metric. Bootstrap resampling of the 152 source-detectable clips yields a mean Kendall τ of 0.936 against the full SeqAcc ranking, supporting ranking stability within the sampled domain.

Reference Editor and Composite Control

ViTeX-Edit-14B is released as an open-source reference editor fine-tuned on the paired training split using motion-aligned glyphvideo conditioning, achieving CharAcc 0.688. Furthermore, a shared Composite post-processing wrapper is applied to all eight baselines and ViTeX-Edit-14B. This control isolates the difference between text synthesis and background reconstruction, showing that for ViTeX-Edit-14B, it raises PSNR loc from 29.08 to 42.95 dB and reduces DreamSim loc from 0.024 to 0.002, indicating that source-pixel restoration accounts for much of the locality gain.

The gist: ViTeX-Bench provides paired training data and a frozen evaluation protocol for video scene text editing, making character correctness, temporal quality, and scene preservation explicit through a three-axis design. The benchmark reveals distinct failure modes across eight baselines and an open reference editor, exposing persistent trade-offs between these critical editing metrics.

Key Metrics Enumerated:

1. Text Correctness: SeqAcc (exact match), CharAcc (partial credit), TTS (temporal text stability).

As a fastidious and diligent researcher, I have analyzed ViTeX-Bench: Benchmarking High-Fidelity Video Scene Text Editing. The core finding is that existing video editing baselines fail by neglecting the simultaneous requirements of exact text correctness, temporal stability (flicker/drift), and scene preservation (edit locality).

Here are specific improvements to AI systems based on this research, categorized by the required capability:

---

)1. Improvement for Temporal Consistency and Text Stability

The paper highlights that models like FLUX-Text achieve high per-frame correctness but suffer from temporal instability (flicker/drift).

FLUX-Text combines high correctness with large text-region residuals (Flickerc = 14.81, Warpc = 13.01), consistent with its independent per-frame edits.

TTS measures whether the decoded string is stable across adjacent detectable frames.

The improvement involves integrating a temporal coherence loss or mechanism directly into the editing backbone.

ViTeX-Edit-14B adapts a pretrained video editor to the paired training split through motion-aligned character conditioning.

Our new glyph branch supplies both character structure and source-aligned motion.

--- System Improvement: Motion-Aligned Glyph Conditioning Module (MAGCM)

Implement a dedicated module, similar to ViTeX-Edit-14B's 'glyph video stream,' that explicitly conditions the text rendering on the source video's optical flow or predicted camera trajectory. This module should take the source text trajectory and projectively warp the target glyph image onto every frame using per-frame homographies derived from tracking (e.g., CoTracker3, as used in ViTeX-Edit-14B).

--- What it can do:

This system will eliminate glyph drift and flicker. It ensures that if a character moves across the screen, its appearance remains consistent with the source text's motion and perspective. The resulting system will produce video edits where the edited text is not only spelled correctly in every frame but also looks physically stable over time, directly addressing the low TTS score observed in models like Wan2.1-VACE-14B.

)2. Improvement for Edit Locality and Background Preservation

The research shows that many editors fail to preserve the surrounding scene, leading to poor edit locality (low DreamSim-loc). For instance, TextCtrl + AnyV2V shows significant drift in the background and scene structure.

Edit locality measures how well a method preserves pixels outside the editable region.

Composite isolates this difference: for ViTeX-Edit-14B, it raises PSNR-loc from 29.08 to 42.95 dB and reduces DreamSim-loc from 0.024 to 0.002.

The improvement involves explicitly training the model or applying a post-processing layer that enforces pixel fidelity outside the mask boundary, similar to the Composite wrapper used in ViTeX-Edit-14B.

--- System Improvement: Locality-Aware Background Restoration (LABR) Layer

Integrate a dedicated restoration layer into any video editing architecture. This layer should calculate a reconstruction loss (e.g., LPIPS or DreamSim) specifically for the unedited regions, using the source video pixels as high-fidelity supervision for those areas.

--- What it can do:

This system will prevent background changes and scene structure drift. It ensures that when text is replaced, the lighting, shadows, and surrounding geometry of the original scene are perfectly reconstructed around the edited text patch. This directly improves Edit Locality metrics (PSNR-loc, DreamSim-loc), ensuring the edit feels like a seamless part of the original video rather than an isolated overlay.

--- 3. Improvement for Robustness via Calibration and Evaluation

The paper emphasizes that performance is highly dependent on OCR calibration and that human evaluation provides crucial context for metric interpretation.

OCR calibration, human evaluation, and annotation-sensitivity analyses support the interpretation of these scores.

The improvement involves developing a self-calibrating pipeline that accounts for script variation (Latin vs. CJK) and font styles, as demonstrated by ViTeX-Dataset's coverage statistics.

--- System Improvement: Adaptive Script/Font Recognition Unit (ASFRU)

Develop a unified pre-processing unit that performs three functions: 1) Dynamic OCR selection based on Unicode blocks; 2) Font style detection; and 3) Confidence thresholding calibrated against human transcription benchmarks.

--- What it can do:

This system will drastically reduce recognition errors across different languages and styles. By dynamically selecting the correct OCR backend (e.g., PP-OCRv5 for Chinese text), it ensures high CharAcc, even when dealing with complex scripts or artistic fonts, providing a more reliable foundation for the subsequent text editing steps.

--- 4. Improvement for Comprehensive Trade-off Optimization

The paper identifies distinct trade-offs between the three primary metrics (SeqAcc ↑, Warpc ↓, DreamSim-loc ↓) and concludes that no single method dominates all three.

The front exposes distinct operating points.

The improvement involves moving beyond simple maximization towards a Pareto optimization framework during training or inference.

--- System Improvement: Pareto-Optimized Editing Controller (POEC)

Design the final editing stage as a controller that optimizes for a weighted combination of the three primary metrics based on the desired outcome (e.g., prioritize SeqAcc for signage, prioritize DreamSim-loc for subtle branding).

--- What it can do:

This system will allow users to select the best editor based on their specific need. For example, a user needing perfect text accuracy in a high-motion scene would select an operating point favoring SeqAcc and Warpc minimization (like FLUX-Text or TextCtrl), while a user needing subtle background preservation would select one favoring DreamSim-loc (like ViTeX-Edit-14B Composite). This provides actionable, tailored AI performance rather than a single monolithic editor.