Beyond Fixed Luminance: Towards Panchromatic and Orthochromatic Image Colorization

Most image colorization systems operate in Lab space by predicting chroma (ab) while preserving an input-derived luminance channel (L). While effective on standard benchmarks, this fixed-luminance design restricts brightness changes and becomes unreliable when grayscale formation deviates from natural-image luminance, as in historical orthochromatic photography. We propose a luminance-agnostic colorization framework that formulates colorization as full-RGB image editing using a foundation image-editing model. To bridge modern panchromatic and historical orthochromatic conditions, we introduce a mixed grayscale objective that trains the model under both standard luminance grayscale and a red-insensitive grayscale formation. Experiments on COCO, ImageNet, and a multi-instance benchmark show that our method is competitive on standard grayscale inputs and substantially more robust under orthochromatic inputs, with qualitative comparisons and a human study indicating fewer visible color artifacts.

The paradigm across modern approaches for image colorization follows a consistent formulation: convert an RGB image to the Lab color space, preserve the luminance channel L, and predict chrominance components ab conditioned on that fixed luminance. This design reduces the learning problem to chroma estimation while treating luminance as ground truth. While effective for reconstruction benchmarks, this formulation imposes a strict and often overlooked constraint: the model is mathematically incapable of altering brightness. Because L is treated as immutable, the network can only produce colors that are consistent with the original grayscale intensity. This restriction fundamentally limits creative recolorization. For example, mapping a dark grayscale region to a vibrant yellow or red surface is physically infeasible under fixed-L constraints, since bright colors require high luminance energy in RGB space. Thus, current methods operate under a constrained prediction regime rather than generative restoration.

The issue becomes more severe when applied to historical photography. Early orthochromatic film stock was insensitive to red wavelengths, producing grayscale images whose luminance distribution does not correspond to human visual perception. Modern colorization models, trained on panchromatic data and conditioned on preserved luminance, inherit a structural domain mismatch. They are forced to assign chroma to brightness values that were never perceptually accurate, leading to suppressed reds. In this sense, current systems fail by design: when the input grayscale image is darker than it should be, the model is incapable of brightening those regions.

Most contemporary approaches that adapt pretrained text-to-image diffusion models do so by changing the architecture to condition them on grayscale. While somewhat effective, these approaches inherit a fundamental mismatch: text-to-image models are not pretrained for image-conditioned tasks. As a result, significant architectural engineering is required to force grayscale structure into a generative pipeline originally designed for synthesis from text alone. Recent foundation models natively trained on image editing provide a superior alternative for colorization without architectural hacking.

In this work, we propose a luminance-agnostic framework for image colorization. Instead of predicting chroma conditioned on fixed L, we fine-tune a foundation image-editing model to predict the full RGB output directly from grayscale input—without luminance swapping. Our approach leverages the strong generative and editing prior of the FLUX.2-klein (4B) model. We choose FLUX.2-klein because it supports high-quality few-step generation, reducing inference time to a few seconds per image. Our key contributions are three-fold: (i) we identify the fixed-luminance bottleneck in Lab-based colorization and show why it is especially problematic for orthochromatic imagery; (ii) we propose a luminance-agnostic colorization framework that formulates colorization as full-RGB image editing using a foundation image-editing model; and (iii) we perform a comprehensive suite of experiments which show that the proposed method, while remaining competitive on standard grayscale inputs, is substantially more robust on orthochromatic inputs and also improves prompt-following behavior compared to baselines.

A core contribution of our work is the Mixed Grayscale Objective, designed to bridge the gap between modern digital intensity and historical film sensitivity. Given a ground-truth colorful image I ∈ RH×W×3 with channels IR, IG, IB, we generate two distinct types of grayscale conditioning images, denoted as Igray. For the Standard Mode, we simulate modern panchromatic film using the standard luminance formulation, conditioned with the text prompt cpan = colorize + Caption: I gray pan = 0.299·IR + 0.587·IG + 0.114·IB. For the Orthochromatic Mode, we simulate historical red-insensitive film using a simplified approximation motivated by the fact that orthochromatic materials are sensitive primarily to blue and green wavelengths and insensitive to red. We therefore exclude the red channel and define I gray ortho = (IB + IG)/2, conditioned with the text prompt cortho = colorize ortho + Caption. Here, colorize acts as a trigger word for the action of colorization, while ortho acts as a trigger word to notify the model that the input is an orthochromatic image.

During training, we encode the ground-truth target image into the latent space using the frozen VAE, yielding the data latent z1 = E(I). Because the Flux architecture is based on Rectified Flow (a formulation of continuous-time diffusion models), the forward process constructs noisy latents zt by linearly interpolating between standard Gaussian noise z0 ∼ N(0, I) and the target latent z1 over timestep t ∈ [0, 1]: zt = t·z1 + (1 - t)·z0. The DiT is trained to predict the vector field (velocity) vθ that transports the noise distribution to the data distribution. The network is conditioned on the text embedding ctext ∈ cpan, cortho and the spatially concatenated grayscale image Igray ∈ Igray pan, Igray ortho. The generative diffusion objective L is minimized as follows: L = E[vθ(zt, t, ctext, Igray) - (z1 - z0) squared]. By training on this objective, the model learns to interpret luminance values differently based on the provided text prompt. Instead of replacing the predicted luminance with a constrained L channel during inference, the network maps the conditional inputs directly to a full RGB representation. This allows it to lift the brightness of dark regions in orthochromatic inputs where a standard L-constrained model would otherwise fail.

We employ FLUX.2-klein-4B as our foundational generative prior, leveraging its robust semantic understanding for high-fidelity color synthesis. To maintain computational efficiency while adapting the model to the colorization task, we utilize Low-Rank Adaptation (LoRA) applied exclusively to the Diffusion Transformer (DiT) blocks. Both the VAE (Image Encoder/Decoder) and the Qwen3-4B Text Encoder remain frozen throughout the process. Our LoRA configuration utilizes a rank of 16 and an alpha of 16, resulting in approximately 23.10 million trainable parameters, representing a mere 0.5% of the total 4.3B parameter count. The model was trained using the AdamW optimizer with a constant learning rate of 1 × 10−4 and a 100-step warmup period. We utilize bfloat16 mixed-precision and enable gradient checkpointing. Training was executed across a distributed setup of three NVIDIA RTX 6000 Ada GPUs with an effective batch size of 12. The model was trained for 10 epochs on a subset of ImageNet from Kaggle, totaling 34,745 images.

To evaluate our proposed method across different visual domains, we utilize subsets of three widely adopted benchmarks: ImageNet (first 5,000 images from the validation set), COCO (first 5,000 images from the test set), and Multi-Instance (the complete validation set, consisting of 7,213 images). For ImageNet and COCO, we used BLIP2 to generate captions, while for Multi-Instance, human-annotated captions were provided with the dataset. We evaluate on both perceptual realism and the statistical distribution of the generated colors. Perceptual Quality Metrics include the Fréchet Inception Distance (FID), spatial FID (sFID), and FID-DINO (same as FID but using DINOv2 features instead of InceptionNet). Chroma and Colorfulness Metrics include the standard Colorfulness metric, Col-diverse (Chroma Diversity), Saturation (Mean LAB Chroma), and ColorNet (Perceptual Colorfulness), a CNN trained on subjective human datasets to predict Mean Opinion Scores of image colorfulness. Baselines include BigColor, COCO-LC, DDColor, DISCO, and UniColor.

The results show a consistent pattern across all three benchmarks. On standard panchromatic grayscale inputs, our method remains competitive with strong recent baselines: DDColor typically achieves the best FID-family metrics, while our approach is often second-best on FID, sFID, and FID-DINO and achieves stronger perceptual color statistics such as Col-diverse and ColorNet. The advantage of our method becomes more pronounced under simulated orthochromatic inputs, where baseline performance degrades more substantially. On COCO, our method achieves the best FID-DINO and the strongest scores on all reported color metrics, while trailing DDColor only slightly on FID and sFID. On ImageNet, our method is the strongest overall in the ortho setting, achieving the best FID, sFID, FID-DINO, Colorfulness, Col-diverse, and ColorNet scores. On the Multi-Instance benchmark, our method again leads on most ortho metrics, including sFID, FID-DINO, Colorfulness, and Col-diverse, while remaining second-best on FID, Saturation, and ColorNet. Overall, these results support our central claim: the proposed luminance-agnostic formulation preserves strong performance on standard grayscale inputs and yields substantially greater robustness when grayscale formation deviates from modern panchromatic luminance.

A potential concern with adapting a generative foundation model is that it may hallucinate content rather than faithfully recolor the input. To verify that our model preserves the structure of the grayscale input, we measure structural fidelity via grayscale reprojection: we convert each colorized RGB output back to grayscale using the corresponding grayscale formation and compare it against the input grayscale image using L1, L2, and SSIM. We compare three variants of FLUX.2-klein-4B: the non-fine-tuned model (Not-FT), a model fine-tuned only on standard panchromatic grayscale (Normal-FT), and our full model trained with the Mixed Grayscale Objective (Mixed-FT). The non-fine-tuned model exhibits substantially weaker structure fidelity (SSIM around 0.36), since it tends to alter image content, whereas both fine-tuned variants preserve structure well (SSIM around 0.81–0.88). This confirms that our luminance-agnostic formulation maintains the grayscale structure while still enabling full-RGB colorization.

To complement the automatic metrics, we conducted a human study focused on visible color artifacts. We sampled the first five images from each of the three evaluation datasets (ImageNet, COCO, and Multi-Instance) under both panchromatic and orthochromatic grayscale settings, resulting in 30 evaluation inputs per method. Each input was colorized by all compared methods and evaluated by 25 human participants. Participants were asked a binary question: whether the displayed colorized image contained noticeable artifacts such as color bleeding, unnatural hue shifts, or spatially inconsistent colorization. We summarize the results using the Artifact-Free Rate. Our method attains the highest Artifact-Free Rate among all compared approaches, suggesting that its outputs are perceived as more stable and less artifact-prone.

We also compare our method against COCO-LC for prompt following. We find that our method is much better at following text prompts, even very difficult ones. When dealing with multiple color words in a text prompt, e.g., the blue cup, the yellow cup, and the cyan cup, COCO-LC incorrectly colors the middle cup red, while our method correctly colors all three cups. Moreover, there are unnatural green spots in the yellow cup for the COCO-LC output, compared to our output, which is clean and more realistic.

We revisited image colorization through the lens of luminance mismatch. Most existing systems predict chroma while preserving an input-derived luminance channel, which works well when grayscale intensities align but can break under domain shifts such as historical orthochromatic imagery. To address this, we proposed a luminance-agnostic colorization framework that predicts full RGB outputs directly using a foundation image-editing model, rather than relying on luminance swapping. Central to our approach is a mixed grayscale objective that trains the model under both standard panchromatic and simulated orthochromatic conditioning, allowing it to reinterpret brightness when the grayscale formation deviates from human-perceived luminance. Across COCO, ImageNet, and a multi-instance benchmark, we find that while our method remains competitive on standard grayscale inputs, it is substantially more robust on orthochromatic inputs, where prior methods degrade more strongly. Qualitative comparisons further indicate cleaner outputs with fewer local color artifacts.

Limitations: We adapt the foundation model using LoRA only; full fine-tuning with larger datasets, longer schedules, and more compute may further improve fidelity and robustness. Moreover, our orthochromatic simulation is a simplified approximation of historical film response, while real archival imagery exhibits diverse capture and aging effects (e.g., blue-biased spectral sensitivity, color fading, and chemical degradation) as well as material-dependent appearance changes. For instance, facial skin can appear harsher under red-insensitive imaging, plausibly due to reduced contributions from red-wavelength subsurface light transport. Future work includes incorporating more physically grounded spectral sensitivity models, training with a broader family of grayscale/color distortions, and extending the framework to jointly address additional restoration factors beyond grayscale formation.

Improvements to AI Systems:

1. Luminance-Agnostic Colorization Architecture

Replace fixed-L (Lab-space) chroma prediction with full-RGB generation via fine-tuned foundation image-editing models (e.g., FLUX.2-klein). This enables brightness changes, allowing dark regions to be recolored as bright objects (e.g., yellow/red surfaces) without physical constraints.

2. Mixed Grayscale Objective for Domain Robustness

Train with two grayscale formations—standard panchromatic (0.299R+0.587G+0.114B) and simulated orthochromatic ((B+G)/2)—using distinct text prompts (colorize vs. colorize ortho). This teaches the model to reinterpret luminance based on input type, preventing red suppression in historical/orthochromatic imagery.

3. Prompt-Conditioned Colorization via Text-to-Image Editing Priors

Leverage native image-editing foundation models (not text-to-image models) to avoid architectural hacking. Use LoRA (rank 16, alpha 16, 0.5% parameters) on DiT blocks only, keeping VAE and text encoder frozen, enabling efficient few-step inference (seconds per image) and strong prompt-following (e.g., correctly colorizing multiple distinct objects with different color words).

4. Structural Fidelity Preservation

Fine-tuning with the mixed objective ensures grayscale reprojection fidelity (SSIM 0.81–0.88 vs. 0.36 for non-fine-tuned), preventing hallucination of content while enabling full-RGB output.

5. Improved Color Statistics and Artifact Reduction

The framework achieves higher perceptual colorfulness (ColorNet, Col-diverse) and fewer visible artifacts (highest Artifact-Free Rate in human studies) compared to baselines like DDColor, COCO-LC, and UniColor, especially under orthochromatic inputs.

6. Unified Handling of Multiple Grayscale Domains

The system can process both modern digital grayscale and historical film inputs without retraining, by using the text prompt to switch interpretation of luminance values—enabling robust performance on archival photos.

What the Improved AI System Can Do:

• Colorize any grayscale image (modern or historical) with accurate, vibrant colors, including brightening dark regions that were previously impossible under fixed-luminance constraints.

• Follow complex text prompts (e.g., blue cup, yellow cup, cyan cup) with precise per-object color assignment, avoiding cross-contamination or unnatural color spots.

• Preserve the original structure of the input (edges, shapes, textures) while generating realistic full-RGB outputs, even when the input grayscale is perceptually distorted (e.g., red-insensitive orthochromatic film).

• Operate efficiently (a few seconds per image) on consumer GPUs, making it practical for large-scale archival restoration or creative editing.

• Adapt to new grayscale formations (e.g., blue-biased spectral sensitivity, fading) by extending the mixed objective with additional simulated distortions, without architectural changes.