Seeing the imagined: latent functional alignment in visual imagery decoding from fMRI data

arXiv:2604.15374 · q-bio.NC, cs.AI, eess.IV · Submitted 2026-04-15 · Read on arXiv

q-bio.NC, cs.AI, eess.IV

Submitted: 2026-04-15

Updated: 2026-09-22

License: http://creativecommons.org/licenses/by-nc-sa/4.0/

The gist: Recent progress in visual brain decoding from fMRI has been enabled by large-scale datasets such as the Natural Scenes Dataset (NSD) and powerful diffusion-based generative models.

Terminology

Abstract

Recent progress in visual brain decoding from fMRI has been enabled by large-scale datasets such as the Natural Scenes Dataset (NSD) and powerful diffusion-based generative models. While current pipelines are primarily optimized for perception, their performance under mental-imagery remains less well understood. In this work, we study how a state-of-the-art (SOTA) perception decoder (DynaDiff) can be adapted to reconstruct imagined content from the NSD-Imagery benchmark. We propose a latent functional alignment (LFA) approach that maps imagery-evoked activity to the pretrained model's semantic content-enriched conditioning space, by adding a simple alignment module, while keeping the original remaining components frozen. To mitigate the limited amount of matched imagery-perception supervision, we further introduce a neural retrieval-based augmentation strategy that selects semantically related NSD perception trials from the same participants. Across four subjects, LFA consistently improves high-level semantic reconstruction metrics relative to the frozen pretrained baseline and a voxel-space ridge alignment baseline, and enables above-chance decoding from multiple cortical regions. These results suggest that semantic structure learned from perception can be leveraged to stabilize and improve visual imagery decoding under out-of-distribution conditions.

Related papers