Hearing Like Humans? Sound Symbolism and Perceptual Alignment in Speech Language Models

Sound symbolism, which describes how people intuitively map speech sounds to perceptual qualities like roundness or sharpness, is being investigated in Speech Language Models (SLMs) to determine if they share this human tendency. The core finding is that while SLMs can sometimes match visual shape perceptions, their auditory judgments often fail to align with human perception because they miss critical acoustic cues, suggesting the weakness lies in how speech is represented rather than vision itself.

The Core Research Questions

The study formulated four research questions to systematically probe this phenomenon:

1. Do SLMs judge speech sounds as round or pointed the way humans do?

2. Are SLMs’ sound-symbolic judgments driven by the same acoustic cues that drive human judgments, such as spectral tilt?

3. Can SLMs match a heard sound to the corresponding shape?

4. If matching fails, can the failure be attributed to the auditory side rather than the visual one?

Experimental Design and Methodology

The researchers conducted four experiments designed to isolate these components of sound symbolism:

1. Experiment 1 (Auditory Channel): This involved a forced-choice task where models chose between a rounded or pointed shape based only on the spoken pseudoword, testing whether they judge sounds as round or pointed.

2. Experiment 2 (Graded Rating): This experiment used a graded rating scale to ask models to rate pseudowords on both roundedness and pointedness, allowing for the investigation of which acoustic cues drive these judgments.

3. Experiment 3 (Crossmodal Matching): This tested whether omni-modal models could match a heard sound to its corresponding shape, which is the closest task to human performance in the bouba/kiki experiment.

4. Experiment 4 (Visual-Only Ablation): This served as a control, removing audio input so models could rate shapes alone, testing whether matching failures are due to auditory limitations or visual perception issues.

Key Findings on Auditory Alignment

The results revealed that SLMs’ auditory judgments diverge from human perception. Specifically:

'on the auditory side, the models’ soundsymbolic judgments diverge from human perception: the best model reaches barely half of the human ceiling.'

Models failed to exploit acoustic cues that drive human intuition, such as spectral tilt. The analysis showed that for humans, spectral tilt dominates (ρ =.431), with the other cues far behind. Furthermore, open-weight models fail specifically at the auditory-visual integration that is central to human sound symbolism.

The Role of Visual Perception and Crossmodal Links

In contrast to their auditory weakness, SLMs demonstrated intact shape perception when visual input was present.

'On the visual side, by contrast, omni models (those that accept both audio and image inputs) closely agree with human shape ratings, demonstrating that they can represent the rounded-pointed distinction with high fidelity when the input is visual rather than acoustic.'

Experiment 4 confirmed this: shape perception is intact, as model correlations against human means ranged from r =.94 to.97 against a human ceiling of.99. This localization suggests that perceptual alignment depends not on stronger vision but on speech representations that capture the cues humans hear.

Localization of Failure and Representation

The study localized the source of failure through representational similarity analysis (RSA) and logit-lens analysis.

'Perceptual decisions form only in the deepest layers.'

For Qwen3-Omni, perceptual decisions form only in the deepest layers, with early argmax trajectories showing a diffuse distribution. The findings indicate a modality gap; while the model lacks the strong ability to separate rounded and pointed in audio, it successfully captures these properties and aligns with the human ratings in the visual domain.

Conclusion on Model Capabilities

The paper concludes that SLMs read the visual structure of sound symbolism much like people do, but their auditory judgments are weak because they miss the acoustic cues that drive human intuitions. The crossmodal link is not entirely broken; a frontier model succeeded at matching, though this success was plausibly via lexical rather than acoustic routes. The ultimate requirement for building perceptually aligned systems is developing audio representations that capture the acoustic cues humans rely on.

Index Terms

Sound Symbolism, Speech Language Model, Crossmodal Correspondence, Perceptual Alignment.

The gist:

SLMs’ auditory judgments align poorly with human perception and miss the acoustic cues that drive human intuitions, while their shape perception remains intact in visual inputs.

Here are specific, actionable improvements for AI systems based on the findings of this paper:

) Improved System Capability: Perceptually Aligned Creative Generation Systems (SLMs)

The primary improvement is shifting Speech Language Models (SLMs) from generating outputs that are merely syntactically correct to generating outputs that align with human perceptual intuitions regarding sound and shape.

1. Acoustic-to-Shape Mapping in Text-to-Image Generation:

2. Improved System Capability: Contextually Perceptual Design Agents

The improved system can perform the following specific functions:

1. Generate images or 3D models based on spoken descriptions that adhere to sound symbolism rules (e.g., generating a sharp, angular robot when prompted with a name like bomu, instead of a round one).

2. Perform crossmodal matching between auditory input and visual output with high fidelity, moving beyond simple lexical association to genuine acoustic-perceptual correspondence.

3. Develop more robust internal representations that explicitly encode low-level acoustic cues (like spectral tilt) into high-level perceptual judgments, rather than relying solely on superficial language patterns or visual features.

) System Enhancement Strategy: Addressing the Modality Gap and Auditory Weakness

To achieve the above capabilities, focus development on mitigating the identified weaknesses in current SLMs:

1. Representation Learning for Acoustic Cues: Implement novel training objectives or fine-tuning strategies that explicitly force models to learn and utilize acoustic features (e.g., spectral tilt, formant structure) as predictors for perceptual qualities (roundness/sharpness), rather than relying on implicit correlations learned from text/image data alone.

2. Deep Layer Alignment Training: Utilize interpretability tools like the logit lens to identify and target specific layers within the SLM architecture where auditory representations begin to align with visual or human cognitive categories, focusing training efforts there to overcome the observed late divergence in open-weight models.

3. Omni-Modal Integration Refinement: For systems designed for multimodal interaction, prioritize architectures (like those found in Qwen3-Omni) that allow for seamless, high-fidelity integration of audio and visual streams to leverage the model's intact shape perception while simultaneously training the auditory component to match human acoustic intuition.

) Specific Technical Interventions Based on Experiment Findings:

1. For Auditory Alignment (RQ 1 & RQ 2): Train models using contrastive learning objectives that specifically reward representations that correlate with known acoustic cues (e.g., spectral tilt) when mapping sounds to target perceptual categories, rather than just rewarding correct sound classification.

2. For Crossmodal Matching (RQ 3): Develop specialized fine-tuning datasets that explicitly pair pseudowords with their corresponding shapes, forcing the model to learn the direct mapping between the auditory signal and the visual geometry, bypassing potential reliance on intermediate lexical nodes.

3. For Localized Failure Analysis (RQ 4): Implement rigorous ablation studies (as performed in Experiment 4) to quantify exactly where failure occurs—auditory judgment vs. shape perception—to guide targeted architectural fixes for modality-specific weaknesses.

Sound symbolism, the human tendency to map speech sounds to perceptual qualities such as roundness or sharpness, arises primarily from the acoustics of speech rather than spelling. Whether Speech Language Models (SLMs) share this tendency remains open, as prior evaluations rely on text or images rather than real speech. We study it using genuine human speech recordings, comparing model judgments against human data across the auditory, crossmodal, and visual components of the effect. We find that SLMs' auditory judgments align poorly with human perception and miss the acoustic cues, such as spectral tilt, that drive human intuitions, and open-weight models cannot reliably link a heard sound to its corresponding shape. With a visual-only control ruling out shape perception, the weakness localizes to how speech is represented, suggesting that perceptual alignment depends not on stronger vision but on speech representations that capture the cues humans hear.