Publication 01 · Audio-based image generation
MARS: Multimodal-Assisted Refined Semantic Alignment.
Audio-to-image generation with fine-grained semantic alignment and aesthetic quality — Mamba audio encoding meets MLLM-guided refinement.
Audio-to-image generation loses semantic detail in two usual ways: direct projection compresses audio too early, and text-intermediary pipelines drop what captions never capture. MARS keeps the audio representation long and structured, aligns it with fine-grained visual descriptions produced by multimodal LLMs, and fine-tunes the generator for aesthetics — improving semantic relevance by 28.73% and aesthetic scores by 127.35% over the strongest AIG baseline.
The question
Two families of methods dominate audio-to-image generation. Direct-projection models encode audio and map it into an image generator's embedding space — efficient, but the audio signal is compressed before alignment is ever learned. Caption-then-generate pipelines lean on pretrained text-to-image models, but the text medium drops auditory nuances no caption was written to carry. Both hit the same two walls: semantic misalignment and loss of visual detail.
MARS asks whether the two walls can be attacked at once — preserving a rich audio representation and controlling the aesthetic outcome of generation.

The approach
Three components divide the labor. A Mamba-based audio encoder handles long audio sequences without the quadratic cost that makes transformers awkward here. A fine-grained alignment strategy uses visual descriptions written by multimodal LLMs as the supervision signal — alignment is learned against rich descriptions, not scarce labels. An aesthetic perception generator then fine-tunes the image generator, so semantic fidelity is not paid for with visual quality.

What it shows
On VGGSound (37,250 records) and VEGAS (9,500 records), MARS outperforms existing methods with average gains of 28.73% in semantic relevance and 127.35% in aesthetic scores against the best AIG baseline.
Cross-domain evaluation on AudioCaps and Clotho — audio captions, not paired images — confirms the representation generalizes, with an average 73.9% improvement on the V2A metric.

Claim boundary
✓ Established
- A Mamba-based audio encoder for long-sequence AIG, aligned with MLLM-derived visual descriptions
- Aesthetic-perception fine-tuning that keeps quality on par with semantics
- State-of-the-art results on VGGSound and VEGAS, plus cross-domain evaluation on AudioCaps/Clotho
- An honest accounting: gains reported as averages against the strongest baseline
✗ Explicitly not claimed
- No claim of real-time generation or on-device deployment
- Benchmark claims are limited to the datasets tested
- No claim of biological-plausibility of the cross-modal mapping
- Therapeutic or accessibility applications are motivation, not validated outcomes
Citation
@article{xu2026mars,
title = {MARS: Multimodal-Assisted Refined Semantic Alignment},
author = {Xu, Junjie and Wu, Xingjiao and Zhang, Zihao and Yang, Shuwen
and Ma, Tianlong and Dong, Daoguo and He, Liang},
journal = {Information Processing \& Management},
volume = {63},
pages = {104292},
year = {2026},
doi = {10.1016/j.ipm.2025.104292}
}