Publication 01 · Audio-based image generation

MARS: Multimodal-Assisted Refined Semantic Alignment.

Audio-to-image generation with fine-grained semantic alignment and aesthetic quality — Mamba audio encoding meets MLLM-guided refinement.

Published · IPM 2026 Audio-to-image Mamba encoder Aesthetic fine-tuning

Junjie Xu, Xingjiao Wu*, Zihao Zhang, Shuwen Yang, Tianlong Ma, Daoguo Dong, Liang He · East China Normal University (* corresponding author)

Audio-to-image generation loses semantic detail in two usual ways: direct projection compresses audio too early, and text-intermediary pipelines drop what captions never capture. MARS keeps the audio representation long and structured, aligns it with fine-grained visual descriptions produced by multimodal LLMs, and fine-tunes the generator for aesthetics — improving semantic relevance by 28.73% and aesthetic scores by 127.35% over the strongest AIG baseline.

37,250VGGSound records used for training and evaluation
9,500VEGAS records in the second benchmark
+28.73%average semantic-relevance gain vs best AIG baseline
+127.35%average aesthetic-score gain vs best AIG baseline
01

The question

Two families of methods dominate audio-to-image generation. Direct-projection models encode audio and map it into an image generator's embedding space — efficient, but the audio signal is compressed before alignment is ever learned. Caption-then-generate pipelines lean on pretrained text-to-image models, but the text medium drops auditory nuances no caption was written to carry. Both hit the same two walls: semantic misalignment and loss of visual detail.

MARS asks whether the two walls can be attacked at once — preserving a rich audio representation and controlling the aesthetic outcome of generation.

t-SNE visualizations of MARS audio embeddings for the airplane-flyby and cow-lowing classes, with sample images from the clustered regions
Fig. 1t-SNE of MARS audio embeddings — (e) airplane flyby, (f) cow lowing. Embeddings of the same acoustic class cluster together; sample images from each cluster region are shown alongside.
02

The approach

Three components divide the labor. A Mamba-based audio encoder handles long audio sequences without the quadratic cost that makes transformers awkward here. A fine-grained alignment strategy uses visual descriptions written by multimodal LLMs as the supervision signal — alignment is learned against rich descriptions, not scarce labels. An aesthetic perception generator then fine-tunes the image generator, so semantic fidelity is not paid for with visual quality.

MARS overall framework: audio input, preprocessing, audio-based image generation, multimodal information integration and understanding
Fig. 2The MARS pipeline — audio in, image out, with multimodal information extraction and understanding bridging the two.
03

What it shows

On VGGSound (37,250 records) and VEGAS (9,500 records), MARS outperforms existing methods with average gains of 28.73% in semantic relevance and 127.35% in aesthetic scores against the best AIG baseline.

Cross-domain evaluation on AudioCaps and Clotho — audio captions, not paired images — confirms the representation generalizes, with an average 73.9% improvement on the V2A metric.

MARS qualitative results: per-class generated images vs reference and Sound2Scene
Fig. 3Per-class qualitative results. Reference images and Sound2Scene outputs are shown alongside MARS generations.
04

Claim boundary

✓ Established

  • A Mamba-based audio encoder for long-sequence AIG, aligned with MLLM-derived visual descriptions
  • Aesthetic-perception fine-tuning that keeps quality on par with semantics
  • State-of-the-art results on VGGSound and VEGAS, plus cross-domain evaluation on AudioCaps/Clotho
  • An honest accounting: gains reported as averages against the strongest baseline

✗ Explicitly not claimed

  • No claim of real-time generation or on-device deployment
  • Benchmark claims are limited to the datasets tested
  • No claim of biological-plausibility of the cross-modal mapping
  • Therapeutic or accessibility applications are motivation, not validated outcomes
05

Citation

@article{xu2026mars,
  title   = {MARS: Multimodal-Assisted Refined Semantic Alignment},
  author  = {Xu, Junjie and Wu, Xingjiao and Zhang, Zihao and Yang, Shuwen
             and Ma, Tianlong and Dong, Daoguo and He, Liang},
  journal = {Information Processing \& Management},
  volume  = {63},
  pages   = {104292},
  year    = {2026},
  doi     = {10.1016/j.ipm.2025.104292}
}