FusionAudit

Abstract

As vision-language models (VLMs) transition from frozen-encoder pipelines to native multimodal architectures, the visual channel increasingly serves as both a perceptual input and a carrier of rendered text that competes with user prompts. Current single-attack, pixel-centric evaluations fail to capture this complexity: they collapse distinct visual attack surfaces into a single scalar and ignore how architectural shifts alter vulnerability. We introduce PaRMA, an evidence-based auditing framework that defines pathway-conditioned risk over two primary image-level interventions: pixel perturbation and rendered-text insertion. Instead of arbitrary attack enumeration, PaRMA uses pathway-specific diagnostics to expose architecture-specific blind spots. Notably, we uncover a see-but-not-deceived signature where native VLMs attend heavily to typographic overlays but visually reject them, and we identify an objective mismatch between encoder-level and answer-level attacks in deep-fusion models. To exploit these findings, we instantiate FusionAudit, introducing two diagnostic-motivated attacks: MANTIS (margin-aware pixel perturbation) and Nat-Evid (host-congruent semantic insertion). Evaluating 18 open-weight VLMs across a novel nativeness taxonomy, we demonstrate that legacy single-attack rankings (e.g., PGD) are highly unstable for native models. Our diagnostic-driven attacks substantially raise the estimated residual risk, showing that PaRMA provides a falsifiable, evidence-driven auditing methodology for uncovering vulnerabilities missed by standard benchmarks.

Nativeness levels

L₁: Alignment-era. A frozen or lightly adapted vision encoder, linked to the language model by a projection.SmolVLM2, BLIP2, InstructBLIP, LLaVA-1.5
L₂: Shallow fusion. Joint multimodal pretraining; visual tokens enter the language model once, as a fixed prefix.Qwen-VL-Chat, Qwen2-VL, Qwen2.5-VL, Gemma3, DeepSeek-VL
L₃: Deep or early fusion. Visual information takes part throughout the language model’s computation.Qwen3-VL, Qwen3.5, Gemma4

Key findings

+25.1pts

risk added by MANTIS and Nat-Evid on deep-fusion (L₃) models

+16.9 on L₂ and +10.6 on L₁, level means

8of 18

models move three or more places when ranked by full risk instead of End-PGD alone

Spearman ρ = 0.837 between the two rankings

+135%

rise in attention on a pasted wrong answer in L₃ models, yet Typo-Evid flips only 13.2% of their correct answers

+138% on L₂, where it flips 20.9%

Risk before and after MANTIS and Nat-Evid

Risk per model (%)
Model	R-hat-0	R-hat-plus
SmolVLM2-2.2B	86.7	100.0
BLIP2-OPT-2.7B	83.5	91.2
InstructBLIP-Vicuna-7B	87.7	99.5
LLaVA-1.5-7B	87.6	97.2
Qwen-VL-Chat	85.3	97.9
Qwen2-VL-2B	74.5	89.8
Qwen2.5-VL-3B	72.0	85.7
Gemma3-4B	66.5	85.4
Qwen2.5-VL-7B	54.2	79.5
Gemma3-12B	60.2	81.5
DeepSeek-VL-7B	87.9	99.2
Qwen3-VL-2B	51.1	82.7
Qwen3-VL-4B	44.7	78.9
Qwen3-VL-8B	46.0	73.1
Qwen3.5-2B	46.5	78.0
Qwen3.5-4B	50.4	68.1
Qwen3.5-9B	43.4	59.5
Gemma4-E4B	71.4	88.7
L1 mean	86.4	97.0
L2 mean	71.5	88.4
L3 mean	50.5	75.6

Share of each model’s correctly answered examples that at least one attack breaks, on 1500 ChartQA, DocVQA and MME questions.

Seen, but not adopted

Attention maps of Qwen3.5-2B on an MME painting, clean and with an injected banner; attention concentrates on the banner.

A typographic banner asserting The correct answer is: yes draws Qwen3.5-2B’s attention (the overlay attention ratio rises from 2.49 to 4.85), but the model keeps the correct visual answer, “no”.

Native models read a pasted wrong answer and seldom adopt it. Nat-Evid writes the wrong answer into the image as host-congruent content, such as a chart annotation or scene text.

BibTeX

@inproceedings{zhang2026fusionaudit,
  title     = {FusionAudit: Pathway-Conditioned Robustness Auditing
               for Native Multimodal Models},
  author    = {Zhang, Jiaming and Li, Zherui and Guo, Siqi and
               Dong, Junhao and Wang, Kun and Lim, Wei Yang Bryan},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026}
}