Catch adversarial content hiding in the vision channel before it reaches your agent.
Vision-language models read pixels, not text — and the attacks on that channel don't look like prompt injection. They look like screenshots. Like screenshots of a "system dialog" overlaid on a UI image. Like a tiny line of OCR-readable text rendered in 6-point gray inside a screenshot. Like a pixel-level perturbation that flips classification. Vigil tracks the five image-channel attack families Vision-Language Models (VLM) and multimodal LLM agents face in production, scores each through the same pattern-weight framework used for text-channel injection, and surfaces the result in the existing alert_signals pipeline so a single monitor covers every input channel.
Five image-channel attack families the vision literature converges on.
Prompt-injection detection engines look for tokens that match jailbreak patterns. Image-channel attacks don't speak the same language — they speak pixels. The five families below are the categories the multimodal-adversarial research consistently converges on, and the categories Vigil scores into the same alert_signals pipeline so a vision-channel attack shows up next to a text-channel one with the same operator workflow applied.
Direct vision prompt injection
Adversarial text — instructions, role assertions, permission manifests — overlaid inside a user-supplied image (large or rendered, often disguised as captions, watermarks, or "page footers") that the vision-language model OCR-decodes and reads as commands. The prompt arrives inside the image; the channel is legitimate.[1][2]
OCR exfiltration & hidden text in screenshots
Small-font, low-contrast, or anti-aliased text rendered inside an image (frequently at sub-12point sizes or against near-identical background colors) so a human observer can't read it but the model's OCR step will. The decoded text surfaces in the prompt as if the user had typed it. Documented in production OCR-exfiltration incidents against vision-enabled assistants.[1][3]
Adversarial visual perturbations
Crafted pixel-level perturbations — imperceptible to humans — designed to flip a vision-language model's classification, route its chain-of-thought, or coerce it into emitting content it would not have generated against the clean image. Belongs to the broader adversarial-ML literature (NIST AI 100-2 family) specialised for vision inputs.[5][2]
Screenshot-of-UI / document-image poisoning
Images that are screenshots of adversarial UIs — fake forms, fake authorization dialogs, fake "Allow access" prompts, fake system-prompt-shaped panels — carrying further instructions inside the rendered pixels. The screenshot is the attack surface; the rendered text inside it is the payload.[4][6]
Multimodal system-prompt smuggling
Image content — overlay text, embedded captioning, or an image whose visual subject is a system-prompt-shaped block — asserting a fake role, an instruction set, or a permission grant that the vision channel surfaces into the model's prompt hierarchy. The image becomes a system-message surrogate and the trust boundary at its source is what gets crossed.[7][6]
Operator cues that surface in the action log, mapped to Vigil's existing detection framework.
The cues below are not a parallel framework — they are the same per-pattern weight → aggregate severity → alert_signals row pipeline used for text-channel prompt injection, applied to the image-channel attack surface. OCR-decoded text re-enters the existing prompt-injection pattern matcher; image provenance is a per-action_logs row attribute; modality-specific perturbation screening lands as a per-modality weight in alert_signals. Same workflow, additional channel.
OCR-decoded text from an image triggers the existing prompt-injection pattern matcher in services/detector.js — a "ignore previous", a permission-broadening request, a hidden role assertion, an instruction-shaped payload the user did not type. The image supplied the text; the matcher catches it; the alert row carries both the source image and the matched pattern.[1][6]
Image channel carrying text at suspicious density or contrast — sub-12point rendering, low-contrast text against near-identical backgrounds, anti-aliased edges, hidden watermarks visible only to OCR. The right control surface is a quarantine bucket on the image + a manual review on the row in action_logs.[3][7]
Image source provenance does not match the user the agent is acting on behalf of — a screenshot taken on an attacker-controlled session flows into a context whose running user identity should never have seen it. Provenance is captured on the action_logs row as a per-image field; mismatch raises the existing cross-session-divergence signal in the alert digest.[4][5]
Vision channel answering in a role or persona inconsistent with the registered identity asserted at session start — a "system" block that wasn't there at registration, a capability grant that wasn't part of the role, a permission breadth that has no declared reason to exist on the image path. Same identity-shift cue the text path catches, but triggered off the vision channel's answer rather than the user's prompt.[7][6]
Decoded OCR text containing permission-broadening requests the system prompt never granted — "from now on you will…", "your new instructions are…", "as a developer-mode assistant you have access to…". Same handler shape as the existing role-play / persona bypass patterns; carried by an image instead of a chat message. Lands in alert_signals with the matching pattern weight applied.[2][1]
Six layered control surfaces for the vision channel.
Image-channel attacks are not a single detection problem. The vision channel has its own failure modes (OCR smuggled text, adversarial pixels, fake UI screenshots, system-prompt smuggling inside image content), and each needs its own control. The six below are the image-channel analogues of the controls the prompt-injection and cross-agent pages already use, scored through the same per-pattern weight framework so an alert on one channel is exactly an alert on another.
Image vs. text channel segmentation
Treat OCR-decoded text from an image as a separate, restricted input — never concatenate it with the user prompt into a single flat instruction stream. Once they're merged, the trust boundary is gone. The vision channel stays a vision channel; the decoded text goes through the existing prompt-injection pattern matcher as its own input.[6][7]
Pattern pre-screening on OCR-decoded text
Run OCR-decoded text back through the existing per-pattern weight matcher in services/detector.js before it reaches the model. A "ignore previous" rendered at 8-point gray inside a screenshot is the same attack shape as the same text rendered in a chat box — score both through the same pattern weights; aggregate severity is identical.[1][2]
Image provenance tagging
Every image carries source + timestamp + serving agent id as metadata on the action_logs row. Untagged images, or images whose source doesn't match the user the agent is acting on behalf of, run at reduced trust — the same control surface used in cross-agent memory provenance, extended to the image carrier.[4][5]
Screenshot / UI-image allowlists
Explicit deny on screenshots of forms, authorization dialogs, system-prompt-shaped panels, and known fake-UI templates. Allowlist of approved UI-image sources only; everything else is quarantined for operator review. Image-channel equivalent of the least-privilege tool scoping control on the text side.[4][3]
Adversarial-perturbation screening
Modality-specific weight in alert_signals: a row raised against the vision channel is scored against the perturbation-screening table in addition to the OCR-text channel. Per-modality weight stack keeps the existing severity scoring intact and adds image-perturbation relevance without a parallel pipeline.[5][2]
Cross-modality divergence scoring
Score the image-channel answer against an expected baseline derived from the text-channel instructions. A vision answer that diverges from the expected baseline by an amount correlated with the OCR-decoded text incident on the image is a high-signal alert — image-channel equivalent of the cross-session-divergence signal on the cross-agent page.[6][7]
Image-input research and sources cited.
The taxonomy, detection signals, and mitigations above are grounded in the multi-modal and vision-adversarial research below. Inline citation chips elsewhere on this page map to the numbered entries here. Image-input citation numbering uses an independent cite-iN page-local namespace — distinct from cite-N on the prompt-injection page and cite-xN on the cross-agent page — so the three detection pages can each stand on their own.
- [1] Prompt injection from an image — multimodal injections and hidden instructions simonwillison.net/2023/Oct/14/multi-modal-prompt-injection
- [2] Abusing Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMs arxiv.org/abs/2403.15591
- [3] Multi-Modal Attack Research — OCR Exfiltration Feasibility in Vision Inputs (research deliverable) simonwillison.net/2023/Oct/14/multi-modal-prompt-injection
- [4] Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection arxiv.org/abs/2302.12173
- [5] Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations (AI 100-2) nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-2.pdf
- [6] Defending Against Indirect Prompt Injection Attacks With Spotlighting arxiv.org/abs/2404.07560
- [7] OWASP Top 10 for LLM Applications — LLM01: Prompt Injection owasp.org/www-project-top-10-for-large-language-model-applications