Defeating Prompt Injections by Design | Summary

This article explains the key points of Defeating Prompt Injections by Design.

• 2025-03-24 (arXiv)
• Debenedetti, Edoardo, Shumailov, Ilia, Fan, Tianqi, Hayes, Jamie, Carlini, Nicholas, Fabian, Daniel, Kern, Christoph, Shi, Chongyang, Terzis, Andreas, Tramèr, Florian.
• Google, Google DeepMind, ETH Zurich
• Paper
• Github

Read this article in Korean

Summary

• Large language model (LLM) agents, systems that use language models to operate tools, can follow malicious instructions hidden in retrieved emails, documents, or tool results. On AgentDojo, a benchmark for tool-using agent utility and security, CaMeL achieves 77.3% benign task success with o3 High versus 84.5% using the same model’s native tool-calling interface; with security policies, this configuration records zero successful attacks in the evaluated attack set.
• CaMeL, short for CApabilities for MachinE Learning, separates trusted planning from untrusted data processing and enforces security policies, rules governing permitted tool calls, through a custom Python interpreter. It tracks control flow, the actions a program executes, and data flow, how values propagate between actions, using capabilities: metadata recording each value’s sources and permitted readers.
• The design introduces model-dependent utility losses, additional token usage, and policy-maintenance costs. Its guarantees depend on policy coverage and correct dependency tracking; misleading textual output and some side channels, leaks through observable behavior such as termination or request counts, remain outside its protection. Formal verification, a proof that the implementation satisfies its security requirements, is left to future work.

1. Introduction

Prompt injection arises when external content introduces instructions that redirect an agent toward data theft or unwanted actions. CaMeL applies software-security controls outside the model: it requires no changes to model weights and assigns policy enforcement to the interpreter.

• The released implementation is available at https://github.com/google-research/camel-prompt-injection.

2. Defeating Prompt Injections by Design

The motivating task is to find meeting notes, extract Bob’s email address and a requested document name, retrieve the document, and send it. The Dual LLM pattern, an architecture separating a trusted planner from a tool-free processor of external content, protects planning but still allows an attacker to manipulate extracted recipients or filenames.

• CaMeL adds value-level provenance, a record of data origin, and reader checks so that an unchanged sequence of actions cannot freely transmit confidential material to an attacker-selected recipient.
• A legitimate recipient extracted from untrusted notes may also require approval when that recipient lacks existing access to the document.

The document-sharing workflow separates the sequence of actions from the values passed between them. An attacker can redirect the workflow by changing either the planned actions or their arguments.

3. Threat Model

The primary adversary controls data that the agent retrieves, such as an email, document, or spreadsheet, and attempts to induce unauthorized actions through that data. The threat model assumes a trusted user prompt, excludes users pasting untrusted instructions into that prompt, and assumes any agent memory has not been compromised.

The tool sequence remains recognizable while the document name and recipient become attacker-selected values. Protecting the planner alone therefore leaves a route for confidential data to reach an unauthorized recipient.

3.1. Explicit non-goals of CaMeL

CaMeL does not prevent text-to-text attacks that alter summaries or recommendations without causing prohibited information flows, including phishing messages presented to the user. It also does not promise complete autonomy: ambiguous requests, uncertain results, or policy restrictions may require human clarification or approval.

• Source annotations could help an interface identify where displayed content originated, but they do not establish that the content is truthful or harmless.

4. The Prompt Injection Security Game

PI-SEC, the paper’s prompt-injection security game, models an adversary choosing an initial tool-accessible state to cause an unsafe execution. The agent produces a Trace of (tool, args, mem_step) tuples, recording each tool call, its arguments, and the state at execution; the adversary wins when an action falls outside Ω_prompt, the allowed-action set for the user prompt.

• Ω_prompt defines security conceptually; CaMeL does not enumerate this set during execution.

The verifier checks recorded tool actions against the permitted set. This defines the security objective without providing a practical method for enumerating every permitted action.

5. CaMeL Design

CaMeL combines two model roles, tools, security policies, value capabilities, and an interpreter that executes generated code while propagating dependencies. Before each tool call, the interpreter checks whether the proposed action and its inputs satisfy the configured policies; a failed check halts execution in the security-game description.

5.1. The Privileged and Quarantined LLMs

The Privileged LLM (P-LLM), the model that writes the execution plan, receives the trusted request and tool specifications but never sees tool-returned values or parsed external content. The Quarantined LLM (Q-LLM), the model that parses untrusted data without tool access, returns structured output conforming to a predefined schema, a specification of output fields and types.

• The additional boolean field have_enough_information can trigger NotEnoughInformationError. The Q-LLM cannot explain what information is missing to the planner because that explanation could carry an injection.
• The two roles can use different underlying models. Running the data-processing role locally is discussed as a possible privacy benefit, rather than an evaluated deployment result.

Planning and external-content parsing occupy separate model roles, while the interpreter maintains dependencies and checks policies. The execution system makes the security decision even if malicious content influences the parsing model’s output.

5.2. Security Policies

Security policies are Python functions that inspect tool names and arguments and return Allowed or Denied with a reason. The calendar example permits sharing when participants come from a trusted source; otherwise, it checks whether participants may read the event’s title, description, location, start_time, and end_time.

• The implementation supplies policies through the CaMeL engine; user-defined, provider-defined, and automatically derived policies are proposed extensions.
• The policy language permits arbitrary Python logic, making policy correctness and conflict resolution part of the security burden.

5.3. Capabilities

Capabilities attach provenance and allowed-reader information to individual values, including derived values. Readers are represented by Public or a set of users, while sources distinguish User literals, CaMeL transformations, particular tools, and optional inner sources such as an email sender.

• This granularity can distinguish information a booking tool legitimately needs, such as names and telephone numbers, from information it should not receive, such as passport numbers.

5.4. CaMeL Interpreter

The custom interpreter executes a restricted Python subset by traversing its abstract syntax tree, the parsed representation of program structure, using Python’s ast library. It tracks value dependencies, checks policies before tool execution, and allows up to 10 planner attempts to repair execution errors while preserving interpreter state.

• Exception text dependent on untrusted values is redacted before reaching the P-LLM.
• STRICT mode propagates dependencies from conditional tests and loop iterables into their blocks; NORMAL mode tracks fewer implicit dependencies, which arise when a value affects whether an operation executes.
• The implementation provides neither atomic execution, in which a sequence completes as one indivisible operation, nor rollback of operations that already produced side effects.

The outgoing email depends on the Q-LLM’s conditional decision as well as the email content. Recording that implicit dependency matters because whether an email is sent can itself disclose information.

6. Evaluation

Evaluation uses AgentDojo’s Workspace, Banking, Travel, and Slack environments. Utility measures successful completion of user tasks, while security measures whether an attacker achieves its adversarial objective.

• The comparisons include native tool-calling interfaces, CaMeL with isolation alone, CaMeL with policies, and existing AgentDojo defenses.
• The principal security comparison uses 949 attacks. These experiments measure behavior on the benchmark and do not formally verify the implementation.

6.1. Utility evaluation

Utility varies substantially by underlying model: o3 High drops from 84.5% to 77.3%, o4 Mini High from 79.4% to 76.3%, and Gemini 2.5 Pro from 73.2% to 41.2%. Both evaluated Gemini models obtain 0.0% benign Travel utility with CaMeL, whereas Claude 4 Sonnet obtains 75.0%. Using Claude 3.5 Haiku as Q-LLM with Claude 4 Sonnet as P-LLM reduces utility by about 1% for an estimated 12% median-task cost reduction.

• 6.1.1. CaMeL has a (small) Impact on Utility: the detailed results support usable performance for some models, but the utility loss is not uniformly small.
• 6.1.2. How does CaMeL impact utility?: failures arise when required actions are specified only in external data, tool-return formats are undocumented, or the planner supplies insufficient context to the Q-LLM. Table 1 attributes 13 Claude 3.5 Sonnet failures to undocumented Travel interfaces.
• 6.1.3. Cheaper Q-LLM Models Have Little Impact on Utility: the paper also reports a similar utility drop with GPT 4.1 Nano as Q-LLM and o4 Mini High as P-LLM.

The suite-level bars expose uneven utility costs across models and environments. Travel utility falls to zero for the two Gemini models, while some Banking and Travel results improve for o3 High and o4 Mini High.

The table attributes 13 Travel failures to undocumented interfaces and identifies five cases each of data requiring action and insufficient Q-LLM context. Its query-misunderstanding row totals 1, although the caption describes 2 instances.

The complete benign utility results support the o3 High comparison of 77.3% with CaMeL versus 84.5% natively. Utility costs differ substantially across models: Gemini 2.5 Pro loses 32.0 percentage points overall.

6.2. Security evaluation

Isolation alone sharply reduces benchmark attack success, and adding policies further constrains tool arguments and information sharing. Table 4 records zero successful attacks for Gemini 2.5 Pro and o3 High with policies, but retains 11 for each Claude 4 Sonnet configuration, 1 for Gemini 2.5 Flash, and 1 for o4 Mini High.

• 6.2.1. Case study: without any Security Policies: residual benchmark successes include document-directed payments and displaying an injected hotel review. The authors distinguish these cases from the protected threat model, particularly when merely printing a review satisfies the evaluator.
• 6.2.2. Case study: with Security Policies: Workspace supports more precise reader annotations and generally triggers fewer policy interventions than Slack or Banking. Figure 10 conditions triggering rates on successfully solved tasks; these rates indicate potential approval burden rather than measured human approval behavior.
• The prose and Figure 9 describe 300 native-interface successes for Gemini 2.5 Pro, whereas Table 4 reports 163 ± 140.1. Both report zero with CaMeL; the native-interface count is internally inconsistent.

Attack counts fall sharply with isolation, with policies adding restrictions on arguments and information sharing. Residual benchmark successes require interpretation against the threat model, and the plotted Gemini 2.5 Pro native count differs from Table 4.

Policy-triggering rates vary by suite and model, reflecting differences in available identity and reader annotations. The denominator includes only successfully solved tasks, and the rates indicate potential intervention burden rather than observed user approvals.

The table reports nonzero CaMeL attack counts for several model configurations, so zero success cannot be claimed across all configurations. Its Gemini 2.5 Pro native count of 163 ± 140.1 conflicts with the main-body value.

6.3. Baseline comparisons

The baselines include Prompt Sandwiching, which repeats the original task after tool outputs; Spotlighting, which marks untrusted content; and Tool Filter, which restricts available tools. With Claude 3.5 Sonnet, Table 7 reports 0 successful attacks for CaMeL with policies, 1 without policies, 24 for Prompt Sandwiching, 5 for Spotlighting, 8 for Tool Filter, and 44 for the undefended model. Table 5 reports benign utility of 63.92% for CaMeL versus 90.72% undefended.

• The prose calls Tool Filter the next-best defense, but Table 7 gives Spotlighting fewer successful attacks: 5 versus 8.
• The paper also reports 276 successful attacks against GPT-4o Mini’s native interface and none with CaMeL, showing that the model’s learned instruction priorities do not provide equivalent protection in this evaluation.

The paired panels show the security and utility tradeoff for Claude 3.5 Sonnet. CaMeL with policies records no successful attacks, while its task utility is lower than the prompting defenses in several suites.

The baseline counts establish CaMeL’s zero-success result with policies for Claude 3.5 Sonnet. Spotlighting records 5 successful attacks versus Tool Filter’s 8, contrary to the prose ranking of the next-best defense.

6.4. Case study: when data flow becomes control flow

A fixed program can still let external data choose among available tools, effectively turning data flow into control flow. The case study examines generated plans that parse email instructions into actions and arguments, demonstrating why model isolation must be supplemented with policies governing those values.

• The authors describe the illustrated attack as a weaponized simplification of generated solutions, rather than the exact original model output.
• Splitting malicious behavior into individually innocuous steps can obscure the overall intent from models processing only local portions of the task.

A program that dispatches tools according to parsed email instructions lets external values determine subsequent actions. A predetermined program structure can therefore still express behavior controlled by untrusted data.

6.5. Overheads

The main text reports median token multipliers of 2.82× input and 2.73× output for Claude 3.5 Sonnet, attributing much of the overhead to code generation and repair. Counts use the GPT-4o tokenizer through tiktoken, a tokenization library, exclude tool-documentation tokens, and include CaMeL’s language-specification prompt.

• The numerical reporting is inconsistent: Table 10 lists benign medians of 2.73× input and 2.82× output, while Table 13 lists under-attack medians of 2.02× input and 2.64× output.
• The paper describes interpreter processing as small relative to model execution, but the quantitative overhead results concern tokens rather than a complete deployment latency benchmark.

The distributions show substantial task-to-task variation in token overhead. The caption’s 2.82× input and 2.73× output claims do not match the under-attack medians reported in Table 13.

7. Side Channels Can Be a Problem

The paper examines side channels through request counts, execution termination, and timing. STRICT mode can block the illustrated request-count leak by making each operation within a loop depend on its private loop bound.

• The timing example would convert a private value into an observable elapsed duration. That example is unavailable in the implementation because the time module is excluded, although other timing channels are not ruled out.

Exception Handling – Inferring private variable based on halted execution

An attacker can make Q-LLM output trigger an exception conditional on a private value, then observe whether a later email or network request occurs to learn one bit. STRICT mode mitigates this route by making every subsequent statement depend on the Q-LLM call’s inputs.

• The mitigation covers Q-LLM-induced exceptions and dependent operations, but exceptions caused through manipulated tool-accessible data remain a possible route.
• The authors propose explicit result-based error handling, where success and failure are represented as values, as an alternative that would expose dependencies more directly.

Both examples make termination depend on a private value, allowing a later observable action to disclose one bit. Extending dependencies beyond a Q-LLM call is the proposed STRICT-mode mitigation for that route.

8. Secondary attack scenarios that CaMeL can help with

The paper discusses two additional applications: preventing an installed spy tool from receiving internal data, and preventing a rogue user from sending confidential documents outside an organization. These scenarios require policies whose authority extends beyond the individual user request and are not explicitly evaluated by AgentDojo.

9. Discussion

The discussion examines the effort required to integrate capabilities, the burden of user approvals, and remaining attack routes. The security argument depends on the surrounding software and policy system as well as model isolation.

9.1. Past problems with adoption of capabilities

Capability-based systems can require substantial implementation effort and adaptation of existing tools and infrastructure. A controlled workspace is more tractable than an environment with heterogeneous third-party services, although the authors suggest that an agent could sometimes manage capabilities centrally.

9.2. De-classification and user fatigue

De-classification, relaxing restrictions to permit an otherwise prohibited disclosure, creates a usability and security tradeoff. Repeated or ambiguous approval requests may desensitize users and lead them to authorize malicious operations, so practical protection depends on policy precision and the approval interface.

9.3. So, Are Prompt Injections Solved Now?

The authors explicitly state that prompt injection and AgentDojo are not fully solved. Policies require specification and maintenance, side channels remain possible, and individually permitted actions might be composed into harmful behavior.

• The analogy to return-oriented programming, chaining permitted code fragments into an exploit, motivates a possible failure mode; it is not a demonstrated general bypass of CaMeL.

10. Future work

Future work includes languages with more explicit error and input/output handling, formal verification of the interpreter and policy enforcement, and automated policy generation informed by context. These are proposed directions; the paper does not prove that the current implementation is fault-free.

11. Conclusion

The conclusion presents CaMeL as a system-design approach that constrains the consequences of vulnerable model behavior through isolation and enforceable policies. It can be combined with training or prompting defenses that improve the underlying model, subject to the policy, implementation, and side-channel limitations examined in the paper.

Contributions

Ilia Shumailov originated the idea and technical design proposal; Edoardo Debenedetti led technical development, and the two co-led CaMeL’s design. Tianqi Fan organized the internship proposal and helped coordinate the project, while the remaining authors contributed technical feedback, reviews, and manuscript development.

Appendix

• A. Extended Related Work and A.1. Background: Software Security Concepts explain the security terminology underlying CaMeL. A.1.1. Computer security nomenclature defines policies, control flow, data flow, and capabilities; A.1.2. Access and Information Flow Controls connects the design to preserving intended program behavior, restricting access, and tracking information to prevent unauthorized disclosure. A.1.3. Tool-calling LLM Agents describes the conventional loop in which tool outputs return to the model and influence subsequent calls.
• A.2. Prompt Injection Defenses reviews delimiters, repeated task reminders, training, detection, monitoring internal model representations, and architectural separation. It identifies Willison’s Dual LLM pattern as foundational and distinguishes CaMeL’s explicit dependency tracking and fine-grained annotations from approaches using classifiers or coarser trust labels.
• B. Full results tables gives benign utility, utility under attack, and attack counts by model and suite. C. Baseline results supplies the Claude 3.5 Sonnet defense comparisons: CaMeL’s under-attack utility is 66.70% versus 86.83% undefended, while its successful-attack count is 0 versus 44. The tables distinguish preventing attacker objectives from completing legitimate tasks.
• D. NORMAL vs. STRICT modes quantifies the additional policy interventions associated with broader dependency tracking. Overall triggering increases from 33.87% to 53.23% without attack and from 26.54% to 45.34% under attack; the accompanying NORMAL-mode dependency graph omits the conditional Q-LLM dependency on send_email.
• E. Security Policy evaluation lists domain-specific policies for payments, password changes, messages, web requests, calendar operations, and file operations. Examples require trusted origins for sensitive changes and reader compatibility for disclosed content; the exact policy code is linked at https://github.com/google-research/camel-prompt-injection/tree/45b656a0aaf688626955b12823ec0bd6af9f7df7/src/camel/pipeline_elements/security_policies.
• F. Extended overheads results provides token totals and per-task multiplier distributions for benign and adversarial conditions. The separation between means and medians shows substantial variation across tasks: Table 13 gives CaMeL input multipliers of 6.17 mean and 2.02 median under attack, and output multipliers of 5.43 mean and 2.64 median. These tables do not consistently reproduce the main text’s input/output figures.
• G. Breakdown for individual failure modes for Claude 3.5 Sonnet v2 maps failures to individual tasks. H. Implementation details and H.1. System prompt document Built-in types, Built-in functions, Built-in methods, Imported classes, Tools functions, and Available types, defining the planner’s supported programming environment. Important notes on the code and Information on the language describe structured parsing, preserved state, and restrictions including no imports, eval, exec, while loops, generators, break, continue, or new functions.
• H.2. Exceptions prompt documents error-based code repair and redaction of untrusted exception content. H.3. Case-study: when control flow turns into data flow from Claude and o3-mini supplies generated examples that extract action names and arguments from email and dispatch tools accordingly. These examples support the main-body warning that an isolated planner can still produce a program whose behavior is selected by external data.
• H.4. Examples of failed tasks includes H.4.1. Workspace 11: Query misunderstanding, where the model uses the wrong temporal reference; H.4.2. Slack 11: Data Requires action, where required contact information resides on an unvisited website; and H.4.3. Workspace 18: Wrong assumptions from P-LLM, whose introductory description mentions an assumed hike time although the displayed transcript repeats the lunch example. H.4.4. Banking 2: Not enough context for Q-LLM asks for payment-account information absent from a rent notice, while H.4.5. Workspace 36: Q-LLM overdoes it/Strict eval expands a packing list instead of reproducing the expected content.
• H.4.6. Banking 14: Ambiguous task illustrates an uncertain suspicious-transaction judgment; its introductory year differs from the displayed transaction date. H.4.7. Travel 19: Underdocumented API shows repeated parsing and repair failures involving an application programming interface (API), while H.4.8. Travel 19: AgentDojo Bug describes an evaluator expecting a day when the response provides operating hours. I. More sophisticated side-channel develops a meeting-summary scenario in which exception-dependent completion can reveal whether a private calendar contains a meeting with Bob, potentially across up to 10 repair attempts.

Brief Thoughts

CaMeL’s main contribution is a concrete architecture that places security enforcement outside the language model. The document-sharing and data-driven dispatch examples explain why both planning isolation and value-level checks are needed. For deployment, the benchmark results should be assessed alongside model-specific utility losses, actual policy coverage, tool annotations, potential approval burden, and implicit information flows; the numerical inconsistencies and absence of formal implementation verification limit stronger claims.