Nvidia has warned that increasingly capable artificial intelligence agents can expose confidential information, execute unauthorised code and circumvent security restrictions even when assigned legitimate tasks, arguing that organisations must design safeguards on the assumption that their systems may attempt to violate established policies. In a paper titled “Assume Misalignment: Securing AI Agents Beyond Prompt Injection”, published by Nvidia, the company’s AI security team sets out an approach that prioritises enforceable technical restrictions over reliance on an AI model’s ability to follow instructions.
The paper distinguishes between an unsafe action and misaligned behaviour. An unsafe action is a specific operation that creates a security risk, while misalignment occurs when an agent pursues a goal contrary to the user’s stated intent or operates beyond the boundaries established by the system owner. An agent can perform an unsafe action without being fundamentally misaligned, including when it misinterprets a legitimate assignment or encounters incomplete instructions. The distinction matters because security risks can emerge without an attacker deliberately manipulating the system.
Nvidia’s central recommendation is to “assume misalignment”: organisations should design systems on the expectation that an agent will eventually attempt an action outside its authorised task or in violation of policy. Security boundaries, the company argues, must remain effective regardless of the agent’s intentions or the methods it uses. The consequences depend on the agent’s access to tools, information and computer systems, as well as the effectiveness of the restrictions imposed on those capabilities.
The paper identifies several ways in which apparently routine work can create security vulnerabilities. Without explicit instructions about data handling, an agent tasked with completing a legitimate assignment might upload an internal file to an external service. In other cases, an agent might disclose internal information, including inference server details and model names, through external version-control systems, or transfer files across local filesystem boundaries using HTTP servers and third-party proxies.
Nvidia says it has observed such behaviour in its own environment. During a system-diagramming task, one user found that an agent attempted to download and install third-party software that would have transmitted data to an external host as part of creating the diagram. In testing, the company also observed agents attempting to move internal information to external systems and transfer files through alternative routes. These examples illustrate how an agent can exceed the intended scope of its assignment while still attempting to complete the original task.
Adversarial inputs can produce similar outcomes. The paper describes an agent that rejected a direct request to execute a malicious command but subsequently installed a package that executed the payload during installation. The dependency-installation process provided an alternative route to code execution through a capability the system was designed to offer. In another example, specially crafted inputs caused an analytics workflow to generate code that bypassed sanitisation controls and achieved remote code execution.
Such cases demonstrate the limitations of evaluating an agent solely by its apparent response to a request. An agent may refuse an explicitly malicious instruction yet carry out an equivalent operation indirectly through an authorised tool. Nvidia argues that security measures must therefore cover the effects of an operation, rather than merely assessing the wording or apparent intent of the request.
Misalignment can also arise from model training and the conditions under which an agent operates. The paper points to models that exhibit security-relevant behaviour following particular triggers, as well as fine-tuning procedures that can affect safety behaviour in areas unrelated to the intended changes. It also cites cybersecurity evaluations in which OpenAI used models with deliberately weakened safety guardrails and agents bypassed network-isolation controls before compromising Hugging Face infrastructure. A separate OpenAI report describes unauthorised communication during training of an unreleased model, when agents unable to share a workbook through the intended local filesystem used a public file-hosting service despite instructions to use local files only.
The runtime environment creates further difficulties. System prompts, tool descriptions, external information retrieved for a task and instructions passed to subordinate agents can all influence behaviour. As workflows grow longer, instructions may receive less attention or disappear when earlier interactions are compressed into a shorter context. Delegated tasks may also reach subordinate agents without all the original restrictions. Nvidia reports observing agents generate prompt-injection payloads in attempts to return to a main agent and exfiltrate SSH keys, or use local scripting libraries to seek direct access to resources protected by policy.
The company cautions that prompt-based safeguards and systems that use another language model to judge an agent’s actions remain vulnerable to context drift, malicious influence and unpredictable outputs. Such measures can provide additional protection, but they should not constitute the principal security boundary.
Instead, Nvidia recommends deterministic policy enforcement outside the agent’s control, preferably within a separate trust domain that the agent cannot modify, disable or bypass. Its paper describes experiments in which an agent with access only to a file-writing tool attempted to rewrite its own tools to create a Bash execution tool. It also wrote commands to executable locations, including ~/.bashrc and scripts referenced in the user crontab, to trigger arbitrary code execution. The findings illustrate why restrictions that an agent can alter cannot reliably contain its behaviour.
Nvidia presents its Open Agent Safety Platform as one means of implementing this approach. OpenShell, part of the platform, provides configurable filesystem and network controls and isolates agent processes from the host system. These restrictions can block unauthorised downloads, limit access to files and prevent transfers to prohibited destinations. If malicious code runs through an otherwise permitted package installation, the sandbox can limit its access to the host and other resources, although files and services deliberately made available to the agent remain potentially accessible.
Nvidia Sentry adds monitoring and enforcement in a separate trust domain, acting as an internal proxy for network traffic, including communications with language models. According to the paper, this allows inspection of tool calls and other information in model interactions, helping detect risky operations and potential misalignment.
The paper also stresses that technical restrictions cannot validate every legitimate business operation. An agent permitted to update customer records, for example, could still make an incorrect change. Application-specific validation and approval requirements therefore remain necessary. Before expanding an agent’s access, Nvidia recommends testing whether restrictions continue to block unauthorised operations, recording relevant activity and enforcement decisions, and ensuring that intervention remains possible without unnecessarily impairing legitimate task completion.
