AI agent audit trail: sandboxes log the denies, not the allows

The sentence I need to correct

In the previous post I wrote that the OpenShell decision log is the only place in your stack where an injection attempt is visible, and that nothing else knows the agent tried. Three denies to unlisted hosts inside a session that started with the agent reading an external document is the shape of an injection, and the log shows it.

That is still true. It is also incomplete, and I found out by attacking our own gateway.

We put an MCP gateway in front of the tools our reference agents use: authentication, a tool manifest, deny-by-default egress. Then we ran two classes of attack against it. Class A: call a tool that is not in the manifest, or make an allowed tool fetch the cloud metadata address 169.254.169.254. Class B: call an allowed tool with a hostile argument, a path traversal in a filename, an injection in a query string.

Class A was refused every time, and every refusal is a span in the trace with error.type = GatewayDenied.egress_host_not_allowed or tool_not_in_manifest. You can see the attack from across the room.

Class B went through, and the trace showed a normal tool call. Correct service, correct tool, correct status. Nothing to look at.

The lesson fits on one line, and it is the reason for this post: the trace records what you blocked, not what got past you. We closed the class B cases the same day by adding argument constraints to the manifest, which turned them into arg_constraint_violated refusals. But notice what actually changed. We did not make the successful attack visible. We made more attacks into policy violations. What no gateway can read is an argument that conforms to its declared shape, and exfiltration through a host you allowed.

That is the authorization layer from the previous post, seen from the evidence side. Every action was permitted. There is no deny to find.

What an AI agent audit trail has to be

If the deny log cannot carry the authorization layer, something else has to, and it has to have four properties.

Complete. Every model call, every tool call, every hop between the agent, the gateway and the tool server, with parent and child relationships intact. Not a summary the agent writes about itself.

Outside the agent's reach. Written by the runtime and the gateway, not by the agent process. An agent that can be talked into exfiltrating a .env file can be talked into editing its own log.

Durable after the agent. Agents are short-lived. The question “what did this agent do on Tuesday” arrives three weeks later, from an auditor who does not know what a trace is.

Free of the payload. This is the one the market gets wrong, and I will spend a section on it.

Here is how we built those four into shoofi, with the measurements.

Complete: the gateway hop is the interesting one

Most agent observability ends at the agent's own spans. The place where policy is applied is the gateway, and the gateway is a separate process. If it does not emit its own spans, with the right span kinds, you get a trace that stops at “the agent called a tool” and never shows what the tool did or whether it was allowed.

In OpenTelemetry a span has a kind. CLIENT means “I called someone”, SERVER means “someone called me”. When a call crosses a process boundary, the caller's CLIENT span and the callee's SERVER span are what let a trace store recognise that two services talked. Without that pair, the gateway's work collapses into the agent's span and the gateway disappears from the trace, which is the one place you needed it.

Our gateway emits two spans per decision: a SERVER span for the leg it answers, a CLIENT span for the leg it initiates. The MCP server behind it parents its own SERVER span on the gateway's CLIENT span, which is what makes the service boundary appear in the trace instead of dissolving into one flat list. Here is a single trace, six spans, as stored:

mcp-probe-client   CLIENT  tools/call read_note
  mcp-rig-gateway  SERVER  execute_tool read_note
    mcp-rig-gateway CLIENT execute_tool read_note
      mcp-rig-server SERVER tools/call
mcp-probe-client   CLIENT  tools/call fetch_url
  mcp-rig-gateway  SERVER  execute_tool fetch_url
                           error.type = GatewayDenied.egress_host_not_allowed

Read a note, then try to fetch an external URL, refused. That is the “read a document, then reach out” shape from the previous post, and it is now one trace instead of a firewall line you have to correlate by timestamp. The client here is a probe rather than a production agent, which is deliberate: you verify the plumbing with a client whose behaviour you control before you trust it with one whose behaviour you do not.

Free of the payload: throw the prompts away

Most LLM observability products store the transcript. Prompt in, completion out, tool arguments in the middle. It makes for a great demo and a terrible thing to hold on someone else's behalf, and it turns your evidence plane into the most valuable exfiltration target in the building.

We made the opposite decision from the start, and it has a real cost: without the transcript you cannot replay an incident word for word. You keep the sequence, the tools, the counts and the verdicts, not the text that produced them. A security reviewer will ask about that trade-off, and the answer is that replaying a prompt requires storing every prompt, and the store then holds the very material a breach would go looking for. We chose to hold the shape of the incident and not its contents. Prompts, completions and tool arguments are removed in the ingestion pipeline, before anything touches disk. Not redacted in the UI. Removed at the edge, by a transform stage in the collector that sits between the public endpoint and the trace store, written in OTTL, the OpenTelemetry transformation language:

processors:
  transform/assert:
    error_mode: propagate
    trace_statements:
      - context: span
        statements:
          - keep_matching_keys(attributes, "^(gen_ai\\.operation\\.name|gen_ai\\.tool\\.name|gen_ai\\.usage\\..+|...)$")

Two things about that snippet. It is an allowlist, not a blocklist: we name what may pass, everything else is dropped, so a framework we have never heard of cannot smuggle a new attribute name through. And error_mode: propagate means it fails closed: a broken statement fails the whole batch and the client gets a 503, rather than a batch that quietly skips the filter.

A configuration is a claim. Here is the measurement. Nine OTLP payloads, sent through the public endpoint, each carrying attributes tagged with a marker string, then read back from the trace store:

The positive controls are the important row. A filter that drops everything also “passes” a test that only checks for absence. The controls prove the pipeline was up and the filter was selective. If your vendor shows you a redaction feature, ask for the positive control.

The last two rows are the edge cases the probe was designed to find. gen_ai.usage\..+is a wildcard, because token counters live under that prefix, and it accepts any suffix. And span names are normalised for digits and hex but not filtered for content. Both reach the trace store, which has a seven-day retention. Neither reaches the run table, because the run writer reads named keys only. The wildcard is being replaced by an enumerated list and span names get the same treatment as attributes. I would rather publish the two cases with their blast radius than the word “guaranteed”.

Below the trace store, the schema has no place for content at all. The run table carries counters, timestamps, a status and an error type. A scan of all 116 text and JSON columns across the 32 tables of the platform database, run twice, fifteen minutes apart, found zero marker strings.

Durable: runs outlive traces

Traces are expensive to keep and boring to read. Ours expire after seven days. An audit question does not.

So every agent run is written to a PostgreSQL row at ingestion, keyed by tenant, trace and agent: model calls, tool calls, tokens, duration, terminal status. The row is kept on its own retention policy, 30 days by default and set per deployment. The trace is the detail view; the row is the record. A run therefore survives its trace by weeks, and the portal says so on the page, because it is not how classic telemetry behaves and an operator should not have to discover it.

How fast does a run become a row? Median 19 seconds, p95 28 seconds over 20 runs, against a 60 second visibility target; eight further runs on the current build sat between 14 and 22 seconds. Small samples at this scale, quoted as such, and the ingestion interval that bounds them is 30 seconds.

The measurement also surfaced the kind of gap this plane exists to surface. Some runs arrived attributed to an agent that had been retired. The writer stored the rows; the read paths hid them, as designed. But the OTLP receiver still accepted and processed them, so the discard happened after the work rather than before it. That is a cost boundary as much as a security one, and it is now enforced at the receiver. You only find that with a record that is written independently of what the UI chooses to show.

Outside the agent's reach: what “unclaimed” means

The inventory page has a section called UNREGISTERED. Anything that emits telemetry into a tenant shows up there, whether or not anyone declared it, with first-seen, last activity and the tools its latest run touched. An operator claims it, or retires it.

That is the snap validation-set idea from the previous post, one level up. Validation sets tell you which revision runs on which host. This tells you which agents exist at all. The agent cannot opt out of appearing, because it is the runtime and the gateway that emit the spans, not the agent. Shadow AI is an inventory problem before it is a policy problem.

Where the edge of the field is

The previous post's honest section was the one to read. Same here, and I am drawing the lines where the whole field currently stands, not where one product does.

A well-formed hostile argument is authorized. Argument constraints in the tool manifest catch a path traversal. They do not catch a syntactically valid query that asks for the wrong customer. The trace shows a normal call. This is the resource-versus-provenance gap from last time. CaMeL and FIDES attack it with data-flow labels; none of that is production infrastructure yet.

Keep the refused destination on the span. evil.example.com and 169.254.169.254 both produce error.type = GatewayDenied.egress_host_not_allowed. Our gateway log keeps the attempted host and the span is being extended to carry it too. If you are building this, do it from day one: the difference between “an unlisted host” and “the metadata service” is the difference between a misconfiguration and an incident.

Span kinds are not enough for a service graph. The trace store's metrics generator registers the gateway edges from the CLIENT/SERVER pairs above; getting those edges reliably into the metrics store as service-graph series is a separate piece of work, so we tell the gateway story through the trace, which is the record, not through the topology, which is a rendering of it.

Framework coverage is a measurement, not a logo wall. LangGraph with OpenLLMetry is proven end to end, from the agent through the public edge, the allowlist, the trace store, the run table and every screen, with pinned fixtures. Pydantic AI is instrumented and in daily use on our writer rig. We add a framework when it has a fixture and a probe, not when it has a name in a list. And the ingestion edge is OTLP/HTTP: gRPC exporters are a separate path we have not opened.

The GenAI semantic conventions have no released version. They moved to their own repository in June and are still marked Development; gen_ai.prompt and gen_ai.completion are already gone in favour of message attributes. So we do not claim conformance to a moving convention. We claim an explicit list of named keys, a fail-closed filter, and a probe you can rerun against the deployed artifact.

A note on decoys

A commenter on the last post proposed planting decoy records so that an agent which does exfiltrate something cannot tell the real one from the fake. The research is real and works unusually well against LLM agents, though the honeywords literature shows it eroding across repeated leaks and shared memory. I see it as a tripwire that sits next to the record described here: the decoy says something was touched, the record shows the sequence that led there, and neither needs the payload.

Where we land: the AI agent audit trail in practice

The previous post ended with the work that does not come in a package: policies tight enough to mean something, revisions consistent across a fleet, and decision logs a compliance team can read. This post is the third item, built.

A sandbox gives you the denies. An evidence plane gives you the allows: complete, written by the infrastructure, durable past the agent, and empty of the content that would make it a liability. That is the layer between “we sandboxed it” and “we can prove what it did”, and it is the one the audit will ask for.

shoofi is AuroraIQ's audit-evidence plane for AI agents, built on open-source tracing, metrics and logging components and deployed inside your own environment. AuroraIQ designs and operates the platforms it runs on, from SRE and production reliability to DevOps engineering and observability and on-call. If you are putting agents in front of systems that matter, get in touch.

You code. We run it.