MCP gateway security: what our traces did and did not record

What we built, and what we ran at it

The gateway is a stateless HTTP forward proxy. It parses each JSON-RPC body, decides, and forwards. It has no session of its own.

Being precise about the path, because it changes how you should read the numbers: there is no agent in it. Our Pydantic AI agent calls its tools as in-process Python functions, so there was no protocol hop to intercept and we had to build a target to have something to attack. The callers here are scripts. This measures a gateway and its telemetry, not an agent framework. One server, LXD virtual machines, traces in Tempo, a test rig rather than a customer deployment.

Class A is what a gateway exists for. A tool that is not on the allowlist. A JSON-RPC method outside the permitted set. An outbound request to the cloud metadata endpoint at 169.254.169.254. All refused, all recorded with error.type = GatewayDenied.<rule> and the tool name. You can query that attribute today and build an alert on it, and the record survives even if the caller never receives the response, because the export is synchronous and sits inside the refusal path:

d = decide(msg, self.egress_allow)
span_id = emit(d, ctx)              # export ATTEMPTED before any response
if not d.allowed:
    self._json(200, {"jsonrpc": "2.0", "id": msg.get("id"),
                     "error": {"code": -32002, "message": "refused by gateway"}})
    return

A crash between the two cannot drop the record of the thing that went wrong, and the cost of that ordering is measurable rather than theoretical: a 3.0s collector delay made the refusal take 3.15s, against 0.00s with export off. The refusal cannot outrun its own record.

Class B is a tool that is on the allowlist, called with an argument it should never accept:

read_note(name="../../../etc/hostname")        -> planted file read
run_report(subject="weekly; cat /etc/shadow")  -> shell string composed
run_report(subject="weekly`id`")               -> shell string composed

All three succeeded. And the spans they produced are byte-identical to the benign calls sitting next to them: execute_tool read_note, tool name read_note, no error type, status unset, status message empty. Two traversals and one analyst reading a note produce the same row three times. Two injections and one legitimate report produce the same row three times again.

Note what the record does not even claim. The status is unset, not ok. It does not say the call succeeded. It says a permitted tool ran and did not error.

Two things are also missing that you would want during an investigation. There is no caller identity, because the gateway authenticates a single shared bearer token, so there is no per-caller identity in the system to carry in the first place. And at the time of the measurement, 26 of 28 decision spans had no parent: each was minted into its own trace. Alerting survives that, since the attribute is queryable either way. Attribution does not. A denial was an event with no subject.

Why this is the default, not our bug

It would be comfortable to file this under implementation detail. It is not.

Prompts, completions and tool arguments are stripped on our traces ingest path before storage, and that is a deliberate decision rather than an omission. But it is also what the specification asks for. In the OpenTelemetry GenAI semantic conventions, model instructions, user messages and model outputs are treated as sensitive: instrumentation should not capture them by default, every content-bearing attribute carries the Opt-In requirement level, and instrumentation that does not support configuration must not populate them at all. Tool arguments are on that list.

So if you follow the conventions, empty content fields are the normal state, not a misconfiguration. Which means the blind spot we measured is not ours. It is the default posture of any spec-compliant agent pipeline, and the attacks that exploit it are the ordinary web vulnerabilities that MCP servers have been shipping with. Anthropic's own Git MCP server took three CVEs in January 2026, one of them argument injection into git_diff. A permitted tool. A hostile argument.

Our own strip is a pipeline stage rather than a flag, which is worth showing because the shape matters more than the regex:

otelcol.receiver.otlp "in" {
  http { endpoint = "0.0.0.0:4318" }
  output {
    traces = [otelcol.processor.transform.assert.input]   # the ONLY output
  }
}

otelcol.processor.transform "assert" {
  error_mode = "propagate"          # fail closed
  trace_statements {
    context = "span"
    statements = [
      `keep_matching_keys(span.attributes, "^(gen_ai\.operation\.name|...|error\.type)$")`,
    ]
  }
}

Three properties do the work. The receiver has one output, so no path through this collector skips the strip. keep_matching_keys is an allowlist, so a content field nobody has heard of is dropped by not being named, rather than needing to be added to a blocklist. And the file is generated from a declared permit set with a checksum header that CI byte-compares, so nobody widens it quietly on a Friday.

We verify it by sending spans carrying a synthetic marker in all six content fields through the real public ingest, then reading the trace back and asserting the fields are absent. Six of six dropped, result committed so a re-run is a diff rather than a fresh set of identifiers. One honest caveat on that: the probe sees what the query API returns, not what is on disk.

Three ways to observe the argument, and one way to stop needing to

We did not want to reach for content capture, so we measured the alternatives instead of assuming them.

Argument length as a signal. Fails. Not for the reason people expect. Our attacks landed in the 0 to 16 and 17 to 64 character buckets and the benign calls only in 0 to 16, so both traversals were longer than every benign argument we measured. The overlap comes entirely from a ten-character injection, weekly`id`. If your intuition was that attacks are long, the data says the short one is the problem.

A hash of the argument. Correlation only. It groups repeat appearances of an argument you have already labelled, and matches a known-bad list you built in advance. For an argument nobody has seen before, which is the case that matters, it tells you nothing.

A bounded shape class rather than the value. A signal, not a verdict. Classify the argument at the gateway into a small closed set and emit the class. Ours declares four and observed three: plain, path_separators, shell_metachars. It separates attack from benign on the calls we ran. It is also a list that lives in your own documentation, which an attacker reads and shapes around.

Constraining the argument at the gateway. Works. The manifest already declares which tools exist. Let it declare the permitted shape of each argument, and let the gateway refuse anything out of shape:

cons = ARG_CONSTRAINTS.get(tool, {})
for name, value in (params.get("arguments") or {}).items():
    pat = cons.get(name)
    if pat is None:
        # deny-by-default extends to arguments, so adding an argument
        # to an existing tool cannot silently reopen the hole
        return Decision(False, "arg_not_declared", tool,
                        f"argument {name!r} has no declared constraint")
    if not re.fullmatch(pat, str(value)):
        # fullmatch, not match: `$` matches before a trailing newline,
        # so `^...$` with .match() accepts "n1\n../etc/passwd"
        return Decision(False, "arg_constraint_violated", tool,
                        f"argument {name!r} does not match its declared shape")

Against the unchanged vulnerable server, with only the gateway moving, four successes became four arg_constraint_violated denials and the benign controls were untouched. The refusal names the argument and never its value, so the fix produces a visible event without opening a content path.

None of that is a new control. Argument-level authorization at the gateway is already sold: Traefik Hub matches policy expressions against mcp.params, MuleSoftvalidates tool parameters against the tool's JSON schema, and Amazon Bedrock AgentCore Policyevaluates Cedar policies per tool call at the gateway boundary under default-deny, generating its schema from the gateway's tool definitions.

What we think is worth adding is the reason to want it that has nothing to do with access control. Constraining arguments does not improve your observability. It moves the interesting event out of the category your telemetry cannot see and into the one it handles well. You stop needing to observe the argument because the gateway now decides on it, and decisions are the thing a gateway records properly.

One deliberate difference from the products above. We derive constraints from our own manifest and not from the server's published inputSchema, because letting the target define the gateway's policy defeats the reason the gateway exists. A compromised server can publish .*.

The content channel we found in our own permit set

While checking the claim above, we found the opposite of what three of our own documents said.

Our permit set filters resource attributes, span attributes and span event attributes. It never touches status.message, and our gateway interpolates the refusal detail into exactly that field. So this is in the store today:

statusMessage = "GatewayDenied.egress_host_not_allowed:
                 host '169.254.169.254' is not on the egress allowlist"

An attacker-supplied value, verbatim, in an unfiltered free-text field, written by the component whose job is enforcement.

The generalisable part is that this is a family, not an accident. OTLP carries string fields that are not attributes, and attribute-based filtering does not reach them. The redaction processor operates on span, log and metric datapoint attributes. Dynatrace documents that the span name is a separate field in the OTLP structure, so attribute-targeted redaction does not affect it. And contrib issue #36633 reports blocked values leaking through in span event attributes.

Span name. Status message. Event attributes. If you run an attribute allowlist and believe you have a content boundary, go and check those three, and check what your own enforcement components put in them. It is the kind of thing that only shows up when somebody actually reads the store, which is why we treat observability as an engineering surface rather than a dashboard you buy.

What the kernel adds, and what it does not

We also joined agent-side eBPF events onto the trace, to see what a view from below the application contributes.

The workload attempts three real escapes out of its root. What the kernel showed:

newfstatat /etc/passwd     escapes_root_to_system_path   trace=9df4b231...
newfstatat /etc/hostname   escapes_root_to_system_path   trace=9df4b231...
newfstatat /etc            escapes_root_to_system_path   trace=9df4b231...

No openat on any target, and the probe traces sys_enter_openat first, so it would have seen one. realpath resolves each component in the kernel, which is why the attempt is real at the syscall layer, and the guard refuses after resolution and before the open. 582 events captured, 29 joined to the run, 553 correctly unjoined.

That absence is the interesting half. The kernel records the attempt, the span records the refusal, and the missing openat is the evidence the guard held. What persists is a bounded path class rather than the path, with a test asserting the raw path is not in the stored record.

Two limits. The join key is a process id plus the run's time window, which is only valid because these runs are one process each. Our attack rig's target is long-lived and serves every caller, so its process id identifies the workload and never the run, and the traversal in this article and the eBPF join have never been combined. Second, and more general: ARMO's write-up puts it the way we would, that eBPF sees the consequences of an injection and not the injected prompt.

There is also a reason to want kernel telemetry that our own setup does not satisfy. In-process instrumentation reports what the code chooses to report, so a compromised tool can strip or falsify its own spans. Our gateway spans are emitted by the gateway rather than by the caller, which is the right side of that boundary. Our agent-side telemetry is not.

What is still open, for everyone

One honest note, because it shapes how you write policy rather than whether you deploy any of this.

Shape-not-content detects policy violations well. It detects argument-borne attacks only once you constrain the arguments, which converts them into policy violations. And it does not detect in-process abuse at all: a model talked into misusing tools it is allowed to use crosses no syscall boundary, violates no manifest, and produces calls that are permitted, in scope and correctly traced.

The precise claim is that no layer reaches that class without capturing content, and the qualifier matters, because there is published work that reaches it by capturing content from outside the application. AgentSight intercepts TLS-encrypted model traffic at the kernel boundary to recover intent, correlates it with kernel events, and reports detection of prompt injection at under 3% overhead. That is a real answer. It is not available to a hosted platform that has promised not to hold prompts, which is a trade we make on purpose rather than a gap we have not noticed.

We are also not alone in the finding. ShieldNet reports the same inversion one layer down: stealthy tool injection preserves benign tool interfaces and input-output behaviour, evading semantic-layer inspection while showing itself only in runtime behaviour, which is why current MCP scanners have systematic blind spots. Different layer, same shape.

Worth noticing how the field usually states this, because it is the mirror image. The standard warning, as Obot puts it, is that a log recording only successful calls misses the blocked ones. In a content-stripped pipeline you get the opposite failure, and it is the more dangerous of the two.

Where we land: MCP gateway security in practice

If you are running agents against tools that matter, three things are worth doing this week and none of them requires a product.

Declare the shape of every tool argument at your gateway and refuse what does not match, with deny-by-default extending to undeclared arguments. Use fullmatch semantics, because ^...$ with a match call accepts a trailing newline and everything after it. Then go and look at what your enforcement components write into span names, status messages and event attributes, since those escape an attribute allowlist.

And ask any vendor selling you agent observability one question: for a successful call to a tool I permitted, with an argument I did not, what does your product show me? If the answer is a tool name and a status, the gap is the same as ours. The difference is only whether it has been named.

AuroraIQ designs and operates the platforms this runs on, from SRE and production reliability to DevOps engineering and observability and on-call. If you are putting agents in front of systems that matter, get in touch.

You code. We run it.