Agent Gateways: the control plane production AI agents shouldn't ignore

As long as an AI agent is limited to searching for information or drafting a response, an error generally remains reversible. The risk changes fundamentally when the agent can modify a database, trigger a payment, send an email, open a ticket, or call a third-party service.

At that point, a core architectural question becomes critical:

Who actually decides whether the action proposed by the model can be executed?

The answer cannot rely solely on the agent's prompt. A probabilistic model should not be solely responsible for authorizing a deterministic action.

That's the role of the agent gateway: placing a control layer between the intention produced by the agent and its execution in the real system.

AWS now describes its AgentCore Gateway as a unified connectivity layer between agents, their tools, and their resources. Its policy engine can intercept requests and evaluate each tool call before authorizing access to a tool. Google also presents its Agent Gateway as a central point for applying policies to tool calls and agentic communications. (AWS Documentation)

The category is still being defined. Not all platforms assign exactly the same scope to the term agent gateway. But the architectural need is already clear:

The moment an agent can produce significant side effects, a deterministic control layer must separate its decision from execution.

What is an agent gateway?

An agent gateway is a point of interception and policy enforcement placed on the execution path of an agent's actions.

Its role is not to make the model smarter. It consists of verifying that the proposed action is authorized, valid, traceable, and compatible with the operational constraints of the system.

A simplified flow works like this:

1. The model proposes using a tool with structured parameters.
2. The orchestrator passes this request to the gateway.
3. The gateway verifies identity, permissions, parameters, quotas, and business rules.
4. The action is authorized, denied, or suspended pending human validation.
5. The tool result is recorded and returned to the orchestrator.

In the case of tools executed client-side with Claude, the model doesn't execute the business code itself. It returns a tool_use block, then the application decides whether or not to execute the tool and passes its result back to the model. This separation provides precisely the interception point needed for policy enforcement. (Claude Platform Docs)

An agent gateway can be:

• a component developed within the orchestrator;
• an independent service;
• a layer built around an existing API gateway;
• a policy engine connected to MCP servers;
• a managed service provided by a cloud platform.

The essential point is not to buy a product called "Agent Gateway". The essential is to have a non-bypassable enforcement point before any sensitive action.

Why prompts and static permissions aren't enough

An instruction like "never refund more than €5,000" is useful for guiding the model. It's not a security rule.

It can be misinterpreted, bypassed by prompt injection, lost in too-long context, or ignored following a reasoning error.

OWASP lists excessive agency among the major risks of LLM-based applications: a model with excessive functionality, permissions, or autonomy can cause damaging actions from unexpected, ambiguous, or manipulated output. Its agent guidance specifically recommends least privilege, independent validation of sensitive actions, and human oversight proportional to their impact. (OWASP Gen AI Security Project)

The problem doesn't come only from a "malicious" agent. An agent can cause an incident while correctly pursuing its objective:

• it repeats an action due to poor retry handling;
• it incorrectly interprets an API response;
• it reuses information from the wrong tenant;
• it chooses a tool too powerful for the requested operation;
• it chains multiple valid actions that collectively produce an undesired result;
• it applies an instruction injected in an email, document, or web page;
• it continues an operation when the first step partially failed.

Protection must therefore sit outside the model's reasoning, in a layer capable of applying deterministic rules.

Six controls for an agent in production

A serious gateway is more than a tool whitelist. It must answer six questions.

1. Identity control

Who is requesting the action?

You must be able to distinguish:

• the agent;
• the application or workflow executing it;
• the end user on whose behalf it acts;
• the organization or tenant involved;
• the session;
• the delegation level granted.

The agent's identity should not be limited to an agent_id field in the payload. It must be attested by a verifiable authentication mechanism.

Google, for example, distinguishes the agent's own identity from its ability to act on its own account or on behalf of a user. Its Agent Identity system relies on cryptographic identity and the SPIFFE standard. (Google Cloud Documentation)

2. Authorization control

Can this agent use this tool in this specific context?

The rule shouldn't just say:

support-agent -> CRM access

It must be able to express:

support-agent -> read orders for current tenant -> no global export -> no deletion -> modification limited to certain fields

The principle of least privilege must apply to tools, operations, resources, and data.

AWS, for example, lets you associate a policy engine with a gateway to evaluate tool calls before execution. Rules can be expressed as deterministic policies, notably with Cedar. (AWS Documentation)

3. Parameter control

Is the action permitted with these particular parameters?

Authorizing the approve_refund tool doesn't mean authorizing all refunds.

The gateway must be able to verify:

• amounts;
• recipients;
• tenant identifiers;
• queried tables or columns;
• accessible domains;
• file types and sizes;
• date ranges;
• data schemas;
• forbidden values.

This validation must be deterministic and ideally based on strict schemas, not a second natural language instruction.

4. Behavioral control

Does the agent's overall behavior remain normal?

An isolated call may be valid while the complete sequence is not.

The gateway must be able to detect:

• loops;
• bursts of calls;
• retries without backoff;
• abnormal costs;
• abrupt tool changes;
• an unusual sequence of actions;
• a volume of refusals revealing a stuck or misconfigured agent.

Rate limits and circuit breakers must be calculated per agent, user, tenant, tool, and action type, not just by IP address.

5. Effects control

Should the action be executed immediately?

Not all actions present the same risk.

A reasonable policy can distinguish:

• read-only actions;
• reversible writes;
• external actions visible to a third party;
• financial operations;
• deletions;
• permission modifications;
• irreversible actions.

For critical operations, the gateway can impose:

• human validation;
• enhanced authentication;
• user confirmation;
• double-approval mechanism;
• prior simulation;
• an idempotency key;
• deferred execution.

The approval mechanisms proposed by agent platforms follow this logic. OpenAI notably plans pauses and validations for calls producing side effects or considered destructive. (OpenAI Developers)

6. Post-execution control

What actually happened?

Authorizing an action is not enough. You must also record:

• the initial request;
• identity and delegation;
• the applied policy;
• the gateway's decision;
• validated parameters;
• the called tool;
• the tool response;
• the final state;
• any errors or compensations.

This trace lets you reconstruct an incident, measure agent performance, and distinguish a poor model decision from external system failure.

The OpenAI Agents SDK tracing, for example, records generations, tool calls, handoffs, guardrails, and custom events from a run. This observability is useful, but it doesn't replace a durable business log adapted to the company's obligations. (OpenAI)

Where to place the gateway in your architecture?

The correct representation isn't necessarily:

LLM -> gateway -> tool

In many systems, the model never directly contacts the tool. It produces a structured call proposal, then the orchestrator decides what comes next.

A more accurate architecture looks like this:

┌────────────────────────┐
│ User / event           │
└───────────┬────────────┘
            │
┌───────────▼────────────┐
│ Agent orchestrator     │
│ LangGraph, n8n, custom │
└───────────┬────────────┘
            │
┌───────────▼────────────┐
│ LLM                    │
│ Tool call proposal     │
└───────────┬────────────┘
            │
┌───────────▼────────────┐
│ AGENT GATEWAY          │
│                        │
│ • identity             │
│ • authorization        │
│ • validation           │
│ • quotas               │
│ • human approval       │
│ • logging              │
└───────────┬────────────┘
            │
┌───────────▼────────────┐
│ Tool executor          │
│ API, MCP, worker       │
└───────────┬────────────┘
            │
┌───────────▼────────────┐
│ Business system        │
│ CRM, DB, email, ERP    │
└────────────────────────┘

The gateway is thus a Policy Enforcement Point placed before the tool executor.

It must be impossible to bypass this point by calling directly:

• the database;
• an MCP server;
• the business API;
• a cloud function;
• a worker;
• a third-party service.

A gateway that only controls one access path while the agent has direct credentials to the tools creates a false sense of security.

Agent gateway, API gateway, and ai gateway: what are the differences?

The three categories overlap partially, but they don't answer exactly the same problem.

An API gateway like Kong, Nginx, or a cloud service can form part of the solution. It already knows how to authenticate, rate-limit, and log requests.

What's generally missing is agentic and business context:

• which agent is acting;
• on whose behalf;
• what intent or task is underway;
• what tool is being invoked;
• what effects the action can produce;
• what business policy applies;
• whether human approval is required;
• whether this invocation is part of an abnormal loop.

The right approach isn't necessarily to replace the API gateway. It can be enriched or complemented by a layer of policies dedicated to agents.

Google, for instance, defines its Agent Gateway as a network abstraction governing client-agent, agent-tool, and agent-agent interactions, with application of security policies and access controls. (Google Cloud Documentation)

Three concrete use cases

Case 1: an agent manages customer refunds

The agent analyzes a request and proposes a refund.

A gateway policy might impose:

Amount ≤ €200:
    automatic execution

€200 < amount ≤ €2,000:
    execution authorized if account meets business criteria

Amount > €2,000:
    mandatory human validation

Amount > €10,000:
    automatic refusal for this workflow

The model's decision remains useful: it can qualify the case and propose an amount. But financial authorization is controlled by an independent rule.

AWS uses precisely a refund-processing scenario in its documentation to show how policies can limit authorized amounts. (AWS Documentation)

Case 2: an agent queries sensitive data

Giving an agent a generic SQL connection is rarely a good idea.

A safer architecture exposes narrow tools:

get_customer_orders(customer_id)
get_order_status(order_id)
get_monthly_sales_summary(period)

The gateway then verifies:

• that the user can access the requested customer;
• that the tenant matches the session;
• that only authorized columns are returned;
• that unnecessary PII is masked;
• that the call remains read-only;
• that the data volume is reasonable.

The best control over a dangerous SQL query often remains never giving the model the ability to generate arbitrary SQL.

Case 3: an agent consumes paid APIs

An agent can produce a significant bill without any individual call being abnormal.

The gateway can apply multiple budgets:

100 calls per minute per agent
1,000 calls per hour per tenant
€200 per day for the workflow
€10 maximum for individual execution

Beyond a threshold, it can:

• slow down calls;
• open a circuit breaker;
• fall back to a cheaper model or provider;
• suspend only the relevant tool;
• request validation;
• stop the run.

A minimal implementation

The first version doesn't need to be a standalone product. A well-designed wrapper around the tool executor can suffice.

from dataclasses import dataclass
from datetime import datetime, timezone
from decimal import Decimal
from typing import Any, Callable


class GatewayDenied(Exception):
    pass


@dataclass(frozen=True)
class InvocationContext:
    agent_id: str
    user_id: str
    tenant_id: str
    run_id: str


@dataclass(frozen=True)
class ToolPolicy:
    handler: Callable[..., Any]
    write_operation: bool = False
    max_amount: Decimal | None = None
    requires_approval: bool = False


class AgentGateway:
    def __init__(self, policies: dict[str, ToolPolicy], audit_sink):
        self.policies = policies
        self.audit_sink = audit_sink

    def invoke(
        self,
        context: InvocationContext,
        tool_name: str,
        params: dict[str, Any],
        approved_by: str | None = None,
    ) -> Any:
        started_at = datetime.now(timezone.utc)
        decision = "denied"
        result_status = "not_executed"

        try:
            policy = self.policies.get(tool_name)

            if policy is None:
                raise GatewayDenied("Tool not allowed")

            self._verify_identity(context)
            self._check_rate_limits(context, tool_name)
            self._validate_schema(tool_name, params)
            self._enforce_tenant_scope(context, params)

            if policy.max_amount is not None:
                amount = Decimal(str(params.get("amount", 0)))

                if amount > policy.max_amount:
                    raise GatewayDenied("Amount exceeds policy limit")

            if policy.requires_approval and approved_by is None:
                raise GatewayDenied("Human approval required")

            decision = "allowed"

            # Use an idempotency key for retryable writes.
            result = policy.handler(
                **params,
                tenant_id=context.tenant_id,
                idempotency_key=f"{context.run_id}:{tool_name}",
            )

            result_status = "succeeded"
            return result

        except Exception:
            result_status = "failed"
            raise

        finally:
            self.audit_sink.write({
                "timestamp": started_at.isoformat(),
                "agent_id": context.agent_id,
                "user_id": context.user_id,
                "tenant_id": context.tenant_id,
                "run_id": context.run_id,
                "tool": tool_name,
                "decision": decision,
                "result_status": result_status,
                # Don't blindly log secrets or PII.
                "params": self._redact(params),
                "approved_by": approved_by,
            })

This code illustrates the interception principle. This is not a production-ready gateway.

Depending on context, it still needs:

• cryptographic authentication;
• distributed quota storage;
• delegation policy;
• strict schema validation;
• secret management;
• sensitive data redaction;
• durable and protected logs;
• timeouts;
• bounded retries;
• compensation mechanisms;
• high availability;
• revocation procedures;
• protection against direct tool access.

Implementation roadmap

Phase 1: inventory actions and reduce permissions

Before building a gateway, list all tools accessible to agents.

For each tool, document:

• accessible resources;
• possible operations;
• side effects;
• reversibility;
• sensitive data exposure;
• potential cost;
• required approval level.

Then start with:

• removing unnecessary tools;
• separating read and write;
• replacing generic tools with narrow business operations;
• adding schema validation;
• logging calls and their results.

Phase 2: centralize policies

Add a structured execution context:

• agent identity;
• user;
• tenant;
• run;
• workflow objective;
• risk level;
• active delegation.

Then centralize:

• authorization rules;
• amount limits;
• quotas;
• approvals;
• temporal restrictions;
• data rules;
• circuit breakers.

Phase 3: industrialize

When multiple teams, agents, or environments duplicate the same controls, transform the layer into a shared service.

Add:

• a policy engine;
• centralized identity management;
• an agent and tool registry;
• versioned policies;
• correlated traces;
• alerts;
• policy testing;
• observation mode without blocking;
• incident procedures;
• compliance controls.

The NIST AI Risk Management Framework recommends identifying functions requiring human oversight, defining roles and responsibilities, and adapting control mechanisms to context and risk level. (NIST)

Most common implementation mistakes

1. Control the tool name, but not its parameters

A whitelist that allows send_email without checking recipient, domain, content, or volume provides only limited protection.

A useful authorization covers:

agent + user + tenant + tool + operation + resource + parameters

2. Leave a bypass path

If the workflow can call the CRM or database directly, the gateway is not an enforcement point.

Business system credentials must be held by the controlled executor, not exposed directly to the model or an alternate worker.

3. Log only the request

An action can be authorized then partially fail.

You must record:

• the decision;
• execution start;
• the result;
• retries;
• final state;
• any compensations.

For asynchronous systems, use a common correlation identifier between the agent run, invocation, job, and business event.

4. Record secrets in traces

Tools often handle tokens, PII, financial data, or customer content.

A useful audit trail doesn't mean recording all parameters and responses without filtering. Redaction and retention rules must be designed from the start.

5. Apply the same limits to all actions

One hundred catalog reads and one hundred transfers don't present the same risk.

Policies must depend on:

• the tool;
• the impact;
• the tenant;
• financial value;
• reversibility;
• trust in the workflow.

6. Block without informing the orchestrator

When an action is denied, the agent must receive an actionable error:

{
  "code": "HUMAN_APPROVAL_REQUIRED",
  "message": "Refunds above 2000 EUR require approval.",
  "retryable": false
}

A vague response like Access denied often pushes the agent to repeat the same action.

7. Confuse observability with authorization

Tracing a call doesn't prevent it from executing.

Tracing helps understand. The gateway must also be able to:

• authorize;
• deny;
• modify;
• suspend;
• request approval;
• open a circuit breaker.

Homegrown gateway or specialized solution?

Agent count is not the right criterion.

A single agent capable of making transfers can justify strong infrastructure. Fifty agents read-only on public data can stay behind a relatively simple wrapper.

The decision should depend on criticality.

Start internally when:

• you have few tools;
• workflows are understood;
• rules are simple;
• you already have solid IAM and observability infrastructure;
• the layer can stay close to the application.

Industrialize or buy when:

• multiple teams duplicate the same controls;
• policies need centralization;
• agents use multiple runtimes;
• MCP servers and tools multiply;
• audit obligations grow important;
• maintaining rules becomes cognitive or operational overhead.

How to know if your agent needs a gateway?

Assign one point for each yes answer:

• Does the agent write to a business system?
• Can it delete or make public data?
• Does it handle personal or confidential data?
• Can it engage an expense?
• Can it contact a customer or third party?
• Does it have multiple tools?
• Does it act for multiple users or tenants?
• Can an action be irreversible?
• Could an error produce a regulatory incident?
• Does the workflow run without direct human supervision?

Interpretation

• 0 to 2 points: a simple wrapper, narrow permissions, and logs may suffice.
• 3 to 5 points: implement an explicit enforcement point and versioned policies.
• 6 points or more: treat the gateway as a critical architecture component.

This grid is not a standard. It helps avoid a decision based only on project size or agent count.

Conclusion

An agent gateway is not a magic proxy that automatically secures an agentic system.

It's an architecture principle:

The model's probabilistic decision must never constitute the sole authorization for a real action.

The moment an agent produces side effects, you must have a control point capable of verifying its identity, delegation, permissions, parameters, behavior, and action impact.

Three principles to remember:

1. The prompt guides; the policy authorizes.
   A natural language instruction doesn't replace a deterministic rule.

2. Control must be non-bypassable.
   All sensitive access must pass through the same enforcement point.

3. Control level depends on impact, not agent count.
   A single agent can require strong governance if it handles payments, permissions, or sensitive data.

You don't necessarily need a specialized platform for the first prototype. But you must define very early where the boundary lies between what the agent proposes and what your system actually authorizes.

response = client.messages.create(
    model=MODEL_NAME,
    max_tokens=1024,
    tools=tool_definitions,
    messages=messages,
)

for block in response.content:
    if block.type != "tool_use":
        continue

    try:
        result = gateway.invoke(
            context=invocation_context,
            tool_name=block.name,
            params=block.input,
        )

        tool_result = {
            "type": "tool_result",
            "tool_use_id": block.id,
            "content": serialize_result(result),
        }

    except GatewayDenied as exc:
        tool_result = {
            "type": "tool_result",
            "tool_use_id": block.id,
            "is_error": True,
            "content": str(exc),
        }

    messages.append({
        "role": "assistant",
        "content": response.content,
    })

    messages.append({
        "role": "user",
        "content": [tool_result],
    })