~/blog/kubernetes-gateway-api-envoy

Gateway API in Practice: From Ingress Migration to Envoy Debugging

•7 min read

The Kubernetes community deprecated Ingress in spirit years ago. Not officially — the API object is still there — but every major implementation has moved on. Contour went Gateway API native. Envoy Gateway was built from scratch for it. The Gateway API SIG produces more commits per month than the Ingress controller ecosystem. The writing is on the wall.

Most platform teams haven't moved yet. They're running nginx-ingress with annotations that look like NGINX config smuggled into Kubernetes via YAML comments. They're fighting annotation conflicts between namespaces. They're explaining to every application team why they can't have host-based routing on the same hostname as their neighbour.

Gateway API solves all of that. But only if you understand what it's actually modelling — because if you treat it as Ingress with more YAML, you'll get more YAML and none of the benefit.

ℹ️

Local Platform Engineering Series

  1. Running Local Kubernetes with k3d: Fast, Ephemeral, and Kind to Your Battery
  2. Gateway API in Practice: From Ingress Migration to Envoy Debugging
  3. Multi-Tenant Observability: LGTM at Platform Scale
  4. Network Control with Cilium and Kyverno: Policies That Actually Work
  5. Observing LLM Inference: The Metrics That Actually Matter
  6. AI Tool Gateways: Sandboxing Agent Access in Kubernetes

What Changed and Why It Matters

Ingress has two fundamental problems:

First, the ownership boundary is wrong. An Ingress resource mixes infrastructure configuration (which load balancer to use, TLS termination, timeouts) with application routing (which service gets which path). The platform team owns the infrastructure; the application team owns the routing. With Ingress, both teams edit the same resource. That's a conflict waiting to happen.

Second, extension is annotation-based and non-portable. Every ingress controller extends Ingress through annotations. nginx.ingress.kubernetes.io/proxy-body-size, traefik.ingress.kubernetes.io/router.middlewares, alb.ingress.kubernetes.io/scheme. They're all different. Migrating controllers means rewriting annotations. There's no standard.

Gateway API solves both by splitting the resource model across the ownership boundary:

ResourceWho owns itWhat it configures
GatewayClassPlatform teamWhich implementation to use (Envoy, NGINX, etc.)
GatewayPlatform teamThe listener (port, protocol, TLS, allowed namespaces)
HTTPRouteApplication teamWhich service gets which traffic

The Gateway and HTTPRoute can live in different namespaces. The platform team provisions the gateway once. Application teams attach routes to it without touching the infrastructure config. Role boundaries enforced by the API, not by convention.

Installing Envoy Gateway on k3d

The k3d-config.yaml from part one disables Traefik. Good — we're replacing it.

Install the Gateway API CRDs first. These ship separately from Kubernetes itself:

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/standard-install.yaml
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/standard-install.yaml

Install Envoy Gateway via Helm:

helm install eg oci://docker.io/envoyproxy/gateway-helm \
  --version v1.3.0 \
  --namespace envoy-gateway-system \
  --create-namespace \
  --wait
helm install eg oci://docker.io/envoyproxy/gateway-helm \
  --version v1.3.0 \
  --namespace envoy-gateway-system \
  --create-namespace \
  --wait

Wait for the controller:

kubectl wait --timeout=3m -n envoy-gateway-system \
  deployment/envoy-gateway --for=condition=Available
kubectl wait --timeout=3m -n envoy-gateway-system \
  deployment/envoy-gateway --for=condition=Available

GatewayClass, Gateway, HTTPRoute

Create the resources in order. The GatewayClass names the implementation. The Gateway is the entry point. The HTTPRoute is the routing rule.

manifests/gateway/gateway.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: infra
---
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: eg
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: platform-gateway
  namespace: infra
spec:
  gatewayClassName: eg
  listeners:
    - name: http
      port: 80
      protocol: HTTP
      allowedRoutes:
        namespaces:
          from: All
manifests/gateway/gateway.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: infra
---
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: eg
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: platform-gateway
  namespace: infra
spec:
  gatewayClassName: eg
  listeners:
    - name: http
      port: 80
      protocol: HTTP
      allowedRoutes:
        namespaces:
          from: All
kubectl apply -f manifests/gateway/gateway.yaml
kubectl wait -n infra gateway/platform-gateway \
  --for=condition=Programmed --timeout=2m
kubectl apply -f manifests/gateway/gateway.yaml
kubectl wait -n infra gateway/platform-gateway \
  --for=condition=Programmed --timeout=2m

The condition=Programmed is the Gateway API signal that Envoy Gateway has accepted the config and programmed the Envoy proxy. If it stays False, the controller has rejected something — check events on the Gateway resource.

Now attach an HTTPRoute. This lives in the application's namespace, not in infra:

manifests/gateway/httproutes.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: my-service
  namespace: my-app
spec:
  parentRefs:
    - name: platform-gateway
      namespace: infra
  hostnames:
    - "my-service.local"
  rules:
    - backendRefs:
        - name: my-service
          port: 8080
manifests/gateway/httproutes.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: my-service
  namespace: my-app
spec:
  parentRefs:
    - name: platform-gateway
      namespace: infra
  hostnames:
    - "my-service.local"
  rules:
    - backendRefs:
        - name: my-service
          port: 8080

The parentRef is what binds the route to the gateway. The gateway's allowedRoutes.namespaces.from: All permits routes from any namespace to attach. In a real multi-tenant platform, you'd restrict this by namespace selector to prevent tenants from squatting on each other's hostnames.

Expose the gateway locally:

kubectl port-forward -n infra svc/envoy-infra-platform-gateway 8080:80 &
echo "127.0.0.1 my-service.local" | sudo tee -a /etc/hosts
curl http://my-service.local:8080/health
kubectl port-forward -n infra svc/envoy-infra-platform-gateway 8080:80 &
echo "127.0.0.1 my-service.local" | sudo tee -a /etc/hosts
curl http://my-service.local:8080/health

Debugging the Gateway API

The Gateway API has a richer status model than Ingress. When something doesn't work, the signal is in .status.conditions on the resources, not just in pod logs.

Check Gateway status

kubectl describe gateway platform-gateway -n infra
kubectl describe gateway platform-gateway -n infra

Look for Programmed: True in the conditions. If it's False, the message will tell you why — usually a missing GatewayClass, a port conflict, or the Envoy proxy pod failing to start.

Check HTTPRoute status

kubectl describe httproute my-service -n my-app
kubectl describe httproute my-service -n my-app

Look for Accepted: True and ResolvedRefs: True. If ResolvedRefs is False, the backend service doesn't exist or uses an incorrect port. If Accepted is False, the route was rejected — often because it references a hostname the parent gateway doesn't permit.

Check Envoy proxy logs

Envoy Gateway creates a managed Envoy proxy deployment in the gateway's namespace. The proxy name follows the pattern envoy-{namespace}-{gateway-name}:

kubectl logs -n infra deployment/envoy-infra-platform-gateway -c envoy --tail=50
kubectl logs -n infra deployment/envoy-infra-platform-gateway -c envoy --tail=50

Traffic errors show up here — upstream connection failures, health check results, TLS handshake failures.

Envoy admin interface

Every Envoy proxy exposes an admin API on port 19000. This is where you can see the full xDS config, active clusters, and real-time statistics:

kubectl port-forward -n infra deployment/envoy-infra-platform-gateway 19000:19000
kubectl port-forward -n infra deployment/envoy-infra-platform-gateway 19000:19000
  • http://localhost:19000/clusters — all upstream clusters and their endpoints
  • http://localhost:19000/config_dump — the full Envoy xDS config as Envoy sees it
  • http://localhost:19000/stats — counter and gauge metrics for every subsystem

The config dump is particularly useful when Envoy is routing to the wrong backend or silently dropping requests. If your HTTPRoute is accepted and the service exists, but requests fail, the answer is almost always in the config dump.

What Envoy Gives Out of the Box

Most people install Envoy Gateway for routing. They're getting a lot more than they realise.

Observability without instrumentation. Envoy emits per-route request counts, latency histograms, and error rates without any code changes. Once you install Prometheus (covered in part three), you get detailed traffic breakdowns per hostname, per route, per upstream cluster — for every service behind the gateway, automatically.

Connection-level retries and timeouts. Configure these on the HTTPRoute, not in application code:

rules:
  - backendRefs:
      - name: my-service
        port: 8080
    timeouts:
      request: 10s
    retry:
      attempts: 3
      perTryTimeout: 3s
      retryOn: "5xx,gateway-error,reset"
rules:
  - backendRefs:
      - name: my-service
        port: 8080
    timeouts:
      request: 10s
    retry:
      attempts: 3
      perTryTimeout: 3s
      retryOn: "5xx,gateway-error,reset"

Header manipulation. Add, remove, or rewrite request and response headers at the gateway layer. Useful for injecting tenant IDs, stripping internal headers before they reach upstreams, or normalising X-Forwarded-For.

Traffic splitting. Send 10% of traffic to a canary backend with a single HTTPRoute spec change — no service mesh required for this use case:

rules:
  - backendRefs:
      - name: my-service
        port: 8080
        weight: 90
      - name: my-service-canary
        port: 8080
        weight: 10
rules:
  - backendRefs:
      - name: my-service
        port: 8080
        weight: 90
      - name: my-service-canary
        port: 8080
        weight: 10

This is a meaningful amount of capability before you've written a single line of sidecar configuration.

The Ownership Model in Practice

In a real platform, this is how you'd structure it:

  • Platform team manages GatewayClass and Gateway in an infra namespace. They control what listeners exist, what TLS certificates are loaded, and which namespaces can attach routes.
  • Application teams manage HTTPRoute in their own namespaces. They declare which hostnames they own and which services back them. They cannot modify the Gateway config.
  • Policy sits on the Gateway — Envoy's EnvoyPatchPolicy or BackendTrafficPolicy resources let the platform team set defaults (timeouts, rate limits, CORS) that apply to all routes unless overridden.

The key result: when an application team deploys a new service, they write one HTTPRoute resource in their namespace. The platform team doesn't touch anything. The gateway picks it up automatically. This scales across hundreds of services in a way that a shared Ingress manifest never does.

What's Next

The gateway gives you routing. The next piece is knowing what's actually flowing through it — metrics, logs, traces, and dashboards, all plumbed correctly for a multi-tenant environment.

Multi-Tenant Observability: LGTM at Platform Scale →