Private by Default Service Mesh

One Istio mesh, two gateways: public traffic through the edge, internal traffic never leaving the VPC.

Private by default

internal APIs never leave the VPC

Wildcard TLS

issued from AWS Private CA

Public / internal

two gateways on one mesh

Pod Identity

no node wide AWS credentials

Overview

A platform usually starts out with one way in: a public load balancer per service. That is fine until some of those services are only ever meant to be called by other services, or by the people running the platform. At that point a public endpoint with an authentication check in front of it is doing more work than it should.

I separated the two. Anything public keeps its route through Cloudflare and an Istio public gateway. Anything internal moves behind a second gateway that only answers inside the VPC, on a private Network Load Balancer with no public IP, using names that only resolve on a Route 53 private hosted zone. Certificates come from AWS Private CA with ACM wildcards, so adding an internal service does not mean issuing another certificate by hand.

The problem

Internal services were exposed the same way public ones were, which made the security story depend entirely on what sat in front of each endpoint.

Internal APIs reachable from the internet, protected only by application level checks
A public load balancer per service, and cost and attack surface growing with each one
Certificates issued by hand for internal hostnames, with expiries nobody tracked
Internal hostnames resolvable publicly, which leaked the shape of the platform
No single place to say what is public and what is not

The solution

Treat public and internal as two separate paths through one mesh. Exposure becomes a property of the route, not an accident of how the service was deployed.

Public edge

Cloudflare with WAF, bot controls and rate limits in front of the public gateway

Public gateway

Istio gateway carrying only the routes meant to be internet facing

Private DNS

Route 53 private hosted zone, so internal names resolve only inside the VPC

Private entry

Internal NLB with no public IP, reachable only from inside the network

Internal gateway

A second Istio gateway terminating TLS for internal routes

TLS

AWS Private CA with ACM wildcards, so new internal hosts need no manual certificate

Identity

EKS Pod Identity and Pod Security Admission instead of node wide credentials

Secrets

Vault and External Secrets Operator syncing into the cluster

Delivery

Argo CD and FluxCD, so routing and policy changes land through git

Architecture

Two paths into the same EKS cluster: internet users pass through Cloudflare into an Istio public gateway, while internal clients resolve a Route 53 private zone and reach a private NLB and an Istio internal gateway holding certificates from AWS Private CA and ACM

mesh.txt

Internet
    → Cloudflare (WAF, bot, rate limits)
    → Istio public gateway
         └── public VirtualServices only

Internal client
    → Route 53 private hosted zone
    → private NLB (no public IP)
    → Istio internal gateway (AWS Private CA + ACM wildcard)
         └── internal VirtualServices

What I implemented

A second Istio gateway for internal routes, so public and internal exposure are separate objects rather than a naming convention
AWS Private CA as the internal certificate authority, with ACM wildcards so a new internal host does not need its own certificate
An internal Network Load Balancer with no public IP, as the only entry point to the internal gateway
Route 53 private hosted zones, so internal hostnames do not resolve outside the VPC
Cloudflare WAF, bot controls and rate limits kept in front of the public gateway
EKS Pod Identity and Pod Security Admission so pods carry their own scoped AWS access instead of borrowing the node's
Vault and External Secrets Operator to sync secrets into the cluster rather than baking them into manifests
Routing and policy managed through Argo CD and FluxCD, so exposure changes are reviewable in git
IAM policy names namespaced per environment, since reusable Terraform modules otherwise collide on AWS global unique names

Before / after

Before

Internal APIs on public load balancers
A load balancer per service
Certificates issued by hand
Internal hostnames resolving publicly
Exposure decided per deployment

After

Internal APIs only reachable inside the VPC
Two gateways on one shared mesh
Wildcard TLS from AWS Private CA
Private hosted zones for internal names
Exposure is a routing rule in git

Results

Internal traffic stays inside the VPC, so an internal API is not one misconfiguration away from being public
Exposing a new internal service is a routing rule, not a new load balancer and a new certificate
Fewer load balancers, and less attack surface to keep track of
Certificates renew without anyone remembering to do it
What is public and what is not is visible in git, not inferred from AWS console settings

Stack

AWS EKSIstioAWS Private CAACMRoute 53Network Load BalancerCloudflareExternal SecretsVaultArgo CD

Have a project in mind?
Let's build something together.

Contact me