Legacy Platform → Consolidated EKS

The problem

The estate had grown service-by-service and account-by-account, not as a platform. Pain: slow new services, inconsistent security, extra load balancers, expensive log plumbing, and no single place to see what is running where.

• Accounts: services running in different AWS accounts, with their own IAM, networking, and deploy habits
• Compute: ECS Fargate stacks per app, plus EC2 servers for leftovers that never made it onto Fargate
• Jobs: cron on EC2, EventBridge + ECS tasks, and Lambda + SQS for pipelines that did not fit cleanly
• Envs: copies that drifted; staging later dropped
• Logs: about 30 CloudWatch → Lambda → ELK forwarders, one per log group
• Secrets: per-stack Secrets Manager, no cluster-wide operator
• Networking: public ALBs as the default — multiple unnecessary load balancers, little consistent private ingress
• Deploys: per-repo, per-stack, or per-account updates — no shared GitOps source of truth
• Scale: always-on EC2, idle Fargate capacity; no cluster-wide bin-packing

The solution

Treat Kubernetes as the runtime for app workloads. Clusters and data plane stay in Terraform. Everything that changes weekly — images, routes, replicas — lives in git and is synced by Argo CD. Istio owns north-south traffic. Defguard split-tunnel VPN plus private hosted zones is how humans reach the internal mesh. Secrets and AWS access are pod-scoped. One ingress path instead of a load balancer per service.

Architecture

App repo push
    → reusable GitHub Actions (OIDC → ECR)
    → patch GitOps manifests
    → Argo CD syncs the cluster
    → Istio VirtualService
         ├── public gateway   (internet / CDN)
         └── internal gateway (VPN + private DNS)

What I implemented

• Terraform for dev and prod EKS — cluster, KMS, add-ons, Karpenter, ESO, IAM, private hosted zones. IAM policy names namespaced per env so reusable modules did not hit AWS’s global unique-name constraint
• Argo CD bootstrap and one Application per workload / env. Chart repositories added in CI (Helm home on the runner), so istio was never “repo not found” because it only existed on a laptop
• Istio public and internal gateways; VirtualServices that split internet vs VPN-only APIs
• Defguard split tunnel: VPC traffic and VPC DNS only — internet stays local
• cert-manager wildcards so new internal services do not need hand-rolled certs
• External Secrets and EKS Pod Identity so pods never use node-wide AWS keys. Access-denied debug splits authentication (who called) from authorization (IAM vs KMS key policy)
• Reusable build → ECR → GitOps tag bump workflows (OIDC, no long-lived AWS keys). Git refs listed before checkout when gitops is both a branch and a path
• In-cluster Actions runners for post-sync verification
• NetworkPolicies, deletion protection, private API CIDR controls
• Cut-over of ECS, EC2, Lambda, EventBridge, extra ALBs, and ELK; archive-only Terraform left as reference

How we cut over

1. Stand up clusters, mesh, GitOps, Defguard VPN, secrets, and observability first
2. Move services one by one — including those in other AWS accounts and on EC2: containerize → GitOps overlay → dual-run if needed → flip DNS
3. Jobs: EC2 cron → CronJobs; EventBridge + ECS and Lambda/SQS → KEDA-scaled Deployments on the same queues
4. Logs: stop per-service forwarders once Loki and Prometheus were live
5. Retire extra load balancers; destroy ECS / EC2 / Lambda / ELK Terraform; keep one reference copy, not live state

Before / after

Results

• One platform instead of a Terraform repo — or AWS account — per service
• Private-by-default APIs — internal hosts only on Defguard VPN and private DNS
• Same deploy path for every backend: GitOps, not ECS vs EC2 vs Lambda vs EventBridge
• Fewer load balancers; Istio owns public vs internal ingress
• Cheaper, simpler ops — no ELK forwarder fleet, no always-on EC2 cron boxes; Karpenter instead of idle Fargate
• Faster new services — overlay + VirtualService + Argo app, not a new stack in another account
• Clear blast radius — two clusters, shared VPC, explicit public vs internal gateways

Stack