EKS Networking Model: VPC CNI, ENI Limits, and Pod IP Planning

What this means in plain terms

How EKS networking works

When a pod is scheduled onto a node, the Amazon VPC CNI plugin runs on that node and assigns the pod an IP address from the VPC subnet. That IP comes from one of the secondary IP slots on an Elastic Network Interface (ENI) attached to the EC2 instance.

The CNI plugin includes a daemon called ipamd (IP Address Management Daemon), which runs as part of the aws-node DaemonSet on every node. Rather than allocating IPs on demand when pods start, ipamd maintains a warm pool of pre-allocated IPs ready to hand to new pods immediately. It watches for available ENI slots and fills them proactively, which keeps pod startup times fast.

When a pod is deleted, its IP is returned to the warm pool and reused by the next pod on that node.

Primary ENI vs secondary ENIs#

Each EC2 instance has one primary ENI that holds the node’s main IP address — the IP Kubernetes uses to identify the node. The VPC CNI attaches secondary ENIs to the instance and uses the secondary IP slots on those ENIs for pod IPs. The primary IP of each ENI is reserved, which is why the pod capacity formula subtracts one IP per ENI.

Pod networking is pure VPC routing. There is no encapsulation layer, no VXLAN tunnel, and no virtual bridge to manage or debug.

Why EKS uses VPC CNI instead of an overlay network

Most self-managed Kubernetes distributions use an overlay network (Flannel, Weave, Calico in encapsulation mode). An overlay gives pods private addresses invisible to the underlying network. Traffic between pods on different nodes is wrapped in a packet the physical network can route, then unwrapped at the destination.

EKS chose the native VPC approach deliberately:

• Pods can reach RDS, ElastiCache, MSK, and other services using security group rules, exactly like EC2 instances — no proxy needed.
• VPC Flow Logs show the pod’s actual IP, making security investigations and compliance audits straightforward.
• No encapsulation means lower latency and less CPU overhead per packet.
• AWS security groups can be applied at the pod level with Security Groups for Pods (covered below).

The cost is IP address consumption. Every pod uses a real VPC IP, which means large clusters need large subnets.

ENI limits and pod density

Pod count on a node is not only limited by CPU and memory. It is also hard-capped by the instance’s ENI and IP limits. The VPC CNI uses this formula:

Max pods = (Number of ENIs x (IPs per ENI - 1)) + 2

The -1 per ENI reserves one IP for the ENI itself. The +2 accounts for the kubelet and the node networking daemon.

The following figures are examples based on current AWS documentation. Verify against the EC2 ENI documentation for your instance type before making capacity decisions, as new instance types are added regularly.

Two things are worth paying close attention to:

• Doubling CPU does not double pod capacity. An m5.xlarge and an m5.2xlarge both cap at 58 pods because they share the same ENI configuration. If you need more pods per node, choose an instance with more ENIs, not just more vCPU.
• Small instances have surprisingly low pod limits. A t3.medium can run at most 17 pods regardless of how much CPU and memory sits unused.

IP address planning and subnet sizing

Because every pod consumes a VPC IP, the subnets for your EKS cluster need to accommodate both nodes and pods. Each node uses one IP, each pod uses one more, and ipamd holds a warm buffer above current demand.

A cluster with 50 nodes running 30 pods each needs at minimum 1,550 IPs before the warm pool overhead. Size your subnets for at least 2x the addresses you expect at peak.

Capacity planning checklist#

Before provisioning a cluster, work through these questions:

• How many nodes will the cluster run at peak? What is the 2x growth target?
• What is the target pod density per node? Check the ENI limits for your instance type.
• Are you deploying across three AZs? Each AZ needs its own subnet sized for the full peak node count.
• Does ipamd’s warm pool fit in the subnet? By default, ipamd pre-allocates a buffer of IPs above current demand.
• Do other workloads share these subnets? Pod IPs and EC2 instance IPs come from the same pool.
• Are you using secondary CIDR blocks to extend address space? These must be attached to the VPC before cluster creation.

Prefix delegation and higher pod density

When secondary IP mode is too constrained, the VPC CNI supports prefix delegation on Nitro-based instance types. Instead of assigning individual IP addresses to ENI slots, prefix delegation assigns entire /28 CIDR blocks (16 IPs each) to every slot. A single ENI slot that previously held one pod IP can now hold 16, substantially raising how many pods can run per node.

Enable it by setting an environment variable on the aws-node DaemonSet:

kubectl set env daemonset aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true

After enabling prefix delegation, update the --max-pods kubelet argument for new nodes. The default kubelet limit does not update automatically; new nodes will still cap at the old secondary IP mode value unless the bootstrap configuration is updated through the node group launch template or managed node group configuration.

Security Groups for Pods

By default, security group rules apply at the node level. All pods on a node share the node’s security group. If any pod on a node needs to reach an RDS instance on port 5432, the entire node has that rule open — which applies to every pod on that node.

Security Groups for Pods (SGP) lets you assign AWS security groups to individual pods rather than to the node. This makes pod-level network segmentation possible without running separate node groups for each pod type.

How SGP works#

SGP relies on trunk and branch ENIs. The VPC Resource Controller (part of EKS) reserves one ENI on each supporting node as a trunk ENI. When a pod needs SGP, the controller attaches a dedicated branch ENI to the trunk, assigns the pod an IP from that branch ENI, and applies the security groups defined in the pod’s SecurityGroupPolicy resource to that branch ENI.

This is distinct from standard VPC CNI secondary IP mode. SGP pods use a separate ENI chain managed by the resource controller, not the main ipamd warm pool.

Enabling SGP#

SGP requires Nitro-based instance types and ENABLE_POD_ENI=true on the aws-node DaemonSet:

kubectl set env daemonset aws-node -n kube-system ENABLE_POD_ENI=true

Assigning security groups with SecurityGroupPolicy#

The correct way to assign security groups to pods is the SecurityGroupPolicy Custom Resource Definition (CRD), installed automatically by the VPC Resource Controller on supported clusters:

apiVersion: vpcresources.k8s.aws/v1beta1
kind: SecurityGroupPolicy
metadata:
  name: allow-db-access
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: payment-processor
  securityGroups:
    groupIds:
      - sg-0123456789abcdef0

Any pod in the production namespace with the label app: payment-processor will be assigned security group sg-0123456789abcdef0. You can also use serviceAccountSelector instead of podSelector, which is useful in IRSA-based architectures where pod identity is already tied to service accounts.

Important caveats#

Supported instances only. SGP requires Nitro-based instances. Verify all node groups in your cluster use compatible instance types before setting ENABLE_POD_ENI=true. Mixed node groups with unsupported instances can cause scheduling errors.

Pod startup latency. Attaching a branch ENI to a trunk takes a few seconds. Pods using SGP start slightly slower than standard pods. For workloads with aggressive scale-out requirements, factor this overhead into your design.

Private subnet behaviour. SGP works correctly in private subnets. For pods in public subnets needing outbound internet access, the branch ENI also needs a route to the internet, typically handled by a NAT gateway.

NodePort and external load balancer traffic. Pods using SGP receive inbound traffic through their branch ENI, which can cause asymmetric routing issues with NodePort services and some LoadBalancer configurations. Test this carefully if your SGP pods receive external traffic through NodePort.

Kubelet health probes. Kubernetes liveness and readiness probes originate from the node, not from inside the pod. Make sure the node’s security group allows the kubelet to reach the pod’s health check port, independent of any SGP rules restricting the branch ENI.

Reserved trunk ENI capacity. Each node reserves one ENI slot for the trunk ENI, slightly reducing total secondary IP capacity available to standard pods on that node.

Security Groups for Pods vs Kubernetes NetworkPolicy

These two features address related but distinct problems. Using one does not replace the other.

Use Security Groups for Pods to control access between pods and AWS services. Use Kubernetes NetworkPolicy to enforce east-west traffic controls within the cluster. The securing EKS clusters guide covers how these fit together into a full security posture.

Pod-to-pod communication and cross-AZ traffic

Because pods have real VPC IPs, pod-to-pod traffic is standard VPC routing with no overlay to traverse:

• Same node: Traffic goes through the local Linux network namespace. No physical network involved.
• Same AZ, different nodes: Standard VPC routing within the AZ. Fast and free.
• Different AZs: Standard VPC routing across AZs. Each GB of data transfer between AZs costs approximately $0.01. For high-throughput workloads like streaming pipelines or real-time analytics, this adds up.

Use Kubernetes topology spread constraints to colocate related pods within the same AZ when latency or cost matters. For Kubernetes services that should prefer local traffic, configure topology-aware routing on the service.

When this model works well, and when it becomes limiting

Good fit#

AWS-native workloads. If pods need to reach RDS, ElastiCache, MSK, or other VPC resources using security groups, the VPC CNI model is a natural match. Clean security group rules, no translation layer.

Teams that need VPC-level visibility. VPC Flow Logs capture real pod IPs. This makes it straightforward to investigate access patterns, connectivity issues, and compliance audit requirements.

Organisations already familiar with AWS networking. If your team understands VPC subnets, ENIs, and security groups, operating the VPC CNI model requires no new mental model.

Well-planned clusters with adequate IP space. If you pre-sized your subnets correctly, the model is operationally simple. No overlay to debug, no encapsulation to reason about.

When it becomes painful#

IP-constrained VPCs. If your organisation has used a large portion of its RFC 1918 address space, carving out large subnets for pods can be difficult. You may need secondary CIDRs or custom networking mode, both of which add complexity.

Very high pod density on small instances. Small and medium instance types have ENI ceilings that prefix delegation can raise but not eliminate entirely. See the managed node groups guide for instance type selection guidance.

Multi-tenant clusters without careful design. The default shared-node model means tenants share a security group. Using SGP for per-tenant isolation requires SecurityGroupPolicy resources per tenant and careful testing of the caveats above.

Environments limited to non-Nitro instances. Prefix delegation and Security Groups for Pods both require Nitro. Non-Nitro instances lock you out of both features, limiting pod density and granular security controls.

Troubleshooting checklist

Pods stuck in Pending

• Run kubectl describe pod <name> and look for failed to assign an IP address to container
• Check available IPs in the node’s subnet via CloudWatch (AvailableIpAddressCount) or the VPC console
• Compare the node’s current pod count against its ENI-based max pods limit
• Check if ipamd is healthy: kubectl logs -n kube-system -l k8s-app=aws-node

Wrong max-pods value after enabling prefix delegation

• The kubelet --max-pods argument must be updated for new nodes; it does not change automatically
• Check the current allocatable pods: kubectl get node <name> -o jsonpath='{.status.allocatable.pods}'

Subnet CIDR exhausted

• Check AvailableIpAddressCount across all AZ subnets in CloudWatch
• Consider adding a secondary VPC CIDR and new subnets, then migrating node groups to the new subnets

Security Groups for Pods not applying

• Verify ENABLE_POD_ENI=true is set on the aws-node DaemonSet
• Confirm the node instance type is Nitro-based
• Check that a SecurityGroupPolicy resource exists in the correct namespace with labels matching the pod
• Check VPC Resource Controller logs: kubectl logs -n kube-system -l app=vpc-resource-controller

Cross-AZ traffic surprises

• Run kubectl get pods -o wide to confirm which nodes pods are running on and which AZ those nodes are in
• Enable VPC Flow Logs and filter by AZ to confirm where traffic is actually flowing

Pod-to-AWS-service traffic not working

• Confirm the pod’s IP belongs to the correct subnet CIDR so security group rules apply as expected
• Check that the target service’s security group allows inbound traffic from the pod’s security group or IP range
• For SGP pods, verify the branch ENI’s security group has the required egress rule

For general VPC connectivity issues, see the troubleshooting network issues guide. For cluster-level monitoring and alerting, see monitoring EKS clusters.