Configuring Cilium Networking for Your Shoot Cluster

This section explains how to enable the Cilium networking extension for a Gardener Shoot and describes every option you can set in the Cilium NetworkConfig. It also covers the routing modes (VXLAN, Geneve and native routing) and when to use each one.

Enabling Cilium

Cilium is currently the only CNI extension provided by OSC and is enabled by default for every Shoot created on OSC. To verify that Cilium is enabled, check the Shoot manifest for the label networking.extensions.gardener.cloud/cilium: 'true' and ensure that the networking section defines all necessary CIDRs.

apiVersion: core.gardener.cloud/v1beta1
kind: Shoot
metadata:
  name: my-cluster
  namespace: garden-my-project
  labels:
  ...
  networking.extensions.gardener.cloud/cilium: 'true'
  ...
spec:
  networking:
    type: cilium
    pods: 100.96.0.0/11
    nodes: 10.250.0.0/16
    services: 100.64.0.0/13
    # ipFamilies: [IPv4]          # IPv4 (default)
    providerConfig:
      apiVersion: cilium.networking.extensions.gardener.cloud/v1alpha1
      kind: NetworkConfig

providerConfig is optional. Without it, Cilium runs with the defaults described here

Shoot fieldMeaning
spec.networking.typeMust be cilium.
spec.networking.podsPod CIDR. Cilium also uses it as the native routing CIDR when native routing is active.
spec.networking.nodesNode CIDR. Cilium uses it for out-of-cluster SNAT.
spec.networking.servicesService CIDR.
spec.networking.ipFamiliesMust be [IPv4]
OSC recommendation

Concrete configuration recommendet by OSC is described here


Defaults applied automatically

AreaDefault behaviour
kube-proxyWhen a new Shoot is created, the extension sets spec.kubernetes.kubeProxy.enabled: false, and Cilium takes over the kube-proxy's job (kube-proxy-replacement: true). Existing Shoots keep their current setting.
Kubernetes API endpoint for CiliumWhen kube-proxy is disabled, Cilium connects to the API server through the Shoot's external advertised address.
RoutingDepends on overlay and tunnel. See Routing modes and tunnels.
MasqueradingIPv4 and IPv6 masquerading are enabled. BPF-based masquerading is off, so iptables does the masquerading.
Load balancingdefault value is snat load balancing mode.
DNSCoreDNS is used for internal in-cluster resolution.
cilium-operator HAThe operator runs 2 replicas only if the minimum values of all worker pools add up to at least 2. Otherwise it runs 1 replica.
MTUDetected automatically.
Hubble, debug, encryption, egress gateway, BGP, L2 announcements, policy audit modeDisabled.
metrics-server policyA CiliumNetworkPolicy called metric-server-cilium is created in kube-system. It allows remote-node ingress to metrics-server.
OSC recommendation

OSC strongly recommends using the default setting for kubeProxy (kubeProxy.enabled: false).


Routing modes and tunnels

Cilium can move pod-to-pod traffic between nodes in two ways:

  1. Encapsulation (tunnel): Cilium wraps each packet from a pod in an extra UDP header and sends it to the other node's IP address. The underlying network only sees node IPs.
  2. Native routing (no tunnel): Packets leave the node with the real pod IP as the source and destination. The underlying network must know how to route the pod CIDRs.

Two fields control this: overlay.enabled and tunnel.

What each tunnel value does

tunnel: vxlan

  • How it works: Cilium wraps each pod packet in a VXLAN header and sends it over UDP. Cilium uses UDP port 8472 by default.
  • Overhead: About 50 bytes per packet, so the pod MTU is lower than the node MTU. Cilium adjusts this automatically.
  • Network requirements: Nodes need IP connectivity to each other, and UDP 8472 must be open between them. The infrastructure doesn't need any routes for pod CIDRs.
  • Use it when: You need something that works on any infrastructure. The underlying network can't or shouldn't know pod IPs, or the nodes are in different L2 segments or subnets.
  • Trade-offs: Encapsulation adds a little CPU cost and lowers the MTU. Outside the cluster, pod IPs are hidden behind node IPs.

tunnel: geneve

  • How it works: Cilium wraps each pod packet in a Geneve header and sends it over UDP port 6081.
  • Overhead: Similar to VXLAN, about 50 bytes or more. Geneve also supports extensible option headers.
  • Network requirements: Nodes need IP connectivity to each other, and UDP 6081 must be open between them.
  • Use it when: You want an overlay but your environment or security policy prefers Geneve, or you plan to use Cilium features that rely on Geneve options.
  • Trade-offs: Same as VXLAN. Some older network devices and firewalls handle VXLAN better.

tunnel: disabled (native routing)

  • How it works: No encapsulation. Cilium sets routing-mode: native, and packets carry the real pod IPs on the wire.
  • Overhead: None. You get the full MTU and the best throughput and latency.
  • Network requirements: The underlying network must route the pod CIDR (spec.networking.pods) to the correct node.
  • Use it when: Performance matters and your infrastructure can route pod IPs.
  • Trade-offs: Pod traffic fails if the network can't route pod CIDRs. Egress Gateway is not available.
OSC recommendation

In OSC infrastructure, native routing is supported and highly recommended. Therefore, unless there is a technical reason to use a different mode, use native routing what means tunnel: disabled. For IPv4 native routing, the Shoot’s Pod CIDR is used as the native routing CIDR. A Variant with tunnel: disabled is the only variant that is fully tested on OSC infrastructure. If it is used a different mode, for example tunnel: geneve or vxlan than OSC cannot guarantee proper performance and reliability.

How overlay and tunnel combine

Cilium decides the effective routing from both fields:

overlay.enabledtunnelEffective routing modeNative routing CIDR set automatically
(not set)(not set)VXLAN tunnelNo
(not set)vxlanVXLAN tunnelNo
(not set)geneveGeneve tunnelNo
(not set)disabledNative routingNo. You must also set ipv4NativeRoutingCIDREnabled: true. Otherwise the Cilium agent won't start.
true(not set) or disabledNative routingyes (pod CIDR)
truevxlanVXLAN tunnelNo
truegeneveGeneve tunnelNo
falsedisabled or (any value)Native routingyes (pod CIDR)
Overlay tunnel config mismatch

In this distribution, overlay.enabled: true without an explicit tunnel value or with tunnel: disabled gives you native routing, not VXLAN. If you need encapsulation together with overlay.enabled: true, set tunnel: vxlan or tunnel: geneve explicitly.

With overlay.enabled: false, any tunnel value you set is overwritten with disabled.

Load balancing modes

These modes only apply when Cilium replaces kube-proxy.

ModeBehaviourWhen to use
snat (default)The node that receives the request SNATs it to the backend. Replies return through the same node.Works on any network.
dsrDirect Server Return. The backend replies to the client directly and keeps the client source IP.Lowest latency, and the backend sees the client IP. Requires native routing.
hybridDSR for TCP, SNAT for UDP.Compromise when UDP traffic doesn't work well with DSR.

OSC recomended setup

OSC recommends network settings that make the most of our infrastructure capabilities. Using native routing with direct server return (DSR) load balancing provides optimized traffic management, improved scalability, and superior network performance without the need for tunneling protocols.

apiVersion: core.gardener.cloud/v1beta1
kind: Shoot
metadata:
  name: my-cluster
  namespace: garden-my-project
spec:
  kubernetes:
    kubeProxy:
      enabled: false
  networking:
    type: cilium
    pods: 100.96.0.0/11
    nodes: 10.250.0.0/16
    services: 100.64.0.0/13
    providerConfig:
      apiVersion: cilium.networking.extensions.gardener.cloud/v1alpha1
      kind: NetworkConfig
      debug: false
      overlay:
        enabled: false     <--- Ensuring native routing is active
      tunnel: disable      <--- Ensuring native routing is active
      bpfSocketLBHostnsOnly:
        enabled: true
      cni:
        exclusive: true
      loadBalancingMode: dsr  <---- Allowing DSR load balancing mode
  provider:
    workers:
      - name: worker
        minimum: 2   # >= 2 enables cilium-operator HA
        maximum: 5

NetworkConfig options in providerConfig section

FieldTypeDefaultDescription
debugboolfalseTurns on debug logging for the Cilium agent.
hubble.enabledboolfalseDeploys Hubble, Hubble Relay and Hubble UI for flow observability.
tunnelvxlan | geneve | disabledvxlanEncapsulation mode. See Routing modes.
overlay.enabledboolnot setSwitches between overlay and native routing - if tunnel field not configured than overlay.enabled: true has the same effect as overlay.enabled: true . See the combination table.
overlay.createPodRoutesboolnot setInstalls direct node-to-node pod routes. Only applies when overlay.enabled: false, and requires a shared L2 network.
storekuberneteskubernetesIdentity store. Only kubernetes (CRD) is supported.
kubeproxy.k8sServiceHoststring (IP or DNS name)Shoot external API addressAPI server host that Cilium uses when kube-proxy is disabled. It only takes effect if k8sServicePort is also set.
kubeproxy.k8sServicePortint (1–65535)—API server port that Cilium uses when kube-proxy is disabled. It only takes effect if k8sServiceHost is also set.
bpfSocketLBHostnsOnly.enabledboolfalseUses socket-level load balancing only in the host namespace. Pods use service load balancing at their interface instead. Required for service meshes such as Istio or Linkerd.
cni.exclusivebooltrueIf false, Cilium won't overwrite other CNI config files, so plugins such as Istio CNI or Linkerd CNI can chain after Cilium.
egressGateway.enabledboolfalseTurns on Cilium Egress Gateway. Requires kube-proxy to be disabled and a tunnel (vxlan/geneve).
mtuint (≥ 0)autoOverrides the MTU that Cilium detects automatically. 0 means auto.
devices[]stringautoNetwork devices that face the cluster or external network. Cilium attaches its BPF NodePort, host firewall and masquerade programs to them.
directRoutingDevicestringautoDevice used for direct routing between nodes, for example by NodePort/DSR.
loadBalancingModesnat | dsr | hybridsnatService load-balancing mode. See Load balancing modes.
ipv4NativeRoutingCIDREnabledbooldepends on overlaySets ipv4-native-routing-cidr to the Shoot pod CIDR. Traffic to this CIDR isn't masqueraded.
ipv6NativeRoutingCIDREnabledbooldepends on overlaySets ipv6-native-routing-cidr to the Shoot's IPv6 pod CIDR.
enableIpv4MasqueradebooltrueMasquerades IPv4 pod traffic that leaves the node towards destinations outside the native routing CIDR.
enableIpv6MasqueradebooltrueSame as above, for IPv6.
enableBPFMasqueradeboolfalseUses eBPF instead of iptables for masquerading. Ignored if snatToUpstreamDNS or snatOutOfCluster is enabled.
snatToUpstreamDNS.enabledboolfalseMasquerades pod traffic to the node's upstream DNS servers (from /etc/resolv.conf). Use this when the infrastructure blocks DNS requests that come from pod IPs.
snatOutOfCluster.enabledboolfalseMasquerades all pod traffic to destinations outside the pod and node CIDRs, using iptables.
l2Announcements.enabledboolfalseTurns on Cilium L2 announcements (ARP replies for LoadBalancer/External IPs).
l2Announcements.leaseDurationdurationCilium defaultMaximum lease duration, for example 15s.
l2Announcements.leaseRenewDeadlinedurationCilium defaultHow long the current leader keeps retrying a lease renewal before it gives up.
l2Announcements.leaseRetryPerioddurationCilium defaultWait time between lease retries.
bgpControlPlane.enabledboolfalseTurns on the Cilium BGP Control Plane.
policyAuditModeboolfalseLogs policy verdicts without enforcing them. Network policies aren't enforced while this is on.
encryption.enabledboolfalseTurns on transparent encryption.
encryption.modewireguard—Encryption backend. Only wireguard is supported.
encryption.nodeToNodeEnabledboolfalseAlso encrypts host-to-host (node) traffic.
encryption.strictModeboolfalseDrops unencrypted pod-to-pod traffic. Only valid with mode: wireguard.

Feature guides

Hubble (observability)

hubble:
  enabled: true

To open the UI locally:

kubectl port-forward -n kube-system deployment/hubble-ui 8081
# open http://localhost:8081

Running without kube-proxy

New Shoots run without kube-proxy by default, and Cilium handles all Service, NodePort and LoadBalancer traffic in eBPF.

Service mesh (Istio )

bpfSocketLBHostnsOnly:
  enabled: true
cni:
  exclusive: false

Istio sidecars often expect traffic handling in a way that can conflict with Cilium’s socket-level load balancing inside pod namespaces.

L2 announcements

l2Announcements:
  enabled: true
  leaseDuration: 15s
  leaseRenewDeadline: 5s
  leaseRetryPeriod: 2s

After that, create CiliumL2AnnouncementPolicy and CiliumLoadBalancerIPPool resources in the Shoot. This feature requires kube-proxy replacement.

Complete examples

1. Minimal (VXLAN tunnel, default)

spec:
  networking:
    type: cilium
    pods: 100.96.0.0/11
    nodes: 10.250.0.0/16
    services: 100.64.0.0/13

2. VXLAN overlay with Hubble

spec:
  networking:
    type: cilium
    pods: 100.96.0.0/11
    nodes: 10.250.0.0/16
    services: 100.64.0.0/13
    providerConfig:
      apiVersion: cilium.networking.extensions.gardener.cloud/v1alpha1
      kind: NetworkConfig
      overlay:
        enabled: true
      tunnel: vxlan
      hubble:
        enabled: true

3. Geneve overlay

    providerConfig:
      apiVersion: cilium.networking.extensions.gardener.cloud/v1alpha1
      kind: NetworkConfig
      overlay:
        enabled: true
      tunnel: geneve

4. Native routing (infrastructure routes pod CIDRs)

    providerConfig:
      apiVersion: cilium.networking.extensions.gardener.cloud/v1alpha1
      kind: NetworkConfig
      overlay:
        enabled: false

6. Native routing with DSR load balancing

spec:
  kubernetes:
    kubeProxy:
      enabled: false
  networking:
    type: cilium
    providerConfig:
      apiVersion: cilium.networking.extensions.gardener.cloud/v1alpha1
      kind: NetworkConfig
      overlay:
        enabled: false
      loadBalancingMode: dsr
      tunnel: disable

Validation rules and common errors

Rule / messageCauseFix
networking type "<x>" is not supportedspec.networking.type isn't cilium.Set type: cilium.
unsupported value ... for tunneltunnel isn't vxlan, geneve or disabled.Use one of the supported values.
unsupported value ... for storestore isn't kubernetes.Remove the field or set it to kubernetes.
unsupported value ... for loadBalancingModeUnsupported value.Use snat, dsr or hybrid.
mtu must be a positive integerNegative MTU.Use 0 (auto) or a positive value.
device name ...Invalid name in devices/directRoutingDevice.Use 1–15 characters, with no / or whitespace.
serviceHost is neither a valid IP address nor a valid domain nameInvalid kubeproxy.k8sServiceHost.Use an IP address or a DNS-1123 name.
servicePort must be between 1 and 65535Port out of range.Fix the port.
strict mode can only be used with wireguardencryption.strictMode: true with a different mode.Set mode: wireguard.
ipFamilies ... cannot have duplicate entries / more than two entriesInvalid ipFamilies.Use [IPv4], [IPv6] or [IPv4, IPv6].
Running kube-proxy with IPVS mode is forbidden in conjunction with node local dns enabledIPVS mode + node-local DNS.Disable kube-proxy, switch to iptables mode, or disable node-local DNS.
cilium must run with an overlay network in order to enable egressGatewayEgress Gateway with native routing.Set tunnel: vxlan/geneve explicitly.
kubeProxyReplacement must be set to 'strict' or 'true' in order to enable egressGatewayEgress Gateway with kube-proxy enabled.Set spec.kubernetes.kubeProxy.enabled: false.

Troubleshooting

# Cilium agent status on every node
kubectl -n kube-system exec ds/cilium -- cilium-dbg status --verbose

# Effective routing mode / tunnel protocol
kubectl -n kube-system get cm cilium-config -o yaml | grep -E 'routing-mode|tunnel-protocol|native-routing-cidr|auto-direct-node-routes'

# Connectivity between nodes
kubectl -n kube-system exec ds/cilium -- cilium-dbg status --all-health
SymptomLikely cause
Pods on different nodes can't reach each other in native routing modeThe infrastructure doesn't route pod CIDRs. Use a tunnel, or createPodRoutes: true on a shared L2 network.
Pods on different nodes can't reach each other in tunnel modeUDP 8472 (VXLAN) or 6081 (Geneve) is blocked between nodes.
Large packets are dropped or connections hangMTU mismatch. Set mtu explicitly.
DNS lookups from pods time outThe upstream DNS server rejects pod source IPs. Enable snatToUpstreamDNS.
Cilium agent doesn't start with tunnel: disabledNo native routing CIDR. Use overlay.enabled: false, or set ipv4NativeRoutingCIDREnabled: true.
Network policies have no effectpolicyAuditMode: true is set.