NVIDIA GPU Operator
Overview
The NVIDIA GPU Operator is a Kubernetes operator that automates the deployment, configuration, and management of the NVIDIA software required to run GPU-accelerated workloads.
Please note that according to the Service Description the use of GPU compute nodes requires a corresponding order and a successful capacity check.
GPU compute nodes are not permanently maintained in the platform's standard capacity pool. Provisioning therefore only takes place after an individual order and approval. Because procurement is required, the usual provisioning times for additional compute nodes do not apply. The actual provisioning time is agreed with the customer as part of capacity planning.
GPUs are allocated to workloads through one of two allocation patterns:
- GPU Allocation Using the Device Plugin —
workloads request GPUs through
nvidia.com/gpuresource requests - GPU Allocation Using DRA — workloads request GPUs and MIG slices through resource claims
Dynamic resource allocation (DRA) is the default allocation pattern on clusters running Kubernetes 1.34.2 or later. On older Kubernetes versions, or when DRA is disabled, the device plugin is used automatically.
Getting Started
Setting Up Your Shoot Cluster with GPU Workers
To use NVIDIA GPUs in your Shoot cluster, at least one worker group must use a GPU-enabled machine type.
- In your project, click Create Cluster to open the new Shoot form in the OSC Dashboard.
- Select a Machine Type with GPU capabilities.
- Set the Machine Image to
NVGPU Garden Linux. - Select a Zone where GPU nodes are available.
- Fill in the remaining fields and click Create.
- Wait for the Shoot cluster to be created and reconciled.
If only a limited number of GPUs is available,
set Autoscaler Max. equal to Autoscaler Min.
and set maxSurge: 0 and maxUnavailable: 1.
Otherwise a rolling update can get stuck,
because no additional GPU node can be provisioned for the surge.
Verifying GPU Availability
Check Node Labels
The Node Feature Discovery component automatically labels nodes with GPU capabilities. You can verify this by checking the node labels:
Check NVIDIA GPU Operator Status
Verify that the NVIDIA GPU Operator components are running:
You should see various NVIDIA operator pods running:
The nvidia-device-plugin-daemonset pod only appears when DRA is disabled — on DRA-enabled
clusters the device plugin is replaced by the DRA kubelet plugin.
Its DaemonSet deploys a nvgpu-dra-driver-kubelet-plugin-<id> pod
on each node supported by DRA;
the plugin is verified separately in
Verifying DRA Availability.
Check NVIDIA GPU Operator Validator
The NVIDIA GPU Operator includes a built-in validator
that sanity-checks driver, device plugin, and CUDA toolkit readiness.
Verify it by inspecting the logs of the nvidia-operator-validator pods:
Expected output:
GPU Allocation Using the Device Plugin
The device plugin advertises GPUs as nvidia.com/gpu resources on the nodes,
and workloads request them through resource requests in the pod specification.
The device plugin is used when the cluster runs a Kubernetes version
older than 1.34.2, or when DRA is disabled.
On DRA-enabled clusters, the nvidia.com/gpu resource is not advertised
and pods requesting it remain Pending.
Use the examples in
GPU Allocation Using DRA
instead.
Verifying Device Plugin Availability
Verify that GPU resources are advertised on your nodes:
On GPU nodes, the nvidia.com/gpu resource appears
in the allocatable resources.
Requesting GPUs with Resource Requests
To use a GPU in your pod, specify the GPU resource and runtimeClassName in the pod specification:
Make sure you have enough resources to satisfy the requests.
Running Sample GPU Applications
To verify that GPU operations are working correctly, run a sample CUDA application.
CUDA VectorAdd Example
This example runs a simple CUDA vector addition operation to verify GPU functionality.
-
Create the
cuda-vectoraddpod:The pod starts, runs the vectorAdd command, and then exits.
-
View the logs from the container:
Expected output:
The
Test PASSEDmessage confirms that the GPU is working. -
Remove the pod:
Simple nvidia-smi Test
For a quicker verification, run a pod that executes nvidia-smi:
-
Create the
nvidia-smipod: -
View the logs from the container:
Expected output:
GPU Allocation Using DRA (Dynamic Resource Allocation)
Dynamic resource allocation (DRA) is the default way to allocate GPUs in Shoot clusters. When the NVIDIA GPU Operator is enabled, the NVIDIA DRA GPU driver is deployed automatically — no manual preparation is required.
With DRA, GPUs are not requested through nvidia.com/gpu resource requests.
Instead, the DRA driver publishes all GPUs and MIG instances
as ResourceSlice objects,
and workloads request them with a ResourceClaimTemplate
referencing one of the available device classes:
gpu.nvidia.com— allocates full physical GPUsmig.nvidia.com— allocates MIG slices
DRA requires Kubernetes 1.34.2 or later. On clusters with an older Kubernetes version, the device plugin is used instead, and GPUs are requested as described in GPU Allocation Using the Device Plugin.
Customers remain responsible for scheduling upgrades on time and ensuring that their applications are compatible with the target version. We strongly recommend staying within the supported lifecycle, as it is necessary for security updates and reliable platform operation.
DRA can be disabled for a Shoot cluster through the NFD extension configuration, for example to keep using the device plugin. For details, refer to the OSC NFD Extension reference.
The examples in this section cover the common GPU allocation scenarios:
- Allocate a Full GPU
- Allocate a MIG Slice
- Select a GPU by Capability (CEL Selector)
- Atomic Multi-Instance Claim (Two MIG Slices)
Verifying DRA Availability
-
Verify that the DRA kubelet plugin is running on your GPU nodes:
Expected output:
-
Check that the device classes are available:
Expected output:
-
List the GPU devices published for your nodes:
Expected output:
Allocate a Full GPU
Use this when your workload needs exclusive access to an entire physical GPU — for example, a large model training job or a workload with high memory pressure.
-
Create a
ResourceClaimTemplate: -
Create a pod that references the template:
-
Verify that the pod is running:
Expected output:
-
Verify that the claim was allocated and bound to the pod:
Expected output:
Expected output:
The claim stays reserved until the pod terminates, at which point the device is released back to the pool.
Allocate a MIG Slice
Use this when multiple tenants or workloads need to share a single GPU with hardware-enforced memory and compute isolation — for example, inference services running alongside each other on an A30.
MIG must be enabled and the desired profiles must be configured on the node before DRA can allocate slices — see Multi-Instance GPU for the configuration and Verifying the MIG Layout for listing the available MIG instances.
After a MIG profile change, the DRA kubelet plugin is restarted automatically and publishes
the updated MIG layout in ResourceSlice objects.
-
Create a
ResourceClaimTemplate: -
Create a pod that uses the MIG slice:
-
Verify that the pod is running:
Expected output:
-
Verify that the MIG instance was allocated:
Expected output:
Multiple pods can each hold their own MIG slice claim on the same physical GPU simultaneously, with hardware-enforced isolation between them.
Select a GPU by Capability (CEL Selector)
Use this when you need a GPU that meets a specific hardware requirement — for example, a minimum architecture generation — without hard-coding a GPU model name.
-
Create a
ResourceClaimTemplatewith a Common Expression Language (CEL) selector expression: -
Create a pod that uses the capability-selected GPU:
-
Verify that the pod is running:
Expected output:
-
Verify that the allocated device satisfies the expression:
Expected output:
If no device satisfies the CEL expression,
the pod stays Pending with a clear scheduler event:
CEL attributes published by the NVIDIA DRA driver (verified on NVIDIA A30 with GPU Operator v25.10.1,
DRA driver k8s-dra-driver-gpu v25.12.0, NVIDIA driver v590.48.1, CUDA 13.1):
GPU memory is published as a device capacity, not an attribute. To select a GPU by minimum memory, use the capacity form:
Run
kubectl get resourceslices -o json | jq '.items[0].spec.devices[0] | {attributes, capacity}'
to confirm the attributes and capacities available on your cluster before writing CEL expressions.
Atomic Multi-Instance Claim (Two MIG Slices)
Use this when a workload needs exactly two MIG slices allocated simultaneously — for example, a pipeline that runs two independent inference stages in separate containers but must guarantee both hardware slices are available on the same node before the pod starts. With the device plugin model, one slice may be reserved while the second is unavailable. DRA solves this atomically.
MIG must be enabled and profiles configured on the node before this claim can be satisfied. For the MIG configuration and profile setup, see Multi-Instance GPU; verify the available slices as described in Verifying the MIG Layout.
-
Create a
ResourceClaimTemplatethat requests two MIG instances in a single claim: -
Create a pod with two containers, each consuming one MIG slice from the shared claim:
-
Verify that the pod is running:
Expected output:
-
Verify that both MIG instances were allocated atomically on the same node:
Expected output:
Both slices are on the same pool, confirming co-location.
If the node does not have two free MIG instances simultaneously,
the pod stays Pending:
Support Status
Multi-Instance GPU
MIG (Multi-Instance GPU) is an NVIDIA technology that slices a single physical GPU into multiple isolated GPU instances. Each instance has dedicated compute, memory, and cache, so different workloads can run securely without resource contention. In Kubernetes, the GPU Operator’s MIG Manager applies the desired layout on nodes, and GPU Feature Discovery advertises the resulting MIG resources for scheduling.
The MIG layout is configured in the same way for both allocation patterns — only the verification and the consumption in workloads differ.
Applying the MIG Configuration
Using a different MIG config is disruptive and causes pods on the node to be restarted or even removed.
You must select the correct MIG profile for the specific GPU model; for the list of profiles, refer to the NVIDIA Supported MIG Profiles reference.
With the default single MIG strategy,
the GPU is partitioned into instances of one identical profile
(for example all-1g.6gb).
To split a GPU into instances with different profiles
(for example all-balanced),
the MIG strategy of the NVIDIA GPU Operator must be changed to mixed first,
through a values override in the providerConfig of the NFD extension
in the Shoot manifest:
For MIG configurations with multiple different profiles,
such as all-balanced,
the mixed strategy must be configured in the NVIDIA GPU Operator
and the Shoot cluster reconciled
before the MIG configuration label is applied to the GPU node.
With the mixed strategy, the MIG instances are advertised
as individual nvidia.com/mig-<profile> resources
instead of a generic nvidia.com/gpu resource count.
For details about the mixed strategy, refer to the
Example: Mixed MIG Strategy
reference.
-
Set the MIG layout via a node label. This example uses the MIG config
all-1g.6gb, which partitions the physical GPU into four isolated MIG instances: -
Wait for the NVIDIA GPU Operator to reconcile.
The
nvidia-mig-managerDaemonSet reconfigures MIG on each node if applicable. -
Verify the new layout as described in Verifying the MIG Layout.
To disable MIG and revert to using the full physical GPU,
change the node label to all-disabled:
Verifying the MIG Layout
With the Device Plugin
The MIG instances appear in the allocatable resources of the node:
Expected output:
With DRA
The DRA driver publishes each MIG instance as a device
in the ResourceSlice objects:
Expected output:
Using MIG Slices in Workloads
- Requesting GPUs with Resource Requests —
with the device plugin and the default
singleMIG strategy, MIG instances are requested like full GPUs throughnvidia.com/gpuresource requests - Allocate a MIG Slice —
with DRA, MIG slices are requested
through the
mig.nvidia.comdevice class
If the MIG strategy is set to mixed —
required for MIG configurations with multiple different profiles,
such as all-balanced —
the MIG instances are requested as individual
nvidia.com/mig-<profile> resources instead.
For changing the MIG strategy,
refer to the values override example in the
OSC NFD Extension reference.
Advanced Configuration
Running the NVIDIA GPU Operator on tainted nodes
Nodes with GPUs can be reserved exclusively for GPU workloads by tainting them, so that regular workloads are not scheduled on these nodes.
Apply a NoSchedule taint to the GPU node. In this example, the taint gpu="true" is used:
From now on, only pods with a matching toleration are scheduled on this node.
The infrastructure resources of the NVIDIA GPU Operator
are deployed with the catch-all toleration operator: Exists,
which tolerates all taints with all effects —
the DaemonSets managed by the NVIDIA GPU Operator
are deployed on tainted GPU nodes automatically.
In previous versions, the toleration for the DaemonSets managed by the NVIDIA GPU Operator had to be configured manually through the NFD extension in the Shoot manifest. This configuration is no longer needed — no action is required from you to run the NVIDIA GPU Operator on tainted GPU nodes.
Tolerations configured manually under daemonsets.tolerations
in the nvgpu-operator values overwrite the default catch-all toleration.
An incorrect custom configuration can prevent the infrastructure pods
of the NVIDIA GPU Operator from being scheduled on the tainted node,
leaving the GPU stack on that node non-functional.
The taint affects all pods, not only the NVIDIA GPU Operator components. GPU workloads scheduled on the tainted node must also specify a matching toleration in their pod specification.
For details about the NFD extension configuration, refer to the OSC NFD Extension reference.
References
- NVIDIA GPU Operator Documentation
- NVIDIA GPU Operator with MIG
- NVIDIA DRA Driver for GPUs (v25.12.0 is deployed with GPU Operator v25.10.1)
- Kubernetes Dynamic Resource Allocation
- OSC NFD Extension documentation