.svg)
Key Takeaways
- Spark on GPU needs the NVIDIA device plugin or GPU Operator, plus explicit GPU requests in both Kubernetes and Spark configuration.
- Kubernetes schedules GPUs as exclusive resources by default, so sharing one across executors takes time-slicing or MIG.
- GPU nodes left running between jobs are a direct cost problem, and elastic node pools that scale to zero solve it.
- The Spark UI cannot see Kubernetes-layer failures such as OOMKilled pods or scheduling delays, which is where most troubleshooting time is lost.
- Full-stack visibility across Spark, GPU scheduling, and Kubernetes ties each failure to one job view.
Your team installed the RAPIDS plugin and configured it two hours ago, yet Spark on GPU is still idle because no executor has landed on a GPU node. The cause could sit in device plugins, taints, GPU scheduling, or Spark settings, and each layer fails differently and leaves its own trail, much of it outside the Spark UI's view. Without a map, root cause investigation becomes a slow, expensive hunt.
What Does it Mean to Run Spark on GPU With Kubernetes?
Running Spark on GPU with Kubernetes means Kubernetes schedules Spark executor pods onto nodes with GPUs attached, while the RAPIDS Accelerator plugin runs the GPU-eligible parts of each job on those GPUs. Kubernetes acts purely as the orchestrator, and the plugin is the acceleration layer.
Most teams arrive at this setup after moving Spark from YARN to Kubernetes, since Kubernetes-based Spark has become the default for teams that need per-job dependency isolation and elastic infrastructure. Teams still planning that move should validate a container and scheduling baseline before adding GPUs.
What Do You Need Before Setting Up Spark on GPU on Kubernetes?
To set up Spark on GPU on Kubernetes, you need four things: GPU nodes, NVIDIA drivers and CUDA, a device plugin, and the RAPIDS Accelerator JAR. Each supports a different layer, from hardware to Kubernetes scheduling to Spark GPU execution, so keep this checklist ready:
- GPU-enabled Kubernetes cluster: Provision worker nodes with physical NVIDIA GPUs to supply capacity for Spark executor pods.
- NVIDIA drivers and CUDA: Install versions compatible with the RAPIDS Accelerator build on every GPU node, since a mismatch causes executor startup failures.
- NVIDIA device plugin or GPU Operator: Deploy it to advertise each GPU as the nvidia.com/gpu resource, choosing the GPU Operator to automate driver and container toolkit installs.
- RAPIDS Accelerator plugin JAR: Bake it into the image or mount it at runtime so Spark's classpath can find it.
GPU node pools are usually separate from CPU node pools and tainted with NoSchedule so only GPU-requesting pods land there, and the device plugin configuration depends on that setup, so set taints and tolerations first. A missing component can leave executor pods stuck in Pending or cause startup failures.
How Do You Install the NVIDIA Device Plugin So Kubernetes Can See Your GPUs?
To make GPUs visible to Kubernetes, run the NVIDIA device plugin as a DaemonSet on each GPU node, most commonly by deploying the NVIDIA GPU Operator, which advertises each GPU to the scheduler as an allocatable resource. Four steps take the cluster from bare GPU nodes to schedulable GPU capacity.
Step 1: Label the GPU nodes
Each GPU node needs a label that marks it as carrying NVIDIA hardware. The GPU Operator applies nvidia.com/gpu.present=true automatically, while a standalone plugin install needs a custom label so a node selector can target those nodes.
Step 2: Deploy the device plugin or GPU Operator with Helm
Helm packages everything that exposes GPUs to Kubernetes into one release, so add the NVIDIA chart repo and install the GPU Operator into its own namespace:
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update
helm install --wait --generate-name -n gpu-operator --create-namespace nvidia/gpu-operatorThe operator manages drivers, the container toolkit, and the device plugin together. Teams that already have drivers installed can deploy the standalone device plugin chart instead.
Step 3: Apply taints and tolerations
A NoSchedule taint on each GPU node keeps general workloads off costly hardware, and pods that request GPUs then need a matching toleration in their spec. The GPU Operator's own DaemonSets already tolerate the nvidia.com/gpu taint, so they still run on those nodes.
Step 4: Verify the GPU shows up as allocatable
Run kubectl describe node <node-name> and look for nvidia.com/gpu under both Capacity and Allocatable, where the count should match the physical GPUs on that node. A missing entry points to a failed driver or device plugin pod, so check pod status in the gpu-operator namespace.
Why Does Kubernetes Schedule GPUs as Exclusive Resources, and How Do You Share One Across Spark Executors?
Kubernetes schedules GPUs as exclusive resources because the device plugin exposes each one as a whole, indivisible unit, so a pod that requests one GPU gets sole access to it while CPU and memory can be divided fractionally.
Sharing one GPU across Spark executors therefore means changing how Kubernetes advertises it, since each executor still requests one GPU through spark.executor.resource.gpu.amount and a second executor on an unshared card waits in Pending.
NVIDIA's GPU Operator documentation confirms exclusive access by default and describes two ways to change it:
How Do You Configure Spark Itself to Use the GPUs Kubernetes Has Scheduled?
Configuring Spark to use the GPUs Kubernetes has scheduled means loading the RAPIDS Accelerator plugin and explicitly enabling GPU execution, because without those settings the executor sits on a GPU node and never touches the GPU.
Verify these settings against the current NVIDIA documentation:
Unsupported operations fall back to CPU execution automatically, so the job still completes, and the physical plan shows whether the work reached the GPU.
How Do You Keep GPU Nodes From Sitting Idle Between Spark Jobs on Kubernetes?
To keep GPU nodes from sitting idle between Spark jobs on Kubernetes, use elastic node pools that scale GPU capacity to zero when idle and back up when a job is submitted, because GPU instances bill whether or not a job runs on them.
Cluster Autoscaler and Karpenter react to the same signal: an executor pod stuck waiting because no GPU is free, and scale-up is not instant since a new node must boot and load drivers, so set idle timers long enough to avoid churn between back-to-back jobs.
Cluster Autoscaler
Set the GPU node group's minimum size to zero. It adds a node when executor pods go Pending and removes it after a configured idle period.
Karpenter
A NodePool consolidation policy removes empty GPU nodes. When executors go Pending, it launches a node sized to the GPU request, which suits spiky Spark workloads.
On managed Spark platforms, idle hours also feed the GPU Spark cost problem that managed platforms created, because a service fee sits on top of already expensive GPU infrastructure.
At scale, a global telecommunications leader cut compute cores by 85% and processing time by 2.5x after Acceldata rebuilt how compute is scheduled and allocated.
How Do You Monitor Spark on GPU Jobs Once They're Running on Kubernetes?
To monitor Spark on GPU jobs on Kubernetes, combine Spark-level metrics (stages, tasks, and executor behavior) with Kubernetes-level metrics (pod status, node GPU allocation, and scheduling events) in one view, because neither alone explains why a GPU job runs slowly or stalls.
Monitoring Spark on Kubernetes starts with the Spark UI, whose SQL tab reveals whether operators ran on the GPU, and the Kubernetes Dashboard, with kubectl describe node showing GPU allocation. Both fall short together: the Spark UI disappears when the driver pod exits unless a history server keeps event logs, and the Dashboard cannot show GPU utilization, which teams add through NVIDIA's DCGM exporter with Prometheus.
Pod logs fill part of the gap, and building a scalable Kubernetes logging platform shows how stdout and stderr streams from every pod can be collected without one noisy service affecting the others.
Why Isn't the Spark UI Enough to See What's Happening to Your GPU Pods on Kubernetes?
The Spark UI is not enough to see what is happening to GPU pods on Kubernetes because it reports only on what happens inside one Spark application, while pod scheduling delays, node-level GPU pressure, and eviction signals occur outside its visibility and often shape a GPU job's runtime before any Spark metric changes.
A Spark job can look slow with no obvious cause while an executor pod waits for a GPU node to scale up.
Three blind spots explain why the Spark UI is not enough for Kubernetes-native profiling:
- Pod scheduling delays: Executors appear in Spark only after they register with the driver, so time spent waiting for a free GPU or a new node never shows up.
- Node-level GPU pressure: Neighboring pods on a shared or time-sliced card slow your tasks, and only node GPU metrics reveal the contention behind the longer task times Spark shows.
- Eviction signals: Kubernetes can evict an executor pod from a pressured node, and Spark logs only a lost executor while the cause sits in Kubernetes events.
Pod events are the first stop: check them for unscheduled executors and evictions, then check node GPU metrics for contention.
Why Are OOMKilled Spark Executors on GPU Nodes Especially Hard to Diagnose?
OOMKilled Spark executors on GPU nodes are especially hard to diagnose because Kubernetes terminates them before Spark can log a reason, so the Spark UI shows only that the executor disappeared while the reason code and timing sit in pod events and kubelet logs.
The reasons why OOMKilled Spark executors on Kubernetes are hard to diagnose multiply on GPU nodes, where executors draw on both host memory and GPU memory, and running out of either looks like a lost executor in the Spark UI.
Host memory exhaustion produces an OOMKilled status, with pinned and off-heap RAPIDS buffers counting toward it, while GPU memory exhaustion usually appears as a CUDA out-of-memory error that DCGM memory usage helps confirm. Check the pod's last terminated state for exit code 137, then compare host and GPU memory at that timestamp.
How Do You Get Full-Stack Visibility Across Spark, GPU Scheduling, and Kubernetes in One Place?
Getting full-stack visibility across Spark, GPU scheduling, and Kubernetes in one place takes a unified orchestration layer that pieces together Spark UI output, pod events, and GPU metrics at the job level, so nobody rebuilds the story by hand from separate tools after every failure.
See how pipeline orchestration on Kubernetes with full visibility works in Acceldata's xLake platform, which offers:
- Per-job runtime observability: Runtime behavior is tracked at the Kubernetes namespace level, so executor losses, pod states, and scheduling delays line up under one job view.
- Coverage across workloads: Spark, Trino, JupyterHub, and Python workloads share one orchestration layer.
- Deployment on your own cluster: xLake deploys directly onto your EKS, AKS, GKE, or on-premises Kubernetes clusters.
Running Spark on GPU on Kubernetes Without the Guesswork with Acceldata
Getting Spark on GPU working on Kubernetes is a multi-layer setup in which device plugins, GPU scheduling, Spark configuration, and cost control all have to be right at once, and a failure can live in any one of them.
Reducing the guesswork comes down to:
- Treating GPU node pools as elastic by default instead of always-on.
- Deciding between time-slicing and MIG before jobs start competing for GPUs.
- Instrumenting Kubernetes and Spark together instead of checking each in isolation after something fails.
Acceldata's pipeline orchestration on Kubernetes with full visibility runs natively on your own Kubernetes cluster and gives per-job visibility across Spark and other workloads in one place.
Book a demo and see per-job visibility on your own infrastructure, without stitching three tools together to find one root cause.
Frequently Asked Questions
Can Spark on Kubernetes use a GPU without the RAPIDS Accelerator plugin?
Yes. Spark on Kubernetes can use GPUs without the RAPIDS Accelerator plugin by configuring Kubernetes GPU resources and Spark’s GPU scheduling and resource settings. RAPIDS is needed specifically to accelerate supported Spark operations on GPUs, not to make GPU access possible.
Does GPU time-slicing on Kubernetes affect the correctness of a Spark job's output?
No. GPU time-slicing changes how GPU resources are shared between workloads, but it shouldn't change Spark's output as long as the applications and GPU operations are deterministic and correctly configured; the main impact is typically on performance and execution time.
What happens to a running Spark job if its Kubernetes GPU node gets reclaimed or preempted?
If the GPU node running a Spark executor is reclaimed or preempted, that executor is lost, and Spark can generally reschedule its tasks on another available executor. The job may slow down or fail if the lost executor's work cannot be recovered or the cluster lacks sufficient GPU capacity to replace it.
Do GPU and CPU Spark executors need separate Kubernetes node pools?
No. GPU and CPU Spark executors don't inherently require separate Kubernetes node pools, but separate pools are often used to isolate workloads, simplify scheduling, and avoid wasting expensive GPU capacity on CPU-only tasks.
Is troubleshooting a GPU-accelerated Spark job on Kubernetes different from troubleshooting a CPU-only one?
Yes. GPU-accelerated Spark jobs add another layer of failure modes involving GPU allocation, device health, memory usage, CUDA or driver compatibility, and GPU scheduling, alongside the usual Spark and Kubernetes issues.
