.svg)
Key Takeaways
- A Spark job can finish successfully and still run twice as long as expected, with no failed task or exception pointing to why.
- Spark profiling connects code-level, stage-level, and cluster-level signals to isolate whether a slowdown comes from application logic, Spark execution, or the infrastructure underneath.
- Persistent shuffle spill, skewed task duration, and high garbage collection time are stronger signals than any single slow-looking metric taken alone.
- On Kubernetes, executor OOM events, node contention, and pod scheduling delays sit outside the Spark UI, so infrastructure telemetry has to be read alongside application metrics to find the real cause.
A Spark job can complete successfully and still take twice as long as it should. There's no failed task or exception to chase, just extra time hiding somewhere in the workload with nothing pointing to where.
Spark profiling is how you find it. By examining code-level metrics, stage and task metrics, and the underlying cluster together, you can isolate the actual bottleneck instead of tuning by trial and error.
This guide covers what to measure at each level, how to read the results, and how to turn them into fixes worth making.
What Does Profiling a Spark Job Involve?
Profiling a Spark job involves measuring where its execution time and resources are spent across stages and tasks, not just confirming that the job finished without errors.
A complete profile examines three levels:
- Code level: Identifies expensive functions and user-defined functions, along with the time or memory each one consumes.
- Stage level: Uses task duration, shuffle read and write, memory or disk spill, and garbage collection time to expose skew and other bottlenecks.
- Cluster level: Tracks executor memory, CPU usage, node contention, and, on Kubernetes, pod behavior.
Examining all three levels together separates a slow piece of code from a Spark execution problem or an infrastructure constraint that none of them would reveal alone.
What Should You Check First in the Spark UI?
Start with the SQL tab, not the stage list. It tells you which operators are burning execution time, and everything else follows from there.
The Spark UI is the fastest way to get a first read on why a job is slow, provided you know exactly where to look. The following checks, done in order, narrow the investigation before you inspect individual tasks:
- Open the SQL tab first and identify which operators consume most of the execution time.
- Follow the DAG from those operators to see which stages they produce and where shuffle boundaries occur.
- Compare stage durations across the job to find the few stages driving most of the runtime.
- Compare task duration within the slowest stage to spot skew between the median and longest-running tasks.
The Spark UI stops at this application-level view. Understanding what the Spark UI can't tell you about the infrastructure underneath is where the investigation goes next, especially on Kubernetes, where the real cause often sits outside Spark entirely.
What Does Shuffle Spill Tell You?
Shuffle spill tells you that Spark could not fit a shuffle's intermediate data into execution memory and wrote it to disk instead, a sign of undersized executor memory or partitions that are too large or skewed.
How that spill is distributed across a stage's tasks determines whether skew or plain undersizing is the cause, as the table below shows.
Once you find an abnormal metric, connect it to the likely cause before changing configurations. Use the profile to narrow the investigation:
How Do You Profile UDF and Code-Level Performance?
You profile UDF and code-level performance by measuring execution time, call frequency, memory use, and hot paths inside individual functions, using Spark's built-in Python profiling tools to pinpoint slowdowns that stage metrics alone cannot explain.
Stage metrics show where a Spark job slows down, but not always which function is responsible. Code-level profiling fills that gap by measuring execution inside individual functions, particularly Python user-defined functions (UDFs) that run outside Spark's JVM.
For PySpark UDF profiling, focus on:
- Function execution time: Find UDFs or function calls consuming a disproportionate share of runtime.
- Call frequency: Spot functions being invoked more often than expected.
- Memory usage: Track allocations and identify lines where memory consumption increases significantly.
- Hot paths: Narrow a slow transformation down to the specific code responsible.
Spark's built-in Python profiling capabilities can capture performance and memory data for Python UDFs, including Pandas and Arrow UDFs, helping narrow a stage-level slowdown to the code responsible.
Profiling adds instrumentation overhead, so use it selectively, after stage-level metrics point toward Python execution or a particular transformation. Once you find the bottleneck, check whether the UDF can be simplified, vectorized, or replaced with a Spark-native operation.
Rerun the workload afterward and compare the stage profile. If the code gets faster but the stage does not, the bottleneck likely sits elsewhere: shuffle, skew, memory pressure, or infrastructure.
What Cluster-Level Signals Does the Spark UI Miss?
The Spark UI can show executor metrics and failures, but it does not provide the full infrastructure context behind them. On Kubernetes, a slow or unstable Spark job may originate at the pod or node level, so the following signals deserve particular attention:
- Executor OOM events: An executor disappearing from the application does not tell you whether the container exceeded its Kubernetes memory limit. Check pod termination reasons and memory usage to distinguish an out-of-memory (OOM) kill from a Spark-level failure.
- Node resource contention: Executors can receive less CPU or memory bandwidth than expected when other workloads compete for resources on the same node, the kind of contention that optimizing clusters through intelligent overcommitment is designed to prevent. Compare Spark performance with node-level CPU, memory, and resource-pressure metrics.
- Pod scheduling delays: Executors requested by Spark may spend time waiting for Kubernetes to place their pods. Resource shortages, scheduling constraints, or unavailable nodes can delay capacity from reaching the application even when Spark is ready to use it.
These signals help explain cases where the Spark profile shows the symptom but not its infrastructure-level cause. This is why Kubernetes-native Spark profiling needs application and Kubernetes telemetry together.
These infrastructure signals can also help trace performance problems back to four common Spark issues: data skew, executor misconfiguration, join/shuffle, and memory issues.
How Do You Turn a Profile Into an Actual Fix?
Once you find an abnormal metric, connect it to the likely cause before changing configurations. Use the profile to narrow the investigation:
Change one variable at a time, then profile the next run against the same workload. This makes it easier to tell whether the change fixed the bottleneck or moved it elsewhere.
For production workloads, profiling should also be repeated over time. Comparing the same signals across runs helps catch regressions as data volumes, code, and infrastructure change.
Two rows in that table, OOMKilled executors and delayed executor starts, point to causes that neither the Spark UI nor a Python profiler can confirm on their own. Both live at the infrastructure layer, so tracking either one down means stepping outside Spark's own tooling to check pod events, node capacity, and container memory limits separately.
That's typically why teams end up stitching an incident together across the four-tool Spark monitoring stack: Spark History Server for the job-level view, kubectl and CloudWatch for the pod-level view, and a custom dashboard to line the two up.
Acceldata's xLake removes that manual stitching by bringing Spark job and stage metrics into the same view as the underlying Kubernetes pod signals, so an OOMKilled executor or a delayed pod scheduling decision shows up right next to the stage it affected instead of requiring a separate lookup. See how xLake correlates Spark job, stage, and infrastructure signals to diagnose performance issues faster.
Book a demo with Acceldata to learn more.
FAQs: Apache Spark Profiling
Does profiling slow down a Spark job while it’s running?
Yes, profiling can slow down a Spark job because collecting detailed execution metrics adds some runtime overhead. The impact is usually small for lightweight profiling but can become noticeable with high-frequency or detailed instrumentation.
What’s the difference between profiling and monitoring?
Profiling examines why a Spark job behaves a certain way, such as identifying slow stages, tasks, or resource bottlenecks. Monitoring tracks what is happening over time, such as job health, resource usage, failures, and performance trends.
Can you profile a Spark job after it has already finished?
Yes, a completed Spark job can be analyzed afterward if its event logs or other execution data were retained. These logs can be reviewed with tools such as the Spark History Server to identify performance bottlenecks, slow stages, and resource usage.
How often should a production Spark job be profiled?
Production Spark jobs generally don’t need continuous profiling. Profiling is most useful after significant code or configuration changes, when performance degrades, or periodically for high-impact jobs to identify emerging bottlenecks.
Do streaming Spark jobs get profiled differently than batch jobs?
Yes. Streaming Spark jobs are profiled with additional focus on continuous performance, such as processing rates, batch duration, input rates, and scheduling delays, using Structured Streaming monitoring metrics. Batch jobs, by comparison, are typically analyzed around individual job runs, stages, tasks, and resource usage.
