.svg)
Key Takeaways
- Spark History Server provides valuable retrospective visibility into Spark application execution through persisted event logs.
- Its Spark-focused view can miss the Kubernetes and infrastructure signals needed to explain why a production failure occurred.
- Real-time observability connects Spark, Kubernetes, and infrastructure signals as incidents unfold, helping teams identify root causes faster.
- Production teams can benefit from using both, with SHS for historical Spark analysis and real-time observability for active incident detection and cross-platform troubleshooting.
At 2 a.m., a production Spark job disappears. The job fails, but there’s no useful error to explain why. You open the Spark History Server, only to find the execution history incomplete or missing. By then, the Kubernetes pod involved may already be gone, taking critical infrastructure context with it.
The Spark History Server can reconstruct application history from persisted event logs. But when a failure crosses Spark, Kubernetes, and infrastructure layers, that history may tell only part of the story. Production teams also need visibility into what’s happening while the evidence still exists.
What Does the Spark History Server Do?
The Spark History Server reconstructs the Spark UI from persisted event logs, giving you a detailed record of what Spark captured during a job. As the application runs, it writes these events to persistent storage, and the History Server later reads them back to rebuild that UI.
This makes it a strong tool for post-run analysis. Once a job finishes, you can review stages, tasks, executors, SQL execution, retries, and shuffle behavior to investigate failures or compare one run against another.
But that record only covers what happened inside Spark. The limitation becomes clear once you consider what the Spark UI can't tell you about the moment a job failed: everything that was happening outside Spark at the same time.
What Happens When the History Server Has Nothing to Show?
When the Spark History Server has nothing to show, the investigation starts with a blind spot: something failed, but there's no record explaining why.
The History Server can only replay the event history available to it, so a crash, misconfiguration, or killed pod that leaves those logs missing or incomplete takes that explanation away before the investigation even begins.
A production incident is rarely explained by one signal alone, and each source answers a different piece of the question:
- Spark event logs record what Spark captured about application execution.
- Driver and executor logs capture what the Spark processes reported at runtime.
- Spark metrics track application and resource behavior over time.
- Kubernetes events log pod scheduling, eviction, restarts, and OOMKills.
- Node and cloud signals reveal memory pressure, disk pressure, and Spot instance terminations.
The Spark History Server can show that an executor disappeared without showing why, and if Kubernetes ended its pod with an OOMKilled status, the Spark-side symptom is all it surfaces while the cause sits in evidence outside the Spark event history.
Why Are Spark Failures Harder to Diagnose on Kubernetes Than on a Traditional Cluster?
Spark failures are harder to diagnose on Kubernetes because the platform itself can kill, evict, or reschedule the pods running Spark, something a traditional standalone or YARN cluster never does to a running executor without warning.
On Kubernetes, Spark runs on ephemeral pods that can disappear quickly. A driver or executor may be OOMKilled, evicted, restarted, delayed by FailedScheduling, or affected by node pressure or a Spot interruption. Spark may record that the loss happened, but the evidence explaining why it happened lives at the Kubernetes or infrastructure layer, one step outside Spark's own view.
Consider a typical failure chain: an executor disappears, Spark records the loss, Kubernetes reports OOMKilled, and node metrics reveal the memory pressure behind it. Looking at Spark alone breaks that chain partway through, right where the real cause sits. It's the kind of disconnect that creates CloudWatch blind spots for Spark on EKS.
Root-cause investigation gets harder still when several Spark jobs share the same nodes or scheduler. Spark monitoring on Kubernetes needs to connect job failures with the infrastructure events causing them, so platform engineers can tell whether an incident is isolated to one application or spreading across the shared environment.
What Does Real-Time Observability Catch That the History Server Can't?
Real-time observability captures Spark and infrastructure signals while a job is still running. Instead of reconstructing an incident later, your team can see failures develop across the application, Kubernetes, and infrastructure layers.
Here’s what that difference looks like in practice:
This is the practical difference between Spark History Server vs. monitoring built for production. Spark real-time observability can flag an executor loss and connect it to the pod, node, or infrastructure event behind it while that evidence still exists.
That context matters most during incidents such as Spark jobs failing at 2 am, when engineers need to find the cause before the workload and its surrounding infrastructure disappear.
Finding that cause sooner can shorten the time a failure disrupts production. Across its customer environments, Acceldata reports 40% less pipeline downtime, according to its Autonomous 2026 event.
For platform teams, that means the value of real-time visibility extends beyond one failed Spark job to maintaining stability across the wider data environment.
Prashanth S., Director of Platform Engineering and Data Operations, describes the same operational outcome:
"Acceldata has been instrumental in enhancing our data platform management capabilities. Their solutions not only ensure platform stability but also drive significant cost savings."
Get the same platform stability that Prashanth's team gained. Explore Acceldata's self-guided product tour to see how real-time observability works on your own Spark and Kubernetes environment
Do You Need Both Spark History Server and Real-Time Observability?
Yes. Many production teams benefit from both Spark History Server and Real-Time Observability.
The Spark History Server remains useful for retrospective, Spark-native analysis. Real-time observability becomes important when you need to catch active incidents, connect them to infrastructure, and understand whether the problem extends beyond one job.
Use Spark History Server when you primarily need:
- Completed-job investigation
- Stage and task analysis
- Retry and shuffle analysis
- Historical job comparison
- Occasional debugging of stable workloads
Add real-time observability when you need:
- Alerts during execution
- Kubernetes pod and lifecycle context
- Node and cloud correlation
- Cross-job incident analysis
- Trend and baseline detection
- Shared-cluster visibility
- Faster root-cause investigation
A small team running a few predictable batch jobs may find SHS sufficient for much of its troubleshooting. The visibility challenge grows when those workloads become part of a broader data environment.
An Acceldata survey of 40 C-level leaders at Fortune 1000 and Global 2000 companies found that 75% run four or more data platforms in active use. In environments like these, investigating Spark jobs separately can make it harder to connect an application failure with issues affecting the wider infrastructure.
xLake addresses that fragmentation through a unified control plane across environments. It brings Spark and infrastructure context into a broader operational view, helping platform teams investigate active failures without relying on event history alone.
SHS and real-time observability therefore serve different parts of the same investigation. SHS preserves the Spark-native history you need after a run, while real-time observability adds the live context needed during production incidents. For teams evaluating that wider monitoring layer, this Spark observability tool comparison compares the available approaches.
Pairing the two is what keeps a gap between Spark-native history and live infrastructure context from turning into hours of manual reconstruction during an actual incident. The next executor that disappears at 2 am shouldn't have to wait for a support ticket to explain itself.
Book a demo with Acceldata and see xLake catch a Spark failure and its infrastructure cause in real time, on your own cluster.
FAQs: Spark History Server and Real-Time Observability
Can you configure Spark to write event logs more reliably?
Yes. Enable Spark event logging, write logs to durable shared storage, and configure appropriate permissions and event-log rolling to reduce the risk of incomplete or lost logs. However, these measures improve reliability rather than guarantee complete logs, as crashes, storage failures, or infrastructure issues can still interrupt event logging.
Does the History Server work the same way on Databricks as on open-source Spark?
No. Open-source Spark uses the Spark History Server to reconstruct application UIs from persisted event logs, while Databricks provides its own Spark UI and platform-managed monitoring experience with different access, retention, and troubleshooting workflows.
How long are Spark event logs typically retained?
Spark event logs have no universal retention period; retention depends on the Spark History Server configuration and the underlying storage lifecycle policy. By default, History Server log cleanup is disabled, but when enabled, spark.history.fs.cleaner.maxAge defaults to 7 days and can be adjusted to fit operational needs.
Is real-time Spark observability worth it for small, infrequent batch jobs?
Not always. For small, infrequent, and predictable batch jobs, the Spark History Server may provide enough visibility for investigating failures after they occur. Real-time observability becomes more valuable when jobs are business-critical, failures are difficult to reproduce, or infrastructure issues need to be detected while the job is running.
What's the actual storage cost of keeping detailed Spark event logs long term?
There is no fixed storage cost, as it depends on the average event-log size, number of applications, retention period, and storage pricing. High-volume or long-running workloads generate more data, while retention policies and event-log rolling can help limit long-term storage growth.
