.svg)
Key Takeaways
- Runtime data quality checks validate data while it moves through a stream, before any consumer reads it.
- Batch checks run after data lands, when bad records have already reached dashboards and models.
- Runtime checks work in three layers: schema enforcement at ingestion, rule validation in a stream processor, and monitoring metrics.
- Failed records belong in a dead-letter queue, and alerts should fire on a failure rate over a short window instead of on each record.
- Acceldata’s data pipeline agent and xReasoning correlate latency, quality, and cost signals across a stream to support automated resolution.
The nightly report comes back clean, yet the finance team has already spent two days working from wrong numbers, because real-time data quality checks never ran on the stream.
Every record passed the rules, but only 11 hours after downstream systems had read them. Runtime data quality checks close that gap by validating records while they move, since a stream does not wait for a validation window, and they catch a malformed event before it is consumed, joined, and written into a report someone acts on.
Why do Static Data Quality Checks Fail on Streaming Data?
Static checks fail on streaming data because they are built to run after data has already landed, while a stream keeps moving, so by the time a fixed rule flags a problem, the bad record has already been consumed downstream.
A batch check assumes a boundary: the load finishes, the table settles, and validation runs against a fixed set of rows. A stream never reaches that boundary, so a check waiting for one cannot act while action still matters.
That timing problem shows up as five failure modes:
- Detection lags impact: A record flagged at 2 a.m. was read by a dozen consumers at 11 p.m., so the alert describes damage instead of preventing it.
- Fixed thresholds break under traffic spikes: A rule set at 10,000 events an hour fires continuously during a promotion and stays silent during an outage.
- Schema drift goes unnoticed: A batch check validates the columns it already knows about, so a producer’s new field or changed type passes because nothing watches for the change itself.
- No lineage awareness: The check reports that a table failed a rule without naming the dashboards, models, or payment flows that read it, which is the information urgency depends on.
- Alert fatigue from rule proliferation: Teams answer each incident by adding a rule, and hundreds of rules later the signal drowns in notifications nobody reads closely.
None of these failures is fixed by writing stricter rules, which is why static data quality checks break in high-velocity systems.
Why Does Traditional Data Governance Break Down in Real-Time Pipelines?
Traditional data governance breaks down in real-time pipelines because it was built around periodic review and manual approval, and retrospective audits, change-control meetings, and centralized gatekeeping all assume there is time between an action and its consequences. A pipeline that never stops moving removes that time.
Retrospective review finds violations after the damage
On a stream, the gap between a bad record arriving and a decision made on it can be seconds, while a quarterly review can only document what already happened. Two historical cases show what that delay costs when a system keeps acting.
The SEC’s 2013 order against Knight Capital describes a loss of more than $460 million in the first 45 minutes of trading on August 1, 2012, traced in part to code Knight moved within its order router in 2005.
Zillow’s Q3 2021 results reported a $304 million inventory write-down as it wound down its home-buying unit, with its CEO citing home-price unpredictability that far exceeded expectations.
Both are history, not current benchmarks, and both show what happens when an automated system keeps acting with no runtime control checking it.
Manual approval creates a bottleneck teams route around
Approval gates work when deployments are infrequent, yet schema changes, new topics, and new consumers arrive continuously on an active streaming platform, so a queue that adds days to each becomes the slowest part of the pipeline.
Engineers then find legitimate ways around it, such as a new topic outside the policy’s scope or a transformation moved into a consumer where no reviewer looks, so governance covers the reviewed paths and misses the rest.
Centralized gatekeeping does not scale across distributed streams
One team cannot hold context for hundreds of topics owned by dozens of producers, and a central reviewer can check structure without checking meaning because the people who know whether a value is plausible are the ones producing it.
Reviewing every change across a large estate turns that body into either a rubber stamp or the platform’s main constraint, which is why traditional data governance breaks in real-time pipelines and why real-time pipelines need runtime enforcement inside the stream itself.
Three Technical Layers of a Runtime Data Quality Check
A runtime data quality check typically runs across three layers: schema enforcement that rejects malformed events at ingestion, business rule validation in a stream processor that checks logic such as valid account IDs or acceptable ranges, and monitoring integration that exports freshness and error-rate metrics so the whole stream stays visible.
Confluent’s technical guide to scaling data quality in streaming architectures sets out this layering, and the table shows what each layer catches.
Schema enforcement is cheap and absolute, but it has no opinion on whether a correctly typed number makes sense: a transaction amount of $4 billion satisfies every structural requirement, so catching it takes the rule layer.
The rule layer evaluates one event at a time, so a topic that normally carries 50,000 events an hour and has carried 400 for two hours raises no alert, since every one of those 400 events passed.
Only exported metrics reveal that pattern, which is why teams that skip the third layer miss a stream that has stopped producing records.
How Do You Handle Records That Fail a Runtime Data Quality Check?
A failed record should be routed to a dead-letter queue instead of being dropped, because dropping it destroys the evidence needed to find the cause. Quarantining it keeps production clean and preserves the record for investigation.
Alerting follows a different rule, because firing on every failed record recreates the alert fatigue runtime checks are meant to solve, so the alert should fire when failures cross a threshold within a defined window.
Confluent’s guide offers an example of more than 2% of events failing validation inside five minutes, though the right threshold belongs to your traffic pattern.
What Grab’s production system looks like
Grab’s platform-level data quality system for Kafka is organized around data contracts that hold the schema definition, the semantic rules the data must comply with, and the ownership details that route an alert to whoever can fix it.
Its Coban platform distinguishes two error types, and the distinction maps onto the layering above:
- Syntactic issues are schema mismatches between producers and consumers that cause deserialization errors, which a registry catches.
- Semantic issues are disagreements between producer and consumer about the expected pattern or meaning of a field, so the record deserializes correctly and means the wrong thing.
Semantic rules are the harder half to write because each one needs field-level knowledge of what a value represents, so Grab used LLM-based recommendations to predict semantic test rules from a stream’s schema and anonymized sample data.
Human-readable rules then compile into an inverse SQL query that captures the records violating a rule, and a test runner executes the tests as FlinkSQL jobs. Violating records go to a dedicated Kafka topic and an S3 bucket, giving both a live stream of failures and a durable record for investigation, and Grab reports the platform covering more than 100 critical Kafka topics.
Together, the dead-letter path, the threshold alert, and the contract ownership turn a failed record into a routed, fixable event.
How Do You Monitor Runtime Data Quality Checks at Scale Across Pipelines?
Monitoring runtime checks at scale means replacing individual pipeline dashboards with a centralized view that correlates quality signals against latency and cost across every stream, since a team running dozens of pipelines cannot watch each one and the per-pipeline view hides the relationships that explain an incident.
A consumer lag spike, a drop in record volume, and a rise in schema failures can be three views of one event, such as a changed payload deployed an hour ago, yet three dashboards owned by three teams turn them into three investigations.
Cost belongs in the same view too, because a stream processor retrying failed records consumes compute, so a quality problem shows up on the bill before anyone files an incident. Treating quality, latency, and cost as one signal set is what separates proactive data quality monitoring across pipelines from three teams comparing notes afterward.
See how Acceldata’s Data Pipeline Agent monitors the producer, broker, and consumer layers of a stream for data quality and pipeline behavior anomalies, while xReasoning correlates those alerts across latency, quality, and cost into a single view that supports automated resolution.
Specialist agents for quality, pipeline, drift, and freshness run in parallel under the architecture and orchestration of agentic data management, which lets them pinpoint an issue such as partition skew and alert the right team.
Detection time decides whether a runtime check helps. A global pharmaceutical leader running Acceldata across more than 15,000 pipelines and 40,000 data assets achieved an 85% year-over-year improvement in mean time to detection, which shows what faster detection looks like at that scale.
How Does Kafka Streams Fit Into a Runtime Data Quality Architecture?
Kafka Streams provides the processing foundation for a runtime data quality architecture, supporting windowing, partitioning, and stateful processing. However, it does not perform data quality validation itself, so a separate quality layer must be built or purchased and integrated with it.
Conflating the two is a common and understandable error, because Kafka Streams can express a quality check clearly, which makes it feel as though adopting the framework delivers the check. What it delivers is the place to put one.
The split looks like this:
- What the substrate gives you: Ordered, partitioned, replayable event delivery, windowed aggregation, local state stores, and exactly-once processing semantics are the primitives a check needs.
- What you still have to build: The schema contract and its enforcement point, the rules that define a valid value, the dead-letter path, the alert thresholds, and the metrics export all remain your job.
Teams that read those guarantees as quality guarantees end up with pipelines that are reliable and still wrong, because a delivery guarantee describes how many times a record arrives and says nothing about whether it was right.
Teams get the most from the framework by harnessing Kafka Streams for real-time processing for what it does well and layering validation on top.
Making Runtime Data Quality Checks Standard Practice with Acceldata
A check that runs after data has already moved through the pipeline works as a postmortem instead of a safeguard. Closing that gap means building validation into the stream instead of bolting it on afterward, and it depends on:
- Enforcing schema at ingestion instead of downstream, so a malformed event is rejected before any consumer can read it.
- Routing failures to a dead-letter queue instead of dropping them, and triggering alerts on a failure rate measured across a short window.
- Monitoring quality signals centrally across every stream, so a quality alert, a latency spike, and a cost increase read as one event instead of three.
Acceldata’s Data Pipeline Agent watches the producer, broker, and consumer layers of a stream, and xReasoning correlates what they report into one view that supports automated resolution.
Book a demo to see how Acceldata catches bad data while it is still moving.
Frequently Asked Questions
Does adding runtime data quality checks slow down a streaming pipeline?
Adding runtime data quality checks adds latency in proportion to their complexity, since a schema validation costs far less than a semantic rule that needs an external lookup. Teams that cannot absorb it run the expensive checks asynchronously on a parallel consumer or sample a share of traffic.
Can runtime data quality checks work with streaming tools other than Kafka?
Yes, runtime data quality checks can work with streaming tools other than Kafka, because the three-layer pattern only needs a schema contract, a stream processor, and a metrics sink. Amazon Kinesis pairs with AWS Glue Schema Registry and Flink, and Pub/Sub supports topic-level schema enforcement with validation in Dataflow.
What happens to data that fails a check but is needed immediately downstream?
Data that fails a check but is needed immediately downstream is usually handled through graceful degradation, where the record passes through with the bad field replaced by a default or null and tagged as substituted. The tag matters more than the substitution, because an untagged default silently becomes a real value in every report that reads it.
How do you test new runtime data quality rules before deploying them to production?
New runtime data quality rules are best tested in shadow mode against live traffic, where the rule evaluates every event and records what it would have flagged without blocking anything. A few days of shadow output reveals the real false-positive rate, which sample records cannot.
Do runtime data quality checks replace batch validation entirely?
No, runtime data quality checks do not replace batch validation entirely, because mature setups keep both to catch different problems. Runtime checks catch bad records as they arrive, while batch validation catches issues that only appear across a complete set, such as a source-to-target total mismatch.
