.svg)
Enterprise data teams routinely spend several hours or days on root-cause investigation when a pipeline fails. The bottleneck is tool switching. The engineer opens Airflow, checks the DAG run, moves to Databricks or Glue to inspect the processing job, queries the catalog to find the affected table, then checks S3 to confirm what landed. Four systems, manual correlation, no connected path from trigger to impact.
That time gap is where data teams lose trust and agents lose accuracy. An autonomous agent consuming a downstream dataset cannot spend minutes verifying provenance. It acts on what it receives, across a hybrid estate spanning cloud and on-prem. If the observability layer cannot connect the full chain, the agent cannot assess what it is consuming. And if the reliability score that gates agent consumption shows a different number depending on which page it is read from, the trust signal breaks before the investigation starts.
Platform-native lineage tools see what happens inside one platform. Log-based observability tools reconstruct history from query logs. Acceldata ADOC v26.9.0 connects all four layers of the stack into one observable, governable surface with active enforcement at each node.
Key highlights
- Cross-layer lineage stitching connects orchestration, processing, catalog, and storage into one graph. When a pipeline fails, the downstream impact through every job, catalog table, and storage location is traceable in a single view.
- AWS Step Functions and Lambda pipelines are now observable in ADOC alongside Airflow, Databricks, and Glue. Teams running serverless orchestration on AWS get the same run history, workflow navigation, and lineage coverage as those running Airflow.
- Reliability scores are computed from one formula and displayed consistently across every surface, including Discover Assets, Asset Overview, Policies, Reliability Explorer, and Data Products.
01 One graph from orchestration to storage
Every feature in this section serves the same outcome: when a pipeline fails or data quality degrades, the full impact is traceable in one view across every technology layer.
Cross-layer pipeline and data lineage stitching
When an Airflow DAG invokes a downstream processing job, the actual input/output lineage lives inside the processing engine. The DAG task has no visibility into what Spark or Glue read or wrote. ADOC previously discovered each layer independently, producing four disconnected lineage segments for what is one data flow.
This release uses metadata relationships already present in underlying catalogs (Glue Catalog, Unity Catalog, Trino Catalog, Redshift External Catalog) to resolve processing-layer assets to catalog entries and storage locations automatically. The connected path runs from orchestration trigger to processing job to catalog-registered assets to storage. The Pipeline Details lineage graph renders this as a multi-level hierarchy with triggered pipelines shown as expandable nodes, so you can drill into upstream and downstream assets without leaving the top-level view. Computed on every run.
For agents consuming downstream datasets, this is the provenance chain they need to assess trustworthiness. For regulated industries, it is a complete audit record from source to consumption, maintained automatically. For Trino environments, the active crawling and profiling approach now extends into the same stitched graph.
Orchestration: Apache Airflow, AWS Step Functions. Processing: Databricks Spark, Trino, AWS Glue, dbt Cloud. Metadata catalogs: AWS Glue Catalog, Unity Catalog, Trino Catalog, Redshift External Catalog. Storage: Amazon S3 and catalog-registered tables.
Available now. No additional configuration beyond existing OpenLineage integrations. → Pipeline Run Details →
AWS Step Functions pipeline integration
ADOC supports Step Functions as a pipeline source. States within a Step Function appear as job nodes on the pipeline canvas, with nested visualization for Map and Parallel states. When one Step Function triggers another, you link the two executions into a single connected view by passing parent run context, the same pattern used for Airflow-to-Spark and Airflow-to-dbt lineage. For enterprises running Airflow for batch and Step Functions for event-driven workflows, both are now traceable in one system. Standard Step Functions only. Express Step Functions are not supported.
Available now. → AWS Step Functions →
Namespace-based filtering for Spark OpenLineage
The Pipelines page now filters by Spark namespace. Pipeline list, summary stats, and charts scope to selected namespaces in one click. Large Spark environments that emit events across multiple clusters, environments, or teams get a focused view instead of one aggregated surface. Brings Spark into parity with Airflow and other OpenLineage sources.
Available now. → Filtering and Searching Pipelines →
dbt Cloud artifact extraction control
When ADOC's dbt Cloud connector already collects model, test, and dataset lineage, Airflow's built-in extraction can produce duplicate or conflicting entries. A new setting in the acceldata-openlineage package (v1.1.0) lets you skip Airflow's built-in extraction when the ADOC connector handles it. Defaults to true, preserving existing behavior. No change required unless you want to disable built-in extraction.
Available now. Requires acceldata-openlineage v1.1.0. → Link Airflow Tasks to dbt Cloud Pipeline Runs →
Configurable delay for pipeline automations
Profiling, Data Quality, and Reconciliation automations now accept a configurable delay (in minutes) before running after their trigger completes. When multiple automations fire simultaneously on a shared Data Plane, this staggers execution and reduces resource contention.
Available now. → Pipeline Run Details | Automation →
02 One score you can trust
A connected lineage graph solves for visibility. When agents and humans check a reliability score before acting on data, that score needs to be correct, computed consistently, and available from any surface. These features make the trust signal reliable.
Consistent reliability scores across ADOC
Before: Asset Overview, Policies tab, Discover Assets, Reliability Explorer, and Data Products computed reliability scores independently using different aggregation windows and methods. The same asset showed different scores on different pages.
Now: all surfaces use one formula. Scores reflect each rule's most recent execution result, aggregated according to the methodology selected under Settings: Policy Based Simple Averaging, Rule Based Simple Averaging, or Rule Based Row Weighted Averaging. The score delta reflects the change from the previous execution and is no longer scoped to a fixed time window. When an agent checks whether a dataset is reliable enough to consume, it reads one number. That number is now correct everywhere.
Available now. → Score Aggregation Methodology →
Absolute Change and Percentage Change for reconciliation
Before: Data Equality reconciliation required exact match between source and target values. Legitimate minor deviations from rounding, currency conversion, or timing differences triggered false mismatches.
Now: two new operators accept tolerance. Absolute Change accepts a fixed deviation from the source value. Percentage Change accepts a percentage of the source value. Both support symmetric, one-sided, or asymmetric tolerance using independent Lower Bound and Upper Bound settings. For teams reconciling financial data across systems where minor variance is expected, this removes the noise without loosening the policy.
Available now. → Threshold-based matching →
03 Governance and access control across the hybrid estate
Cross-layer visibility and trustworthy scores need a governance layer that applies consistently across clouds, engines, and on-premises systems. In hybrid estates, where different teams own different portions of the pipeline and data landscape, governance that fragments along technology boundaries does not hold. These features enforce standards, control access, and keep infrastructure resilient.
Resource-Based Access Management (RBAM) for Pipelines
Before: pipeline access was all-or-nothing. No way to scope visibility by domain, team, or data source.
Now: admins group pipelines into Pipeline Resource Groups by data source, individual selection, or regex pattern matching on pipeline names. Domain Roles grant view or modify access to specific teams via new Pipeline Management permissions. New pipelines automatically join any resource group whose pattern matches. Every tenant retains a default group covering all pipelines, so existing access is unaffected until an admin narrows it.
Available now. → Pipeline Groups →
Data Product governance: Custom Fields and Tags
Administrators define tenant-level custom fields on Data Products (Text, Number, Date, DateTime, Email, Checkbox, Pick List, Multi-Select), mark them required, set defaults, and surface them in the Marketplace or as filters. Organizations that need consistent classification standards or business metadata (owner, cost center, SLA tier) across every Data Product now have the enforcement mechanism for it. Extends Data Products from ADOC 26.5.0.
Data Products also join ADOC's unified tagging framework, the same model already used for assets, policies, and catalog objects. Associate tags from the shared library on any Data Product detail page. Filter the Manage listing by tag.
Available now. → Data Products | Custom Fields → · → Data Products | Detail Page →
Redshift Serverless S3 persistence on Pushdown
Teams that migrated Redshift data quality policies from Spark to Pushdown for cost and performance lost S3 result persistence, breaking compliance and BI workflows that depend on good/bad row records. The connector now persists results to S3 via Redshift's native UNLOAD mechanism. Good and bad row records are accessible through the Data Violations view and directly from your S3 bucket. Engine flexibility across the hybrid stack without losing the audit trail.
Available now. → Configure Redshift Spectrum for External Tables →
Crawler resilience and performance
Two changes to crawler behavior across S3, Kafka, and Pub/Sub data sources.
The S3 crawler no longer deletes assets when underlying files are temporarily missing during a crawl (delayed upload, brief storage interruption). Assets are retained in their last-known state. Policies stay active and continue running on schedule without manual intervention.
Crawlers for S3, Kafka, and Pub/Sub now process multiple assets per batch (default: 5) instead of one at a time. A single asset failure no longer stops the entire crawl. Batch size is controlled by CRAWLER_ASSET_BATCH_SIZE per data source or globally under Data Plane Application Config.
Available now. → S3 Crawler Behavior → · → Amazon S3 → · → Apache Kafka → · → Google Cloud Pub/Sub →
In this release
Full details for each capability are available in the documentation links below.
🔗 Cross-Layer Lineage Stitching, New, 26.9.0
Four technology layers, one connected graph. ADOC resolves the relationships that orchestrators, processing engines, catalogs, and storage systems cannot see about each other, producing end-to-end lineage from the DAG that scheduled the work to the S3 location where the data landed.
→ Read more
⚡ AWS Step Functions Pipeline Integration, New, 26.9.0
Serverless orchestration enters the same observable surface as Airflow and Databricks. Step Functions states render as job nodes with nested visualization, and linked executions produce a single connected pipeline view.
→ AWS Step Functions
📊 Consistent Reliability Scores, Improvement, 26.9.0
One formula, every surface. Discover Assets, Asset Overview, Policies, Reliability Explorer, and Data Products all compute from the same methodology, with a score delta that reflects change from the previous execution.
→ Score Aggregation Methodology
Full release notes on docs.acceldata.io →
Evaluating Acceldata? Book a demo →
.png)
