Databricks Open Sources Spark Declarative Pipelines for Scalable Data Engineering

Databricks has taken a significant step by open-sourcing its core declarative ETL framework, known as Apache Spark Declarative Pipelines. Originally launched as Delta Live Tables (DLT) in 2022, this framework is designed to simplify the creation and operation of reliable, scalable data pipelines, supporting both batch and streaming workloads seamlessly.

Traditionally, data engineering involves complex pipeline authoring, heavy manual operations, and maintaining separate systems for batch and streaming data. Spark Declarative Pipelines changes this by allowing engineers to declare their pipelines using SQL or Python, while Apache Spark manages execution details like dependency tracking, table management, parallelism, checkpoints, and retries automatically.

This declarative approach supports modern data realities such as change data feeds, message buses, and real-time analytics, making it ideal for powering AI systems. It integrates smoothly with object storage platforms like Amazon S3, Azure Data Lake Storage, and Google Cloud Storage, enabling unified processing of batch and streaming data through a single API.

The framework’s effectiveness is proven at scale. Enterprises like Block have cut development time by over 90%, while Navy Federal Credit Union reduced pipeline maintenance by 99%. These improvements translate into faster time-to-insight, lower operational costs, and enhanced performance tailored to specific latency requirements, including sub-second streaming.

Databricks’ open-source move also sets it apart from competitors like Snowflake, whose recent Openflow service focuses mainly on data ingestion into its platform. In contrast, Spark Declarative Pipelines empower users to build end-to-end pipelines from raw data sources to transformed, usable data, all within an open ecosystem accessible beyond Databricks’ own platform.

The upcoming integration of Spark Declarative Pipelines into the Apache Spark codebase marks a milestone in open data infrastructure. While the exact release date is pending, the technology is already commercially available through Databricks Lakeflow Declarative Pipelines, offering enterprise-grade features and support.

By open-sourcing this framework, Databricks reinforces its commitment to simplifying data engineering and fostering an open ecosystem where users can innovate without vendor lock-in. For organizations aiming to streamline their data pipelines, reduce operational overhead, and accelerate AI initiatives, Spark Declarative Pipelines represent a powerful new tool in the data infrastructure landscape.