bombercorny/declarative_pipelines — explained in plain English
Analysis updated 2026-05-18
Define a Databricks pipeline's tables and transformations as rows in a config table instead of code.
Add data quality checks that drop or fail rows failing a condition.
Change pipeline behavior by editing configuration data rather than redeploying code.
Register tables, materialized views, and temporary views dynamically at runtime.
| bombercorny/declarative_pipelines | 0-bingwu-0/live-interpreter | 010zx00x1/faresnipe | |
|---|---|---|---|
| Stars | 2 | 2 | 2 |
| Language | Python | Python | Python |
| Setup difficulty | hard | moderate | easy |
| Complexity | 4/5 | 2/5 | 2/5 |
| Audience | data | general | general |
Figures from each repo's GitHub metadata at analysis time.
Requires a configured Databricks CLI profile and Databricks Connect to seed test data before deploying.
declarative_pipelines is a framework for building data pipelines on Databricks without writing pipeline logic directly in code. Instead, you describe each step of your pipeline as a row in a configuration table stored in Databricks, and the framework reads that table at runtime and builds the pipeline for you automatically. A pipeline is made up of steps. Each step names the source tables it reads from, points to a transformation function that processes the data, and defines a target, which can be a regular table, a materialized view, or a temporary view. Steps can also include quality checks, called expectations, that verify the data meets certain conditions, such as a value being positive, and can drop or fail rows that do not pass. To use it, you install the project's dependencies, then run included scripts to create example source data and write a sample pipeline configuration into your Databricks catalog. After updating a deployment file with your own catalog and schema names, you deploy the project as a Databricks bundle and run the pipeline job. The framework then reads the configuration table, works out what each step needs to do, and registers the corresponding datasets and checks with Databricks. This approach means changing what a pipeline does, such as adding a new transformation step or a new quality rule, mostly involves editing rows in the configuration table rather than changing and redeploying code. The project includes a documented schema for the configuration table and a full working example script showing how a pipeline entry is structured. The project is written in Python and depends on the Databricks CLI and Databricks Connect for setup and testing. It is a small, early-stage project with a narrow, specific purpose: making Databricks pipelines configurable through data rather than through hardcoded code.
A Databricks framework that lets you define and run data pipelines by configuring steps in a table instead of writing pipeline code.
Mainly Python. The stack also includes Python, Databricks, Delta Lake.
Setup difficulty is rated hard, with roughly 1h+ to a first successful run.
Mainly data.
This repo across BitVibe Labs
double-check against the repo, no cap.