git404hub

what is declarative_pipelines fr?

bombercorny/declarative_pipelines — explained in plain English

Analysis updated 2026-05-18

2PythonAudience · dataComplexity · 4/5Setup · hard

tl;dr

A Databricks framework that lets you define and run data pipelines by configuring steps in a table instead of writing pipeline code.

vibe map

mindmap
  root((declarative_pipelines))
    What it does
      Reads pipeline config from a table
      Builds Databricks pipelines dynamically
      Applies data quality expectations
    Tech stack
      Python
      Databricks
      Delta Lake
    Use cases
      Configure pipelines without code changes
      Add quality checks to steps
      Register tables and views at runtime
    Audience
      Data engineers
      Databricks users

Code map

Detail Auto

An interactive map of this repo's files and how they connect — its source is parsed live in your browser. Click Visualize to build it.

filefunction / class

what do people make with this?

VIBE 1

Define a Databricks pipeline's tables and transformations as rows in a config table instead of code.

VIBE 2

Add data quality checks that drop or fail rows failing a condition.

VIBE 3

Change pipeline behavior by editing configuration data rather than redeploying code.

VIBE 4

Register tables, materialized views, and temporary views dynamically at runtime.

what's the stack?

PythonDatabricksDelta LakeUnity Cataloguv

how it stacks up fr

bombercorny/declarative_pipelines0-bingwu-0/live-interpreter010zx00x1/faresnipe
Stars222
LanguagePythonPythonPython
Setup difficultyhardmoderateeasy
Complexity4/52/52/5
Audiencedatageneralgeneral

Figures from each repo's GitHub metadata at analysis time.

how do i run it?

Difficulty · hard time til it works · 1h+

Requires a configured Databricks CLI profile and Databricks Connect to seed test data before deploying.

in plain english

declarative_pipelines is a framework for building data pipelines on Databricks without writing pipeline logic directly in code. Instead, you describe each step of your pipeline as a row in a configuration table stored in Databricks, and the framework reads that table at runtime and builds the pipeline for you automatically. A pipeline is made up of steps. Each step names the source tables it reads from, points to a transformation function that processes the data, and defines a target, which can be a regular table, a materialized view, or a temporary view. Steps can also include quality checks, called expectations, that verify the data meets certain conditions, such as a value being positive, and can drop or fail rows that do not pass. To use it, you install the project's dependencies, then run included scripts to create example source data and write a sample pipeline configuration into your Databricks catalog. After updating a deployment file with your own catalog and schema names, you deploy the project as a Databricks bundle and run the pipeline job. The framework then reads the configuration table, works out what each step needs to do, and registers the corresponding datasets and checks with Databricks. This approach means changing what a pipeline does, such as adding a new transformation step or a new quality rule, mostly involves editing rows in the configuration table rather than changing and redeploying code. The project includes a documented schema for the configuration table and a full working example script showing how a pipeline entry is structured. The project is written in Python and depends on the Databricks CLI and Databricks Connect for setup and testing. It is a small, early-stage project with a narrow, specific purpose: making Databricks pipelines configurable through data rather than through hardcoded code.

prompts (copy fr)

prompt 1
Help me set up the Databricks CLI DEFAULT profile needed to run declarative_pipelines.
prompt 2
Walk me through seeding test data and config with the create_test_data and create_test_config scripts.
prompt 3
Explain how to write a new pipeline step with sources, a transformation, and a target in the config table.
prompt 4
Show me how to add an expect_or_fail quality expectation to a pipeline step.

Frequently asked questions

what is declarative_pipelines fr?

A Databricks framework that lets you define and run data pipelines by configuring steps in a table instead of writing pipeline code.

What language is declarative_pipelines written in?

Mainly Python. The stack also includes Python, Databricks, Delta Lake.

How hard is declarative_pipelines to set up?

Setup difficulty is rated hard, with roughly 1h+ to a first successful run.

Who is declarative_pipelines for?

Mainly data.

peek the repo → explain another one

This repo across BitVibe Labs

double-check against the repo, no cap.