Databricks Asset Bundles: Deploying Lakehouse Workflows from CI/CD, Not the UI

Data teams and platform teams have historically lived in different operational universes.

While software engineers spend years establishing strict Git branching strategies, automated linting, containerization, and immutable deployment pipelines, data engineering frequently relied on the Databricks Workspace UI. An engineer would write a Python notebook, click "Schedule Job" in the web console, attach an interactive cluster, and call it a day.

That workflow is fine for interactive exploratory data analysis. For production lakehouse pipelines processing financial transactions, machine learning features, or business-critical analytics, it is an enterprise disaster waiting to happen:

• No Auditable Source of Truth: Who modified the cron schedule at 3 a.m.? Which version of the notebook is currently executing in production?
• Configuration Drift Between Workspaces: Dev, UAT, and Production workspaces fall out of sync. A job that runs in Staging fails in Production because a secret scope or cluster policy was created manually in one workspace and forgotten in the other.
• Excessive Permissions (The Security Nightmare): To update a production job, developers demand administrative write access to the Production Databricks workspace—violating SOC2, ISO 27001, and basic principle-of-least-privilege boundaries.

Databricks Asset Bundles (DABs) are Databricks' official answer to this crisis: a declarative Infrastructure-as-Code (IaC) framework that brings software engineering rigor to lakehouse delivery.

1. What Are Databricks Asset Bundles?

A Databricks Asset Bundle (DAB) is an infrastructure-as-code package defined in YAML that encapsulates everything required to run a data workload:

• Source code (Python files, SQL scripts, notebooks, dbt projects).
• Orchestration definitions (multi-task Databricks Workflows and schedules).
• Data pipelines (Delta Live Tables / DLT).
• Compute definitions (serverless compute, cluster policies, and libraries).
• Target environments (Dev, Staging, and Production workspace configurations).

Instead of clicking through the UI, your entire data workload is declared in a root configuration file: databricks.yml.

my-lakehouse-bundle/
├── databricks.yml            # Main bundle configuration & targets
├── resources/
│   ├── ingestion_job.yml     # Databricks Workflow & schedule definition
│   └── dlt_pipeline.yml      # Delta Live Tables pipeline spec
└── src/
    ├── transform.py          # Pure Python business logic / wheel
    └── dbt_project/          # dbt models and tests

2. The Core Mechanics: Targets and Modes

The defining superpower of Databricks Asset Bundles is how it separates development testing from production deployments through target definitions:

# databricks.yml
bundle:
  name: customer-analytics-pipeline

include:
  - resources/*.yml

variables:
  catalog:
    description: "Unity Catalog target"
    default: "dev_catalog"

targets:
  dev:
    mode: development
    default: true
    workspace:
      host: https://adb-dev-123456789.azuredatabricks.net

  uat:
    workspace:
      host: https://adb-uat-987654321.azuredatabricks.net
    variables:
      catalog: "uat_catalog"

  prod:
    mode: production
    workspace:
      host: https://adb-prod-555555555.azuredatabricks.net
      root_path: /Shared/.bundle/${bundle.name}
    variables:
      catalog: "prod_catalog"

1. Development Mode (mode: development)

When a developer runs databricks bundle deploy -t dev on their local laptop, Databricks automatically isolates their deployment:

• Job names are automatically prefixed with [dev ${workspace.current_user.short_name}].
• Workflows are paused by default so schedules do not fire and collide with real data pipelines.
• Code is uploaded to the developer's personal home folder (/Users/sourabh@company.com/.bundle/...).
• Multiple data engineers can test changes simultaneously against the shared Dev workspace without overwriting each other's jobs.

2. Production Mode (mode: production)

When deploying to production:

• Job names use canonical production names without user prefixes.
• Schedules and triggers are activated.
• Assets are deployed to a shared, governed workspace directory (/Shared/.bundle/...).
• Crucially: Production deployments enforce headless service principal execution. No individual human developer's credentials are bound to the production job.

3. Defining a Production Workflow in YAML

Instead of configuring tasks in the UI canvas, workflows are version-controlled in modular YAML files:

# resources/ingestion_job.yml
resources:
  jobs:
    nightly_customer_etl:
      name: "nightly-customer-etl"
      schedule:
        quartz_cron_expression: "0 0 2 * * ?"
        timezone_id: "UTC"
      
      job_clusters:
        - job_cluster_key: "cost_optimized_cluster"
          new_cluster:
            spark_version: "14.3.x-scala2.12"
            policy_id: "A0B1C2D3E4F5" # Enforced Platform Cluster Policy
            autoscale:
              min_workers: 1
              max_workers: 4

      tasks:
        - task_key: "extract_raw_data"
          job_cluster_key: "cost_optimized_cluster"
          spark_python_task:
            python_file: "../src/extract.py"
            parameters:
              - "--catalog=${var.catalog}"

        - task_key: "transform_silver"
          depends_on:
            - task_key: "extract_raw_data"
          job_cluster_key: "cost_optimized_cluster"
          spark_python_task:
            python_file: "../src/transform.py"

Because this YAML is stored in Git, any change to the schedule, cluster size, or dependencies goes through standard peer code review.

4. The Enterprise CI/CD Pipeline Architecture

To achieve true operational reliability, no engineer should ever run databricks bundle deploy -t prod from their personal workstation.

Production deployments must execute strictly through an automated CI/CD pipeline powered by dedicated Service Principals:

[ Developer Branch ] ──► (PR Open) ──► [ databricks bundle validate ]
                                               │
                                               ▼
[ Merge to Main ]    ──► [ Deploy to UAT ] (via UAT Service Principal)
                               │
                               ▼
                    [ Manual Approval Gate ]
                               │
                               ▼
[ Deploy to Prod ]   ──► [ databricks bundle deploy -t prod ]
                         (via Production Service Principal)

Complete GitHub Actions Production Pipeline:

# .github/workflows/deploy-databricks-bundle.yml
name: "Databricks Bundle CI/CD"

on:
  pull_request:
    branches: [main]
  push:
    branches: [main]

jobs:
  validate:
    name: "Validate Bundle Configuration"
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v4

      - name: Install Databricks CLI
        uses: databricks/setup-cli@v0.220.0

      - name: Validate Dev Bundle
        run: databricks bundle validate -t dev
        env:
          DATABRICKS_HOST: ${{ secrets.DATABRICKS_DEV_HOST }}
          DATABRICKS_TOKEN: ${{ secrets.DATABRICKS_DEV_SP_TOKEN }}

  deploy_uat:
    name: "Deploy to UAT"
    needs: validate
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    environment: uat
    steps:
      - uses: actions/checkout@v4
      - uses: databricks/setup-cli@v0.220.0
      - name: Deploy UAT Bundle
        run: databricks bundle deploy -t uat
        env:
          DATABRICKS_HOST: ${{ secrets.DATABRICKS_UAT_HOST }}
          DATABRICKS_TOKEN: ${{ secrets.DATABRICKS_UAT_SP_TOKEN }}

  deploy_prod:
    name: "Deploy to Production"
    needs: deploy_uat
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    environment: production # Requires manual peer review approval gate
    steps:
      - uses: actions/checkout@v4
      - uses: databricks/setup-cli@v0.220.0
      - name: Deploy Production Bundle
        run: databricks bundle deploy -t prod
        env:
          DATABRICKS_HOST: ${{ secrets.DATABRICKS_PROD_HOST }}
          DATABRICKS_CLIENT_ID: ${{ secrets.DATABRICKS_PROD_SP_CLIENT_ID }}
          DATABRICKS_CLIENT_SECRET: ${{ secrets.DATABRICKS_PROD_SP_CLIENT_SECRET }}

5. Security & Governance Rules for Enterprise Bundles

1. Zero Hardcoded Secrets: Never put database passwords, API tokens, or webhook URLs inside databricks.yml. Store them inside Databricks Secret Scopes backed by Azure Key Vault or AWS Secrets Manager, and reference them using standard runtime secrets:

   # In Python task
   db_pass = dbutils.secrets.get(scope="lakehouse_secrets", key="db_password")
   

2. Cluster Policies Over Raw Clusters: Never allow bundle definitions to create arbitrary, unbounded multi-node clusters. Force every job to specify a policy_id governed by the Platform Engineering team. This guarantees that auto-termination, allowed VM node types, and tag enforcement are strictly applied.
3. Isolate Service Principals per Account: The service principal used to deploy into the Dev workspace must have zero permissions in the Production workspace. A compromised dev token should never be able to mutate production pipelines. This mirrors the exact isolation model we use in Terraform across Dev, UAT, and Production.

The Engineering Bottom Line

The Databricks web UI is an unmatched canvas for data discovery and rapid prototyping. But when a prototype graduates into an operational business pipeline, it must leave the UI behind.

Databricks Asset Bundles bridge the gap between Data Science and Platform Engineering:

• Data engineers write declarative YAML describing what they want to run.
• Git provides history, peer review, and rollback capabilities.
• Automated CI/CD pipelines handle environment promotions through locked-down service principals.

Done right, deploying a mission-critical Databricks lakehouse workflow becomes as safe, boring, and repeatable as shipping a microservice.