dbt

Connect a dbt project to Lightdash, keep it in sync, and use metrics defined in dbt

dbt is the industry-standard tool for transforming data in your warehouse. You write modular transformations as plain SQL select statements, and dbt handles the dependencies between them, builds the resulting tables and views, and tests them.

Lightdash reads a connected dbt project and builds your semantic layer from it. Your dbt models define what the tables are; the Lightdash metadata in their YAML defines what they mean — which columns are dimensions, which aggregations are metrics, how models join.

dbt is not required. If you don’t use it, define the same semantic layer in Lightdash YAML instead. This area covers the dbt integration itself: connecting a project, keeping it in sync, moving definitions between Lightdash and your repository, and what changes when you upgrade dbt.

For how to shape models so they explore well, see modeling strategies.

Set up a dbt project

Before you start, you need a dbt project on your machine and a Lightdash account. If you haven’t used dbt before, follow dbt’s getting started guide first.

Install and log in to the CLI

Install the Lightdash CLI with Homebrew on macOS, Linux, or Windows through WSL:

brew tap lightdash/lightdash
brew trust lightdash/lightdash
brew install lightdash
lightdash --version

For other install methods, such as a direct binary download or npm, see Install the Lightdash CLI.

Lightdash generates a personal access token for you during onboarding and shows the login command to copy into your terminal:

lightdash login https://{{ lightdash_domain }} --token my-super-secret-token

{{ lightdash_domain }} is app.lightdash.cloud on Lightdash Cloud, or your own domain if you self-host. For other login methods, see Authenticate the CLI.

Generate your first Tables

If you use dbt Cloud, clone your repository to your local machine first.

From the folder that contains your dbt_project.yml file, run:

lightdash dbt run

The command documents every column in your models as a dimension and generates a Table for each model. For example, a leads.yml file becomes a Table called Leads with the dimensions Lead id, User id, and Sdr.

If you see Error: Not a dbt project (or any of the parent directories). Missing dbt_project.yml file, change into your dbt project folder and run the command again. To generate Tables for only some models, pass dbt’s model selection syntax. The examples under Choose which models become Tables below show common patterns.

Create your project

Create the Lightdash project and follow the link the CLI prints to start exploring:

lightdash deploy --create

Lightdash creates the project with the connection details from your local profiles.yml. Next, update your connection settings to switch to a service account and connect your version-controlled dbt project, so Lightdash stays in sync with your repository. To compare CLI deploys with a git connection, see Syncing dbt projects.

What are Tables?

Tables are the starting point for any data exploration in Lightdash — they’re the data you can query. Lightdash stays in sync with your dbt project, so Tables come from the dbt models defined in your project’s YAML files. If a dbt model has a YAML definition, it shows up in Lightdash as a Table.

Not sure what a YAML file is? Check out dbt’s docs on model properties to learn about adding YAML files for your dbt models.

Choose which models become Tables

lightdash dbt run documents every column in your models as a dimension and generates a Table for each one. It supports dbt’s model selection syntax, so you can generate Tables for your whole project at once or narrow to a subset — for example, everything tagged lightdash:

lightdash dbt run                                        # every model with a YAML definition
lightdash dbt run -s tag:lightdash                       # all models with the lightdash tag
lightdash dbt run -s payments                            # just payments
lightdash dbt run -s payments+                           # payments and all children
lightdash dbt run -s +payments                           # payments and all parents
lightdash dbt run -s payments+ +customers tag:lightdash  # mix and match