Skip to content

Getting Started

Info

This walk-through intends to give you a detailed understanding of how to use dbtvault and the provided macros to develop a Data Vault Data Warehouse from the ground up. If you're looking to quickly experiment and learn using pre-written models, take a look at our worked example.

In this section we teach you how to use dbtvault step-by-step, explaining the use of macros and the different components of the Data Vault in detail.

We will:

  • process a raw staging layer.
  • create a Data Vault with hubs, links and satellites using dbtvault.

Pre-requisites

  1. Some prior knowledge of Data Vault 2.0 architecture. Have a look at How can I get up to speed on Data Vault 2.0?

  2. A Snowflake account, trial or otherwise. Sign up for a free 30-day trial here

  3. You must have downloaded and installed dbt and set up a project.

  4. Sources should be set up in dbt (see below).

  5. We assume you already have a raw staging layer, PSA (Persistent Staging Area) or Data Lake.

  6. Some raw vault structures need to be loaded iteratively to ensure record deltas (changes) over time are correctly processed. Currently, these are the following structures:

    • Transactional Links
    • Satellites
    • Effectivity Satellites

    To correctly load these structures, we have developed custom materialisations which iteratively load data:

    These macros are described in mre depth in their respective sections

  7. You should read our best practices guidance.

Setting up sources (in dbt)

We will be using the source feature of dbt extensively throughout the documentation to make access to source data much easier, cleaner and more modular.

We have provided an example below which shows a configuration similar to that used for the examples in our documentation, however this feature is documented extensively in the documentation for dbt.

We recommend that you place the schema.yml file you create for your sources, in the root of your models folder, however you can place it wherever needed for your specific project and models.

schema.yml

1
2
3
4
5
6
7
8
9
version: 2

sources:
  - name: my_source
    database: MY_DATABASE
    schema: MY_SCHEMA
    tables:
      - name: raw_orders
      - name: ...

Installation

Read the installation instructions on dbt hub


Last update: 2021-01-25