Changelog (Stable)¶
All stable and notable changes to this project will be documented in this file.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
Note
To view documentation for a specific version, please click the 'docs' badges under the specific changelog entry.
[v0.8.3] - 2022-05-10¶
All existing macros are now supported by all platforms!
New¶
General¶
- dbt 1.1.x support
Google BigQuery and MS SQL Server¶
- PITs (pit macro)
- Bridges (bridge macro)
Fixed¶
Effectivity Satellites¶
Staging¶
- Fixed an issue where hashed columns with lower-case columns provided to an
exclude_columns
config, behaved incorrectly (#110)
[v0.8.2] - 2022-03-14¶
More Google BigQuery and MS SQL Server support, plus fixes!
New¶
Google BigQuery and MS SQL Server¶
- T-Links (t_link macro)
- Effectivity Satellites (eff_sat macro)
- Multi-Active Satellites (ma_sat macro)
- Extended Tracking Satellites (xts macro)
See our Platform support matrix for more details.
Fixed¶
- Fixed a bug where
vault_insert_by_period
would give an error during incremental loads (#108) - Fixed
vault_insert_by_x
issues for MS SQL Server - Fixed (increased) datetime precision in
max_datetime
for Google BigQuery
[v0.8.1] - 2022-02-22¶
HOTFIX RELEASE
Fixed¶
- Fixed a bug where
vault_insert_by_rank
unintentionally used logic fromvault_insert_by_period
when in full-refresh mode or replacing an existing view.
[v0.8.0] - 2022-02-21¶
This is a big release for dbtvault. It's the first time we are releasing support for new platforms! Please welcome to the dbtvault family, Google BigQuery and MS SQL Server!
This is just the start, and we're excited to bring even more platforms (and further support for existing platforms) to you in the future!
New¶
Google BigQuery and MS SQL Server¶
- Hubs (hub macro)
- Links (link macro)
- Satellites (sat macro)
All platforms¶
- Column Escaping (#28, #23) - Docs: dbtvault now automatically surrounds all column names with quotes. This is to allow for columns with reserved words, spaces, and other oddities. The type of quotes is configurable, please refer to the docs linked above.
[v0.7.9] - 2021-12-13¶
Dependencies¶
Breaking¶
- dbt v1.0.0 Support
- Support for earlier dbt versions (dbt <=0.21.0) removed. This is as a result of the upgrade to dbt v1.0.0. How do I upgrade my project?
- Updated to dbt_utils v0.8.0 (for dbt 1.0.0 compatibility)
New¶
Table structures¶
- Point in Time tables Tutorial - Macro docs
- Bridge tables Tutorial - Macro docs
- Extended Tracking Satellites (XTS) Tutorial - Macro docs
Materialisations¶
Behind the Scenes¶
- More test coverage for incremental loading.
- Improved consistency and support for Composite PKs.
- Significantly simplified Multi-Active Satellite (MAS) logic.
Bug Fixes¶
- Multi-Active Satellite record duplication under some circumstances #50
[v0.7.8] - 2021-10-25¶
Dependencies¶
- dbt 0.21.0 support
- dbt utils package dependency now has a version range (sorry!)
Fixes¶
- Effectivity Satellites with auto-end-dating off now handle the use case where records may already be end-dated in the staging layer, as a result of loading data 'manually' end-dated by business rules.
Features¶
Rank column configurations in stage macro (ranked_columns):¶
- Provide ASC or DESC for an
order_by
column Read More - Configure the ranking to use
DENSE_RANK()
orRANK()
Read More
Configuration for hash strings¶
- Concatenation string can now be user defined
- Null placeholder string can now be user defined
[v0.7.7] - 2021-08-24¶
- Re-release of v0.7.6.1 to ensure deployment to dbt Hub
[v0.7.6.1] - 2021-07-14¶
- Hotfix for 0.7.6 to remove unintentionally added macros from the beta branch. #36
Installing¶
Note: This version cannot be installed via dbt hub, please install as follows:
1 2 3 |
|
[v0.7.6] - 2021-07-13¶
- Updated to dbt 0.20.0 and incorporated
adapter.dispatch
changes (#32)
[v0.7.5] - 2021-06-10¶
New structures¶
- Multi-Active Satellites Read More
Bug Fixes¶
- Fixed a bug where an Effectivity Satellite with multiple DFKs or SDKs would incorrectly handle changes in the corresponding link records, meaning one-to-many relationships were not getting handled as intended.
Improvements¶
- Added support for multiple
order_by
orpartition_by
columns when creating ranked columns in thestage
orranked_columns
macros. - Performance improvement for the Satellite macro, which aims to reduce the number of records handled in the initial selection of records from the source data.
[v0.7.4] - 2021-03-27¶
Bug Fixes¶
- Fixed NULL handling bugs in Hubs, Links and Satellites (#26)
- Fixed a bug where Effectivity Satellites would incorrectly end-date (with auto-end-dating enabled) records other than the latest, resulting in duplicate end-date records for previously end-dated records.
Improvements¶
- Added check for matching primary key when inserting new satellite records in the sat macro. This removes the requirement to add the natural key to the hashdiff, but it is still recommended. Read More
Quality of Life¶
- Payload in Transactional (Non-Historised) Links now optional
- Effective From in Satellites now optional
[v0.7.3] - 2021-01-28¶
- Updated dbt to v0.19.0
- Updated dbt utils to 0.6.4
[v0.7.2] - 2021-01-26¶
New¶
-
Derived columns can now be provided lists, for creating composite column values. (#20) Docs
-
The hashed_columns exclude flag in staging can now be provided without a list of columns, and dbtvault will hash everything. Docs
-
Rank Load Materialisation: Iteratively load your vault structures over a configured ranking Read More
-
The stage macro now has a new
ranked_columns
configuration section to support the above materialisation. Read More
Improved¶
- Optimised Satellite SQL for larger loads (billions) seen in the wild.
- For non-hashdiff composite hashed_columns: If all components of the key are NULL, then the whole key will evaluate as NULL. Read more
- Hashing concatenation now uses
CONCAT_WS
instead ofCONCAT
; this is more concise. - The stage macro has received a big overhaul, and the SQL should now be more efficient and easier to read.
- Optimised table macro SQL across to board by reducing the number of CTEs
Fixed¶
- Fixed multiple (minor) bugs in the stage macro (#21)
- Fixed and improved the
adapter.dispatch
implementation (#22) - Fixed a bug in the vault_insert_by_period materialisation (#19)
Docs¶
- Added examples of different ways to provide metadata to the metadata reference
- Added a short guide on extending dbtvault
- Updated all SQL snippets to reflect changes
[v0.7.1] - 2020-12-18¶
New¶
- exclude_columns flag for hashdiffs - Inverse the columns selected for creating a hashdiff to select ALL columns except those listed in the metadata. This is very useful for large multi-column hashdiffs.
Note
See the new stage macro configurations section of the macro docs for more information on the change above.
Improved¶
-
The stage macro now generates CTE-based SQL instead of one big block. This makes it easier to read and debug. See here for more information on why we've moved to CTEs.
-
Multi-dispatch implementation now supports a package override variable, providing a smoother experience for users wishing to override macro implementations. Documentation will be made available in due course. See Issue #14 for more details.
-
Hashed columns now 'see' columns defined as derived columns. Allowing you to use them in your hashed column definitions. Issue #9
Fixed¶
-
Fixed a bug in the vault_insert_by_period materialization which caused orphaned temporary relations under specific circumstances. Issue #18
-
Stage macro conversion to CTE fixes Issue #17
-
dbt_utils dependency is now explicit Issue #15
[v0.7.0] - 2020-09-25¶
New¶
-
Effectivity Satellites: A newly supported Data Vault 2.0 structure.
Read more
Macro Reference -
Period Load Materialisation: Iteratively load your vault structures over a configured period Read More
- dbt Docs: The built-in dbt docs site (
dbt docs serve
) now includes documentation for dbtvault*. - dbt v0.18.0 support dbt v0.18.0 Release Notes
Info
*This is intended as quick reference and for completeness only, the online documentation is still the main reference documentation.
Improved¶
- All table macros now make more use of CTEs to reduce nested SQL and improve readability and debugging potential. Why CTEs?
- All macros have had the licence header removed. This was a little messy and unnecessary.
Removed¶
- Support for dbt versions prior to v0.18.0 Upgrading to v0.18.0
[v0.6.2] - 2020-08-06¶
Fixed¶
dbt_project.yml
config-version: 1
caused an error in any dbt version prior to 0.17.x
. We only put this config
in for users making use of variables in their dbt_project.yml
file.
Note: If using vars
in dbt_project.yml
, you still need to specify config-version: 1
in your own project's dbt_project.yml
.
Guidance will be released for alternatives to model-scoped dbt_project.yml
vars in the next major release of dbtvault (0.7.x
)
Read more about the config-version setting.
[v0.6.1] - 2020-06-24¶
Added¶
-
dbt 0.17.0 support WARNING This comes with a caveat that you must use
config-version: 1
in yourdbt_project.yml
-
All macros now support multiple dispatch. This update is to make way for additional platform support (BigQuery, Postgres etc.)
Changed¶
- A hashdiff in the stage macro now uses
is_hashdiff
as a flag instead ofhashdiff
, this is to clarify this config option as a boolean flag.
Improved¶
Macros¶
- Minor macro re-factors to improve readability
Removed¶
Macros¶
- Cast macro (supporting) - No longer used.
- Check relation (internal) - No longer used.
[v0.6] - 2020-05-26¶
MAJOR UPDATE
We've added a whole host of interesting new features.
Read our v0.5 to v0.6 migration guide
Added¶
-
Staging has now been moved to YAML format, meaning dbtvault is now entirely YAML and metadata driven. See the new stage macro and the staging tutorial for more details.
-
Renamed
source
metadata configuration tosource_model
to clear up some confusion. A big thank you to @balmasi for this suggestion. -
HASHDIFF
aliasing is now available for Satellites Read More
Upgraded¶
- hub and link macros have been given a makeover. They can now handle multi-day loads, meaning no more loading from single-date views. We'll be updating the other macros soon, stay tuned!
Fixed¶
- Fixed
NULL
handling when hashing. We broke this in v0.5 (see related issue) Read more
Removed¶
- Deprecated macros (old table template macros)
- A handful of now unused internal macros
- Documentation website from main repository (this makes the package smaller!) New docs repo
[v0.5] - 2020-02-24¶
Added¶
- Metadata is now provided in the
dbt_project.yml
file. This means metadata can be managed in one place. Read Migrating from v0.4 for more information.
Removed¶
- Target column metadata mappings are no longer required.
- Manual column mapping using triples to provide data-types and aliases (messy and bad practice).
- Removed copyright notice from generated tables (we are open source, duh!)
Fixed¶
- Hashing a single column which contains a
NULL
value now works as intended (related to: hash, multi_hash, staging).
[v0.4.1] - 2020-01-08¶
Added¶
- Support for dbt v0.15
[v0.4] - 2019-11-27¶
Added¶
- Table Macros:
- Transactional Links
Improved¶
- Hashing:
- You may now choose between
MD5
andSHA-256
hashing with a simple yaml configuration Learn how!
- You may now choose between
Worked example¶
- Transactional Links
- Added a Transactional Link model using a simulated transaction feed.
Documentation¶
- Updated macros, best practices, roadmap, and other pages to account for new features
- Updated worked example documentation
- Replaced all dbt documentation links with links to the 0.14 documentation as dbtvault is using dbt 0.14 currently (we will be updating to 0.15 soon!)
- Minor corrections
[v0.3.3-pre] - 2019-10-31¶
Documentation¶
- Added full demonstration project/worked example, using snowflake.
- Minor corrections
[v0.3.2-pre] - 2019-10-28¶
Bug Fixes¶
- Fixed a bug where the logic for performing a base-load (loading for the first time) on a union-based Hub or Link was incorrect, causing a load failure.
Documentation¶
- Various corrections and clarifications on the macros page.
[v0.3.1-pre] - 2019-10-25¶
Error handling¶
- An exception is now raised with an informative message when an incorrect source mapping is provided for a model in the case where a source relation is also provided for a target mapping. This caused missing columns in generated SQL, and a misleading error message from dbt.
[v0.3-pre] - 2019-10-24¶
Improvements¶
-
We've removed the need to specify full mappings in the
tgt
metadata when creating table models. Users may now provide a table reference instead, as a shorthand way to keep the column name and date type the same as the source. The option to provide a mapping is still available. -
The check for whether a load is a union load or not is now more reliable.
Documentation¶
- Updated code samples and explanations according to new functionality
- Added a best practices page
- Various clarifications added and errors fixed
[v0.2.4-pre] - 2019-10-17¶
Bug Fixes¶
- Fixed a bug where the target alias would be used instead of the source alias when incrementally loading a Hub or Link, causing subsequent loads after the initial load, to fail.
[v0.2.3-pre] - 2019-10-08¶
Macros¶
- Updated hash and multi-hash
- hash now accepts a third parameter,
sort
which will alpha-sort provided columns when set to true. - multi-hash updated to take advantage of the hash functionality.
- hash now accepts a third parameter,
Documentation¶
- Updated hash and multi-hash according to new changes.
[v0.2.2-pre] - 2019-10-08¶
Documentation¶
- Finished Satellite page
- Added Union sections to Hub and Link pages
- Updated staging page with Satellite fields
- Renamed
stg_orders_hashed
back tostg_customers_hashed
[v0.2.1-pre] - 2019-10-07¶
Documentation¶
- Minor additions and corrections to documentation:
- Fixed website URL in footer
- Added contribution page to docs
- Corrected version in dbt_project.yml
[v0.2-pre] - 2019-10-07¶
Improved¶
Read the linked documentation for more detail on how to take advantage of the new and improved features.
-
Table Macros:
- All table macros now no longer require the
tgt_cols
parameter. This was unnecessary duplication of metadata and removing this now makes creating tables much simpler.
- All table macros now no longer require the
-
Supporting Macros:
- add_columns
- Simplified the process of adding constants.
- Can now optionally provide a dbt source to automatically retrieve all source columns without needing to type them all manually.
- If not adding any calculated columns or constants, column pairs can be omitted, enabling you to provide the source parameter above only.
- hash now alpha-sorts columns prior to hashing, as per best practices.
- add_columns
-
Staging Macros:
- staging_footer renamed to from and functionality for adding constants moved to add_columns
- multi-hash
- Formatting of output now more readable
- Now alpha-sorts columns prior to hashing, as per best practices.
[v0.1-pre] - 2019-09 / 2019-10¶
Added¶
-
Table Macros:
- Hub
- Link
- Satellite
-
Supporting Macros:
- cast
- hash (renamed from md5_binary)
- prefix
-
Staging Macros:
- add_columns
- multi_hash (renamed from gen_hashing)
- staging_footer
Documentation¶
- Numerous changes for version 0.1 release