All stable and notable changes to this project will be documented in this file.
To view documentation for a specific version, please click the 'docs | passing' badges under the specific changelog entry.
[v0.7.8] - 2021-10-25¶
- dbt 0.21.0 support
- dbt utils package dependency now has a version range (sorry!)
- Effectivity Satellites with auto-end-dating off now handle the use case where records may already be end-dated in the staging layer, as a result of loading data 'manually' end-dated by business rules.
Rank column configurations in stage macro (ranked_columns):¶
- Provide ASC or DESC for an
order_bycolumn Read More
- Configure the ranking to use
Configuration for hash strings¶
- Concatenation string can now be user defined
- Null placeholder string can now be user defined
[v0.7.7] - 2021-08-24¶
- Re-release of v0.7.6.1 to ensure deployment to dbt Hub
[v0.7.6.1] - 2021-07-14¶
- Hotfix for 0.7.6 to remove unintentionally added macros from the beta branch. #36
Note: This version cannot be installed via dbt hub, please install as follows:
1 2 3
[v0.7.6] - 2021-07-13¶
- Updated to dbt 0.20.0 and incorporated
[v0.7.5] - 2021-06-10¶
- Multi-Active Satellites Read More
- Fixed a bug where an Effectivity Satellite with multiple DFKs or SDKs would incorrectly handle changes in the corresponding link records, meaning one to many relationships were not getting handled as intended.
- Added support for multiple
partition_bycolumns when creating ranked columns in the
- Performance improvement for the Satellite macro, which aims to reduce the number of records handled in the initial selection of records from the source data.
[v0.7.4] - 2021-03-27¶
- Fixed NULL handling bugs in Hubs, Links and Satellites (#26)
- Fixed a bug where Effectivity Satellites would incorrectly end-date (with auto-end-dating enabled) records other than the latest, resulting in duplicate end-date records for previously end-dated records.
- Added check for matching primary key when inserting new satellite records in the sat macro. This removes the requirement to add the natural key to the hashdiff, but it is still recommended. Read More
Quality of Life¶
- Payload in Transactional (Non-Historised) links now optional
- Effective From in Satellites now optional
[v0.7.3] - 2021-01-28¶
- Updated dbt to v0.19.0
- Updated dbt utils to 0.6.4
[v0.7.2] - 2021-01-26¶
The hashed_columns exclude flag in staging can now be provided without a list of columns, and dbtvault will hash everything. Docs
Rank Load Materialisation: Iteratively load your vault structures over a configured ranking Read More
The stage macro now has a new
ranked_columnsconfiguration section to support the above materialisation. Read More
- Optimised Satellite SQL for larger loads (billions) seen in the wild.
- For non-hashdiff composite hashed_columns: If all components of the key are NULL, then the whole key will evaluate as NULL. Read more
- Hashing concatenation now uses
CONCAT; this is more concise.
- The stage macro has received a big overhaul, and the SQL should now be more efficient and easier to read.
- Optimised table macro SQL across to board by reducing the number of CTEs
- Fixed multiple (minor) bugs in the stage macro (#21)
- Fixed and improved the
- Fixed a bug in the vault_insert_by_period materialisation (#19)
- Added examples of different ways to provide metadata to the metadata reference
- Added a short guide on extending dbtvault
- Updated all SQL snippets to reflect changes
[v0.7.1] - 2020-12-18¶
- exclude_columns flag for hashdiffs - Inverse the columns selected for creating a hashdiff to select ALL columns except those listed in the metadata. This is very useful for large multi-column hashdiffs.
See the new stage macro configurations section of the macro docs for more information on the change above.
The stage macro now generates CTE-based SQL instead of one big block. This makes it easier to read and debug. See here for more information on why we've moved to CTEs.
Multi-dispatch implementation now supports a package override variable, providing a smoother experience for users wishing to override macro implementations. Documentation will be made available in due course. See Issue #14 for more details.
Hashed columns now 'see' columns defined as derived columns. Allowing you to use them in your hashed column definitions. Issue #9
Stage macro conversion to CTE fixes Issue #17
dbt_utils dependency is now explicit Issue #15
[v0.7.0] - 2020-09-25¶
Period Load Materialisation: Iteratively load your vault structures over a configured period Read More
- dbt Docs: The built-in dbt docs site (
dbt docs serve) now includes documentation for dbtvault*.
- dbt v0.18.0 support dbt v0.18.0 Release Notes
*This is intended as quick reference and for completeness only, the online documentation is still the main reference documentation.
- All table macros now make more use of CTEs to reduce nested SQL and improve readability and debugging potential. Why CTEs?
- All macros have had the licence header removed. This was a little messy and unnecessary.
- Support for dbt versions prior to v0.18.0 Upgrading to v0.18.0
[v0.6.2] - 2020-08-06¶
config-version: 1 caused an error in any dbt version prior to
0.17.x. We only put this config
in for users making use of variables in their
Note: If using
dbt_project.yml, you still need to specify
config-version: 1 in your own project's
Guidance will be released for alternatives to model-scoped
dbt_project.yml vars in the next major release of dbtvault (
Read more about the config-version setting.
[v0.6.1] - 2020-06-24¶
dbt 0.17.0 support WARNING This comes with a caveat that you must use
config-version: 1in your
All macros now support multiple dispatch. This update is to make way for additional platform support (BigQuery, Postgres etc.)
- A hashdiff in the stage macro now uses
is_hashdiffas a flag instead of
hashdiff, this is to clarify this config option as a boolean flag.
- Minor macro re-factors to improve readability
- Cast macro (supporting) - No longer used.
- Check relation (internal) - No longer used.
[v0.6] - 2020-05-26¶
We've added a whole host of interesting new features.
sourcemetadata configuration to
source_modelto clear up some confusion. A big thank you to @balmasi for this suggestion.
HASHDIFFaliasing is now available for Satellites Read More
- hub and link macros have been given a makeover. They can now handle multi-day loads, meaning no more loading from single-date views. We'll be updating the other macros soon, stay tuned!
- Deprecated macros (old table template macros)
- A handful of now unused internal macros- Documentation website from main repository (this makes the package smaller!) New docs repo
[v0.5] - 2020-02-24¶
- Metadata is now provided in the
dbt_project.ymlfile. This means metadata can be managed in one place. Read Migrating from v0.4 for more information.
- Target column metadata mappings are no longer required.
- Manual column mapping using triples to provide data-types and aliases (messy and bad practice).
- Removed copyright notice from generated tables (we are open source, duh!)
- Hashing a single column which contains a
NULLvalue now works as intended (related to: hash, multi_hash, staging).
[v0.4.1] - 2020-01-08¶
- Support for dbt v0.15
[v0.4] - 2019-11-27¶
- Table Macros:
- You may now choose between
SHA-256hashing with a simple yaml configuration Learn how!
- You may now choose between
- Transactional Links
- Added a transactional link model using a simulated transaction feed.
- Updated macros, best practices, roadmap, and other pages to account for new features
- Updated worked example documentation
- Replaced all dbt documentation links with links to the 0.14 documentation as dbtvault is using dbt 0.14 currently (we will be updating to 0.15 soon!)
- Minor corrections
[v0.3.3-pre] - 2019-10-31¶
- Added full demonstration project/worked example, using snowflake.
- Minor corrections
[v0.3.2-pre] - 2019-10-28¶
- Fixed a bug where the logic for performing a base-load (loading for the first time) on a union-based hub or link was incorrect, causing a load failure.
- Various corrections and clarifications on the macros page.
[v0.3.1-pre] - 2019-10-25¶
- An exception is now raised with an informative message when an incorrect source mapping is provided for a model in the case where a source relation is also provided for a target mapping. This caused missing columns in generated SQL, and a misleading error message from dbt.
[v0.3-pre] - 2019-10-24¶
We've removed the need to specify full mappings in the
tgtmetadata when creating table models. Users may now provide a table reference instead, as a shorthand way to keep the column name and date type the same as the source. The option to provide a mapping is still available.
The check for whether a load is a union load or not is now more reliable.
- Updated code samples and explanations according to new functionality
- Added a best practises page
- Various clarifications added and errors fixed
[v0.2.4-pre] - 2019-10-17¶
- Fixed a bug where the target alias would be used instead of the source alias when incrementally loading a hub or link, causing subsequent loads after the initial load, to fail.
[v0.2.3-pre] - 2019-10-08¶
- Updated hash and multi-hash
[v0.2.2-pre] - 2019-10-08¶
- Finished Satellite page
- Added Union sections to Hub and Link pages
- Updated staging page with Satellite fields
[v0.2.1-pre] - 2019-10-07¶
- Minor additions and corrections to documentation:
- Fixed website URL in footer
- Added contribution page to docs
- Corrected version in dbt_project.yml
[v0.2-pre] - 2019-10-07¶
Read the linked documentation for more detail on how to take advantage of the new and improved features.
- All table macros now no longer require the
tgt_colsparameter. This was unnecessary duplication of metadata and removing this now makes creating tables much simpler.
- All table macros now no longer require the
- Simplified the process of adding constants.
- Can now optionally provide a dbt source to automatically retrieve all source columns without needing to type them all manually.
- If not adding any calculated columns or constants, column pairs can be omitted, enabling you to provide the source parameter above only.
- hash now alpha-sorts columns prior to hashing, as per best practises.
[v0.1-pre] - 2019-09 / 2019-10¶
- Numerous changes for version 0.1 release