Data Contracts and CI/CD
How do data contracts fit into your CI/CD pipeline?
By Dr. Simon Harrer · · 2:40 video · 9 slides
In short
A data contract is a file in the Git repository of the data product. Each change goes through a CI/CD pipeline. The pipeline checks the data contract, tests the data in each environment, and publishes the results to the data marketplace.
Watch on YouTube
A one-line change
A data engineer renames one column in a pull request: customer name becomes full name. The review looks harmless. After the deployment, a claims report, a churn model, and an AI agent fail. The data contract had the information, but nothing checked it.
The data contract lives in Git
Keep the data contract in the same Git repository as the pipeline code. The file name shows the dataset, the major version, and the platform. The code and the data contract change together. Each commit starts the CI/CD pipeline.
Company rules
First, the lint job checks that the file is valid ODCS. Then a small Python script checks your company rules. For example, each field of an active data contract must have a classification. The build stops until each field has one.
Breaking changes stop the build
The next job compares the changed data contract with the version on the main branch. The Data Contract CLI finds that a column was removed. This is a breaking change, so the build fails before the merge.
One schema for each pull request
Before the merge, the pipeline deploys the change into its own environment. The server in the data contract takes its schema name from a variable. So each pull request gets its own schema, and the tests do not touch the data of other people.
Test again in staging
After the merge, the pipeline deploys to staging and runs the data pipeline. Then it tests the schema, the quality rules, and the freshness again. The pipeline uses the same data contract for each environment. Only the server in the data contract changes from one environment to the next.
Publish when all jobs are green
Then the pipeline deploys to production and tests again. Only when each job is green, it publishes the data contract and the test results to the data marketplace. Data consumers can then see the current state of the promise.
Monitor the promise each night
The code does not change overnight, but new data arrives all the time. So a second pipeline tests the production data each night. Here, the customers table is 30 hours old, but the limit is 24 hours. The data marketplace tells the owner and the downstream consumers.
Each commit and each night
Each commit goes through the pipeline. Each night, a scheduled job tests the data again. So problems become visible before a data consumer uses incorrect data. The data contract is checked on each change and each day.