Data Contracts and Data Quality
How do you define, test, and measure data quality?
By Dr. Simon Harrer · · 2:56 video · 7 slides
In short
Write the quality rules and the service levels into the data contract. The Data Contract CLI tests them automatically. A quality score for each dimension shows where you must improve the data.
Watch on YouTube
Nobody wrote it down
The revenue dashboard shows that revenue doubled overnight. The cause is duplicate rows in the orders table. Nobody wrote down that each order ID must be unique, so nobody checked it. Data quality starts when you write down what good data means.
Start in plain language
The business and the data consumers write down the rules in plain language. Then a data engineer converts the rules to SQL or to library metrics, with help from AI. Some rules stay text, because you cannot automate them or the work is too much.
Quality rules in the data contract
The quality rules go into the schema of the data contract, next to the columns that they check. Library metrics cover the basic checks, for example null values, duplicates, and row count. SQL covers all other checks. Each rule also names its quality dimension.
Service levels in the data contract
The data contract also contains service levels. Freshness says that the newest order is less than 24 hours old. Retention says that no order stays longer than one year. Both service levels use the order timestamp.
Tested automatically
The Data Contract CLI tests the real table against each rule. Here, all other checks pass, but freshness fails, because the newest order is 31 hours old. Run the tests in CI on each change and on a schedule in production. With the publish option, the results go to the data marketplace.
Seven quality dimensions
ODCS has seven quality dimensions: accuracy, completeness, conformity, consistency, coverage, timeliness, and uniqueness. Tag each rule with one dimension. Then you can compare the results across all data contracts. Each dimension shows a different type of problem.
Quality score by dimension
The data marketplace changes the results into a quality score from 0 to 100. The score for each dimension shows where the quality is low. You can see the score for a data contract, a data product, a team, a domain, and the organization. Low values show where you must invest.