Skip to main content

Data Contracts

What is a data contract, and why do you need one?

By Dr. Simon Harrer · · 2:02 video · 8 slides

In short

A data contract is a written promise from a data producer to its data consumers. It describes the data and the rules that the data must follow. Without it, a small change can break the data consumers and nobody sees it in time.

Watch on YouTube
Slide 1 of Entropy Explains episode 1, A small change breaks three consumers: One team produces the orders table.

A small change breaks three consumers

One team produces the orders table. A dashboard, an ML model, and an AI agent use this table. The data producer renames one column from amount to total. All three data consumers break, because nobody knew that they used this column.

Slide 2 of Entropy Explains episode 1, Write the promise down: The data producer writes the promise in one document.

Write the promise down

The data producer writes the promise in one document. The data consumers read the same document and rely on it. This document is a data contract. It is the one place where both sides agree on the data.

Slide 3 of Entropy Explains episode 1, What goes into a data contract: A data contract has eight parts.

What goes into a data contract

A data contract has eight parts. The fundamentals give the id, the version, and the status. Other parts describe the schema, the data quality rules, and the team. The last parts give the terms of use, the service levels, the servers, and the context for AI agents.

Slide 4 of Entropy Explains episode 1, One YAML file: In practice, the data contract is a YAML file in the Open Data Contract Standard, ODCS.

One YAML file

In practice, the data contract is a YAML file in the Open Data Contract Standard, ODCS. People can read this file. Tools can also read it. The same file shows the schema, the quality rules, and the freshness promise.

Slide 5 of Entropy Explains episode 1, One standard and two open-source tools: ODCS is the de facto standard for data contracts.

One standard and two open-source tools

ODCS is the de facto standard for data contracts. Bitol, a Linux Foundation project, maintains it. With the open-source Data Contract Editor, you edit the file visually. The open-source Data Contract CLI checks the file and tests the data in your CI pipeline.

Slide 6 of Entropy Explains episode 1, Machines check the promise: The Data Contract CLI reads the data contract and checks the real data against it.

Machines check the promise

The Data Contract CLI reads the data contract and checks the real data against it. It checks that the order id is present and that the types match. It also checks that no amount is negative and that the data is less than one hour old.

Slide 7 of Entropy Explains episode 1, The test fails first: Now a bad record arrives with a negative amount.

The test fails first

Now a bad record arrives with a negative amount. The test fails before a data consumer sees the incorrect numbers. The CLI shows which check failed and which value is incorrect. The data producer can fix the problem before it causes damage for the data consumers.

Slide 8 of Entropy Explains episode 1, An API for data: A data contract is an API for data.

An API for data

A data contract is an API for data. The data producer makes promises, and the data consumer relies on them. The data contract helps with communication, trust, and discovery for people and for AI agents.

Read more