Data Contracts and Data Products
What is a data product, and how does it relate to data contracts?
By Dr. Simon Harrer · · 2:38 video · 9 slides
In short
A data product is the data, the code that makes it, and the metadata that describes it. One team owns it. Each output port of the data product has its own data contract, which is the promise for that port.
Watch on YouTube
More than a table
Many people think that a data product is only a table. A data product also includes the code that makes the data. It includes the metadata that tells who owns it, which ports it has, and which data contracts apply. One team owns all of it.
Input ports and output ports
Data comes into the data product through input ports. Here it comes from the web shop and from the payments data product. The data product gives its data through output ports. Each output port refers to a data contract, the promise for exactly that port.
A YAML file in ODPS
The data product is also a YAML file. It uses the Open Data Product Standard, ODPS. The file lists the owner team, the input ports, and the output ports. Each output port points to its data contract in ODCS.
Code and description in Git
The data product file lives in Git, next to the pipeline code and the data contracts. Code and description change together, in the same pull request. On each change, the CI pipeline publishes the data product to a data marketplace.
Request access with a purpose
In the data marketplace, data consumers find the data products that they need. A data consumer requests access to one output port and gives a purpose. The owner team approves, or a policy approves automatically. An access agreement then connects the two teams under the terms of use.
A breaking change, a new version
The data producer wants to rename amount to total. This is a breaking change. So version 1 stays as it is, and a new output port with version 2 starts next to it. The Data Contract CLI shows the breaking change before it goes live.
Consumers migrate at their own pace
Both output ports run in parallel. Each data consumer moves from version 1 to version 2 when it is ready. No data consumer must change on the same day as the data producer. The data producer can see which data consumers still use version 1.
Retire the old version
When no data consumer uses version 1 anymore, the data producer shuts it down. Only version 2 remains. The data product changes without an outage for its data consumers. Each data consumer had time to test version 2 before the change.
Ownership and guarantees
Each data product has an owner team. The data contract gives the guarantees for each output port. An access agreement connects the data product to each data consumer. Together, they make data that people and AI agents can find, use, and trust.