Skip to main content

Building Data Products with AI

How do you build a data product with AI?

By Dr. Simon Harrer · · 2:08 video · 7 slides

In short

A coding agent writes the pipeline code of the data product. It needs three inputs. The data contract tells what to build, and a skills repository tells how to build it. The data marketplace gives the data to build on.

Watch on YouTube
Slide 1 of Entropy Explains episode 4, A coding agent writes the pipeline: Each data product needs pipeline code: models, tests, and a CI workflow.

A coding agent writes the pipeline

Each data product needs pipeline code: models, tests, and a CI workflow. Today, a coding agent writes this code. The agent itself is a commodity. The result depends on the inputs that you give to the agent.

Slide 2 of Entropy Explains episode 4, The what: the data contract: The first input is the data contract.

The what: the data contract

The first input is the data contract. It tells what the data product must deliver. The business, the data producer, and the data consumers write it together in a workshop. They change it until all agree. This part stays human work.

Slide 3 of Entropy Explains episode 4, The how: a skills repository: Without instructions, each run builds the data product in a different way.

The how: a skills repository

Without instructions, each run builds the data product in a different way. A skills repository gives the agent your conventions, templates, and policies. For example, it tells the agent how to use dbt on Snowflake, how to test, and which security rules apply.

Slide 4 of Entropy Explains episode 4, The data to build on: Data products use other data products.

The data to build on

Data products use other data products. You find them in a data marketplace. The agent uses a CLI to search the data marketplace and to read the data contracts. When it needs data that it cannot read yet, it requests access.

Slide 5 of Entropy Explains episode 4, One prompt, then tests: Then you give one prompt to the agent.

One prompt, then tests

Then you give one prompt to the agent. The agent checks its own work with the Data Contract CLI. It changes the code until all tests pass. After about five minutes, the data product is built and tested.

Slide 6 of Entropy Explains episode 4, Change the contract first: The new data product goes back into the data marketplace, so other data products can use it.

Change the contract first

The new data product goes back into the data marketplace, so other data products can use it. To change the data product, you change the data contract first. Here, the data contract gets a new impact score, and the agent builds it with the same prompt.

Slide 7 of Entropy Explains episode 4, Humans define the what: The prompt, the skills, and the data marketplace stay the same.

Humans define the what

The prompt, the skills, and the data marketplace stay the same. The human work moves to the data contract and to the check of the result. The agent builds the data product. People decide what the data product must deliver, and they check that the result is correct.

Read more