Declarative pipelines: the hidden metric behind Gen Z's physical media revival

The Trend Nobody's Store Can Confirm

Vinyl, CDs and DVDs have been climbing for years. Most retailers can't say why, because their reporting was never built to ask.

For over a decade the metrics pointed towards streaming. Music became Spotify. Film, TV and series became Netflix. Shelves once stacked with CDs and DVDs slowly emptied out, at home and on the high street.

Then the numbers stopped cooperating with that story. Vinyl sales kept climbing. CDs quietly came back. Even DVDs and Blu-rays, written off years ago, started finding buyers again. Industry reports increasingly point to Gen Z, a generation raised entirely on subscriptions, are apparently now drawn to ownership, scarcity and something they can physically hold.

That's an interesting trending topic. It's also not a very useful one on its own, because it raises a far more specific question: how many individual shops could actually confirm what's happening with their own customers?

Most can't. Their reporting tells them what sold. It has never told them who bought it, not because the information wasn't worth having, but because nobody thought to ask that question when the reporting was first built.

Why This Isn't a Curiosity Problem

Every shop owner has a hunch about their customers. What's usually missing isn't the question, it's a way to afford asking it.

Testing a new question against existing data almost always means new engineering work, a new ingestion process, a new dependency wired in by hand, someone signing off on the time it'll take. That's true whether the business is a record shop or a logistics firm. None of it is difficult in isolation. It's just enough friction that most organisations only take it on once they're already confident the answer is worth having, and confidence like that is hard to justify for a question nobody's tested yet.

That's the actual reason most promising questions never get built. Not a lack of imagination. A cost nobody wants to carry for something that might reveal nothing at all.

What the Scout Bee Gets Right

Around 80% of a honeybee colony's foraging trips follow the waggle dance, directions back to a source another bee already found. The other 20% search on their own, and most come back empty.

It's an analogy the advertising strategist Rory Sutherland has returned to often, in his book Alchemy and in talks on decision-making under uncertainty. Judged purely on efficiency, that 20% looks like waste. A colony that spent every trip chasing certainty would look leaner on paper, right up until the flowers it already knew about disappeared.

What This Looks Like for an Actual Shop

A record shop notices sales climbing across every format and has no way to confirm whether the Gen Z story applies to its own customers.

So, it launches a simple student discount scheme. Not because the discount itself changes much, but because every sign-up captures something the till never has: an age bracket, a reason to come back, a growing picture of who's walking through the door.

Once that data exists, genuinely different questions become answerable. Are younger customers behind the increase, or is it the same regulars simply buying more? Which format brings someone in first? Do vinyl buyers eventually come back for Blu-rays? None of those questions could be answered yesterday. Today they're just another table, sitting alongside the sales report that was already running.

Why Lakeflow Declarative Pipelines Changes the Outcome

A hand-built pipeline makes every new question expensive. Lakeflow Declarative Pipelines makes most of that cost disappear.

In a traditionally orchestrated pipeline, adding a new data source such as loyalty scheme sign-ups often means writing new ingestion logic, manually managing execution dependencies, and introducing another workflow that someone has to build, monitor and maintain. That's real engineering time spent before anyone knows if the question is worth answering.

A declarative pipeline inverts that. You describe the table you want, a materialized view segmenting sales by age bracket and visit frequency, for example, and the platform resolves the dependency graph, the execution order, the retries and the incremental updates on its own. Lakeflow Declarative Pipelines, Databricks' managed implementation of Spark Declarative Pipelines (originally built as DLT, and later donated to open source Apache Spark), works exactly this way: you declare what should exist, and it works out how. This is the same medallion-architecture thinking behind a properly built data warehouse, just applied to a single new question instead of the whole platform.

The practical effect is that a new customer attribute stops being a mini rebuild of the warehouse. It becomes one more table dropped in alongside everything already running. Some of those new tables need to update the moment new data lands rather than waiting for the next scheduled run, which is its own decision, and its own service, separate from the pipeline question itself.

Two Shops, Same Growth, Different Outcome

Two record shops see identical increases in physical media sales. Only one of them finds out why.

The first keeps reporting revenue by category, exactly as it always has. Sales are up, the numbers are accurate, nobody's questioning the report. The second adds the loyalty scheme and extends its pipeline with a handful of new tables. Six months later, it knows something the first shop doesn't, not just that sales rose, but which customers are actually behind it and what they're likely to buy next.

That shop starts a student night. Adjusts what it stocks. Tinkers with the marketing to see what actually lands. And because the pipeline is already built to make new tables cheap, each of those changes gets measured the same way the original hunch did, another table, another dependency the platform resolves on its own, tracking whether the student night actually brought people back or the new stock actually sold. The scouting doesn't stop once the first question gets answered. It just moves on to the next one.

From the outside, it looks like a sharp strategic call. It wasn't. It was simply cheaper for that shop to stay curious.