The Taps Metaphor
Background
We recently had our bathroom renovated and the number of decisions you have to make is overwhelming!! Particularly about everyday items you previously took for granted. Take taps for instance, who knew the world of taps was so vast!
This got me thinking about what's important. Imagine the scenario where you have two sinks:
Sink A with world-class, high-pressure plumbing connected to a cheap, wobbly plastic tap that drips and is stiff to turn.
Sink B with perfectly adequate plumbing connected to a beautiful weighted chrome waterfall tap which releases a stream of water at an easy-to-control temperature and perfect pressure.
If it was your bathroom, which one would you choose?
The Data Plumber
This metaphor is a good one for the world of data teams. The plumber is the data engineer. A highly skilled individual focused on the back end. They look after ingestion from the data source, transformation of the source data, storage and warehousing, and all and any optimisation required. It must be engineered well enough so it works at-scale where the amount of data is very large. It must be engineered in a way that it is repeatable without breaking so it works day after day, time after time. Their work is highly complex as they must also cater for the universe of all possibilities and edge cases and, for the most part, it is all hidden from view.
Quite rightly, they take pride in being able to achieve all this with the end result a series of data pipelines that don't leak.
The Tap Whisperer
The tap whisperer is the craftsperson focused on the last-mile. Where the data leaves the warehouse and meets the user. Their scope is the semantic model, UI and UX considerations, report clarity, robustness and usability and the performance of front-end metrics such as DAX.
Their pride comes from building an intuitive, trustworthy user interface that is an enjoyable experience for the end-user.
When Plumbers Choose Taps
In describing the two different roles, it is clear that there is a separation of skillset and focus between them. And as long as that remains, the system is harmonious and the data product created will remain high quality.
However there can be instances when the data engineer, who thinks in terms of data pipelines and logic struggles to understand the experience from the user's point of view.
I saw this play out recently. We had a column chart where each column represented a month of the year and the customer had fed back that they wanted a dashed average line be added to the visual. The immediate reaction from the data engineers in the room was bewilderment: "Why? There is already a card above the chart which displays the average."
Whereas I heard the (unsaid) user's plea: "It would be so much easier and I wouldn't have to think at all if I had a line representing the average on the chart itself making comparisons between numbers instant and unmissable"
The data engineers were thinking like plumbers! The data was already delivered, the pipe had done its job and the number already existed in the report.
But the end user was thinking like someone at the sink. They didn't want the cognitive load of having to hold the number in memory and compare it to each of the numbers the columns were displaying. They wanted the benchmark integrated into the visualisation itself. They wanted to be able to easily turn the tap and have warm water instantly flow. They were asking for a better experience.
The plumbers saw redundancy, the end-user wanted a more joined up experience.
This small story highlights the danger and the core disconnect. The team was focused on the efficiency of the pipework (avoiding redundancy) while completely missing the usability of the stiff, old tap (reducing cognitive load).
If you let plumbers design taps, you get a system that is logically sound but humanly frustrating.
The Bottom Line
From the user's point of view the tap is the product. It is the entire system and they don't see or understand the plumbing. Their trust and satisfaction in the data product relies completely on their interaction with the tap. If the tap is a joy to use and gives them what they want when they want it, the feedback for the product will be positive. If it squeaks or is hard to turn or has low-pressure the product will receive complaints or have low adoption.
Millions can be spent on data infrastructure, but if the tap is bad, the return on investment is zero. Users will abandon the tool.
And go back to Excel, probably!
The Solution: Hire a Tap Whisperer
Just as you wouldn't ask a master plumber to design a luxury tap, you shouldn't expect a data engineer to be a master of front-end craft. They are distinct and equally valuable disciplines.
There is a need for more recognition for the front-end craftsperson. The analytics engineer or data visualisation engineer who is able to craft a user-centric experience that reduces friction and brings joy to the user. This person acts as the translator between the complex world of data engineering and the practical world of the users. Their goal is to ensure the incredible power of the plumbing is delivered through a flawless, intuitive interface.
User Experience is King
To keep users happy, to keep them engaged and coming back for more, we have to celebrate the taps more. Yes the invisible work of the data engineer is a wonder, no doubt, but the work doesn't stop once the data mess has been streamlined into the database / data warehouse. The magic wand should then be passed on to the front end expert to wield their version of magic on the user interface.
The Final Turn of the Tap
The most intricate and sophisticated data pipeline in the world is a failure if the user can't get a clean drink of water from it. Let's design for thirst, not just for the pipe.