The symptom
Dashboards are built on the assumption that if you can see a problem, you can manage it. The symptom this article examines is the specific failure of that assumption: a dashboard that shows green while the thing it is meant to represent is visibly, measurably, going wrong by another route entirely. This is not a case of a dashboard being technically broken. The data displayed is usually accurate, in the narrow sense that it correctly reports what was collected. The gap lies between what was collected and what actually needed to be known, and a dashboard, however well designed, cannot show what it was never given.
The recognisable form of this symptom is two internally consistent but contradictory signals sitting side by side, both correct, neither explaining the other. It is a more unsettling version of an ordinary missed target, because a missed target at least tells you where to look. Two correct numbers that disagree tell you that somewhere, the measurement itself has a blind spot — and a dashboard, by design, only shows you the numbers it was built to collect, not the blind spot around them.
Why it happens
Steven Kerr’s research on reward systems offers one relevant piece of the mechanism: people respond to what is actually measured and rewarded, not to the intention behind the measurement. Applied to instruments like surveys and dashboards, this extends further — people respond not only to what is measured, but to the conditions under which they believe their response will be used, seen, and potentially attributed back to them. A survey can be well designed, methodologically sound, and still fail to capture honest input if the conditions surrounding it make candour feel costly to the person answering. The instrument is not broken. The environment around it is not one where its intended function — surfacing honest signal — can actually operate.
This is a distinct failure mode from a badly designed survey or a poorly built dashboard, and it is easy to conflate the two. A badly designed instrument produces noisy or meaningless data regardless of who is answering. A well-designed instrument operating in the wrong conditions produces clean, consistent, entirely plausible-looking data that is nonetheless systematically biased — which is considerably harder to detect, because nothing about the output looks wrong on its face.
What leaders typically conclude when two dashboards disagree is that one of the instruments has a flaw — a question worded poorly, a data source that needs auditing. That conclusion is sometimes correct and always worth checking. What the evidence more often points to, particularly when the disagreement is this stark, is not an instrument problem but a conditions problem: the environment in which one of the two measures is collected does not actually support the honest signal the instrument was designed to capture.
What it looks like in practice
Consider an organisation where an engagement survey returns strong, broadly reassuring results across the business. In the same period, attrition is the highest it has been in five years. Leadership’s first explanation is that the survey instrument itself must be flawed — perhaps the questions are not calibrated to catch the real issues, or the survey is simply not a reliable predictor of who actually leaves.
A closer look at how the survey was administered complicates that story. The survey is genuinely anonymous in design, but it is administered and reported by team, and several teams are small enough that an individual’s response pattern feels identifiable to that person, even if it technically is not attributable by anyone reviewing the results. Response rates in the smallest teams — where identifiability concerns would be strongest — are also the lowest, a detail easy to miss if response rate is not examined by team size. The instrument itself is sound. The conditions for candour are simply absent in exactly the places where candid answers would matter most, and the survey, faithfully reporting what people were actually willing to say under those conditions, produces a result that looks reassuring while the reality it was meant to capture is moving in the opposite direction.
This pattern generalises beyond engagement surveys to any dashboard or reporting mechanism that depends on people volunteering honest information about problems, risks or dissatisfaction. The dashboard shows what people are willing to report, filtered through their own judgement about what reporting it will cost them. Where that judgement leans toward caution, the dashboard will look calmer than reality regardless of how well the underlying questions were designed — and no amount of refining the questions will fix a problem that lives in the conditions surrounding the answer, not in the answer’s wording.
The organisational instinct that follows a mismatch like this one is often to add another dashboard — a new instrument, meant to catch what the first one missed. This instinct treats the problem as an information gap, when the actual gap sits in the conditions surrounding disclosure rather than in the range of questions being asked. A second, third or fourth dashboard, layered on top of the first without addressing why candour was scarce in the original one, tends to inherit the same limitation for the same underlying reason, which is one explanation for why organisations sometimes accumulate a large number of reporting mechanisms without a corresponding increase in genuine visibility into the problems those mechanisms were meant to surface.
The same pattern appears across many kinds of organisation; see Other Sectors for how it is read elsewhere.
Questions worth considering
- What would someone need to believe about the consequences of an honest answer before they would actually give one on this particular instrument?
- Where two measures of the same underlying reality disagree, has the disagreement been investigated, or has one simply been assumed to be the more trustworthy of the two?
- Are response rates on any of the organisation’s key feedback instruments examined by team size or group, or only in aggregate?
This perspective is part of Performance.
Your team is facing similar challenges?
Subscribe to The Riddle Ledger
Perspectives are sent as they are published, with no promotional email, and you can unsubscribe at any time.