Field report 13 · Operations and observability
When the dashboard goes quiet: observability needs cross-checks
How to tell that monitoring actually monitors, and why “green” is not proof.
- Status
- From practice Experience from our own operations.
- Published
- Publisher
- SIMO GmbH, Aschaffenburg, Germany
Starting questionHow do we know that our monitoring actually monitors?
What we experienced
While building 56 charts across five dashboards, one central chart stayed empty. The metric had been renamed in the version in use, and the query reported success with zero results.
Elsewhere, the bot protection of the upstream network service injected challenge pages into data requests. It looked like a broken backend.
What it means for business architecture
“Green” is not proof. Anyone using metrics for decisions needs a check against the actual data source, not against the documentation.
Learnings
- “Successful but empty” is the most treacherous state. It has to count as an error.
- Check metric names against the live interface instead of copying them from examples.
- Several protective layers can get in each other's way. When something fails, check the upstream layer first.
- monitoring
- observability
- metrics
- operations