Your Phishing Reports Aren’t Showing the Whole Story

Ever wonder why your phishing reports feel like they’re missing half the story? Most dashboards just show surface-level numbers, but behind those simple stats is a constant stream of real threats slipping through cracks. Today, I’ll show you how to transform Microsoft Defender data into living dashboards that actually tell you what’s happening in your environment — and what you’re not seeing yet.The Hidden Layer: What Defender Knows That Your Reports Don’tIf you’ve ever looked at your security dashboard and thought, “Looks good to me,” you’re not alone. Execs love a tidy chart—blocked emails, a drop in reported phishing, maybe one or two suspicious sign-ins. It’s comforting, right? But here’s the catch: the data sitting right underneath is almost never as simple as those friendly graphs make it seem. In most orgs, the actual story is far more complicated, largely because those dashboards pull from the same handful of exportable stats. A lot rides on whatever filter you set in your mail flow reports or security tool. Most people stick to what’s easy to get out of Exchange Online or the built-in phishing report from their email provider. If a user flagged something, tick mark. If an email was blocked, bar goes up. End of story—or so it appears.But Microsoft Defender for Office 365 is sitting on a goldmine of details most teams skip over completely. It’s the classic iceberg: everything you show in a regular incident review covers about twenty percent of what actually gets picked up in the background. What Defender captures is almost embarrassingly detailed. It logs every click your users make on links inside emails—even when Safe Links steps in to stop a detonation. It tracks those silent “near miss” moments when a phish was one click away from success. Automated Investigation & Response runs playbooks in the background, picking up on correlated signals your manual review would probably never spot until the situation escalates for real. Most dashboards? They just don’t bother to look under the surface. We all know those emails that get blocked right away get counted, but a targeted attack that blends into a newsletter and is manually reported by one vigilant user? Often lost in the noise.Let’s talk reality for a second. I saw this firsthand last summer. Security had a dashboard that looked flawless—trendline of blocked phishing up, reported incidents down, execs all happy. Meanwhile, a low-volume spear-phishing campaign was targeting the finance team. Defender tagged it with a high severity, ran an automated investigation, and quietly bundled up the event in the backend logs. None of it landed in the weekly cybersecurity summary because nobody was pulling data from the Automated Investigation & Response logs. It wasn’t even a blip for execs until someone got suspicious about a calendar invite. That’s the gap—Defender caught the signal, but the dashboard never showed it.If you crack open Defender’s portal, there are three sources that almost always get left out: Threat Explorer, Automated Investigation & Response, and User Submissions. Threat Explorer is not just a list of threats—it maps relationships between malicious files, sender infrastructure, and user behavior. It tracks attack campaigns, figuring out who else in your org saw the same phish, even if no one reported it. AIR, that’s Automated Investigation & Response, does more than block an obvious threat. It pieces together what your automated policies did: what devices were checked, how compromised accounts were flagged, which mailboxes were scanned for ‘potentially harmful’ content long before a breach is visible to end users. And user submissions—probably the least appreciated signal—layer something valuable on top: human reporting of suspicious items that the filters missed. Defender takes those and sometimes surfaces genuine threats by combining user intel with backend analytics.Research from Microsoft regularly shows data gaps between what’s available in Defender logs and what actually gets piped into exec-facing tools. Even in mature security programs, you’ll see dashboards showing blocked mail totals but skipping over AIR investigations, user-reported near-miss phishes, or campaign mapping data from Threat Explorer. In many tenants, nobody’s wiring up the automated investigation tables to reports at all—it’s an extra export, another click, something to fill next quarter’s backlog. The net effect is that leaders walk into security reviews seeing “zero incidents” when what actually happened is much more complicated. They miss context—what threats got close but were caught at the last second, how many users actually clicked something dangerous before the block, or which attack vectors are being tested by threat actors right now.This isn’t just a technical shortcoming—it’s an awareness problem that can leave the business exposed. Say you’re only catching two out of five signals that matter. Maybe you’ve got blocks and reports—but nothing from AIR or Threat Explorer. Leaders end up believing that the risk is low because those details never make it to the dashboard. But the most useful dashboards surface signals most people miss: who’s being targeted and how often, how employees respond to sophisticated lures, and whether automated policies are actually working or just hiding problems until they escalate.The gap between what Defender knows and what hits the regular reports is bigger than most orgs think. Those glossy, high-level metrics end up creating a kind of invisible shield where executive teams assume their controls are better than they are. And all the while, the real signals—those near-misses, automated investigation results, and full campaign data—get lost in the shuffle because nobody wired them into the story. So if all this data is right there in Defender, what’s stopping us from using it? The answer: almost no one is building frameworks that take advantage of it. That’s what needs to change, and that’s exactly what I want to get into next.Beyond the One-Off: Building a Repeatable Security Dashboard FrameworkIf you’ve ever watched your shiny new dashboard fall apart the moment Microsoft Defender changes a field name, you already know how fragile these setups really are. Teams get excited, spin up Power BI, connect to that first export, and within a week they’ve got a handful of pretty charts. Job done—for now. But fast forward to the next Defender update, or worse, the next round of phishing attacks using totally new lures and attacker infrastructure. Suddenly columns are missing, charts break, and the data just doesn’t line up. The reality is, it’s straightforward to pull a phishing summary for this month, but building something that adapts to whatever the threat landscape throws at you? That’s where most dashboards fall flat.We’ve all been there: your team spends hours every quarter scrambling through spreadsheets, manually fixing broken queries and swapping in new attack types that didn’t exist when you built the last report. Someone pulls an export from AIR, another from Threat Explorer, and now you’ve got two sources that don’t even speak the same language. In the background, Defender itself is updating; Microsoft tweaks schemas, new API endpoints arrive, and suddenly all those beautiful visuals are out of sync. If your dashboards rely on manual steps and one-off metrics, you’re not just chasing attackers—you’re chasing your own tools.That cycle happens because most orgs treat dashboards like fixed artifacts, not living systems. We see a lot of patchwork: tables copied out of Excel, mismatched metrics stitched together, and visuals meant to impress more than inform. The result? Dashboards that tell you what happened last month, but can’t keep up with what’s happening now because they break every time Defender evolves. When executive reporting time comes, teams rush to update everything by hand because automation was always “tomorrow’s problem.” It’s familiar, but it’s also kind of exhausting. And risky.This is where the idea of a dashboard framework comes in—a repeatable, modular system that’s designed to connect to the real Defender data, model how everything relates, and standardize the critical metrics that actually indicate risk. A real framework isn’t a template you download and forget about. Instead, it’s a collection of core building blocks: reliable connectors that pull Defender’s freshest data automatically, a resilient model that adapts when the source data structure shifts, a shortlist of KPIs that matter for threat response, and flexible visuals focused on what matters most, not just what looks pretty.Let’s break that down. First, reliable data connectors. Too many teams grab a CSV from the portal, build out a dashboard, and call it a day. Until next week, when they need a new CSV. Instead, you want direct connections—using Defender’s API, set up in a way that survives authentication changes and schema updates. Power BI’s connectors can do this, but only if you invest the time upfront to map how each table and field relates to real threat signals.Second, that resilient data model. Think of all the ways Defender can adjust its logging—new columns, renamed fields, sudden additions for a brand-new detection policy. If all you’ve got is a pile of flat tables, every change is a ticket to go fix broken dashboards. But if your model relates incidents, users, mailboxes, devices, and actions in a unified schema, Defender’s tweaks don’t derail your narrative. Microsoft’s own security ops guidance pushes this approach: invest first in structuring your data before painting any visuals.Third, prioritized KPIs. Not all metrics deserve equal attention. Executive teams don’t need ten flavors of “email blocked.” What they want: time to incident resolution, users clicking on threats, high-risk accounts targeted repeatedly, and which attack vectors got closest to succeeding. Defining these KPIs up front, based on both operationa

Become a supporter of this podcast: https://www.spreaker.com/podcast/m365-fm-modern-work-security-and-productivity-with-microsoft-365–6704921/support.

If this clashes with how you’ve seen it play out, I’m always curious. I use LinkedIn for the back-and-forth.