I built the tool .. but forgot the skill..

So.. here is a train of thoughts…

I’ve been doing this telemetry stuff for a while now. KQL snippets, ADE (or what I recently learned: the real abbreviation is “ADX” ) dashboards, the Troubleshooting Buddy (with Dmitry) (where it all started to go into an interesting direction) and of course now BC Telemetry Buddy. Of course..

At some point I kind of thought: yeah, I’ve figured this shit out.

But then I started prepping my sessions for Days of Knowledge in Birmingham .. and something clicked. Or rather: something annoyed me. Because the more I thought about what I wanted to tell people, the more I realized that the quality of the questions really do make for better answers. And that BC Telemetry Buddy – as much as I love it – might not be quite there yet.

So now I’m thinking.. I still have shit to figure out.

Because I keep struggling with people asking the wrong questions – or .. not being critical enough. They have Application Insights, have dashboards, have AI… and still spend hours just figuring out where to even start .. or just believe anything the AI spits out.

So, in a way, the access problem has been solved. Not the thinking problem.

AI made this painfully obvious. Ask “is my system slow?” and you’ll get a beautiful, generic answer, without too much depth and/or full of assumptions that sounds so believable that you have the tendency to just go with it!
Ask “Give me a detailed report of the change in SQL statement durations over the past 7 days, and see if certain patterns in SQL statements or callstacks stand out. Explain like I’m a junior.” and now we are somewhere.

Same tools. Completely different result.

The variable isn’t the tooling. It’s the question.

BC Telemetry Buddy is great, but if you feed it vague nonsense, it’ll confidently hand you vague nonsense back. Or in other words – AI scales your thinking.. good AND bad. So if your team was bad at asking questions before… well. Did we solve anything?

The other angle to this: not being critical enough about what comes back.

Because when BC Telemetry Buddy – or any AI for that matter – returns what looks like a detailed, structured analysis.. it’s tempting to just.. accept it. It looks smart. It uses the right words. It has bullet points and everything!

Or .. what Tine said (something in the lines of): AI doesn’t hallucinate 20% of the time – it’s hallucinating 100% of the time, and happens to be right 80% of it.

So here’s the uncomfortable truth: to use AI well on telemetry, you still need to know telemetry. Not KQL necessarily. But enough to “smell” when something doesn’t add up. Enough to say “wait, that conclusion doesn’t match what I know about this customer.”

When I’m doing an analysis, I pretty much always challenge everything. Like:

• “Prove with SQL Statements and/or callstacks“
• “Create a decent KQL so I can look for myself“
• “Read this analysis, and check with new telemetry data if it’s true”
• “Explain it like i’m five“
• …

And that’s what’s been on my mind lately. And I have been wondering what to do …

Like right now BCTB is great at answering questions. But what if it got better at suggesting them? What if it nudged you towards the things you should be asking – even if you didn’t know to ask? What if it could figure out (based on your first question) the different paths to investigate, give you a choice, and suggest depth and direction along the way?

And on the critical side – what if it didn’t just hand over an answer, but challenged you a bit? “Does this conclusion make sense given what you know about this customer?” “Here’s what I assumed – is that correct?” Something that forces a moment of reflection instead of just producing output and calling it a day.

I don’t know if any of this would even make sense, or be possible for that matter. But it IS something to think about. It reminds me of our session at TechDays where Dmitry introduced the first step of “rephrase the question“. Maybe BCTB should spend more time understanding what the user actually wants, based on the first question, may be even based on the data in the system, before diving into the problem. And maybe it should spend more time making sure the user actually validates what comes out.

I don’t know. Something to think about, I guess.. feedback and comments always appreciated

this train of thought came to happen while waiting for my delayed flight .. in a conversation with Claude (you might know .. it) ..
Claude also helped me draft some of the text – sorry not sorry

Original Post https://waldo.be/2026/03/28/i-built-the-tool-but-forgot-the-skill/