BC Telemetry Buddy – 84 commits later..

I did a thing … well .. I guess .. I kept doing things ..

If you haven’t read my previous post, I recommend to do so: Analyzing BC Telemetry with AI with the “BC Telemetry Buddy”. Consider this one the be the “sequel”

TL;DR

In the last 4 weeks, my BC Telemetry Buddy went from “nice demo” to “this might actually be getting somewhere”.

Roughly 84 commits, 25 release-bumps later, we’re at something that I’d say is mature and useable.

What did I do ..

Architecture Transformation from “bundled” to “modular”

Remember when the MCP was “glued” inside the extension and you had to install everything, even if you just wanted to run a query? When it only worked for the chat in VSCode (pretty much only for Github Copilot)?

Yeah… that’s gone.

With v1.0.0, I ripped the whole thing apart (in a good way):

The extension now runs completely standalone: commands & KQL go straight to Azure/Kusto. No MCP, no extra process, just VS Code and your telemetry.
The MCP server is now a separate, optional NPM package (bc-telemetry-buddy-mcp) purely for chat/AI scenarios. KB tested it – it should work .

So now you have the choice on where to use the MCP, which client (or that’s the theory – I’m not going to claim I tested all clients). And if you have KQL to run separately, the VSCode extension will take the same settings, and just run it for you. I don’t think too many people will do that – but it’s a nice gadget ;-).

Do take into mind – the intent is still the same: You create a folder with settings, that folder acts as a workspace to create, save, update, follow up, … on any Telemetry analysis you wish for.. .

Discovery-first approach

Business Central telemetry is amazing… and also a bit of a jungle. Different tables, 2000+ events, customDimensions that behave like they’re allergic to schemas… you get the idea.

So I wanted the “BC Telemetry Buddy” to do more of a “discovery first, query later” kind of approach:

An Event Catalog lets you browse event IDs (like RT0001, RT0005, …) with descriptions, Learn links, and frequency. So .. less guessing .. that’s what I’m aiming for.
Schema discovery tools sample your actual telemetry to show which fields really exist, their types, and example values – before you write a single line of KQL.
From there, you get ready-made KQL patterns with proper type conversion instead of “convert everything to string and hope for the best”.

The initial version of the buddy had this approach already, but this new one should apply it more consistently .. well .. If there are necessary improvement necessary, the “framework” is there to do so ..

The expert

When I learned about the Chat Participant and the Chatmodes – I just knew I had to do something with that.

The initial version of the buddy already had a “poor man’s” implementation. Now .. It’s better! And I intend to improve it along the next releases as well.. .

The chat participant “@bc-telemetry-buddy” is now a little 13KB “expert brain” that knows:

When you just want data:
@bc-telemetry-buddy show me errors for CustomerX today
→ It runs the right query, returns a nice table, no questions asked.
When you want analysis:
@bc-telemetry-buddy analyze CustomerX’s slow SQL problems
→ It discovers events (like RT0005), looks at patterns, runs multiple queries, and comes back with insights.

It should understand multiple customers/profiles, maps company names to tenant IDs, and only asks for confirmation when things are genuinely ambiguous.

On top of that, there’s a performance-focused chatmode that goes full detective: it hunts for deadlocks, timeouts, slow SQL, builds an analysis document and even suggests charts. To be honest, this chatmode is just a start – and the intention is to make it much more of an expert than it already is.. .

Do understand, there is a difference between a chatmode and the chat participant. The chatmode is an “Agent” you can activate from the dropdown in the chat:

The Chatparticipant is basically an “expert” you can talk to from whatever agent:

How to use them? Well, I use “@bc-telemetry-buddy” for a quick question where I know I want to dive into telemetry”. If I want to go deep, create charts, follow up, … I use the chatmode. And yes – I also make the buddy remind you of that

Setup Wizard & Multi-Profiles

Feedback that I got was: hey, it’s easy to set up, but in my case, I need to be able to access multiple endpoints, because customers have their own endpoint.

So, I changed it a bit, and although I realize that there is still room for improvement, this is what we have now:

A setup wizard that has updated prerequisites
Support for having more than one endpoint in one config.

So when you start fresh, just run through the Setup Wizard, and it should guide you:

I won’t go through the entire wizard, but what is the significant change here is the second tab, where I start with two examples: a single profile, and a multiple profile.

Quite honestly, what I do to make it easier on me, is to simply copy the example, create a file, fill in all details, and paste the result back into the wizard.

Future versions should make this even more easy.

One attempt to improve to handle multiple profiles, is this command in VSCode:

Try it out – hope it helps ;-).

The agent is aware of the multiple profile (there is an extra tool for that), and you can simply mention the profile in your prompt to analyze against it .. .

Usage Telemetry

At some point I had to ask myself: are people actually using it in the wild? How can I improve? Are there many issues?

So in v1.2.0+, the extension started sending anonymous usage telemetry to a separate Application Insights resource (not your BC telemetry): extension lifecycle events, commands used, performance, error signals – but no KQL queries, no business data, no PII, and it fully respects VS Code’s telemetry setting.

A few fun takeaways from the first weeks:

10am is the “Power Hour”: most people are investigating Telemetry in the morning
Get_event_catalog has a 27% failure rate – need to see where that is coming from, because it doesn’t fail on my end, so I’m hoping users will report their story!
20 continuous hours of activity .. it really lives out there!
Belgium (probably me) is battling Germany for first place Most usage in Germany though

I would like to stress here – GDPR and all PII compliance is completely maintained here. I created a chapter that describes the Usage Telemetry and Privacy:

https://github.com/waldo1001/waldo.BCTelemetryBuddy/blob/main/docs/UserGuide.md#telemetry–privacy

Product hardening

Version 0.1.0 was a cute proof of concept. Version 1.2.9 is something I’m comfortable telling partners to use in their daily Telemetry story.

Behind the scenes:

A few hundred tests cover extension, MCP server and shared logic, with overall coverage somewhere in the 80% range – and CI refuses to merge if I drop below a minimum threshold. I’m planning to up this threshold every time I see hallucinations …
I test intent detection, profile switching, setup wizard behavior, auth flows, MCP tools, caching, config discovery… basically all the painful edge cases I’ve personally hit.

Current status about the tests:

737 tests covering critical paths
84.87% overall code coverage
1.01:1 test-to-code ratio
100% function coverage in MCP
Fast CI/CD (7s test execution)
Zero failing tests

Some of the more “fun” bug stories:

Logging from the MCP server to stdout (instead of stderr) broke the JSON-RPC stream for some clients .. “some” ..
Config loading got more robust: it now honours workspace paths, environment variables and the expected discovery order, instead of occasionally pretending your config doesn’t exist. I can tell you – the config-file stuff in some kind of MCP that needs to work on generic chat agents .. oh-dear-lord .. Is THAT complicated .

With two separate components (extension + MCP) in one repo, doing releases manually was… let’s call it “educational”.

So I wired everything up:

GitHub Actions handle CI: tests on multiple OSes and Node versions, security scans, and build checks for every push and PR. Thanks Copilot, for getting me going on this
Separate release pipelines publish:
- the extension to the VS Code Marketplace,
- the MCP server to NPM as bc-telemetry-buddy-mcp.

A small PowerShell script ties it together: bump version, update changelog, create tags, push – and CI takes over from there.

And .. I just informed the copilot-instructions on how I want a release to happen .. so when releasing, all I need to do .. is ask copilot :-).

The nice side effect: shipping small, frequent releases is now boringly safe. In this 4-week sprint alone, that meant well over a dozen extension releases and about ten MCP releases – including a few same-day hotfixes when telemetry showed something odd.

Conclusion

Since the initial version I previously blogged about, BC Telemetry Buddy went from a cool idea with a lot of potential to a modular, test-covered, telemetry-driven, AI-powered tool that’s actually holding up in real-life partner scenarios.

Next up: better visualizations, more specialized chat flows, and .. of course .. whatever crazy ideas come out of the telemetry and your feedback.

My next blogpost will drill more into some use cases that I’m applying in the real world. Yes, I’m using this pretty much on a daily basis, helping our partners, our consultants and our developers with insights to improve their and our software!

It’s open source. If you have ideas, issues, or just want to lurk through the code .. knock yourself out