The Hidden Problem with AI Agents: Too Much LLM, Not Enough Engineering with Karthikeyan VK (MVP)

Artificial Intelligence is moving faster than almost any technology wave we have seen before. Every week brings new models, new copilots, new frameworks, new AI agents, and endless promises about autonomous systems replacing repetitive work across the enterprise. But beneath all the hype lies a deeper engineering problem. Too many organizations are building AI systems with Large Language Models at the center of everything — while completely ignoring architecture, orchestration, state management, observability, governance, and deterministic engineering principles. In this episode of the m365.fm podcast, Mirko Peters sits down with Microsoft AI MVP, CTO, international speaker, and author Karthikeyan VK to discuss one of the most important realities of enterprise AI today: why most AI agent architectures are fundamentally flawed from an engineering perspective. This conversation goes far beyond AI hype and dives deep into what actually matters when building scalable, reliable, enterprise-grade AI systems with Microsoft Azure AI Foundry, orchestration patterns, memory management, evaluation pipelines, multi-agent architectures, and domain-specific AI solutions.

WHY MOST AI AGENTS ARE BUILT WRONG

According to Karthikeyan, one of the biggest mistakes organizations make today is trying to use Large Language Models for everything. Instead of treating the LLM as a reasoning engine or orchestration layer, many teams try to make the model itself perform every business operation directly. The result is often a probabilistic system attempting to replace deterministic engineering. And that creates serious reliability problems. Karthikeyan explains that enterprise systems cannot behave unpredictably. If an AI system returns different results for the same financial transaction, customer workflow, or approval process, organizations immediately lose trust. That is why AI agents must still be engineered like traditional enterprise software systems — with architecture, orchestration, retries, validation, observability, and governance built into the foundation. 

THE REAL ROLE OF LLMs IN ENTERPRISE SYSTEMS

One of the strongest insights from the episode is the distinction between probabilistic and deterministic systems. Large Language Models are probabilistic by nature. They generate outputs based on probability distributions, context windows, and token prediction patterns. Enterprise workflows, however, are often deterministic:

• Financial calculations
• Inventory management
• Identity systems
• Compliance workflows
• ERP integrations
• Security processes

According to Karthikeyan, organizations should stop trying to make LLMs replace deterministic engineering logic. Instead:

• The LLM should act as the reasoning layer
• Deterministic tools should execute workflows
• Business logic should remain controlled
• Orchestration should drive execution
• Validation should happen continuously

This architectural mindset dramatically improves reliability and scalability.

WHY ORCHESTRATION IS THE REAL SECRET

One of the biggest missing components in enterprise AI systems today is orchestration. Karthikeyan explains that many organizations simply connect an LLM to a chatbot framework and assume they have built an AI agent platform. But real enterprise systems require orchestration patterns. For example:

• Which tools should execute first?
• Which workflows run in parallel?
• Which actions require validation?
• Which systems are allowed to be called?
• Which failures require retries?

Without orchestration, AI systems become unreliable and difficult to scale. The intelligence lies in:

• Tool orchestration
• Workflow selection
• Context awareness
• State management
• Evaluation logic
• Memory handling

This distinction becomes critical when organizations attempt to move AI systems from proof-of-concept into production environments.

MEMORY MANAGEMENT IS MORE IMPORTANT THAN PEOPLE REALIZE

Another major focus of the episode is memory handling inside AI systems. Most users do not realize that every conversation with an LLM becomes a growing token context window. As conversations grow:

• Token costs increase
• Latency increases
• Context quality degrades
• Important information gets lost
• Systems hallucinate more easily

Karthikeyan explains that enterprises must actively engineer memory strategies:

• Session memory
• Persistent memory
• Conversation summarization
• Context compression
• State tracking
• Token optimization

Without proper memory engineering, AI systems eventually lose reliability.

THE BIGGEST PROBLEM: LACK OF OBSERVABILITY

One of the strongest warnings throughout the discussion is around observability. Many AI systems today cannot explain:

• Why decisions were made
• Which tools were called
• Which prompts executed
• Which memory state existed
• Which reasoning path was taken

This creates major problems in enterprise environments where debugging, compliance, and traceability are essential. Karthikeyan strongly recommends tracing reasoning paths, tracking memory states, monitoring token usage, evaluating decision quality, and building proper debugging dashboards from day one. Without observability, enterprise AI becomes impossible to operate safely at scale.

WHY AZURE AI FOUNDRY MATTERS

A major part of the discussion focuses on Microsoft Azure AI Foundry and why Karthikeyan sees it as one of Microsoft’s strongest AI platform evolutions so far. According to him, Foundry solves several foundational AI engineering challenges by providing:

• Built-in orchestration
• Evaluation pipelines
• Governance tooling
• Memory handling
• Observability features
• Secure enterprise integration

He explains that Azure AI Foundry is not just another AI toolset — it represents Microsoft’s shift toward becoming a true enterprise AI platform provider.

Become a supporter of this podcast: https://www.spreaker.com/podcast/m365-fm-modern-work-security-and-productivity-with-microsoft-365–6704921/support.