The Billion-Vector Problem: HNSW vs. DiskANN in Azure AI Search

Most architects default to HNSW because it’s the industry standard. It’s the algorithm used by most vector databases, the one featured in tutorials, and the option many teams deploy without a second thought.For small and medium-sized workloads, that’s often the right decision.But at enterprise scale, a hidden problem begins to emerge.The moment organizations start dealing with hundreds of millions—or even billions—of embeddings, the economics of vector search change dramatically. What looked like a straightforward architectural decision suddenly becomes a conversation about infrastructure budgets, memory consumption, scalability, and long-term sustainability.In this episode of the M365 FM Podcast, we explore one of the most important design decisions facing enterprise AI architects today: when should you use HNSW, and when does DiskANN become the better option?More importantly, we examine how this decision impacts Azure AI Search, Azure Cosmos DB, Microsoft 365 Copilot-style architectures, Retrieval-Augmented Generation (RAG) systems, and the future of large-scale enterprise search.
WHY VECTOR SEARCH CHANGES EVERYTHING

Traditional search systems rely on keywords. They look for exact matches between a query and the words stored inside documents. While this approach works reasonably well for structured content, it struggles when users describe concepts differently than the documents themselves.Vector search solves this challenge by converting both documents and queries into embeddings—high-dimensional numerical representations of meaning. Instead of searching for matching words, vector databases search for semantic similarity.This is the foundation of modern AI-powered search experiences, enterprise copilots, and Retrieval-Augmented Generation systems. It allows users to find information based on intent rather than exact terminology, dramatically improving discovery across large knowledge repositories.
THE REAL CHALLENGE ISN’T SEARCH—IT’S SCALE

Most conversations about vector search focus on retrieval quality, embeddings, and similarity algorithms.Far fewer discussions focus on the infrastructure required to make those searches happen.Every vector must be stored somewhere. Every nearest-neighbor calculation requires an index. Every index consumes resources.At smaller scales, those requirements are manageable.At enterprise scale, they become the dominant factor in architectural decisions.The episode explores how the physical location of your vector index—whether it lives entirely in memory or partially on disk—ultimately determines the economics of large-scale AI systems. This seemingly technical distinction becomes one of the most important variables affecting cloud costs, scalability, and long-term platform viability.
UNDERSTANDING HNSW

Hierarchical Navigable Small World (HNSW) has become the gold standard for approximate nearest neighbor search.The algorithm uses a sophisticated graph structure that enables extremely fast vector retrieval with impressive recall rates. By organizing vectors into interconnected layers, HNSW can navigate large vector spaces with remarkable efficiency.Its strengths are easy to understand:

• Extremely low latency
• Excellent recall quality
• Mature ecosystem support
• Broad industry adoption

For small and medium-sized vector workloads, HNSW remains one of the best options available.However, the algorithm is built around a critical assumption: the entire graph must remain in memory.That assumption becomes increasingly expensive as datasets grow. What begins as a performance advantage eventually becomes a scalability challenge, particularly when organizations move into the hundreds of millions of vectors.
THE HNSW MEMORY WALL

One of the most eye-opening discussions in this episode focuses on what happens when vector indexes reach massive scale.Memory consumption grows alongside the graph, and eventually organizations encounter what many architects now call the memory wall.At this point, infrastructure requirements shift from ordinary compute resources to specialized memory-optimized environments. Replication, disaster recovery, regional deployments, and high-availability architectures multiply those requirements even further.The result is that an algorithm originally selected for performance can eventually become one of the largest cost drivers within an AI platform.This isn’t a failure of HNSW.It’s simply a consequence of the architectural assumptions that made HNSW successful in the first place.
ENTER DISKANN

DiskANN was developed by Microsoft Research to address the scaling limitations associated with memory-heavy vector search architectures.Rather than keeping the entire graph in RAM, DiskANN uses a hybrid approach that combines memory-resident navigation structures with SSD-based storage for full-precision verification.The result is a system capable of maintaining high retrieval quality while dramatically reducing memory requirements.This architectural shift fundamentally changes the economics of large-scale vector search.Instead of paying premium prices for massive memory footprints, organizations can leverage significantly cheaper SSD storage while still delivering enterprise-grade search experiences.DiskANN wasn’t created because HNSW stopped working.It was created because enterprise-scale workloads eventually outgrow the assumptions that HNSW depends upon.
DISKANN INSIDE THE MICROSOFT ECOSYSTEM

One of the most fascinating parts of the discussion explores where DiskANN appears across Microsoft’s broader AI portfolio.The technology powers several large-scale Microsoft services and plays a key role in enabling semantic retrieval at massive scale.We examine how DiskANN is implemented within:

• Azure Cosmos DB
• SQL Server Vector Search
• Azure AI Search architectures
• Microsoft 365 Copilot-scale retrieval systems

Understanding these implementation patterns provides valuable insights into how Microsoft itself approaches large-scale retrieval challenges and why certain architectural recommendations continue to evolve.
COST, LATENCY, AND THE ENTERPRISE TRADE-OFF

One of the central themes throughout the episode is that architecture is ultimately about trade-offs.HNSW offers extraordinary speed and simplicity for workloads that comfortably fit within memory constraints.DiskANN introduces slightly higher retrieval latency while dramatically reducing infrastructure requirements.The key question isn’t which algorithm is universally better.The key question is which algorithm aligns best with your workload.Factors discussed include:

• Dataset size
• Growth projections
• Update frequency
• Latency requirements
• Infrastructure budgets
• Multi-region deployments
• Compliance requirements

By evaluating these variables together, architects can make decisions based on long-term operational realities rather than short-term benchmarks.
RAG, HYBRID SEARCH, AND RETRIEVAL QUALITY

The conversation also explores how vector indexing choices fit into modern Retrieval-Augmented Generation architectures.A critical takeaway is that retrieval quality depends on far more than the underlying ANN algorithm.Chunking strategies, metadata design, hybrid retrieval pipelines, reranking models, and evaluation frameworks all play a larger role in overall answer quality than most organizations realize.Whether you’re using HNSW or DiskANN, the surrounding retrieval architecture ultimately determines whether your AI assistant delivers accurate answers or confident hallucinations.The discussion highlights why modern enterprise AI systems increasingly combine vector retrieval, keyword search, metadata filtering, semantic reranking, and agentic workflows into a single retrieval pipeline.
MULTI-TENANT AI AND GOVERNANCE AT SCALE

As organizations deploy AI across multiple departments, regions, and business units, governance becomes just as important as performance.This episode examines how retrieval architectures support:

• Departmental isolation
• Security trimming
• Metadata filtering
• Compliance controls
• Multi-tenant AI deployments
• Enterprise-scale governance

These considerations become increasingly important as AI systems move beyond experimentation and become part of everyday business operations.
KEY TAKEAWAYS

The HNSW versus DiskANN discussion is not simply an algorithm comparison.It is a conversation about scale, economics, infrastructure design, and the future of enterprise AI.By understanding the strengths and limitations of both approaches, architects can build retrieval systems that remain performant, cost-effective, and scalable as vector counts grow from millions to billions.Whether you’re designing Azure AI Search solutions, building enterprise copilots, deploying Retrieval-Augmented Generation platforms, or planning the next generation of knowledge management systems, understanding this trade-off is becoming an essential architectural skill.The billion-vector problem isn’t a future challenge.For many organizations, it’s already here.

Become a supporter of this podcast: https://www.spreaker.com/podcast/m365-fm-modern-work-security-and-productivity-with-microsoft-365–6704921/support.