Vector RAG vs Hybrid RAG vs Agentic RAG: one benefits question, three answers
Similarity search finds the right document. A graph finds the connected ones. Only a routed workflow with live data answers what applies to this employee. A production comparison with numbers, failure modes and a real case.
An employee types: "Is my MRI covered, and how much will I pay?"
Three assistants answer.
The first one returns a paragraph about imaging coverage from a plan document. It is accurate, well written and useless: it is the right topic for the wrong person, because the employee is on a different plan.
The second one finds the right plan, follows it to the imaging coverage and the prior-authorization policy, and explains the rules correctly. It still cannot say how much this employee will pay, because that depends on how much of the deductible they have already met this year.
The third one routes the question, fetches the plan rules and the employee's live eligibility and deductible status from the system of record, checks the answer against policy and replies with a number, or hands the conversation to a person when it is not sure.
All three are called RAG. Only one of them resolves the question. This article compares the three, what each one costs, and where each one breaks.
The problem: three kinds of knowledge behind one question
"Retrieval-augmented generation" sounds like one technique. In production it is a decision about where each part of the answer comes from. The MRI question needs three kinds of knowledge:
- Documents: what the plan says about imaging. Unstructured text, changes a few times a year.
- Relationships: which coverage belongs to which plan, which policy applies to which coverage, which FAQ overrides which clause. Structure that lives between documents.
- Personal, live state: this employee's plan, eligibility and claims. Structured data that changes daily and is governed per employer.
A design that only retrieves documents can answer the first kind. Most enterprise questions mix all three.
Requirements
- Correct for this person: an answer about someone else's plan is a wrong answer, however fluent.
- Tenant isolation: employer A's documents and data must never reach employer B's employees, under any prompt.
- Permissions enforced at the source: the assistant must not see more than the employee could see in the portal.
- Auditable: every answer traceable to the passages and data it used.
- Escalation: when confidence is low or the topic is sensitive, a person takes over with full context.
- Conversational latency: a few seconds per turn, not tens.
Scale estimate
bswift reported 750,000+ Emma messages across 150,000 chat sessions in 2025: about 5 messages per session and roughly 2,000 messages a day on average, with open enrollment concentrating load into a few weeks. Using that as the anchor, with illustrative assumptions for the rest:
| Input | Value |
|---|---|
| Messages per year | 750,000 (reported) |
| Average / open-enrollment day | ~2,000 / ~10,000 messages |
| LLM calls per message | 3 (route, compose, validate) |
| Tokens per message | ~7,000 (mostly retrieved context) |
| Corpus (illustrative) | 1,000 employers × 20 plan documents × 30 pages |
That gives:
- ~5 billion tokens a year, dominated by context, not by the answer. Retrieval precision is a cost lever: every irrelevant chunk is paid for three times.
- ~1.2 million chunks (two per page), which at 1,536 dimensions is ~7 GB of vectors. Size is not the hard part; filtering by tenant on every query is.
- ~30,000 LLM calls on a peak day. Routing every turn through the largest model is wasteful: a small, fast model can classify intent, and the expensive one only composes.
Approach 1: Vector RAG
Scroll sideways to see the full diagram →
Documents are split into chunks, each chunk becomes an embedding, and the vectors go into a store with metadata (tenant, plan, document). At question time the question is embedded, the top-K most similar chunks are retrieved, filtered by tenant, and passed to the LLM as context.
Where it shines
- Simple to build and cheap to run. One index, one query, one model call.
- Excellent for questions whose answer lives in one passage: "what is a qualifying life event?"
- Easy to refresh: re-embed a document when it changes.
Where it breaks
- Similarity is not relevance. "Imaging coverage" chunks from five plans look alike; the retriever cannot know which plan is this employee's.
- Relationships are invisible. The answer often needs the coverage and the policy it points to and the exception in an FAQ. Top-K rarely brings back all three.
- Personal state is out of reach. Deductibles and claims are not documents. Copying them into the vector store creates a second, stale, ungoverned copy of sensitive data.
- Tenant filtering is a single point of failure. One missing metadata filter and another employer's plan shows up in the context.
Approach 2: Hybrid RAG (vectors + knowledge graph)
Scroll sideways to see the full diagram →
Similarity search still finds the entry passages, and a knowledge graph adds what is connected to them: plan → coverage → policy → FAQ. The question's entities (plan name, coverage type) are looked up in the graph, their neighborhood is pulled in, and both result sets are merged and reranked before the LLM answers. Many teams also add keyword search (BM25) for exact terms like plan codes.
Where it shines
- Correct context, not just similar context. The answer about imaging comes with the prior-authorization policy that governs it.
- Explainable retrieval. A graph path is a readable justification: this plan, this coverage, this policy.
- Better recall on multi-document questions without inflating chunk size.
Where it breaks
- The graph is a product to maintain. Entity extraction, schema and updates are real engineering, and a stale edge is worse than no edge.
- Still no personal state. The answer is right for the plan, not for the person: it cannot say what this employee will pay.
- More latency and moving parts: two retrieval systems, a merge step and a reranker on every turn.
Approach 3: Agentic RAG
Scroll sideways to see the full diagram →
The model no longer receives a fixed context. A router classifies each turn and plans which sources it needs, usually more than one: documents, the graph, and live business data fetched from the system of record through APIs scoped to the employer and the employee's permissions. The selected sources are queried in parallel, their results are merged into one context that keeps track of where each fact came from, a compose step writes the answer, and a validation step checks grounding, policy and confidence before replying or handing off to a person.
The MRI question is the typical case: it needs all three at once. The plan documents say imaging is covered, the graph links that coverage to the prior-authorization policy, and the live API says how much of the deductible is left. Multi-source turns bring two problems single-source RAG never has:
- Partial results. One source times out or fails. The answer either states what it could not confirm ("coverage confirmed, cost estimate unavailable right now") or goes to a person; it never fills the gap.
- Conflicts. The documents say one thing and the live record another. Precedence is a design decision, not a model decision: live data over documents, the employee's plan over generic text, the most specific rule over the general one.
There are two ways to build it, and the difference matters more than the name:
Scroll sideways to see the full diagram →
- A free-form agent loop gives the model every tool and lets it decide the next step on every turn until it chooses to stop. Flexible, and hard to bound: cost, latency and paths vary per conversation.
- An explicit workflow (a state graph, such as LangGraph) fixes the nodes and edges. The model still classifies and composes, but where the conversation can go is designed, observable and testable. For health and money questions, this is the version that survives review.
Where it shines
- Answers for this person. Live eligibility and claims make the answer specific and current.
- One source of truth. Personal data stays in the system that owns it; permissions are enforced there, not re-implemented in the AI layer.
- Resolution, not containment. Validation plus explicit handoff means the assistant stops when it should, with context, instead of guessing.
Where it breaks
- Routing becomes the critical path. A wrong route sends a claims question to the documents and returns a confident, irrelevant answer.
- Latency and cost add up. Three or more model calls plus API calls per turn; without a small router model and parallel retrieval, turns get slow.
- Testing is harder. Every path needs coverage; without a regression set, a prompt change silently breaks a route.
Side by side
| Vector RAG | Hybrid RAG | Agentic RAG (workflow) | |
|---|---|---|---|
| Answers | What the documents say | What the documents say, connected | What applies to this person, now |
| Sources | Vector store | Vectors + graph (+ keyword) | Vectors, graph and live APIs, several per turn |
| Personal data | No (or a stale copy) | No | Yes, from the system of record |
| Model calls per turn | 1 | 1-2 | 3+ |
| Main risk | Similar but wrong context | Graph drift | Wrong route, untested paths |
| Best for | FAQ-style, single-passage questions | Policy questions across documents | Account-specific questions with consequences |
They are layers, not rivals. Production assistants usually contain all three: vector search inside the graph-aware retrieval, both inside a routed workflow that also calls live data.
Failure modes in production
- Cross-tenant leakage. A missing filter in one retrieval path mixes employers. Enforce tenancy in the tool layer and in the data stores, and test it with adversarial queries, not just unit tests.
- Stale embeddings. Plans change at renewal; old chunks keep answering. Version documents and re-index on change, with the effective date in the metadata.
- Hallucinated personal facts. If live data is unavailable, the model fills the gap with something plausible. Make "data unavailable" an explicit state that routes to handoff.
- Misrouting. Track route accuracy as a metric; review misrouted conversations every week.
- Over- and under-escalation. Too eager and the assistant is a phone menu; too shy and it guesses about money. Tune thresholds against real outcomes, not intuition.
- Prompt injection through documents. Retrieved text is untrusted input. Keep instructions out of the context channel and validate outputs against policy.
- Partial multi-source context. One of several sources fails and the answer is composed from the rest as if it were complete. Track which sources each answer needed and which ones actually returned.
- Source conflicts. Documents and live data disagree. Encode precedence explicitly and log every conflict; they are often data-quality bugs upstream.
- Silent regressions. Every prompt, model or retrieval change runs against a golden set of real questions before it ships.
Choosing a stack
| Layer | Common choices | Note |
|---|---|---|
| Vector search | pgvector, OpenSearch, Pinecone, Weaviate | Pick the one that filters by tenant efficiently at your scale |
| Knowledge graph | Neo4j, Amazon Neptune | Worth it when answers span related documents |
| Orchestration | LangGraph, LlamaIndex workflows, custom state machines | Prefer explicit graphs for regulated domains |
| Evaluation & tracing | LangSmith, Ragas, Arize Phoenix | A golden-query set matters more than the tool |
| Models | OpenAI, Anthropic, open models via a provider-agnostic layer | Small model to route, larger one to compose |
Real-world case: Emma at bswift
Situation. bswift runs benefits administration for large employers: thousands of employees per client, each with their own plans, eligibility and claims. Support ran through forms, phone and email. The answers lived across plan documents, relational and non-relational stores, and real-time business rules in bswift's .NET platform, all under per-employer tenancy and governance.
Task. Build an assistant that resolves benefits questions for each specific employee, not one that deflects them with generic FAQ text, without moving sensitive data out of the platform that governs it.
Action.
- I designed the end-to-end architecture as Principal Engineer: a Next.js conversational interface, services on AWS, a provider-agnostic model layer over OpenAI and Anthropic, and an explicit LangGraph workflow instead of a free-form agent.
- An entry router node classifies each turn and selects the retrieval paths it needs: plan documents in a vector store, a Neo4j knowledge graph linking plans, coverages, policies and documents, and live business rules (eligibility, claims, enrollment) fetched in real time from the .NET backend.
- Live data never gets copied into AI stores, and every tool call is scoped to the employer and the employee's permissions, enforced by the platform that owns the data.
- Escalation triggers on low confidence, sensitive topics or a request for a person, and passes a conversation summary plus the retrieved context so nobody repeats themselves.
- Quality is measured, not assumed: LangSmith tracing, a golden-query regression set and human review before and after every change.
Result. bswift launched the new Emma Chat publicly in February 2026 and reports an 80% autonomous resolution rate, with 750,000+ messages across 150,000 chat sessions in 2025. Answers are plan-specific and grounded in claims data, and transfers to human representatives keep the full context.
The lesson I keep from it: decide where each part of the answer comes from before choosing a retrieval technique. Documents, relationships and live personal state need different machinery, and the router that chooses between them is the most important component in the system.
Read the full case: how Emma was designed for resolution, not containment →
Where to start
- Classify your real questions by what they need: documents, relationships, personal state. Count them; the mix decides the architecture.
- Start with vector RAG for the document-only share, with tenant filters enforced and tested.
- Add the graph only when multi-document questions fail measurably.
- Bring personal data through APIs, never copies, scoped to the user's permissions.
- Make the workflow explicit: router, retrieval paths, compose, validate, handoff.
- Build the golden set early and run it on every change.
- Measure resolution, not conversations handled.
The takeaway
Vector RAG finds the right document. Hybrid RAG finds the right connected documents. An agentic workflow with live data answers for the actual person, and knows when to stop. The step up is not a smarter model; it is deciding, explicitly, which source owns each part of the answer.
References
- Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020).
- Edge et al., From Local to Global: A Graph RAG Approach to Query-Focused Summarization (2024).
- LangGraph documentation.
- bswift, Enhanced AI-native Emma Chat announcement (2026).