Vector database vs application database: what AI agents need from each

The agent counted seven orders. There were four.

An account manager asks a sales agent a simple question before a renewal call: "How many open orders does Acme have, and what's outstanding?" The agent searches its vector index for Acme orders, gets the eight most similar chunks back, and answers with confidence: seven open orders, $412,000 outstanding.

The real answer was four orders and $268,000. Two of the seven had been cancelled last week, but the index still held the confirmation emails that created them. One was a duplicate: the same order appeared in a PDF and in a CRM note, embedded twice. And one belonged to Acme's European entity, which the account manager isn't allowed to see. A fifth open order, created that morning, hadn't been indexed yet, so it wasn't counted at all.

Nothing malfunctioned. The vector database did its job well: it found text that resembled the question. The design failed because it asked a similarity index to act as a system of record.

What is the difference between a vector database and an application database?

A vector database ranks stored content by how similar it is to a query. An application database stores the current, authoritative state of business records and answers exact questions about them under transactions. One finds candidates. The other holds the truth.

Vector search turns text into embeddings and returns the nearest neighbours. That's powerful for meaning: it finds the clause that matches a question even when the words differ. But "nearest" is a ranking, not an answer. It returns the top k results, it has no notion of whether a document is still current, and two near-identical chunks look like two pieces of evidence.

An application database (usually relational, sometimes document or key-value) is built for the opposite job. It knows that order 4471 exists once, that its status is cancelled, that it belongs to a specific legal entity, and that the change committed at 09:42. Ask it for a count or a total and you get one exact number.

Five things similarity search can't give you

A vector index isn't a weak database. It's a different tool, and five properties of a system of record are simply outside its job.

1. Transactions. An agent that approves a refund and updates a balance needs both changes to happen together or not at all. Retrieval indexes are written in batches and read by ranking; they don't coordinate writes.
2. Current state. Embeddings are a copy, taken at indexing time. When a case closes or an order is cancelled, the old text stays in the index until someone re-embeds or deletes it.
3. Authorization by record. Permission is a property of each record (this customer, this entity, this region) and it changes. Chunks often inherit coarse metadata, or none.
4. Exact answers. Totals, counts, averages and "is anything still open?" need every matching record, not the eight most similar.
5. Identity. A record has one key. Its text may appear in an email, a PDF, a ticket and a note, and each copy becomes its own vector.

Why do AI agents get totals and counts wrong?

Because the agent is summing a ranked sample, not querying a set. Top-k retrieval returns the most similar chunks, so anything not retrieved is missing and anything retrieved twice is counted twice.

The model is good at producing a fluent total from whatever it's given. It has no way to know that the ninth matching order fell just outside the cut-off, or that two chunks describe the same invoice. Raising k doesn't fix it. A bigger sample is still a sample, and stale or duplicate chunks get more chances to appear.

The same applies to status. "Is the claim approved?" is a lookup against one field on one record. Answered from retrieval, it becomes a guess from documents that mention approval: the draft that said "pending approval", the email that said "approved subject to conditions", the note that said "approval withdrawn". The model picks the most plausible reading. The database would simply have returned the value.

Authorization belongs inside retrieval, not after it

The European order in the opening scenario is the dangerous failure. A wrong count is embarrassing; a leaked record is a breach. OWASP's 2025 list for LLM applications names vector and embedding weaknesses, including weak access control and leakage across contexts, as a top risk.

Two patterns fail here. The first puts everything in one index and asks the model to "only use what the user may see". The model has already read the restricted content, and an instruction isn't a control. The second filters the answer afterwards, which is too late for the same reason.

Doing this well needs permission data that's as current as the records themselves. That's another argument for keeping operational questions in the application database, where authorization can be checked against live attributes, as attribute-based access control describes. See permission-aware retrieval for how to carry those checks into the index side too.

Agents need both, used for different questions

The fix isn't to drop vector search. It's to route each question to the store built for it.

A useful test: if two people asking the same question at the same moment should always get the same answer, it's a query. If they might reasonably get different supporting passages, it's retrieval.

Make operational queries typed, scoped and recorded

Sending the agent to the database raises its own risk: an agent writing free-form SQL against production tables. The safer pattern is a set of typed queries the agent can call, each one authored and reviewed in advance.

• Typed collections. Expose business entities (orders, customers, cases) with declared fields and types, not raw tables.
• Join and result contracts. Declare how entities relate and what shape and grain the result has, so a total can't silently double-count.
• Scope on every call. Apply the requester's tenant, entity and record permissions inside the query, not in the prompt.
• A receipt for every answer. Record which query ran, with which parameters, under whose authority, against which data version, and what it returned.

That receipt is what lets you defend a decision later. "The agent said four orders" becomes "query open_orders_by_account returned four rows at 10:14 under the account manager's authority." For more on why free-form text-to-SQL isn't enough, see governed queries for AI agents.

Keep the index honest about what it is

Retrieval still needs discipline, because its outputs end up in front of the model next to exact data.

• Link chunks to records. Store the source record's key and version with each chunk, so retrieved text can be checked against current state.
• Prefer the record when they disagree. If a chunk says "approved" and the record says "withdrawn", the record wins, and the conflict is worth logging.
• Expire and re-index on change. When a record changes status or permission, update or remove the chunks derived from it.
• Collapse duplicates. Group chunks by source record before they reach the model, so one invoice can't look like three.
• Label it as evidence. Retrieved text informs the answer. It never instructs the agent and never overrides a field in the system of record.

Common anti-patterns

Seven rules for vector and application data

1. Use similarity search to find candidates, never to state facts about current business state.
2. Answer totals, counts and status with exact queries on the system of record.
3. Apply the requester's authority inside retrieval and inside every query.
4. Give agents typed, reviewed queries, not raw tables.
5. Record a receipt for every query: what ran, for whom, against which data, with what result.
6. Link every chunk to its source record and version; collapse duplicates.
7. When retrieved text and the record disagree, the record wins.

The result: candidates and truth, each in its place

Vector databases earned their place in agent architectures. They make unstructured knowledge usable, and nothing else finds the right passage in ten thousand contracts as well. The mistake is asking them to do a second job they were never designed for.

Agents that act on business data need both stores, each answering the questions it's built for: permission-aware retrieval for evidence, and typed, scoped, recorded queries for anything that must be exactly true right now. Put that split in the architecture rather than in the prompt, and the agent's answer about Acme becomes one you can check, repeat and defend.