Retrieval Engineering for Agents

Prompt engineering influences how a model reasons. Retrieval engineering determines what it has to reason about.

Vespa.ai13 minute read

Introduction: Agents Depend on the Evidence They Find

An AI agent can reason carefully and still reach the wrong conclusion. It may miss a critical passage, rely on outdated information, or retrieve something that appears relevant but concerns the wrong product, customer, or policy. The reasoning may be sound, but the supporting evidence may be incomplete.

For organizations building agents around proprietary information, this is a fundamental engineering challenge. The knowledge that makes an agent useful, including product specifications, customer records, research, policies, and operational data, must be accessible in a form that supports the task. Connecting an agent to that information is the starting point. Finding the right evidence, at the right time, determines how effectively it can use it.

One Request, Many Retrieval Decisions

Consider a shopping agent asked to find a dress similar to a photograph, compare its quality with alternatives, confirm availability in the customer’s size, and check whether it can arrive before Saturday.

Each step requires a different kind of search. Visual similarity helps discover alternatives. Exact matching identifies a specific product. Review passages provide evidence about quality and fit. Structured filters enforce price, size, and delivery requirements. Ranking determines which options deserve further investigation.

The process also changes as the agent learns. Reviews suggesting that a dress runs small may trigger a sizing investigation. The agent may reject a promising alternative because it cannot arrive in time. Each result helps the agent decide what to search for next.

Figure 1. One request, many retrieval decisions: the shopping agent’s request leads to visual similarity, exact matching, review passages, structured filters and ranking, and each result helps it decide what to search for next.

This makes retrieval part of the agent’s decision-making process. An early omission or misleading result can influence subsequent searches and steer the task in the wrong direction. Agents can cross-check and recover, but recovery costs time, extra searches, and model tokens.

Relevance, Freshness, and Speed Become Task-Level Requirements

These workflows place connected demands on retrieval infrastructure. Information must be relevant to the immediate question, sufficiently detailed to support a decision, and current enough to act on. A product match is of limited value if its availability has changed. A relevant policy passage can produce the wrong answer if it has been superseded.

Performance matters across the entire task. Sequential searches add latency at each step. Parallel searches can leave the agent waiting for the slowest response. Expensive ranking models may improve evidence selection, but applying them indiscriminately increases cost and response time.

The agent also needs to understand how to search. Field names alone may be ambiguous: “available” could mean stocked somewhere or available in the requested size. Different tasks may require different retrieval methods, filters, and ranking strategies. Developers need to make those choices accessible through documented, tested interfaces with clear limits.

From Search Capabilities to an Engineered System

Keyword search, vector similarity, hybrid retrieval, and reranking each address part of this problem. Their value depends on how they work together with application data, business rules, learned models, and live updates, and whether that combination delivers the evidence an agent needs within its latency and cost constraints.

Evaluating retrieval therefore requires looking beyond individual search results. Does the system help the agent complete the task correctly? Can failures be traced to missing evidence, inappropriate filters, ranking decisions, or misunderstood data? Can the retrieval strategy evolve as the application becomes more demanding?

Answering these questions requires a systematic approach to designing, measuring, and improving how agents access information. That is the role of retrieval engineering.

What is Retrieval Engineering?

Prompt engineering influences how a model reasons. Retrieval engineering determines what it has to reason about.

Search engineering has always balanced competing priorities: relevance, performance, scalability, and cost. AI raises the stakes considerably. Retrieval systems increasingly assemble the context that large language models and AI agents use to reason, generate, and act. Every retrieval decision becomes part of an automated workflow where quality, latency, and freshness directly influence the outcome.

Retrieval Engineering is the practice of designing, optimizing, and operating retrieval workflows that balance search quality, latency, freshness, scalability, and infrastructure cost. It treats the retrieval workflow as a single system whose components must work together efficiently.

Retrieval workflows orchestrate multiple retrieval techniques, ranking models, and relevance signals, including:

Hybrid retrieval combining keyword search, semantic search, and structured data
Structured filtering and business rules
Query rewriting and expansion
Personalization
Machine-learned ranking
Real-time updates
Machine learning inference
Access control and privacy policies
Context assembly for language models and AI agents

Each capability can improve retrieval, but also adds computational cost and architectural complexity. Retrieval Engineering determines where these techniques add value and how to execute them efficiently at scale.

For agents, the workflow may be invoked repeatedly within a single task. Retrieval Engineering must therefore account for both the quality of individual searches and their contribution to the agent’s overall progress. A relevant result is useful only if it provides evidence appropriate to the decision the agent is making.

Figure 2. Retrieval Engineering: a query and your data lead into one workflow to retrieve, rank, and return, with continuous freshness and evaluation under it, powering AI applications.

The Relationship to RAG and Context Engineering

Retrieval-augmented generation (RAG) grounds model responses in retrieved information. Production implementations require engineering across the path from source data to the context supplied to the model: deciding what to retrieve, applying permissions and business rules, ranking candidates, incorporating current signals, and selecting the final evidence.

For AI agents, retrieval engineering is a core part of context engineering: the broader discipline of designing and managing what a model receives at each step of a task. Retrieval engineering supplies relevant, current, and permitted evidence from external information sources. Context engineering also encompasses system instructions and examples, tool definitions and usage guidance, conversation history, persistent memory, and the agent’s working state, including its goals, progress, and intermediate results. It determines how the model selects, structures, and updates these elements within its context window, including when to summarize or remove material as the task progresses. Together, these functions give the agent the evidence, guidance, and continuity it needs to reason and act effectively.

Figure 3. Retrieval engineering inside context engineering, supplying evidence from external information sources.

Providing more information is not necessarily the solution to missing context. Irrelevant or redundant material consumes the context window, increases processing cost and latency, and can obscure important evidence. The objective is to deliver sufficient, relevant, current, and permitted information for the task.

The Limits of Fragmented Architectures

Vector databases solved an important problem: making semantic retrieval practical at scale. They allow applications to retrieve information based on meaning, extending what can be found through exact keyword matching.

As AI applications mature, their requirements often extend beyond semantic retrieval. High-quality systems combine keyword search, structured filtering, business rules, machine-learned ranking, real-time updates, and inference to assemble useful context. The engineering challenge becomes coordinating these capabilities throughout the workflow.

Many organizations address this by integrating specialized technologies. A vector database provides semantic retrieval. A search engine handles keyword matching. Other services provide reranking, personalization, or machine-learning inference. These architectures can work well, especially when teams need specialized capabilities or must build on existing infrastructure.

However, each additional service can introduce data movement, network latency, operational dependencies, and synchronization work. The costs depend on how the services interact, where data resides, and how often the workflow runs.

Agents can amplify these costs. Repeated retrieval invokes the same integrations many times. Parallel searches increase demand on shared services. Sequential searches accumulate latency. An update that has reached one system but not another can produce inconsistent evidence within the same investigation.

Semantic similarity alone also rarely determines the best result. Precise terminology, proprietary vocabularies, structured business data, behavioral signals, freshness, and authority can all matter. Retrieval quality depends on how effectively these signals work together.

Vector databases and Lucene-based search engines address many parts of this workflow. The architectural question is whether the complete system remains efficient, scalable, and manageable as agent activity and retrieval complexity grow.

The Integration Tax

The integration tax is the ongoing cost of coordinating a fragmented stack: infrastructure, engineering effort, data synchronization, network overhead, and operational complexity.

For an agent application, assess that cost per completed task as well as per query. A workflow that looks inexpensive in isolation may become costly when an agent invokes it repeatedly or needs additional searches to recover from incomplete results.

An integrated architecture can reduce this overhead by executing retrieval, ranking, and inference closer to the data. Measure the benefit against the application’s actual workload, including relevance requirements, update frequency, and expected concurrency.

The AI Search Platform

AI Search Platforms provide an execution layer for intelligent applications, from search and recommendations to answer engines and agents. They bring retrieval, ranking, machine-learning inference, and real-time serving into a common architecture.

This approach can reduce the operational complexity and data movement involved in coordinating independent services. It also gives developers a common framework for combining retrieval methods, relevance signals, and models as application requirements evolve.

Vespa.ai develops an AI Search Platform built for Retrieval Engineering. It supports customer-facing intelligent applications through a distributed serving engine that combines text, vectors, structured data, programmable ranking, and model inference.

For agent developers, this provides a foundation for exposing several tested retrieval strategies through one serving framework. The agent can select the strategy appropriate to its next investigation while the platform executes the configured retrieval and ranking logic.

Figure 4. Two Approaches to AI Retrieval: fragmented retrieval across separate services, and Vespa running the same steps in one engine.

Building the Retrieval Workflow

Retrieval workflows combine keyword search, semantic retrieval, structured filtering, machine-learned ranking, real-time updates, and business logic into an execution path. The challenge is making those capabilities work together with predictable latency, operational simplicity, and infrastructure efficiency.

Vespa brings these capabilities into a single distributed serving engine. Its architecture supports four connected engineering requirements: unified retrieval, intelligent ranking, continuous freshness, and performance at scale.

1. Unified Retrieval

AI applications depend on retrieving enough relevant evidence to support the next reasoning step. Modern retrieval workflows combine dense vector search, keyword search, structured filtering, metadata, and business rules to identify suitable candidates.

These methods serve different purposes. Semantic retrieval can discover related concepts even when wording differs. Keyword matching preserves exact terminology. Structured filters enforce requirements such as price, location, or availability. Combining these methods enables more precise retrieval than relying on a single signal.

Vespa executes hybrid retrieval natively, combining vectors, text, and structured data within a distributed query. Retrieval operates where the data resides, reducing the need to move candidate sets between separate retrieval services.

Retrieval representations can also go beyond a single vector per item. Multi-vector retrieval, late interaction, multimodal retrieval, and visual document understanding use richer structures to represent content. Vespa’s tensor support allows labelled numerical information, multiple embeddings, and other features to participate in retrieval and ranking within the same platform.

For an agent, this means different investigations can use different combinations of evidence. The shopping agent may use an exact product identifier to establish which item it is investigating, image embeddings to find alternatives, and keyword and semantic signals to locate reviews about fabric quality.

The same principle applies to information-intensive applications. A research agent may need an exact company name, a reporting-period filter, semantic matching for the topic, and an authority signal for the source. The retrieval method should match the evidence required at that step.

2. Intelligent Ranking

Ranking determines which information receives prominence and ultimately reaches the user or model. For agents, this influences both the immediate decision and the direction of subsequent searches.

Applying sophisticated models to every candidate can become prohibitively expensive. Multi-phase ranking progressively refines candidate sets, using inexpensive calculations broadly and applying more expensive models to a smaller set of promising results.

Vespa supports first-phase and second-phase ranking on content nodes, with global-phase ranking after candidates are gathered across nodes. Developers can combine text relevance, vector similarity, business features, and learned model outputs while controlling how many candidates each stage evaluates.

Figure 5. Multi-phase ranking in Vespa: first-phase and second-phase ranking on content nodes, then global-phase ranking after candidates are gathered across nodes.

This makes compute allocation part of the ranking strategy. An exact lookup may require little additional ranking. A comparison involving visual similarity, customer preferences, and product quality may justify more sophisticated evaluation.

Agents also need evidence at the right level of detail. Returning an entire document or every review can bury the passage that answers the question. A workflow can rank parent items and select relevant passages within them, preserving the relationship between the evidence and its source.

For the shopping agent, that might mean selecting a promising dress and returning the review passage about breathable fabric. For a research agent, that might mean identifying the relevant report and returning the passage that supports a specific claim. The objective is to select evidence that supports the task while keeping model input and ranking costs under control.

3. Continuous Freshness

AI applications operate in changing environments. Documents are revised, inventory changes, customer behavior evolves, and new information arrives. The delay between a change and its availability to retrieval affects the evidence an agent can use.

Vespa supports ongoing indexing and updates to text, vectors, and structured data while serving live traffic. For properly configured attribute fields, partial updates let you update signals without reading and rewriting the whole document.

This is useful for inventory, popularity, customer preferences, and other frequently changing features. Updated values can influence subsequent searches and ranking decisions without waiting for a periodic recommendation batch.

For agents, freshness also has a task-level dimension. A longer investigation may span several searches while the underlying data changes. Developers need to determine which decisions require a fresh check and which evidence should remain tied to a known version or point in time.

The shopping agent may discover a dress early in the task, but availability and delivery eligibility should be checked again before making a recommendation or initiating a transaction. Retrieval supplies evidence for the decision; the system performing an action must still validate the conditions required for that action.

4. Performance at AI Scale

Agents change the economics of retrieval. A single user request may trigger multiple searches, model evaluations, and follow-up investigations. The resulting load depends on task complexity, agent behavior, and the number of users.

Sequential searches add to total response time. Parallel searches create bursts of demand, and a slow response can delay the next step when the agent needs all results. Retries and recovery add further work.

Vespa distributes data and executes retrieval, filtering, and content-node ranking close to that data. Global ranking can then refine candidates gathered across nodes. This architecture supports scaling retrieval and ranking together while reducing unnecessary movement of large candidate sets.

Capacity planning must reflect the complete workload: data size, query complexity, model cost, update traffic, concurrent tasks, and the number of retrieval operations each task generates. Average query latency alone is insufficient. Tail latency, throughput under concurrency, and task completion time also matter.

The relevant economic measure is the cost of completing a useful task at the required quality. A cheaper query that forces an agent to repeat searches or recover from missing evidence may increase the total cost.

Giving Agents the Right Retrieval Controls

A capable retrieval engine still needs an interface the agent can use correctly. Different steps in an investigation require different retrieval strategies, and the agent needs enough guidance to select among them.

Package tested strategies

Developers can define strategies for common needs: exact lookup, semantic discovery, visual similarity, evidence gathering, or value comparison. Each strategy should specify its inputs, filtering requirements, ranking behavior, and operating limits.

Vespa separates the retrieval expression from the rank profile. Vespa’s query language, YQL, expresses the search, while named rank profiles define how candidates are evaluated. An application can expose tested combinations so an agent selects a suitable strategy and supplies the relevant inputs.

This allows the agent to adapt its investigation while keeping scoring logic, model execution, and computational limits under developer control.

Figure 6. Tested strategies, bounded autonomy: the agent chooses among tested combinations, scoring logic, model execution and computational limits stay under developer control, and the application enforces the rest.

Explain the meaning of the data

A schema tells an agent which fields exist, but field names alone may not explain how to use them. Guidance should clarify units, business meaning, relationships, and applicable constraints.

For example, does “available” mean stock exists anywhere, or that the item can be purchased in the requested size and location? Is a date the dispatch date or the expected arrival date? Does a review apply to the current product version?

Application-specific instructions can document these distinctions alongside valid query examples and guidance on choosing retrieval strategies. A schema-to-skill approach packages schema information, Vespa usage guidance, and application explanations into a Markdown skill an agent can consult.

Such guidance can help an agent construct requests directly. It still requires evaluation and server-side validation; documentation cannot guarantee that every query will be appropriate or correct.

Enforce permissions and operating limits

Permissions and privacy policies determine which evidence an agent can access and which information it can pass to a model. The application and retrieval infrastructure should enforce these controls rather than leaving them to the agent’s interpretation of instructions.

The surrounding application must also enforce timeouts, search budgets, and limits on expensive operations. Logs should capture the chosen strategy, inputs, filters, ranking configuration, and relevant results so developers can diagnose failures.

These controls support bounded autonomy: the agent decides what to investigate next within an interface whose behavior and limits have been defined and tested.

Evaluating Retrieval Across the Agent’s Task

Retrieval Engineering needs evidence that the workflow works. Evaluation should cover individual searches and the agent tasks those searches support.

Begin with representative tasks and judgments about the evidence needed to complete them. Include exact lookups, ambiguous requests, fresh-data requirements, restricted information, and cases where no suitable result exists.

At the retrieval level, assess whether the necessary evidence is found, ranked prominently, sufficiently current, and associated with the correct source. Check that filters and permissions work as intended and that context contains enough information without unnecessary duplication.

At the task level, assess whether the agent reaches a supported conclusion, recognizes missing evidence, and makes appropriate follow-up searches. An agent can retrieve individually plausible results while still comparing the wrong products, confusing reporting periods, or treating an empty result set as proof of absence.

Operational evaluation should include completion time, retrieval volume, model calls, retries, and cost. Test under realistic concurrency and update traffic so relevance improvements can be assessed alongside their production impact.

Use failures to identify the next engineering change.

Missing candidates may require different retrieval methods or query interpretation.
Poor ordering may require ranking changes.
Outdated evidence may reveal update delays.
Invalid searches may point to ambiguous schema guidance or inadequate validation.

This connects retrieval tuning to the outcome the application needs to deliver.

Putting Retrieval Engineering into Practice

Progress can begin with one valuable agent workflow. Map the path from source data through retrieval, ranking, and context assembly to the agent’s decision. Identify the systems involved, the controls applied, and the points where latency or evidence quality degrades.

Establish measurable requirements for relevance, freshness, permissions, response time, and cost. Then address the most consequential gaps and evaluate the effect on completed tasks.

A useful starting assessment asks:

Evidence: What information does the agent need, and what determines whether it reaches the model?
Retrieval: Which steps need exact matching, semantic retrieval, structured filters, or multimodal search?
Ranking: How are relevance signals and models combined, and where is expensive evaluation justified?
Freshness: How quickly do changes become searchable, and which decisions need revalidation?
Control: Where are permissions, privacy policies, timeouts, and search budgets enforced?
Agent interface: Can the agent understand field meanings and choose appropriate, tested strategies?
Architecture: How much data movement, integration, and synchronization does each task require?
Evaluation: Can failures be traced from the agent’s conclusion back to the retrieval decisions that shaped it?
Economics: What are the latency and cost of a completed task at the required quality?

The lessons from one workflow can guide improvements across other applications. As data, models, and agent behavior evolve, retrieval becomes an ongoing production discipline with clear ownership and a repeatable improvement process.

Conclusion: Engineer the Evidence Behind the Agent

An agent’s ability to reason and act depends on the information it can access, understand, and use. Reliable performance requires relevant evidence, current data, appropriate controls, and retrieval strategies suited to each step of the task.

Retrieval Engineering brings these requirements together. It treats retrieval, ranking, inference, freshness, and context assembly as a connected workflow, evaluated for both evidence quality and production efficiency.

Vespa.ai provides a programmable serving framework for building these workflows, combining hybrid retrieval, multi-phase ranking, model inference, and real-time updates. Developers can use that framework to expose tested strategies that agents select as their investigations evolve.

For teams building agent applications, the practical next step is to examine the complete path from a user’s request to the evidence supporting the final decision—and engineer that path for the work the agent needs to do.