State-of-the-art e-commerce search requires more than hybrid retrieval and better ranking models. It also requires offline data that supports trustworthy experiments. Teams must be able to reconstruct catalog, inventory, and feature data as of a given point in time. They must also be able to replay index mutations and requests against that state. Without this foundation, offline evaluation for learning to rank and automated model research can produce misleading results.
Based on lessons from Etsy’s migration from Solr and multiple other retrieval engines to a single Vespa-based system, this talk explains how model building and evaluation should shape data ingestion. It shows why Vespa should consume model- and evaluation-relevant data from a point-in-time-correct ML data platform instead of relying on separate pipelines that are difficult to reproduce.
The talk also covers three related practices. First, we explore what LLMs need in order to perform migrations from other retrieval engines successfully. Second, we discuss how to distill strong teacher models into fast GBDT rankers that run within the Vespa content nodes. Third, what makes for successful and impactful autoresearch, including the durable execution building blocks that allow optimization pipelines to run for weeks on end.
These lessons form a practical guide for teams building and evolving production e-commerce search with Vespa.
