Bypassing inference bottlenecks: Accelerating complex

What happened

Modern search or recommendation applications are increasingly expected to return a coherent set of results rather than a single best match. For example, when a user searches for "camping gear", they don’t want ten slight variations of four-person tents. They want a coherent, complementary slate that includes essential camping gear, such as a tent, sleeping bag, portable stove, and headlamp.

For our open-ended abstract retrieval tasks, this composite reward is a weighted balance of three competing pillars: During training, we optimize the fan-out language model against these geometric realities using group relative policy optimization (GRPO) with soft proximal policy optimization (PPO). This specific triad of rewards is critical because they act as mutual counter-anchors.

If a model is optimized purely for groundedness, it will reward-hack the system by generating degenerate, nonsensical strings that happen to mathematically map to a specific database coordinate. If alignment is added to fix the nonsense, the policy simply cheats by collapsing into repetitive paraphrases of the user's prompt. By injecting the Vendi Score as a counter-anchor, Retrieve-for-Train effectively closes off these shortcut solutions.

To achieve a high-reward state, the policy is forced into a balanced region of the embedding space where it must discover valid, strictly grounded, yet semantically distinct variations of the original intent. To evaluate the Retrieve-for-Train framework, we used a combination of frozen, dataset-specific multimodal embedding backbones and open-source language models optimized for query expansion.

Why it matters

To do this, systems use a query fan-out technique that breaks a single broad prompt into several related sub-queries to cover potential user interests. However, teaching an LLM to perform database-aware query decomposition dynamically drains a massive thinking budget. By design, zero-shot LLMs are general autoregressive text predictors; they aren’t optimized to navigate the specific, geometric manifold of a target corpus.

By distilling these optimized exploration behaviors into a lightweight diffusion retriever, we enable highly efficient, single-pass query fan-out at inference time. This achieves mathematically formulated, set-level properties without the overhead of test-time thinking tokens. When tasked with brainstorming a complex group of search terms, it’s tempting to simply deploy a standard, off-the-shelf LLM at inference time to handle the job.

However, relying on generic models for database-aware query decomposition introduces two critical challenges: The Retrieve-for-Train treats the AI's training like an offline practice session rather than a test it has to take on the spot while a user is waiting.

Instead of forcing the AI to slowly figure out the rules of a good search and drain a massive processing budget every single time someone types a query, Retrieve-for-Train runs an offline RL training program once. This program uses a rigorous reward system to turn abstract goals like "ensure the results are diverse and actually in stock" into an exact step-by-step instruction manual.

Once that manual is built, the AI can execute it instantly during a real search without delay. The success of the Retrieve-for-Train framework hinges entirely on how we define "good" search behavior. Traditional supervised training evaluates pointwise relevance via learning to rank, scoring each retrieved item in isolation. However, a truly expert search slate is defined by non-decomposable, set-level properties.

You can’t measure the diversity or complementarity of a single item; these properties only exist mathematically when evaluating the entire collection of retrieved results. Rather than relying on ambiguous natural language instructions to enforce these fan-out properties, Retrieve-for-Train fine-tunes the 4B open-source language models (Gemma3-4B and Qwen3-4B) via reinforcement learning using a strict mathematical composite reward.

What to watch

We evaluated this setup across two distinct set-valued retrieval regimes: For the multimodal embedding backbones, we conducted experiments across two domains: A large-scale fashion dataset of user-curated outfits used for text-to-image experiments (evaluated using a CLIP-based retriever), and a proprietary industrial dataset of expert-generated music playlists used for text-to-music evaluations (evaluated using MuLan).

For the language models, the query fan-out process was driven by 4B open-source models, specifically Gemma3-4B and Qwen3-4B, which were tasked with generating exactly 10 sub-queries for every single main search prompt they processed. We implemented the RL training for these fan-out models via Soft-GRPO, an approach that uses group relative policy optimization with soft PPO regularization.