Perplexity Research Lead: Transforming Qwen3.5 into an Efficient Search Agent with GRPO and Gated Rewards
The Perplexity research team published a technical article detailing the post-training process of its web search agent: based on the open-source MoE models Qwen3.5-122B-A10B and Qwen3.5-397B-A17B, a "two-stage post-training" approach is employed. First, supervised fine-tuning (SFT) establishes essential deployment behaviors such as instruction adherence, language consistency, refusal, and safety. Then, online policy reinforcement learning (RL) optimizes search accuracy and tool usage efficiency in a real tool environment. The RL phase uses the GRPO (Grouped Relative Policy Optimization) algorithm, with training data consisting of two parts: first, a self-developed multi-hop verifiable Q&A dataset, constructing 2-4 hop entity chain questions from internal seed queries and verifying answer uniqueness with multiple independent web solvers; second, general dialogue data based on rubrics, breaking down format constraints, instruction adherence, and safety norms into automatically checkable atomic conditions to continuously constrain behavior during the RL phase, preventing the degradation of the "good habits" established during SFT.
The core of the reward design is "gated aggregation": preference rewards are only included in the calculation when baseline tasks are correct (e.g., correct Q&A or passing all rubric criteria), preventing answers that are more pleasing in tone or style from receiving high rewards when factually incorrect. Efficiency penalties are applied using an intra-group anchoring method, where multiple correct trajectories within the same question group are anchored by the least tool calls/shortest outputs, imposing smooth penalties on excess, thereby suppressing excessive tool calls and verbose answers without sacrificing accuracy. On the FRAMES benchmark, the post-trained Qwen3.5-397B-SFT-RL achieves 57.3% accuracy under a single tool call budget, 5.7 percentage points higher than GPT-5.4 and 4.7 percentage points higher than Sonnet 4.6; with a medium budget (4 tool calls), the accuracy is 73.9%, with a cost of approximately $0.02 per query based on public API pricing, while GPT-5.4 under the same budget achieves 67.8% / $0.085 and Sonnet 4.6 achieves 62.4% / $0.153, demonstrating a clear advantage on the "accuracy-cost" curve.
Source: Public Information
ABAB AI Insight
The key idea of this post-training scheme is to optimize "compliance/format/safety" and "search capability/tool efficiency" in two separate stages: first using SFT to "teach the model to behave well," and then using RL to "teach it to search" in a tool environment. Single-stage RL can easily lead the model to sacrifice the compliant behaviors established by SFT in pursuit of rewards, such as oversimplifying answers or ignoring format or safety constraints; Perplexity mitigates this reward hacking by incorporating rubric-based general dialogue into the RL data and ensuring that "being correct comes first, then being good" through a gated reward mechanism.
The synthetic multi-hop verifiable Q&A dataset addresses the "supervisability" issue: real user search Q&As often lack a unique standard answer, making it difficult to construct clear RL signals. However, questions generated through entity chains can be cross-verified among multiple web solvers, retaining only samples with unique answers, providing a high signal-to-noise ratio training target for the search agent. Meanwhile, the rubric-based general dialogue data ensures that the model is continuously "reminded" to adhere to format, citation, and safety constraints during the RL phase, even when handling non-pure retrieval tasks (rewriting, planning, long dialogues)—this allows the agent to search in real product environments without easily "forgetting" deployment requirements.
In terms of efficiency, the intra-group anchoring penalty design aligns well with engineering realities: it does not simply penalize all tool calls or long outputs but rather makes relative comparisons between "multiple correct strategies" for the same question, using the most economical trajectory as a benchmark and applying gentle pressure on redundant calls and lengthy answers. This design avoids the cliff-like drop in accuracy caused by a "one-size-fits-all" reduction in tool calls while guiding the model towards a "sufficient but not excessive" tool strategy—this is one reason it achieves higher accuracy under relatively low tool budgets on benchmarks like FRAMES.
From an industry perspective, this article elevates the route of "open-source large models + self-built RL training pipeline" to a new height: Perplexity does not rely on its own closed-source base but instead utilizes commercially available open-source models like Qwen3.5-122B/397B, transforming them into specialized agents that can match or even surpass GPT-5.4 and Sonnet 4.6 in search tasks through complex post-training engineering. This offers two insights for the future ecosystem: first, the importance of the model itself is marginally decreasing, with data and reward design becoming the main sources of differentiation; second, the combination of "vertical tasks + open-source base + heavy post-training" presents a realistic opportunity to outperform general closed-source models in specific tracks while significantly reducing costs—illustrated by the comparison of "+6 percentage points accuracy / cost only 1/4 of GPT-5.4" in the search scenario.