Building Reliable Sim Driving Agents by Scaling Self-Play
archive: archived pipeline: cataloged verified
Get this paper ↗ (full text — opens at the source; we link to it, we don't host it)
Summary
This paper addresses the critical need for reliability in simulation agents used for autonomous vehicle (AV) development. While generative models have improved the realism of simulated road users, they often suffer from high rates of unintended collisions and off-road movements, which introduce noise into safety evaluations and benchmarking. The authors propose that scaling self-play reinforcement learning (RL) can close this reliability gap, producing agents that consistently adhere to designer-defined criteria, such as staying on the road and avoiding collisions, while navigating to target positions. The study utilizes the GPUDrive simulator, a GPU-accelerated, data-driven environment containing over 160,000 scenarios from the Waymo Open Motion Dataset. Agents are trained using Proximal Policy Optimization (PPO) with self-play, where up to 64 agents per scenario interact under a shared, decentralized policy. The agents operate under semi-realistic constraints, including partial observability (a 50-meter view radius), no historical context, and discrete action spaces for acceleration and steering. The reward function incentivizes goal achievement while penalizing collisions and off-road events. Training was conducted on a single NVIDIA A100 GPU, processing 2 billion agent steps over 24 hours. The results demonstrate that self-play scales effectively with data volume. Agents trained on 10,000 scenarios achieved a 99.8% goal completion rate with less than 0.8% combined collision and off-road incidents on 10,000 held-out test scenarios, effectively eliminating the train-test gap. This performance significantly outperforms state-of-the-art supervised models from the Waymo Open Sim Agent Challenge, reducing collision and off-road rates by approximately fourfold. Although the agents showed partial robustness to out-of-distribution events, they struggled with rare behaviors like navigating backward. However, the authors demonstrated that fine-tuning the pre-trained model on just 13 hand-designed scenarios for 15 minutes enabled the agents to master these rare tasks with near-perfect success. The significance of this work lies in establishing a framework for highly reliable simulation agents that can be tuned for specific collision rates, supporting both nominal and safety-critical testing. By open-sourcing the agents and training scripts, the authors provide a reproducible baseline that improves the signal-to-noise ratio in AV evaluation pipelines. The findings suggest that while self-play excels at reliability, balancing this with human-like realism remains an area for future improvement. The approach also has broader implications for agent-based modeling in other fields, such as neuroscience, where reliable digital twins are essential for controlled experimentation.
Key finding
Scaling self-play reinforcement learning enables simulation agents to achieve near-perfect reliability, with less than 0.8% collision and off-road rates on unseen test scenarios, significantly outperforming state-of-the-art generative models.
Methodology
simulator
Provenance
The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed. Discovered via bulk_ingest_rosap on 2026-05-23 (6 acquisition events logged).
| Stage | Outcome | Tool | Model | Prompt | Attempts | Completed |
|---|---|---|---|---|---|---|
| discover | success | rosap | — | — | 2 | 2026-05-23 |
| archive | success | — | — | — | 1 | 2026-05-23 |
| extract | success | cached | — | — | 2 | 2026-06-10 |
| clean | success | — | — | — | 1 | 2026-06-01 |
| chunk | success | — | — | — | 1 | 2026-06-01 |
| embed | success | — | — | — | 1 | 2026-06-02 |
| enrich | success | — | — | — | 1 | 2026-05-23 |
| promote | success | — | — | — | 1 | 2026-05-23 |
| summarize | success | llm | qwen3.6-27b-prismaquant | summ-v5 | 3 | 2026-06-10 |
| tag | success | vector_similarity | — | — | 24 | 2026-06-11 |
| verify | success | — | — | — | 2 | 2026-06-10 |
Summary generated by qwen3.6-27b-prismaquant on 2026-06-10; verification: verified.
Topics
Ranked by relevance to this paper. Hover a topic for its definition.
Information type
What kind of knowledge this paper contributes, grouped by family — independent of topic (what it is about) and method (how it was studied).
- Methodological Resource: tool software, validation psychometrics
- Theoretical Contribution: computational model