Photorealism in Driving Simulations: Blending Generative Adversarial Image Synthesis With Rendering

Yurtsever, Ekim; Yang, Dongfang; Koc, Ibrahim Mert; Redmill, Keith · 2022 · OpenAlex-citations

DOI: 10.1109/tits.2022.3193347

archive: archived pipeline: cataloged verified

Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)

Summary

This paper addresses the critical need for high visual fidelity in driving simulations, which are essential for developing intelligent vehicle systems and conducting human-in-the-loop experiments. Conventional computer graphics pipelines rely on labor-intensive 3D modeling, texturing, and physics-based rendering, often resulting in imagery that lacks photorealism and suffers from repetitive patterns that break immersion or cause machine learning algorithms to overfit. To solve this, the authors propose a hybrid generative neural graphics pipeline that blends generative adversarial image synthesis with partial physics-based rendering. The method utilizes the CARLA driving simulator to generate a 2D semantic layout from simple, textureless 3D models. This semantic map serves as a conditional input for a Generative Adversarial Network (GAN), specifically a SPADE-based conditional GAN trained on the Cityscapes dataset, to synthesize photorealistic background imagery. Simultaneously, critical objects of interest, such as vehicles and lane markings, are rendered individually using Unreal Engine 4 to ensure precise control over their appearance and physics. These partially rendered foreground elements are then blended with the GAN-synthesized background using a blending GAN (GP-GAN), which refines the composite image to ensure realistic integration. Experimental evaluation demonstrates that the proposed hybrid approach outperforms both conventional full-rendering and pure GAN-based synthesis. Using Frechet Inception Distance (FID) measurements against real-world datasets like Cityscapes and KITTI, the hybrid method produced images with higher visual fidelity and closer similarity to real-world data. Semantic retention analysis using DeepLabV3 further confirmed that the blended images maintained the structural integrity of the scene better than GAN-only outputs, which often distorted vehicle shapes, or full-render outputs, which produced unrealistic shadows. The study also validated that replacing repetitive synthetic textures with generative surfaces improves the diversity and realism of the simulation environment. The significance of this work lies in its ability to reduce the manual labor required for creating high-fidelity simulation assets while enhancing the realism of the output. By combining the controllability of physics-based rendering for critical objects with the photorealistic generation capabilities of GANs for backgrounds, the pipeline offers a scalable solution for driving simulations. This improvement in visual fidelity is crucial for training robust vision-based algorithms and ensuring that human subjects exhibit natural driving behaviors during simulation-based experiments, thereby bridging the gap between synthetic training data and real-world deployment.

Provenance

The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed.

Summary generated by qwen3.6-27b-prismaquant on 2026-06-26; verification: verified.

Topics

Ranked by relevance to this paper. Hover a topic for its definition.

• simulator validity fidelity

Information type

What kind of knowledge this paper contributes, grouped by family — independent of topic (what it is about) and method (how it was studied).

• Methodological Resource: tool software