DepthVision: Enabling Robust Vision-Language Models with GAN-Based LiDAR-to-RGB Synthesis for Autonomous Driving

Kirchner, Sven; Purschke, Nils; Greer, Ross; Knoll, Alois · 2025 · ArXiv.org

DOI: 10.48550/arxiv.2509.07463

archive: archived pipeline: cataloged verified

Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)

Summary

This paper addresses the challenge of maintaining reliable autonomous vehicle operation when visual inputs are degraded by poor lighting, motion blur, or sensor failure. While Vision-Language Models (VLMs) have advanced perception and planning, they rely heavily on RGB data, which is scarce in low-light conditions and lacks the geometric robustness of LiDAR. However, integrating LiDAR directly into VLMs is difficult due to architectural incompatibilities and the scarcity of large-scale LiDAR datasets. The authors propose DepthVision, a multimodal framework that enables off-the-shelf, frozen VLMs to exploit LiDAR data without retraining. The system synthesizes dense, RGB-like images from sparse LiDAR point clouds and adaptively fuses them with real camera feeds, effectively turning LiDAR into a drop-in visual surrogate to extend the operational envelope of existing models. The methodology consists of three primary components. First, LiDAR point clouds are projected into the camera frame using known extrinsics and intrinsics, creating a sparse depth map that is cropped and interpolated via nearest-neighbor methods to preserve geometric discontinuities. Second, a conditional Generative Adversarial Network (GAN) with a U-Net generator and PatchGAN discriminator translates this sparse depth map into a dense RGB image. A lightweight residual refiner network iteratively corrects artifacts and sharpens structural details. Third, a Luminance-Aware Modality Adaptation (LAMA) module fuses the synthesized LiDAR-derived image with the real RGB input. LAMA employs two strategies—global and pixelwise weighting—based on scene luminance. It dynamically down-weights unreliable RGB data in dark regions while preserving real camera information in well-lit areas, ensuring the final fused image remains compatible with standard VLM visual encoders. The authors evaluate DepthVision on real and simulated datasets, including nuScenes and CARLA, across multiple VLMs and safety-critical tasks such as visual question answering and scene understanding. The results demonstrate substantial improvements in low-light scene understanding compared to RGB-only baselines. The framework successfully maintains high performance even when real camera signals are degraded or unavailable. Ablation studies and vehicle-in-the-loop experiments confirm the system's robustness under adverse illumination conditions and its practical deployability. The approach preserves full compatibility with frozen VLM architectures, requiring no modifications to the downstream language or vision transformers. The significance of this work lies in providing a practical pathway for integrating range sensing into modern vision-language systems without the need for expensive retraining or architectural changes. By synthesizing RGB-like data from LiDAR, DepthVision bridges the gap between the semantic richness of camera data and the geometric reliability of LiDAR. This allows autonomous systems to leverage the vast pretraining of web-scale visual models while remaining robust in challenging environments where cameras fail. The findings suggest that LiDAR-guided synthesis is a viable strategy for enhancing the safety and reliability of VLM-based autonomous driving stacks, particularly in unstructured real-world scenarios.

Key finding

Synthesizing RGB images from LiDAR data and adaptively fusing them with camera inputs based on luminance significantly enhances the performance of frozen Vision-Language Models in low-light and degraded visual conditions.

Methodology

simulation_modeling

Provenance

The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed. Discovered via author_sweep_intake on 2026-05-28.

Summary generated by qwen3.6-27b-prismaquant on 2026-06-10; verification: verified.

Topics

Ranked by relevance to this paper. Hover a topic for its definition.

• generative ai voice assistants