Multi-Frame, Lightweight & Efficient Vision-Language Models for Question Answering in Autonomous Driving

Gopalkrishnan, Akshay; Greer, Ross; Trivedi, Mohan M. · 2024 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2403.19838

archive: archived pipeline: cataloged verified

Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)

Summary

This paper addresses the computational inefficiency of current Vision-Language Models (VLMs) in autonomous driving, which typically rely on large language model (LLM) backbones exceeding one billion parameters. These large models incur high memory costs and inference latencies, making them unsuitable for real-time deployment in vehicles with tight resource constraints. To solve this, the authors introduce EM-VLM4AD, a lightweight, multi-frame VLM designed for Visual Question Answering (VQA) that achieves high performance with significantly reduced computational overhead. The proposed architecture utilizes a pre-trained Vision Transformer (ViT-B/32) to encode multi-view traffic images (front, left, right, back, etc.) into patch embeddings. These individual view embeddings are aggregated into a unified representation using gated pooling attention, which introduces non-linearity to extract relevant visual information across views. This combined image embedding is concatenated with text embeddings and fed into a lightweight T5 language model. The authors explore two backbone configurations: a fully fine-tuned T5-Base (~223M parameters) and an 8-bit quantized T5-Large (~750M parameters) fine-tuned using Low-Rank Adaptation (LoRA). Training is conducted on the DriveLM dataset using a two-stage process: first aligning image embeddings with the frozen LM, then fine-tuning the LM while keeping the image encoder frozen. Experimental results demonstrate that EM-VLM4AD outperforms the existing DriveLM-Agent baseline on ROUGE-L and CIDEr metrics, despite having at least three billion fewer parameters. Specifically, the T5-Base variant achieved a ROUGE-L score of 71.98 and a CIDEr score of 3.20, compared to 66.79 and 2.79 for the baseline. Computational analysis reveals that EM-VLM4AD requires at least 10 times less memory and floating-point operations (FLOPs) than comparable models like DriveMLM or Drive-GPT4. The T5-Base model uses only 0.94 GB of memory and 9.47B FLOPs, while the quantized T5-Large variant uses 0.77 GB. Qualitative evaluations show the model effectively answers questions regarding perception, planning, and traffic agent behavior by dynamically focusing on relevant camera views, though it struggles with ego-vehicle behavior prediction tasks that require temporal context. The significance of this work lies in demonstrating that efficient, sub-billion parameter models can achieve superior or comparable performance to massive LLMs for autonomous driving VQA tasks. By drastically reducing memory and computational requirements, EM-VLM4AD enables real-time inference on accessible hardware, such as single GPU instances, thereby improving the accessibility of interpretable AI systems for researchers and the feasibility of deploying VLMs in actual autonomous vehicles.

Key finding

EM-VLM4AD achieves higher ROUGE-L and CIDEr scores than the DriveLM-Agent baseline while requiring at least 10 times less memory and floating-point operations.

Methodology

simulation_modeling

Provenance

The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed. Discovered via author_sweep_intake on 2026-05-28.

Summary generated by qwen3.6-27b-prismaquant on 2026-06-10; verification: verified.

Topics

Ranked by relevance to this paper. Hover a topic for its definition.

• generative ai voice assistants