Multi-Modal Sensor Fusion-Based Semantic Segmentation for Snow Driving Scenarios

Vachmanus, Sirawich; Ravankar, Ankit A.; Emaru, Takanori; Kobayashi, Yukinori · 2021 · OpenAlex-citations

DOI: 10.1109/jsen.2021.3077029

archive: archived pipeline: cataloged verified

Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)

Summary

This paper addresses the critical challenge of autonomous vehicle perception in snowy environments, where poor visibility and low contrast between snow-covered surfaces and objects severely degrade the performance of standard computer vision algorithms. The authors propose a multi-modal sensor fusion approach for semantic segmentation, combining color (RGB) images with thermal maps (T) to improve the detection of roads and pedestrians in winter conditions. The motivation stems from the high accident rates and transportation delays caused by snow, particularly in regions like Hokkaido, Japan, where distinguishing pedestrians from snow piles is difficult for RGB-only sensors. The proposed method utilizes a fully convolutional encoder-decoder architecture. The encoder consists of two parallel branches based on the ResNet-50 model, one processing RGB inputs and the other processing thermal inputs. To integrate these modalities, the authors introduce a novel fusion module adapted from AdapNet++, which merges feature maps from both branches using convolutional branches and a skip connection, employing a softplus activation function to handle temperature features effectively. The network also incorporates an Atrous Spatial Pyramid Pooling (ASPP) module to capture multi-scale context and uses a pyramid supervision training scheme to enhance learning. The decoder, based on ResNet-34, expands the feature maps back to the original image size for pixel-wise classification. The model was evaluated on a newly collected snow dataset and publicly available datasets, including Cityscape and Synthia. The results demonstrate that the fused RGB-T input significantly outperforms RGB-only inputs, particularly in segregating human subjects in snowy environments. The combination of the proposed fusion module and the pyramid supervision path achieved the highest mean accuracy and mean intersection over union (mIoU) across all tested datasets. The study confirms that thermal information compensates for the loss of color gradient features in snow, allowing the network to distinguish living subjects from the ambient environment more effectively than state-of-the-art single-modal networks. The significance of this work lies in its contribution to robust autonomous driving systems capable of operating in adverse weather conditions. By demonstrating that multi-modal fusion improves semantic segmentation accuracy in snow, the paper provides a viable solution for enhancing road safety and reducing accident risks in winter scenarios. The proposed architecture offers a reliable method for integrating diverse sensor data, addressing a key limitation in current deep learning approaches for autonomous vehicles.

Provenance

The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed.

Summary generated by qwen3.6-27b-prismaquant on 2026-06-26; verification: verified.

Topics

Ranked by relevance to this paper. Hover a topic for its definition.

• distraction detection algorithms