SynTrackThinking improves multimodal multi-object tracking for autonomous driving through frequency-aware fusion and temporal contrastive learning.

Su, Z; Chang, W · 2026 · PubMed Central

DOI: 10.1038/s41598-026-44182-4

archive: archived pipeline: cataloged verified

Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)

Summary

This paper addresses the challenges of Multimodal Multi-Object Tracking (MM-MOT) in autonomous driving, specifically targeting the limitations of existing fusion strategies and temporal instability. Current methods often rely solely on spatial processing, neglecting frequency-domain cues essential for fine-grained cross-modal correlation, and struggle with cumulative tracking drift over long sequences. To resolve these issues, the authors propose SynTrackThinking, a unified framework that integrates frequency-aware representation learning with contrastive temporal reasoning. The methodology centers on two core components: the Explainable Multimodal-Domain Cross-Attention Fusion Module (EMCFM) and the Multimodal Contrastive Tracking Learning (MCTL) strategy. EMCFM enhances feature expressiveness by decomposing visual, textual, and auditory inputs into multi-resolution frequency components via Wavelet Transform. It employs Adaptive Wavelet Sub-band Attention to selectively amplify informative spectral bands and utilizes tri-modal cross-attention to bridge spatial and frequency domains. MCTL projects heterogeneous modality embeddings into a shared latent space using focal contrastive objectives, explicitly enforcing semantic and temporal coherence to suppress identity drift. The system is built upon a deformable DETR backbone with a 6-layer encoder–decoder and 300 object queries, optimized using AdamW. To evaluate the framework, the authors constructed three new large-scale benchmarks—Multi-KITTI, Multi-KITTI+, and Multi-BDD—incorporating visual, textual, and speech modalities, alongside their text-only counterparts. Experiments were conducted on six datasets, comparing SynTrackThinking against state-of-the-art baselines such as TransTrack, TrackFormer, and TransRMOT. The results demonstrate that SynTrackThinking achieves average gains of 9.14% in HOTA and 67.67% in MOTA over TransRMOT on challenging MM-MOT benchmarks. The framework also showed superior precision and interpretability compared to FFT-based or conventional cross-attention schemes, while maintaining stable performance under linguistic ambiguity, diverse speech patterns, and noisy environmental conditions. Ablation studies confirmed the complementary contributions of EMCFM and MCTL, highlighting favorable accuracy-efficiency trade-offs. The significance of this work lies in its provision of a robust, scalable solution for multimodal tracking in complex autonomous driving environments. By jointly integrating frequency-aware fusion and long-range contrastive reasoning, SynTrackThinking effectively mitigates the suboptimal fusion and temporal semantic inconsistency that hinder previous approaches. The introduction of new multimodal benchmarks further supports comprehensive evaluation under diverse conditions. The authors position SynTrackThinking as a foundational perception layer that complements higher-level multimodal reasoning systems, offering a reliable mechanism for continuous spatiotemporal localization and identity preservation.

Provenance

The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed.

Summary generated by qwen3.6-27b-prismaquant on 2026-06-26; verification: verified.

Topics

Ranked by relevance to this paper. Hover a topic for its definition.

• distraction detection algorithms