Semi Real-time Data Cleaning of Spatially Correlated Data in Traffic Sensor Networks

Rollo, Federica; Bachechi, Chiara; Po, Laura · 2022 · Crossref

DOI: 10.5220/0011588500003318

archive: archived pipeline: cataloged verified

Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)

Summary

This paper addresses the challenge of maintaining data reliability in Internet of Things (IoT) traffic sensor networks, where sensor faults and unusual traffic conditions (e.g., accidents) can degrade the performance of urban traffic models. The authors propose a semi-real-time data cleaning methodology that distinguishes between sensor malfunctions and genuine traffic anomalies by leveraging spatial correlations among sensors. The study is motivated by the need to improve traffic management systems in smart cities, specifically using data from induction loop sensors that measure vehicle flow and speed. The methodology consists of three sequential steps: correlation analysis, anomaly detection, and anomaly classification. First, the system identifies groups of spatially correlated sensors using Detrending Cross-Correlation Analysis (DCCA). Second, it employs three complementary anomaly detection techniques: a flow-speed correlation filter to remove physically inconsistent measurements, the Forgetting Factor Iterative Data Capture Anomaly Detection (FFIDCAD) algorithm to identify deviations from normal multivariate distributions, and an Autoregressive Integrated Moving Average (ARIMA) model to detect temporal deviations. Third, anomalies are classified as either sensor faults or unusual traffic conditions based on their spatial distribution; anomalies occurring simultaneously in multiple nearby sensors are classified as traffic conditions, while isolated anomalies are treated as sensor faults. The approach was validated using real-world data from approximately 400 sensors in Modena, Italy, covering over 550 million observations from 2018 to 2022. The authors tested the system on two specific days with reported car accidents (November 8, 2018, and April 15, 2019). Results showed that the three detection techniques were complementary, with minimal overlap in detected anomalies. The flow-speed filter primarily identified high-speed outliers, while FFIDCAD detected low-flow anomalies. The system successfully identified sensor faults, such as sensors reporting constant speeds or unrealistic high values, and replaced them with proximal averages. Crucially, the methodology detected unusual traffic conditions corresponding to the locations and times of reported accidents, demonstrating its ability to distinguish environmental events from hardware failures. The significance of this work lies in its ability to enhance the accuracy of traffic models by providing clean, reliable data streams. By effectively filtering sensor noise and identifying genuine traffic disruptions, the method supports more robust real-time traffic management and surveillance systems. The study highlights the importance of combining spatial correlation analysis with multiple statistical detection methods to handle the non-stationary and complex nature of traffic data. Future work aims to further analyze the relationship between detected slowdowns and specific accident types to improve predictive capabilities.

Provenance

The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed.

Summary generated by qwen3.6-27b-prismaquant on 2026-06-26; verification: verified.

Topics

Ranked by relevance to this paper. Hover a topic for its definition.

• naturalistic crash near crash