Benchmarking Advanced Tabular AI Models for Classifying SAE Automation Levels in Crash Data

Somvanshi, Shriyank; Dutta, Anandi; Das, Subasish · 2026 · Crossref

DOI: 10.1007/s42421-026-00162-8

archive: archived pipeline: cataloged verified

Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)

Summary

This study addresses the challenge of classifying Society of Automotive Engineers (SAE) automation levels in police-reported crash data, where metadata regarding vehicle automation is often incomplete or inconsistent. The research is motivated by the growing deployment of automated vehicles and the need for robust classification frameworks to support crash database quality assurance, label inference, and automation-stratified safety research. The authors clarify that the classification task identifies the vehicle’s designed automation capability (derived from make, model, and year) rather than confirming active system engagement at the time of the crash. The researchers benchmarked five tabular machine learning and deep learning models: random forest, XGBoost, MambaAttention, prior-data fitted network (TabPFN), and TabTransformer. The dataset consisted of 4,649 structured crash records from the Texas Crash Record Information System (CRIS) for 2024, covering SAE Levels 1 through 5. To prevent data leakage, records were partitioned at the crash level using GroupShuffleSplit, resulting in 3,721 training and 928 test records with zero overlap in Crash IDs. To address a severe 21:1 class imbalance, Synthetic Minority Over-sampling Technique for Nominal and Continuous features (SMOTE-NC) was applied exclusively within the training partition. Performance was evaluated using fivefold stratified cross-validation, with SMOTE-NC applied independently within each fold. XGBoost achieved the highest macro-F1 score (0.573; 95% CI 0.514–0.633), while TabPFN attained the highest macro-AUC (0.786) and accuracy (82.9%) without requiring task-specific training. MambaAttention achieved a macro-F1 of 0.509 and the highest recall for advanced automation (27.6%), though this estimate carried substantial uncertainty due to only 29 test cases for that class. McNemar’s tests indicated that TabPFN and XGBoost formed a statistically equivalent performance tier, whereas MambaAttention’s error profile differed significantly from other models. TabTransformer exhibited persistent instability and near-chance discriminative performance, constituting a reproducible negative finding for vanilla transformer architectures under extreme class imbalance. Feature contribution analysis using TreeSHAP identified population group, vehicle body style, roadway class, and crash timing as dominant predictors, revealing distinct associative feature profiles for each SAE category. The findings demonstrate that tabular machine learning models can classify SAE-coded vehicle capabilities in crash records with meaningful accuracy. This supports applications in data quality assurance and the design of safety research stratified by automation level. The study provides methodological guidance for handling imbalanced crash classification tasks and highlights the limitations of certain deep learning architectures in this context. The authors caution that identified feature associations are exploratory and reflect data collection patterns rather than causal crash mechanisms.

Provenance

The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed.

Summary generated by qwen3.6-27b-prismaquant on 2026-06-26; verification: verified.

Topics

Ranked by relevance to this paper. Hover a topic for its definition.

• telematics crash prediction

Information type

What kind of knowledge this paper contributes, grouped by family — independent of topic (what it is about) and method (how it was studied).

• Empirical Findings: crash risk outcomes