The L3Pilot Data Management Toolchain for a Level 3 Vehicle Automation Pilot

Hiller, Johannes; Koskinen, Sami; Berta, Riccardo; Osman, Nisrine; Nagy, Ben; Bellotti, Francesco; Rahman, Ashfaqur; Svanberg, Erik; Weber, Hendrik; Arnold, Eduardo H.; Dianati, Mehrdad; De Gloria, Alessandro · 2020 · Crossref

DOI: 10.3390/electronics9050809

archive: archived pipeline: cataloged verified

Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)

Summary

This paper addresses the critical challenge of managing and analyzing large-scale experimental data from the L3Pilot project, the first comprehensive European test of Society of Automotive Engineers (SAE) Level 3 and 4 automated driving functions on public roads. With 13 vehicle owners conducting tests across 10 countries, the project required a robust data management toolchain capable of handling heterogeneous data sources while respecting strict confidentiality and intellectual property constraints. The primary motivation was to establish a standardized, scalable architecture that supports diverse research questions ranging from technical system performance to user acceptance and societal impact, thereby facilitating pre-competitive collaboration among industry and academic partners. The authors developed a multi-layer data management toolchain implemented in a desktop/cloud environment, designed to evolve into an edge-to-cloud reference architecture. Central to this system is the L3Pilot Common Data Format (CDF), an open-source standard that converts proprietary vehicular signals into a unified structure stored in HDF5 files. This format ensures portability and compatibility across various analysis tools. The methodology follows the FESTA framework for field operational tests, structuring research into four clusters: technical/traffic evaluation, user acceptance, impact evaluation, and socio-economic impact. Data processing involves rigorous quality checking using a custom Java tool called L3Q, which validates data structures and signal plausibility. Additionally, the system employs pseudonymization via SHA-256 hashing to protect driver privacy and vehicle owner confidentiality while maintaining data traceability. The toolchain facilitates iterative data enrichment through MATLAB scripts, generating derived measures (e.g., time headway), identifying driving scenarios (e.g., lane changes, cut-ins), and computing performance indicators (e.g., statistical descriptors of specific events). Video annotation is integrated into this workflow to support scene detection and data validation. The architecture supports four distinct layers of data access: proprietary owner data, restricted analysis partner data, aggregated de-identified data for consortium-wide analysis, and public summary indicators. This structure allows for seamless cross-vehicle data management and efficient storage, enabling the consortium to answer complex research questions without exposing sensitive raw data. The significance of this work lies in the creation of a comprehensive, open-source reference architecture for automated vehicle testing. By standardizing data formats and processing workflows, the L3Pilot toolchain enhances research reproducibility and scalability. The authors conclude that this framework not only supports the current pilot but also provides a valuable blueprint for future large-scale field operational tests, addressing the growing need for efficient, privacy-preserving data management in the rapidly advancing field of automated driving.

Provenance

The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed.

Summary generated by qwen3.6-27b-nvidia on 2026-08-10; verification: verified.

Topics

Ranked by relevance to this paper. Hover a topic for its definition.

• naturalistic crash near crash

Information type

What kind of knowledge this paper contributes, grouped by family — independent of topic (what it is about) and method (how it was studied).

• Methodological Resource: tool software, dataset resource, measurement protocol