Safety Assessment of Automated Vehicles: How to Determine Whether we Have Collected Enough Field Data?

Reference

E. de Gelder, J.-P. Paardekooper, O. Op den Camp, and B. De Schutter, "Safety assessment of automated vehicles: How to determine whether we have collected enough field data?," Traffic Injury Prevention, vol. 20, pp. S162-S170, 2019.

Abstract

Objective: The amount of collected field data from naturalistic driving studies is quickly increasing. The data are used for, amongst others, developing automated driving technologies (such as crash avoidance systems), studying driver interaction with such technologies, and gaining insights into the variety of scenarios in real-world traffic. Since the collection of data is time consuming and requires high investments and resources, questions like “do we have enough data?”, “how much more information can we gain when obtaining more data?”, and “how far are we from obtaining completeness?” are highly relevant. In fact, deducing safety claims based on collected data, e.g., through testing scenarios based on collected data, requires knowledge about the degree of completeness of the data used. We propose a method for quantifying the completeness of the so-called activities in a dataset. This enables us to partly answer the aforementioned questions.
Method: In this paper, the (traffic) data are interpreted as a sequence of different so-called scenarios that can be grouped into a finite set of scenario classes. The building blocks of scenarios are the activities. For every activity, there exists a parametrization that encodes all information in the data of each recorded activity. For each type of activity, we estimate a probability density function (pdf) of the associated parameters. Our proposed method quantifies the degree of completeness of a data set using the estimated pdfs.
Results: To illustrate the proposed method, two different case studies are presented. First, a case study with an artificial dataset, of which the underlying pdfs are known, is carried out to illustrate that the proposed method correctly quantifies the completeness of the activities. Next, a case study with real-world data is performed to quantify the degree of completeness of the acquired data for which the true pdfs are unknown.
Conclusion: The presented case studies illustrate that the proposed method is able to quantify the degree of completeness of a small set of field data and can be used to deduce whether sufficient data have been collected for the purpose of the field study. Future work will focus on applying the proposed method to larger datasets. The proposed method will be used to evaluate the level of completeness of the data collection on Singaporean roads, aimed at defining relevant test cases for the autonomous vehicles’ road-approval procedure that is being developed in Singapore.

Publisher page

• Access full text  

Downloads

• PDF of corresponding technical report (350 KB)

BibTeX

@article{deGPaa:19-019,
   author  = {de Gelder, Erwin and Paardekooper, Jan-Pieter and Op den Camp,
              Olaf and De Schutter, Bart},
   title   = {Safety Assessment of Automated Vehicles: {How} to Determine
              Whether we Have Collected Enough Field Data?},
   journal = {Traffic Injury Prevention},
   volume  = {20},
   pages   = {S162--S170},
   year    = {2019}
   }

• Download BibTeX entry