SAFE TEST Implemented at the American Center for Mobility (ACM)

---

The road to autonomous vehicle (AV) deployment just got much shorter. Over the past several months, the CCAT team has been hard at work implementing a state-of-the-art toolbox at the American Center for Mobility (ACM). The Safe AI Framework for Trustworthy Edge Scenario Tests, or SAFE TEST, aims to radically improve the efficiency of AV performance testing. It does so by combining two pieces of technology developed by CCAT Director Dr. Henry Liu – the Augmented Reality Environment and the Naturalistic and Adversarial Driving Environment.

The Augmented Reality Test Environment was developed under the CCAT grant in 2017 and was led by Drs. Henry Liu and Yiheng Feng to address the lack of vehicle interactions available to researchers that use test tracks such as Mcity. Currently, these facilities merely offer empty roadways to run tests and despite the COVID-19 pandemic, U.S. roads are anything but empty. To add real vehicles to a test scenario, companies would have to spend thousands of dollars and hours to coordinate and control. The team developed an augmented reality environment which allowed them to add virtual, background traffic to Mcity that the test vehicle saw as “real”. The simulated vehicles are easily controlled so specific test scenarios can be repeated perfectly each time.

The Naturalistic and Adversarial Driving Environment (NADE) was part of a research paper published by Drs. Henry Liu and Shuo Feng that was recently published in Nature Communications. This research addresses the lack safety-critical scenarios a test vehicle might experience on the roadways. Known as “long-tail events”, these rare situations such as a vehicle merging in front of you or hard braking, are important for an AV to respond to provide public trust. The NADE inserts background vehicles that conduct these adversarial and rare maneuvers at a much higher rate. Simultaneously, the environment uses naturalistic driving data from the University of Michigan to ensure unbiasedness.

By combining these two cutting-edge technologies, the SAFE TEST is able to increase the efficiency of AV performance testing, reduce operational costs, and accelerate product validation. As part of CCAT’s 2021 research, Dr. Liu was awarded funding to implement the SAFE TEST at ACM, located at the historic Willow Run site. Led by President and CEO, Reuben Sarkar, ACM is a collaborative effort comprised of government, industry and academic organizations focused on accelerating the mobility industry through research, testing, standards development and educational programming. With the proposed system, the testing miles in the ACM test track can be converted approximately into equivalent mileage with public roads.

Images of the SAFE TEST Implementation at the American Center for Mobility (ACM)

On Wednesday, September 1st, ACM hosted a free webinar to discuss this recently added technology with three speakers: Ralph Buckingham from Intertek, Dr. Henry Liu, and Dhiren Verma from ACM. To start the discussion, Ralph covered one of the major reasons behind implementing this technology – time savings. According to research from Nidhi Kalra and Susan M. Paddock, to get an AV to drive 20% better than humans, a fleet of 100 AVs driving at 25 miles per hour would need to drive for 28 million miles to avoid crashes, 170 million miles to avoid injuries, and 5 billion miles to avoid fatalities. In all, this work would take hundreds of years. They estimate that one mile on the ACM track with SAFE TEST equates to 5,000 real-world miles. Another big factor is cost savings. It is estimated to reduce costs by 50 – 100 times through decreased development costs and cycles.

---

“[SAFE TEST] is helping us bring an advanced capability and toolset to ACM that wouldn’t otherwise be available in the market. Really, it’s a force multiplier.”

Reuben Sarkar, President & CEO, American Center for Mobility

---

To validate the effectiveness of this solution, Dr. Liu highlighted the large-scale naturalistic driving data used from the Safety Pilot Model Deployment program under the University of Michigan Transportation Research Institute (UMTRI). Besides the generation of long-tail events, the NADE can also provide systematic safety performance analysis. This includes counterfactual simulation which compares what actually happened with what would have happened in relevant situations. Long-tail events can also be augmented to meet the needs of the user. Other important analyses conducted by NADE include safety metrics, accident type, accident responsibility, and a breakdown of the strengths and weaknesses of the AV. The Augmented Reality Testing platform can communicate the virtual traffic to the vehicle in two different ways: using roadside units on the test track or directly. Using roadside units utilizes dedicated short-range communication (DSRC), to communicate with an onboard unit (OBU) in the test vehicle.

To conclude the September 1st event, the CCAT team provided private demos of the SAFE TEST for ACM VIPs. These guests received a first-hand look at the various scenarios that the NADE could offer in the Level 4 AV Lincoln MKZ. One of these guests included Jonathan Crocker, Senior Manager of Vehicle Test Engineering at Toyota Research Institute (TRI). “I was impressed with the seamless integration of Augmented Reality traffic into the AV camera feed and driving stack, when riding in an equipped car on the ACM Highway Loop. The system continuously exposed the AV to realistic adversarial traffic scenarios, allowing the team to monitor the response of the vehicle to situations that would be difficult to perform using physical vehicles. Augmented reality will be a useful asset in the verification toolchain to accelerate testing of adversarial edge cases in a realistic and safe manner.”

Henry Liu and Jonathan Crocker discussing the SAFE TEST after their private demo

If you are interested in learning how the SAFE TEST can help you validate your products at the American Center for Mobility, you can watch this free webinar (requires registration). If you want to find out how your CAV test team can access this innovative technology, you may contact Kevin Kelly, Sales Director at ACM (Kevin.Kelly@acmwillowrun.org).