Towards Safe and Efficient Autonomous Driving: A Synergistic Approach with Human Expertise and Multimodal Large Language Models

Principal Investigator(s):

Sikai Chen, Assistant Professor of Civil and Environmental Engineering – University of Wisconsin–Madison
Yiheng Feng, Assistant Professor of Civil Engineering – Purdue University
Assistant Director – Center for Road Safety (CRS)

Project Abstract:
Autonomous driving (AD) systems often face challenges with corner cases due to limited scene comprehension and insufficient learning of human knowledge in safety-critical situations. To address this, we propose a dual-stage approach integrating multimodal large language models (MLLMs) and human expertise. The MLLM will employ Chain-of-Thought (CoT) reasoning for improved decision-making and be continuously fine-tuned through reinforcement learning (RL), with human expertise injected through human-AI interaction supported by an accident warning system. Additionally, a unified platform will be developed to integrate scenario generation, algorithm development, and testing. Comprehensive closed-loop evaluations across benchmarks will demonstrate the model’s lightweight, fast, and reliable performance in end-to-end AD applications.

Institution(s): Purdue University
University of Wisconsin-Madison

Award Year: 2025

Research Focus: Safety, Mobility

Project Form(s):