Safe and Personalized Control of Autonomous Vehicles with On-Board Vision Language Models: System Design and Real-World Validation

Principal Investigator(s):

Ziran Wang, Assistant Professor of Civil Engineering – Purdue University

Project Abstract:
This project focuses on enhancing autonomous vehicle control systems by integrating on-board Vision-Language Models (VLMs) for safe and personalized driving experiences. Building on the previously awarded CCAT project on “CAV Pilot Development and Deployment in Midwest Winter,” this research addresses critical challenges in autonomous vehicle development regarding limited on-board computational resources by implementing lightweight VLM frameworks and Retrieval-Augmented Generation (RAG)-based memory modules. The project will validate the system’s ability to handle challenging urban scenarios, reduce human takeover rates, and adapt to diverse environmental conditions.

Institution(s): Purdue University

Award Year: 2025

Research Focus: Safety, Mobility

Project Form(s):