AI Learns to Detect Driver Impairment from Gaze, Motion, and Video

AI Learns to Detect Driver Impairment from Gaze, Motion, and Video

John Gideon, Anshul Gupta, Megan Applegate-Kenton, Kayli Battel, Todd J Rowell +5 more

7 min read•Nov 18, 2026

An AI model that learns from driver gaze, vehicle motion, road video, and impairment labels improves impairment detection while also predicting where drivers look and how vehicles move. The work pairs this model, ID-MAE, with DIVERT, a 103-hour simulator dataset spanning 109 participants and multiple controlled impairment scenarios, giving researchers a richer testbed for systems that need to recognize risky human behavior.

What did the researchers build?

The researchers built two things: a multimodal AI model called Impaired Driving MAE, or ID-MAE, and a driving dataset called DIVERT. ID-MAE is designed to detect driver impairment by learning from several signals together: eye gaze, vehicle trajectories, video of the driving scene, and labels describing the driver’s impairment state. It also learns to predict gaze and vehicle movement, rather than training only to classify impairment.

DIVERT expands the data available to develop and evaluate such models. The researchers collected it in the CARLA driving simulator, with controlled driving tasks and hazards. The full dataset contains 103.3 hours of driving from 109 participants. Of that, 46.42 hours involve induced impairments across ten techniques and four categories: cognitive distraction, frustration and stress, visual distraction, and cognitive load.

This combination is intended to address gaps in earlier datasets, which could be small, lack video or motion data, use different levels of vehicle automation, or cover only a narrow range of impairment types. DIVERT supplies varied conditions while keeping a consistent set of rich data modalities.

Overview of ID-MAE's multimodal encoder-decoder architecture

What are the key results?

ID-MAE outperforms the baselines evaluated by the researchers on both DIVERT and the existing Impaired Driving Dataset (IDD). The paper also reports that reconstructing masked gaze and trajectory data improves impairment detection compared with training on impairment labels alone. In other words, asking the model to learn patterns in driver behavior provides a useful training signal beyond the classification task itself.

The authors also examine results by impairment type and assess the benefits of training across multiple types together. This matters because a detector trained on one impairment category might not transfer well to another. Joint training lets the model learn from shared behavioral patterns while still evaluating specific impairment categories.

The supplied paper text does not include the numerical detection scores or AUC values, so the size of the performance gains cannot be reported here. The dataset figures do provide useful scale: DIVERT includes 109 participants and 103.3 total hours, while IDD contains 23.7 hours from 52 participants and covers three impairment conditions—statement understanding, a 1-back task, and alcohol intoxication. The findings support the approach, but the missing score details limit direct comparison of how much better it performs.

Example of a driving hazard scenario represented in the DIVERT data

How does ID-MAE work?

ID-MAE uses an encoder-decoder transformer: a neural network that processes sequences of information and learns relationships among their parts. Instead of treating each camera frame or gaze measurement as an isolated input, it brings video, gaze, vehicle trajectory, and impairment information into one model.

The training approach is a form of masked diffusion. In broad terms, parts of the input are hidden or corrupted, and the model learns to recover the underlying data. This gives it a self-supervised learning task: it can learn structure from the sensor signals themselves, rather than depending entirely on impairment labels. The impairment state is handled as a discrete token, while gaze and trajectory are continuous signals.

The model uses geometric relative attention to help connect tokens based on their spatial relationships. Attention is the mechanism that lets a transformer weigh which inputs are relevant to each other; the added geometry-based bias gives it information about how tokens are positioned relative to one another. This is intended to help the model relate signals from different parts of the driving scene and behavior.

Separate output heads predict different targets. The impairment head produces a score for each impairment type. The gaze head predicts each eye’s yaw and pitch—the directions of eye movement. The trajectory head predicts acceleration and yaw rate, or how quickly the vehicle changes heading, then integrates those values into estimates of velocity, position, and heading. Predicting these clean behavioral targets alongside impairment labels encourages the model to learn a more complete picture of driving behavior.

Why does this matter for robotics?

Robots that operate around people often need to interpret more than the physical scene: they also need to account for human attention, workload, and behavior. ID-MAE is a driving research system, not a general-purpose robot safety product, but its core idea—combining visual context with behavioral signals and learning from partially hidden data—could inform monitoring in other human-machine settings.

For autonomous vehicles, detecting signs of impairment could support safety features that alert a driver or adjust how the vehicle operates. The study does not demonstrate a deployed intervention, but it offers a way to develop detectors using more than labels alone. Its dataset also gives researchers a structured starting point for testing whether a model recognizes different kinds of impairment rather than just one.

The same broader challenge appears in industrial environments, where people and automation share tasks and operators’ attention can affect safe coordination. Teams evaluating warehouse robots can consider human-state monitoring as one part of the wider safety system, alongside robot sensing and operating procedures. Before such methods are used in practice, they would need validation with real operators and real-world sensors.

What are the limitations and open questions?

DIVERT is collected in a simulator, where researchers can control tasks and hazards more consistently than on public roads. That makes experiments repeatable, but it does not establish that the model will work equally well in real vehicles, with different cameras, lighting, driver populations, or natural forms of impairment. The study covers four induced categories, which is broader than many datasets, but it does not represent every cause or degree of unsafe driving.

The reported results show qualitative improvements, but the supplied text omits exact benchmark scores, making it difficult to judge the size and reliability of the gains or compare them numerically with other systems. Further questions include how quickly the model runs, how it handles missing or noisy sensors, and whether its predictions are dependable enough to support safety-critical decisions. Those are important steps between a promising benchmark result and an operational tool.

What are the frequently asked questions?

What does ID-MAE detect? It classifies impairment states using gaze, vehicle trajectories, scene video, and impairment labels. The paper evaluates it on controlled impairment conditions, not as a clinical diagnostic system.

What is DIVERT? DIVERT is a CARLA simulator dataset with 103.3 hours of driving from 109 participants. It includes ten induction techniques spanning four impairment categories.

Does ID-MAE only classify impairment? No. It also predicts gaze direction and vehicle motion, including acceleration and yaw rate. These extra prediction tasks help the model learn from behavior as well as labels.

Was the model tested on real roads? The described DIVERT data comes from a driving simulator. The supplied text does not report a real-road deployment test.

What is the takeaway?

ID-MAE combines impairment classification with learning to reconstruct and predict driver behavior, and it outperforms the reported baselines on two driving datasets. DIVERT provides a larger, more varied simulator resource for studying how such models perform across different controlled impairment conditions.