Dolphins: Multimodal Language Model for Driving | Summary

This article explains the key points of Dolphins: Multimodal Language Model for Driving.

• 2023-12-01 (arXiv)
• Ma, Yingzi, Cao, Yulong, Sun, Jiachen, Pavone, Marco, Xiao, Chaowei.
• University of Wisconsin-Madison, NVIDIA, University of Michigan, Stanford University
• Paper
• Github

Read this article in Korean

Summary

• Dolphins addresses how a driving assistant can interpret video, answer questions, and reason about vehicle behavior in varied road scenes. In one qualitative tunnel example, it says the time cannot be determined when given an example in its prompt, whereas it answers that it is night without one; the paper reports no comparative accuracy score.
• The vision-language model, which combines visual input with language, builds on OpenFlamingo and is tuned first on grounded reasoning examples and then on driving instructions. Its demonstrations cover perception, planning, conversation, and responses to corrective feedback.
• The driving dataset contains 32k video-instruction-answer triplets, including 11k detailed conversations generated using ChatGPT. The authors report an average inference time of 1.34 seconds on an NVIDIA A100 for DriveLM and identify computational cost as a deployment constraint.

1 Introduction

The authors identify scene interpretation, adaptation to unfamiliar situations, and correction from feedback as challenges for autonomous driving systems. They propose Dolphins as a conversational assistant that accepts video or images, instructions, and historical control signals to support high-level driving decisions.

The diagram connects the two tuning stages to the abilities later illustrated in the demonstrations, including adaptation and feedback-driven correction.

2 Related Work

The paper situates Dolphins among language-based driving systems, vision-language models, and multimodal in-context learning, where examples are supplied in the prompt. The authors describe prior driving models as limited in task diversity or video input and use OpenFlamingo’s interleaved-input design to address these limitations.

3 Method

The method first trains on general image questions with step-by-step grounded responses, then transfers the resulting capabilities to a driving video instruction dataset. Retrieved examples are included during driving instruction tuning to support adaptation to new tasks.

3.1 GCoT Instruction tuning

Grounded Chain of Thought (GCoT) uses ChatGPT to generate an image description, identify the queried object and its position, and provide reasoning when needed, followed by the reference answer. These responses train the model on general visual question answering, which answers questions about images, before driving-specific fine-tuning.

The surfboard example shows how captions, object annotations, a question, and an answer are converted into a staged training response.

3.2 Devised Instruction Tasks for Autonomous Driving

The four driving tasks are Behavior Understanding, Behavior Reasoning, Prediction with Control Signals, and Detailed Conversation. The control task predicts the ego car’s speed and turning angle for the next second from historical signals; the ego car is the vehicle whose camera supplies the scene.

BDD-X supplies roughly 7,000 videos and approximately 25,000 annotated examples. The resulting 32k triplets comprise 21k from BDD-X labels and 11k ChatGPT-generated detailed conversations that expand descriptions with traffic rules, risks, and precautions.

The examples distinguish short behavior and control tasks from the longer ChatGPT-enriched conversation task.

3.3 Multi-modal In-context Instruction Tuning in Autonomous Driving

The retrieval procedure describes selecting examples by cosine similarity between video or text representations, but training uses text similarity only, with at most k = 3 examples per triplet. The text representations are produced by text-embedding-ada-002; retrieved examples precede the current instance in the prompt.

4 Training

Training proceeds in two stages: image instruction tuning with GCoT templates, followed by driving-video instruction tuning. The paper omits the general video-text alignment stage used by the video models it discusses.

4.1 Model Architecture

Dolphins combines a CLIP vision encoder, a perceiver resampler that processes visual features, and a language model with gated cross-attention connecting visual and language features. Learned temporal and media position embeddings encode frame and example order; the encoders are frozen while the resampler, cross-attention layers, and low-rank adaptation (LoRA) modules are fine-tuned.

The architecture places temporal and media embeddings on visual inputs and connects resampled features to language blocks through gated cross-attention.

4.2 Implementation Details

Training uses cross-entropy loss on answer tokens, DeepSpeed, and AdamW with β1 = 0.9, β2 = 0.999, and weight decay 0.01. The authors use 4 NVIDIA A100 GPUs, batch size 128, a second-stage learning rate of 2 × 10−5, maximum sequence length 1024, and BF16 precision for training and inference.

The template places a task definition and example exchanges before the current video and computes loss on answer sequences.

5 Demonstration

The demonstration section presents selected zero-shot examples, without task examples in the prompt, and few-shot examples, with a small number of examples, across scene understanding, prediction, planning, adaptation, conversation, and correction. Although the introduction mentions quantitative metrics, this version principally presents qualitative examples rather than a comparative performance table.

5.1 Holistic Understanding and Reasoning

In one rainy-intersection example, Dolphins describes the scene and attributes the ego car’s stop to a red light; in another, it recommends slowing or stopping if a van ahead turns or decelerates. The authors acknowledge limited relevant training instructions for prediction and planning and recommend supporting these tasks with examples, dialogue, and control signals.

5.2 Human-like Capabilities

In the tunnel example, an in-context demonstration changes the answer from night to uncertainty about the time. In a rainy-intersection dialogue, the model answers successive questions about weather, driving caution, and the traffic light; in another interaction, it revises a prediction from going forward to turning right after a user explains a flashing light.

The tunnel comparison provides a within-example contrast: without an example the model says night, while with one it says the time cannot be determined.

The correction follows an explicit user explanation of the turn signal, demonstrating response revision with supplied feedback.

6 Conclusion and Future Directions

The authors report a mean inference time of 1.34 seconds on an NVIDIA A100 for the DriveLM dataset and identify latency and vehicle power consumption as obstacles to deployment. They suggest more efficient customized or distilled models as future directions.

Appendix

• A Data: The first-stage image dataset has approximately 10.7k examples, including 9,645 visual question answering examples with ChatGPT-generated GCoT responses. It draws on VQAv2, OK-VQA, GQA, and TDIUC, supplements annotations with Visual Genome, and includes LLaVA-instruct-80k and SVIT data.
• B Prompts: The appendix provides the prompt used to generate three-step GCoT responses from captions, object locations, a question, and a reference answer. Its illustrated example identifies a surfboard and gives the reference answer 1926.

Brief Thoughts

The examples show that prompt context and explicit feedback can change individual responses, but they do not establish how often those changes are correct across a test set. Figures 6 and 9 also mark hallucinated text, and the reflection examples supply corrective information, limiting claims about independent error detection or driving safety.