Sekai: A Video Dataset towards World Exploration | Summary

This article explains the key points of Sekai: A Video Dataset towards World Exploration.

• 2025-06-18 (arXiv), 39th Conference on Neural Information Processing Systems (NeurIPS 2025)
• Li, Zhen, Li, Chuanhao, Mao, Xiaofeng, Lin, Shaoheng, Li, Ming, Zhao, Shitian, Xu, Zhaopan, Li, Xinyue, Feng, Yukang, Sun, Jianwen, et al.
• Shanghai AI Laboratory, Beijing Institute of Technology, Shanghai Innovation Institute, Shenzhen MSU-BIT University, The University of Tokyo
• Paper
• Github
• Project Page

Read this article in Korean

Summary

• Training video models for explorable environments requires long, diverse first-person footage annotated with camera trajectories, the camera’s positions and orientations over time. Compared with the baseline camera-controlled video model, fine-tuning on Sekai-Real drone footage reduces translation error from 28.32 to 17.19 and rotation error from 27.2 to 19.89.
• Sekai combines over 5,000 hours of real-world walking and drone footage across 101 countries and regions and more than 750 cities with game footage from Lushfoil Photography Sim. Its collection, filtering, annotation, and sampling pipeline supplies location, scene, weather, time of day, crowd density, captions, and camera trajectories.
• Annotation checks report location issues in less than 5% of 500 sampled videos and category agreement above 90% on 500 examples, with indoor scenes excluded from the latter analysis. Generation experiments show improved camera following and higher aggregate scores at later training checkpoints, but use 50 selected test clips and do not establish long-duration interactive exploration. Ordinary video generation results compare fine-tuning checkpoints without an unfine-tuned baseline.

1 Introduction

The paper defines world exploration as constructing a dynamic, realistic environment from image, text, or video for interactive navigation. Camera-controlled video generation offers a route to this goal because keyboard and mouse inputs can be converted into camera motion, but existing datasets often lack the duration, geographic diversity, and annotations needed to train this behavior.

• Sekai-Real contains YouTube footage, while Sekai-Game contains recorded gameplay with directly captured camera poses, each specifying a camera position and orientation. Walking footage retains audio.
• The contributions are the dataset, its curation pipeline, and validation through statistics, annotation checks, and generation experiments. The project URL is https://lixsp11.github.io/sekai-project/.

The example connects a walking video with its camera path, location, environmental labels, and detailed caption. It illustrates how movement supervision is paired with descriptions of the environment.

The map illustrates worldwide coverage through walking and drone examples, while the lower panel separates the smaller game collection. The reported 5064 hours of Sekai-Real and 36 hours of Sekai-Game show the difference in their scale.

2 Related Work

The related work covers models that generate explorable worlds and datasets used to train video generators. Sekai combines extended first-person movement with annotations describing the environment and camera motion.

2.1 World Generation Model

World generation research includes video generation, three-dimensional scene generation, and four-dimensional generation, meaning three-dimensional scenes that change over time. The paper discusses estimating depth to expand images into three-dimensional representations and generating temporally evolving environments, then focuses on data for interactive video generation.

2.2 Video Generation Dataset

The paper distinguishes datasets restricted to specific scenarios from datasets covering open scenarios. It describes the listed specific-scenario datasets as containing less than 800 hours and averaging less than 20 seconds per video, while larger open-scenario datasets still lack the combination of long sequences, camera poses, and environmental labels.

• OpenSoraPlan-V1.0 contains 274 hours with detailed captions, and OpenVid-1M contains 2100 hours but averages no more than 25 seconds per video. MiraData averages 72.1 seconds and lacks the exploration and world annotations emphasized here.
• This section describes Sekai videos as lasting 1 to 39 minutes, averaging 2 minutes. Figure 2 reports different duration statistics, and the PDF does not clearly explain how these descriptions relate.

3 Dataset Curation

Dataset curation consists of video collection, pre-processing, annotation, and sampling. Real-world videos require inferred annotations, while the game pipeline can capture camera poses directly and obtain environmental information from game-map descriptions.

The two branches distinguish inferred annotation of YouTube footage from direct camera-pose capture during gameplay. Quality and diversity sampling further select the real-world training subset; starred trajectory statistics apply only to a subset of annotations.

3.1 Video Collection

The authors identify 10471 hours of walking videos and 628 hours of drone videos from YouTube through manual collection and keyword searches. After download failures and broken videos, the downloaded material contains 8409 hours of walking footage and 214 hours of drone footage, totaling 8623 hours.

• Source videos were released during the preceding three years, last 30 minutes to 12 hours, and provide at least 1080P resolution at 30 to 60 frames per second. The pipeline downloads the 1080P version with the highest bitrate, the amount of encoded data per second.
• Lushfoil Photography Sim supports walking and first-person drone exploration of landscapes. The authors record 40 hours using OBS Studio, video-recording software, at 1080P and 30 frames per second, with an 8 to 12 Mbps bitrate and varied locations and weather.

3.2 Video Pre-processing

Pre-processing removes openings, endings, discontinuities, low-quality footage, subtitles, and implausible estimated camera motion. Shots, continuous sequences between detected transitions, are divided into one-minute clips, with shorter fragments discarded; contiguous clips can later be stitched for training.

• The pipeline trims two minutes from both ends of each YouTube video, detects shot boundaries with TransNetV2, a shot-transition detection model, at threshold 0.4, and trims five seconds from both ends of each shot. Acceleration using graphics processing units, or GPUs, with PyNVideoCodec for decoding and CVCUDA for frame operations is reported to make shot detection five times faster than the original implementation.
• Clips are encoded at 720p and 30 frames per second using H.265 video compression in MP4 files at 4 Mbps. Peak signal-to-noise ratio, a reconstruction fidelity measure, exceeds 35 in the reported transcoding evaluation; walking audio is encoded using AAC audio compression at 48 kHz.
• Luminance filtering removes clips with more than 15 consecutive frames of extremely high or low average brightness and excludes 300 hours. COVER, a video quality evaluator, removes the lowest-scoring 10% by technical quality; VideoSubFinder, a subtitle detector, removes approximately 5% containing subtitles visible for more than 0.75 seconds in the bottom one-third of frames.
• On the selected trajectory-annotated subset, clips are excluded for multiple direction reversals exceeding 150 degrees within 10 seconds, a viewpoint change greater than 60 degrees between consecutive frames, or a positional displacement greater than five times the average displacement over the containing 30-frame window.

3.3 Video Annotation of Sekai-Real

Sekai-Real annotations combine source metadata, visual interpretation, and estimated camera geometry. Location extraction uses GPT-4o, a language model, to structure YouTube chapter descriptions with ISO 3166 standardized country or region codes; timestamp matching assigns clips to chapters, and approximately 8% of clips are discarded because they cannot be uniquely matched.

• Qwen2.5-VL-72B, a model that processes visual and textual inputs, first labels scene, weather, time of day, and crowd density, abstaining when uncertain. It then uses those labels, location, and frames sampled every two seconds to generate time-ordered captions averaging over 176 words.
• The authors compare camera-motion estimation methods and select MegaSaM, a system for estimating scene structure and camera motion from video. They replace its single-image depth estimator with Video Depth Anything, a video depth-estimation model, to improve temporal consistency and adapt inference for multiple machines and GPUs.

3.4 Video Annotation of Sekai-Game

The Sekai-Game pipeline uses RE-UE4SS, a scripting system for accessing Unreal Engine objects, together with OBS Studio to capture video and annotations. Its tools standardize camera settings, record camera poses in real time, and hide the graphical user interface.

• Location and category information come from game-map descriptions, while caption prompts are adapted to gameplay. Recorded poses are calibrated for delay and interpolated to align with video frames; these captured poses and game-derived metadata have a more direct reference than annotations inferred from real-world footage.

3.5 Video Sampling

Video Sampling produces a 400-hour subset named Sekai-Real-HQ to fit the available training budget. The procedure first selects for quality, then sequentially samples for diversity in content, location, category, and camera motion.

• 3.5.1 Quality Sampling sums COVER’s aesthetic and semantic quality scores, measuring visual harmony and content completeness or consistency, and retains the highest-scoring 70% of clips.
• 3.5.2 Diversity Sampling uses InternVideo2, a video representation model, to extract embeddings, numerical representations of clip content. Mini-batch K-Means, a clustering method, groups these representations within each country or region; quality-ranked selection and removal of similar clips retain 70% at the content stage. Location sampling retains 60%, reallocating quotas when cities contain too few clips.
• Category sampling retains 60% using probabilities inversely proportional to label frequencies across weather, scene, time of day, and crowd density. It multiplies probabilities across these categories under an independence assumption, normalizes them, and samples without replacement.
• Camera sampling retains 75% by jointly grouping clips according to start-to-end movement direction and jitter, defined as the Euclidean norm of positional variance computed every 30 frames. It samples across these groups to diversify camera motion.

4 Dataset Statistics

Sekai-Real spans 101 countries and regions, but its geographic distribution remains concentrated: the top eight countries account for about 60% of total duration. Outdoor footage and daytime scenes dominate, while crowd-density labels span sparsely populated environments through crowded streets.

• Figure 2 reports 5064 hours of Sekai-Real and 36 hours of Sekai-Game across nine game locations. It gives a Sekai-Real duration range of 1 minute to 5.9 hours and an 18.5-minute average; the relationship to the one-minute processing clips and Section 2.2’s duration statistics is unclear.
• More than half of Sekai-Real videos score above 0.9 on the reported overall quality measure. Sekai-Real-HQ has higher mean quality and lower variance, a more balanced geographic distribution, and a smoother camera-jitter distribution.
• Sekai captions have a higher average token count than OpenVid-1M captions, where tokens are the text units counted in the comparison. This establishes longer captions, while the plot does not establish greater caption accuracy.

The segmented distribution shows that geographic breadth coexists with concentration in a few countries and dominant scene or lighting conditions. Crowd-density labels span different levels of human presence.

The quality and location panels show higher quality scores and a more balanced geographic distribution after selection into Sekai-Real-HQ. The caption panel shows longer captions than OpenVid-1M, while the jitter panel describes camera-motion redistribution rather than geometric accuracy.

5 Experiments

Experiments evaluate annotation quality, text-conditioned and image-conditioned video generation using Sekai-Real-HQ, and camera-controlled generation using camera annotations. The ordinary generation study fine-tunes SkyReels-V2, a pretrained video generator; the interactive study fine-tunes Wan2.1-Fun-V1.1-1.3B-Control-Camera, a camera-conditioned video generator.

5.1 Evaluation of Annotation Quality

Annotation validation checks location, camera trajectories, and category labels. The numerical evidence concerns sampled location issues and category agreement; the trajectory comparison reports relative smoothness and annotation quality without a numerical error table against known camera poses.

• Co-authors inspect 500 randomly sampled videos using online maps for omitted or incorrectly merged segments, timestamp mismatches, and locations inferred without explicit source support. The paper reports such issues in less than 5%.
• MegaSaM and VGGT, a model predicting three-dimensional quantities from images, are compared on 100 walking and 100 drone-view clips. The authors report smoother trajectories from MegaSaM and faster inference from VGGT, motivating their choice of MegaSaM.
• For 500 examples sampled to maintain category variety, agreement between Qwen2.5-VL and co-author labels exceeds 90%. Indoor scenes are excluded from the analysis, and weather errors mainly confuse cloudy or foggy conditions with rain, whose evidence may require consecutive frames.

5.2 Video Generation

5.2.1 Settings uses SkyReels-V2-T2V-14B-540P for text-to-video generation, abbreviated T2V, and SkyReels-V2-I2V-14B-540P for image-to-video generation, abbreviated I2V. Training uses 544×960×49 video dimensions, 16 frames per second, batch size 1, learning rate 1e-5, Adam, an adaptive gradient optimizer, and 8 NVIDIA H100 GPUs, with a stated total of 20,000 iterations. Inference uses the same dimensions and frame rate with 50 steps.

• Sekai-Real-HQ is randomly split into ten folds, with the final two excluded from training. Independent annotators select 50 diverse clips from these held-out folds for evaluation.
• VBench, a suite of video generation metrics, measures subject appearance consistency, background stability, motion smoothness, motion extent through Dynamic Degree, visual appeal through Aesthetic Quality, and distortions through Imaging Quality. VBench++, an extended evaluation suite, adds I2V Subject and I2V Background to assess correspondence with the input image.

5.2.2 Quantitative Results reports rising Overall Score across the listed training checkpoints: T2V increases from 4.26 at 5000 steps to 4.34 at 20000, and I2V increases from 6.10 at 5000 to 6.29 at 17500. Table 1 demonstrates improvement over earlier fine-tuning checkpoints but does not quantify improvement over the original unfine-tuned models.

• T2V subject consistency rises from 91.93% to 96.02%, aesthetic quality from 48.38% to 52.09%, and imaging quality from 57.04% to 60.90%. Dynamic Degree falls from its 94.67% peak at 10000 steps to 78.66% at 20000.
• I2V Subject increases from 93.74% to 97.27%, and I2V Background from 93.85% to 96.65%. Imaging quality peaks at 68.54% at 10000 steps before falling to 66.54% at 17500, while Dynamic Degree declines from 100.00% to 85.00%.
• The authors interpret reduced motion as a shift toward cleaner frames and sharper details. The table shows declining motion extent alongside increases in some quality metrics, but does not isolate the cause.

Overall Score increases across the listed fine-tuning checkpoints for both tasks, while Dynamic Degree declines at later stages. I2V imaging quality falls after its intermediate peak, so improvement varies by metric; the table contains no zero-step comparison.

5.3 Interactive Video Generation

5.3.1 Settings fine-tunes Wan2.1-Fun-V1.1-1.3B-Control-Camera using an initial image, text prompt, and camera trajectory. Separate runs train for two epochs, two passes through the training data, on Sekai-Real drone footage and the entire Sekai-Game dataset, respectively. Evaluation uses 50 test videos selected through the previously described procedure.

• The baseline converts discrete camera actions into poses and then into Plücker embeddings, the per-frame camera representation supplied to the model. Fine-tuning instead uses Sekai’s annotated per-frame poses.
• TransErr and RotErr measure translation and rotation discrepancies between the input trajectory and the trajectory estimated from generated video using MegaSaM. Lower values indicate closer camera following; the tables do not specify physical units.

5.3.2 Quantitative Results reports that Sekai-Real drone fine-tuning reduces TransErr from 28.32 to 17.19 and RotErr from 27.2 to 19.89, absolute reductions of 11.13 and 7.31. Sekai-Game fine-tuning reduces these errors from 7.64 to 4.22 and from 8.36 to 6.22, respectively.

• For real-world drone generation, background consistency improves from 94.85% to 95.56%, aesthetic quality from 58.25% to 59.18%, and imaging quality from 74.73% to 75.83%. Subject consistency falls from 97.61% to 97.34%, and motion smoothness from 99.22% to 99.11%, so gains are not uniform across metrics.
• The camera-error reductions support improved trajectory following in the tested settings. MegaSaM supplies both real-world training annotations and generated-video trajectory estimates, so this evaluation does not independently establish absolute geometric accuracy.

Sekai-Real drone fine-tuning lowers both camera-trajectory errors. Background consistency, Dynamic Degree, aesthetic quality, and imaging quality improve, while subject consistency and motion smoothness decline slightly.

The game-data experiment also improves camera following: translation error decreases from 7.64 to 4.22 and rotation error from 8.36 to 6.22. Only these two errors are reported, leaving accompanying visual-quality changes unmeasured in this table.

6 Conclusion

The paper presents Sekai as a dataset of walking and drone footage with environmental and camera annotations, together with a sampling procedure that selects training data for a compute budget. Its experiments demonstrate utility for the tested video generation and camera-following tasks; unrestricted world exploration remains a broader objective.

Appendix

• The supplied 14-page PDF contains no appendix. Pages 11–14 contain acknowledgments and references.

Brief Thoughts

Sekai combines geographically diverse first-person footage with camera and environmental supervision. The camera-control experiments offer direct baseline comparisons, while the ordinary generation study compares fine-tuning checkpoints without an unfine-tuned baseline or an alternative training dataset. Long recordings do not establish long-duration generation: the SkyReels-V2 experiments use 49-frame clips at 16 frames per second. Persistent navigation, revisiting locations, and sustained scene consistency are not evaluated. The manually selected 50-clip evaluation sets and lack of uncertainty estimates limit conclusions about generalization. The paper describes random folds but does not state whether all clips from an original recording remain in one fold. Duration statistics and intermediate game-data totals differ across the text and figures, leaving the dataset accounting unclear. Caption length and camera-path smoothness describe the annotations but do not directly establish semantic correctness or geometric accuracy.