Monthly AI Papers: September 2025

Contents

Trends
Top Papers in September 2025
Sources

Trends

September 2025’s theme across these three papers is measuring useful model capabilities under explicit data and evaluation conditions: LIMI studies learning interactive task execution from curated demonstrations, Qwen3-Omni tests integration of text, images, audio, and video, and SimpleQA Verified revises how factual recall is graded. Their shared engineering concern is what a reported improvement actually measures. Demonstration counts need trajectory-length context, integrated models need comparisons against specialized capabilities, and factuality scores need rules for correct answers and abstention, responses that do not attempt an answer. 1, 2, 3

Data construction is a concrete intervention in both training and evaluation. LIMI’s 78 demonstrations contain complete trajectories, sequences of reasoning, tool calls, and observations, averaging 42.4k tokens each. SimpleQA Verified reduces 4,326 questions to 1,000 through deduplication, source checks, balancing, and difficulty selection. These results make example count an incomplete description of either training resources or benchmark coverage: LIMI does not match training tokens or collection effort across datasets, while SimpleQA Verified’s filters change the knowledge distribution being tested.

Aggregate scores also require task-level inspection. Qwen3-Omni broadly preserves text and vision capabilities in a matched base-model comparison, with declines on individual tasks. LIMI’s reported averages with and without its execution environment cover different benchmark sets, preventing that comparison from isolating the environment’s contribution. SimpleQA Verified reports overall accuracy, accuracy among attempted answers, and attempt rate, exposing how willingness to answer affects factuality rankings. Together, these studies support reading headline results alongside their comparison conditions and component metrics.

Top Papers in September 2025

1. LIMI: Less is More for Agency

https://arxiv.org/abs/2509.17567
LIMI: Less is More for Agency | Summary
LIMI: Less is More for Agency studies agency, the capacity to discover problems, formulate hypotheses, and execute solutions through environments and tools. Supervised fine-tuning, training on recorded demonstrations, adapts GLM-4.5 using 60 real-world queries and 18 queries synthesized from GitHub pull requests, proposed repository changes. GPT-5 and four human collaborators collect successful trajectories, including feedback and recovery from intermediate failures. For Supervised fine-tuning, see Towards a Unified View of Large Language Model Post-Training | Summary.
On AgencyBench, an execution-based benchmark containing four collaborative coding tasks and six research or information-processing tasks, LIMI raises the reported average from 45.1% for GLM-4.5 to 73.5%. First-Turn Functional Completeness, the percentage of requirements correctly implemented initially, rises from 37.8% to 71.7%; success within three allocated rounds rises from 47.4% to 74.6%.
The same base model fine-tuned on 10,000 AFM-CodeAgent-SFT-Dataset samples reaches 47.8%, compared with LIMI’s 73.5% from 78 demonstrations. The difference is 25.7 percentage points, approximately 53.7% relative improvement. Trajectories range from 13k to 152k tokens, so the comparison establishes an advantage by example count without establishing equivalent savings in training tokens, annotation effort, or collection cost.

2. Qwen3-Omni Technical Report

https://arxiv.org/abs/2509.17765
Qwen3-Omni Technical Report | Summary
Qwen3-Omni Technical Report combines text, image, audio, and video inputs with text and speech outputs. Its Thinker interprets inputs and generates text, while its Talker generates speech. Both use Mixture-of-Experts architectures, which selectively activate expert components. Multiple codebooks, sets of discrete codes representing each audio frame, support streaming waveform generation through a decoder that uses current and preceding context.
The controlled comparison matches contemporaneous 30B-A3B base models, identifiers distinguishing total and active parameter counts, on training schedules, shared modality datasets, batch sizes, effective modality-specific epochs, and compute. Results support broad capability preservation at this scale: multidisciplinary visual-question performance on MMMU validation rises from 57.22 to 59.33, while video understanding on MVBench falls from 71.87 to 69.50. The study provides no comprehensive scale sweep or uncertainty estimates.
Reported theoretical cold-start first-packet latency, the time to produce the first speech packet without prior context, is 234 ms for audio and 547 ms for video at one concurrent request. At six concurrent requests, these increase to 1172 ms and 2284 ms. The generation real-time factor, processing time divided by produced audio duration, remains below 1 at 0.66 for six requests; the table does not identify inference hardware.

3. SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge

https://arxiv.org/abs/2509.07968
SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge | Summary
SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge evaluates parametric factuality, recall of facts stored in model parameters without external assistance. Its 1,000 short-answer questions come from 4,326 SimpleQA questions after source and question deduplication, publisher-access filtering, answer-type and topic balancing, source reconciliation, and difficulty selection.
With standard application programming interface parameters and no tools, Gemini 2.5 Pro scores 55.6% F1-Score, compared with 52.3% for GPT 5 and 51.9% for o3. Here F1-Score is the harmonic mean of accuracy over all questions and accuracy over attempted answers. Claude Opus 4 illustrates the effect of abstention: 54.1% accuracy among attempted answers combines with a 35.5% attempt rate to produce 19.2% overall accuracy and 28.3% F1-Score.
The revised autorater, a language model prompted to grade answers, uses question-specific numeric acceptance ranges and clearer rules for direct answers, uncertainty, and abstention. Evaluation uses gpt-4.1-2025-04-14 as the autorater. The supplied examples motivate the revisions, but no aggregate validation against independent human labels is reported. Because grading ignores extra information that does not contradict the reference answer, the score measures targeted factual recall rather than the accuracy of every statement in a longer response.