LLM360: Towards Fully Transparent Open-Source LLMs | Summary

This article explains the key points of LLM360: Towards Fully Transparent Open-Source LLMs.

• 2023-12-11 (arXiv)
• Liu, Zhengzhong, Qiao, Aurick, Neiswanger, Willie, Wang, Hongyi, Tan, Bowen, Tao, Tianhua, Li, Junbo, Wang, Yuqi, Sun, Suqi, Pangarkar, Omkar, et al.
• Petuum, MBZUAI, USC, CMU, UIUC, UCSD
• Paper
• Github

Read this article in Korean

Summary

• LLM360 addresses the difficulty of reproducing and studying large language models (LLMs), models trained on large text collections, when releases omit training data, code, and intermediate states. Its initial release provides two approximately 7-billion-parameter models and their training artifacts; under the paper’s evaluation that equally weights the language-task and code-task averages, CRYSTALCODER scores 41.65 versus CodeLlama-7B’s 39.94.
• The framework connects data and its training order, implementation and configuration, checkpoints, and metrics throughout pretraining, the models’ initial training process. Checkpoints are saved model states: AMBER has 360 released checkpoints, while CRYSTALCODER has 143 and uses three stages of English-and-code training totaling approximately 1382 billion tokens, the text units processed by the model.
• The released artifacts support tracking benchmark performance and exact reproduction of training continuations across AMBER checkpoints. AMBER lacks saved optimizer states, the training algorithm’s accumulated history, and its early checkpoints used reduced precision, limiting exact training recovery; its reported four-benchmark average of 45.20 also trails LLaMA-7B’s 49.69.

1 Introduction

The introduction identifies three problems with incomplete releases: Data Provenance, the origins and composition of training data; Reproducibility, the ability to repeat reported training and evaluation; and Open Collaboration, research enabled by access to intermediate training artifacts. Undisclosed data makes bias and benchmark leakage, overlap between training and evaluation data, harder to investigate, while missing code and checkpoints forces researchers to repeat expensive training. LLM360 proposes releasing exact pretraining data, model and training configurations, saved checkpoints, and collected evaluation results.

2 Related Work

Pythia is the closest predecessor, providing 154 checkpoints for models ranging from 70 million to 12 billion parameters to study learning dynamics and scaling. LLM360 extends this approach to trillion-token training and staged language-and-code mixtures; the authors state that AMBER surpasses Pythia-12B on public benchmarks, although this section supplies no numerical comparison. Table 1 compares disclosure across seven artifact categories, but its checkmark for AMBER optimizer checkpoints conflicts with Section 4.1.5, which states that these states were not saved.

The comparison separates openness into seven artifact categories, showing which releases expose the materials needed to examine training. AMBER and CRYSTALCODER receive checkmarks throughout, but AMBER’s optimizer-state checkmark is contradicted by Section 4.1.5.

3 The LLM360 Framework

The LLM360 Framework specifies four coordinated release components: Training Dataset and Data Processing Code; Training Code, Hyperparameters, and Configurations; Model Checkpoints; and Metrics. The framework recommends releasing model weights and applicable optimizer history, alongside system utilization, training loss, gradient norms, and downstream evaluations aligned with data chunks and checkpoints. Here, loss measures the training objective, while gradient norms summarize the magnitude of the gradients used to update model parameters.

• The authors report that tuned data-model-pipeline parallelism, which distributes data, model computation, and successive model stages across devices, provides up to 15% greater throughput than PyTorch FSDP, fully sharded data parallel training, on their Nvidia A100 clusters. The report does not provide a detailed throughput comparison.
• They also report that the inverse-frequency matrix used by rotary positional embeddings (RoPE), the model’s positional representation, should remain in FP32, 32-bit floating-point precision.
• Training metrics are linked at https://wandb.ai/llm360/projects, and analysis code is linked at https://github.com/llm360/analysis360.

The diagram aligns model checkpoints, data chunks, and metrics across training and groups their supporting code alongside them. This correspondence enables checkpoint-specific analysis; the schematic does not give the literal checkpoint count for every released model.

4.1 Amber

The Initial Model Release introduces AMBER, a general-purpose English model, together with AMBERCHAT and AMBERSAFE, variants adapted through additional training. The report calls AMBER a 7B model, while its architectural table specifies 6.7 billion parameters.

The amber illustration serves as AMBER’s model identity beside the introduction of the English model and its fine-tuned variants.

4.1.1 Details on Data Preparation and Model Architectures

AMBER combines RefinedWeb, StarCoder, and RedPajama-v1 data, including C4, without additional cleaning, filtering, or subsampling; a global shuffle divides the reported 1259.13 billion tokens into 360 equal chunks. It uses the LLaMA-7B architecture with 32 layers, 32 attention heads, hidden size 4096, intermediate size 11008, maximum sequence length 2048, and vocabulary size 32000.

• Table 2 lists Arxiv 30.00B, Book 28.86B, C4 197.67B, Refined-Web 665.01B, StarCoder 291.92B, StackExchange 21.75B, and Wikipedia 23.90B tokens, where B denotes billion. Refined-Web is the largest component.
• The AdamW optimizer, the algorithm used to update model parameters, uses β1 = 0.9, β2 = 0.95, a cosine learning-rate schedule from 3e−4 to 3e−5, weight decay 0.1, gradient clipping at 1.0, and 2,000 warmup steps.
• The batch size is 2,240, calculated as 224 × 10, rather than LLaMA’s 2,048. Table 3 specifies RMSNorm, the architecture’s normalization operation, with ε = 1e−6.

Refined-Web contributes 665.01 billion tokens and StarCoder contributes 291.92 billion, making them the two largest sources in the reported 1259.13-billion-token mixture. These counts specify the data composition used for AMBER.

The configuration identifies AMBER’s parameter count as 6.7B and specifies its depth, widths, attention heads, context length, and vocabulary. It supports architectural comparison with LLaMA-7B, while the surrounding sections document differences in training data and execution.

4.1.2 Details on the Pre-training Infrastructure

AMBER is trained on 56 DGX A100 nodes, each containing four 80GB A100 graphics processing units, for 224 devices in total. The cluster uses four NVLink connections per device and two-port 200 Gb/sec InfiniBand connections across nodes; these are the device and node interconnects, respectively, and the reported throughput is approximately 582.4k tokens per second. Its lit-llama implementation, based on PyTorch Lightning, uses mixed precision with BF16, a 16-bit floating-point format, for activations and gradients and FP32 for model weights.

• Figure 3 shows a sharp initial decline in training loss followed by a slower decline with fluctuations across checkpoints. The numerical failures and checkpoint problems are documented separately in Section 4.1.5.
• The implementation is linked at https://github.com/Lightning-AI/lit-llama.

Training loss falls sharply near initialization and then improves more gradually with visible variation. The curve summarizes optimization progress across checkpoints; the separately reported skipped chunks and checkpoint problems qualify interpretation of the training process.

4.1.3 Finetuned AMBER models

AMBERCHAT is described here as fine-tuned on the evolved instruction data used by WizardLM, using the FastChat training framework for 3 epochs, or passes through the data, on eight 80GB A100 devices with FSDP, learning rate 2 × 10−5, 16 gradient accumulation steps, and warmup ratio 0.04. Gradient accumulation combines gradients from multiple steps before updating parameters. AMBERSAFE is trained on ShareGPT 90K and further optimized on SafeRLHF, a human-preference dataset used for safety alignment, with β = 0.1, four gradient accumulation steps, and learning rate 5 × 10−7.

• The report calls the alignment procedure “Direct Parameter Optimization (DPO),” while its cited reference [39] is titled “Direct preference optimization: Your language model is secretly a reward model.” It is the preference-based optimization procedure used for AMBERSAFE.
• The ShareGPT 90K footnote specifies checkpoint 355 as the base rather than the final checkpoint. Section 4.1.4 also describes an AMBERCHAT experiment using ShareGPT 90K, leaving its relationship to the WizardLM-data recipe unclear.

4.1.4 Results and Analysis

All 360 AMBER checkpoints are evaluated on the Open LLM Leaderboard’s four benchmarks: ARC, the AI2 Reasoning Challenge; HellaSwag and MMLU, two language-evaluation datasets in the suite; and TruthfulQA, its truthfulness evaluation. Figure 4 specifies 25-shot ARC, 10-shot HellaSwag, 5-shot MMLU, and 0-shot TruthfulQA, where the shot count denotes the number of examples supplied for evaluation. ARC and HellaSwag improve overall, MMLU initially declines before recovering, and TruthfulQA declines as pretraining proceeds.

• Table 4 reports AMBER scores of 41.89 ARC, 74.14 HellaSwag, 30.76 MMLU, and 34.00 TruthfulQA, averaging 45.20. Its average exceeds RedPajama-INCITE-7B-Base’s 44.65 but trails OpenLLaMA-v2-7B’s 48.18 and LLaMA-7B’s 49.69.
• AMBERCHAT averages 49.12, and AMBERSAFE averages 53.15. AMBERSAFE’s TruthfulQA score is 55.44 versus AMBER’s 34.00; these evaluations do not measure safety comprehensively.
• The authors describe ARC and HellaSwag as increasing monotonically. The plotted checkpoints show local fluctuations, supporting an overall upward trend.

ARC and HellaSwag rise overall, MMLU falls initially and improves later, and TruthfulQA declines. The respective 25-shot, 10-shot, 5-shot, and 0-shot settings specify the evaluation conditions.

AMBER’s average of 45.20 is below LLaMA-7B’s 49.69, while AMBERCHAT and AMBERSAFE reach 49.12 and 53.15. AMBERSAFE’s 55.44 TruthfulQA result is the highest in the table, but this benchmark does not measure safety comprehensively.

4.1.5 Issues Encountered During Pre-training

AMBER encountered NaN loss, a nonnumeric loss value, in four of its 360 data chunks; these chunks were skipped and still failed when moved later in training. The authors completed the learning-rate schedule by repeating the first four chunks. They also disclose missing optimizer states, which required AdamW history to be reinitialized on resumption, and an early bug that saved checkpoints in BF16 despite maintaining training weights in FP32.

• The authors suggest that missing optimizer history may have contributed to instability and early checkpoint precision may have reduced accuracy. Neither explanation is isolated by a controlled experiment.
• Saved weights support analysis and further training, but the missing optimizer history and reduced-precision early checkpoints limit reconstruction of the original training states.

4.2 CRYSTALCODER

CRYSTALCODER combines English and code across three stages totaling approximately 1382B tokens: around 345B tokens from half of SlimPajama; approximately 927B tokens from its remaining half plus two epochs of StarCoder; and 100B tokens of Python and web-related code plus 10B SlimPajama tokens. Its LLaMA-7B-like architecture incorporates maximal update parameterization (muP), a parameterization associated with hyperparameter transfer in reference [44], applies RoPE to the first 25% of hidden dimensions, and uses sequence length 2048 and LayerNorm, a normalization operation supported efficiently by its hardware. The report describes 32032 as an “embedding dimension,” and training runs on Cerebras Condor Galaxy 1 (CG-1), a 4 exaFLOPS, or 4 × 10^18 floating-point operations per second, 54 million core, 64-node system.

• Figure 6 tracks the four language benchmarks plus HumanEval and MBPP, the two code-generation benchmarks, using pass@1, the single-attempt success metric. Stage transitions coincide with changes in benchmark trajectories, including marked code-score improvements after code enters training.
• Table 5 gives CRYSTALCODER a language-task average of 50.92 and a code-task average of 32.38, producing the equally weighted overall score 41.65. CodeLlama-7B scores 42.42, 37.45, and 39.94 respectively: CRYSTALCODER has the higher language average and lower code average.
• Mistral-7B scores 48.68 overall and exceeds CRYSTALCODER on every listed individual benchmark. CRYSTALCODER’s overall score exceeds several listed baselines, but does not lead the comparison.

The crystal illustration serves as CRYSTALCODER’s model identity beside the description of its English-and-code training mixture.

Dashed lines mark the two transitions between three training stages. Code benchmark scores rise strongly after the first transition, while HellaSwag drops at stage boundaries and MMLU improves substantially later in training. These changes coincide with changes in the training mixture, without isolating their causal effects.

CRYSTALCODER’s 50.92 language average and 32.38 code average produce 41.65 overall, compared with CodeLlama-7B’s 39.94 and Mistral-7B’s 48.68. The overall score weights the two task groups equally. OpenLLaMA-v2-7B’s reported code average is inconsistent with its two listed code scores.

4.3 ANALYSIS360

ANALYSIS360 demonstrates checkpoint-based research through memorization, measured as token accuracy when generating a training sequence’s continuation from its prefix. The experiment samples 1000 sequences from each of 360 data chunks, takes the first 64 tokens, and uses a 32-token prompt to generate a 32-token continuation; a score of 1 makes the sequence 32-extractible, meaning the continuation is reproduced exactly.

• The score is \(\operatorname{score}(k,l)=\frac{1}{l}\sum_{i=1}^{l}\mathbf{1}[S_{k+i}=G_{k+i}]\), where S is the training sequence and G is the generated continuation; the experiment sets k = l = 32.
• Figure 7 evaluates ten selected checkpoints using only chunks already encountered at each checkpoint. The annotated exact-match proportion rises from 0.8% at checkpoint 35 to 1.3% at checkpoint 355, with intermediate fluctuations.
• Figure 8 shows stronger memorization of recently seen data and an initial decrease followed by later increases for a given chunk group. Figure 9 shows strong correlations across checkpoints in both token-level memorization scores and exact extraction.

The distributions use previously encountered training chunks for each selected checkpoint and highlight exact 32-token continuation matches. Their annotated proportion increases from 0.8% to 1.3% between the first and last displayed checkpoints, with fluctuations in between.

Markers identify the latest chunk group seen at each checkpoint, and points to their right concern unseen data. The curves support the authors’ observations of stronger recent-data memorization and an initial decline followed by later growth for previously encountered data.

The two matrices show strong cross-checkpoint correlations in sequence-level token accuracy and exact extraction. They describe persistence in which sequences are memorized, without identifying its cause.

5 Summary and Take-home Messages

The take-home messages emphasize numerical stability, hardware-dependent parallelism, and data cleaning, mixture ratios, and scheduling. AMBER’s performance gap from LLaMA despite similar hyperparameters motivates investigating data and implementation differences, although this report does not isolate their respective effects. Proposed uses include branching from intermediate checkpoints to change data mixtures, developing domain-specific models, and initializing training methods that require partially trained weights.

• The authors propose inspecting exact data and bias analyses before deployment and using reproducible training traces to study safeguards. The report does not demonstrate reductions in deployment risk from these uses.
• The throughput advantage of tuned hybrid parallelism is presented as especially relevant when intra-node bandwidth is limited.

6 Conclusion and Future Work

The conclusion presents LLM360 as a release framework instantiated by AMBER and CRYSTALCODER, with training code, configurations, intermediate weights, data order, processing code, and analyses. Its statement about releasing optimizer states conflicts with AMBER’s documented omission in Section 4.1.5. More detailed model evaluations, a larger model, and studies of optimal data mixtures are described as future work.

Appendix

• The supplied 15-page report contains no appendix; References follows the conclusion and acknowledgements. The available implementation and analysis links include https://github.com/llm360 and https://github.com/LLM360/Analysis360, with the project linked at https://www.llm360.ai/.

Brief Thoughts

The correspondence among data chunks, intermediate weights, and evaluations enables the learning and memorization analyses in Figures 4 and 7–9. Exact replay is limited by AMBER’s missing optimizer history and reduced-precision early checkpoints, while the absence of controlled ablations leaves the causes of performance differences unresolved. Table 5 also contains an arithmetic inconsistency: OpenLLaMA-v2-7B’s listed HumanEval 15.32 and MBPP 12.69 average to 14.005, rather than the reported code average 28.01, so that row’s aggregate comparison requires caution.