RedPajama: an Open Dataset for Training Large Language Models | Summary

This article explains the key points of RedPajama: an Open Dataset for Training Large Language Models.

• 2024-11-19 (arXiv), 38th Conference on Neural Information Processing Systems (NeurIPS 2024) Track on Datasets and Benchmarks
• Weber, Maurice, Fu, Daniel, Anthony, Quentin, Oren, Yonatan, Adams, Shane, Alexandrov, Anton, Lyu, Xiaozhong, Nguyen, Huu, Yao, Xiaozhe, Adams, Virginia, et al.
• Together AI, Stanford University, University of Chicago, EleutherAI, Ontocord.ai, Princeton University, ETH Zurich, Mila, Montréal, Canada, Université de Montréal, Ohio State University, Caltech
• Paper
• Github

Read this article in Korean

Summary

• RedPajama addresses limited access to documented training corpora and the difficulty of measuring how text filtering affects large language models (LLMs), models pretrained on large text collections. With 468M-parameter models each trained on 100B tokens, the text units processed during training, its filtered 2023-14 web subset achieves a Rank-Score, a normalized aggregation of task rankings, of 0.700 versus the filtered web corpus RefinedWeb’s 0.650, while its average benchmark score is lower at 37.6 versus 37.9. This RedPajama-V2 recipe combines Gopher rules, heuristics for natural-language quality and repetition, with fuzzy deduplication, removal of near-duplicate documents.
• RedPajama-V1 reconstructs approximately 1.2T tokens from seven source groups following the published recipe for LLaMA, an open language-model family. RedPajama-V2 provides an estimated 123.7T tokens from 84 snapshots of Common Crawl, a public web archive, in five languages. Its 46 quality measures accompany the approximately 50T-token head+middle subset, documents in the lower two of three perplexity buckets measuring how unexpected text is under a Wikipedia-trained language model.
• Filtering choices affect model quality, but the experiments do not establish a universally best recipe: at 1.6B parameters and 350B training tokens, the stronger tested RedPajama-V2 configuration scores 50.0 on average versus RefinedWeb’s 52.0. The authors identify limited experimental scale and the absence of thorough benchmark decontamination, checking for evaluation data in training corpora, and analysis of personally identifiable information as limitations.

1 Introduction

The introduction identifies three obstacles to open language-model development: opaque data curation, limited access to large high-quality corpora, and insufficient artifacts for analyzing and filtering those corpora. RedPajama responds through Transparency, Scale, and Versatility: public documentation and code, large accessible collections, and document annotations that let users choose filtering criteria.

• RedPajama-V1 supports reproduction of a published training mixture, while RedPajama-V2 supplies lightly processed web text and reusable quality metadata for a subset.
• The reported ecosystem includes models trained with RedPajama and datasets derived from it, particularly SlimPajama, a cleaned and deduplicated version of RedPajama-V1.

The diagram distinguishes direct use of RedPajama-V1 and V2 from downstream use through SlimPajama. It documents adoption and dataset derivation relationships, without measuring how much model performance is attributable to RedPajama.

2 Related Work

The related work distinguishes web-only corpora, including C4, RefinedWeb, and FineWeb, from composite corpora that combine multiple domains, including the Pile, Dolma, ROOTS, and RedPajama-V1. RedPajama-V2 contributes a large, mostly unfiltered collection with reusable quality signals for constructing different training subsets.

• Table 1 reports RedPajama-V1 at 3.0 TB and RedPajama-V2 at 270.0 TB, with open access and open code for both.
• The comparison also lists DCLM-Pool at 340.0 TB, so it does not support an unqualified claim that RedPajama-V2 exceeds every listed dataset in storage size.

V1 combines multiple domains, while V2 exposes raw multilingual web data. The storage figures qualify the paper’s scale claims: DCLM-Pool is listed at 340.0 TB compared with V2’s 270.0 TB.

3 RedPajama-V1: An open Reproduction of the LLaMA Training Data

RedPajama-V1 is a best-effort reproduction of the training data described for the original LLaMA models. The authors document missing recipe details and their replacement decisions; the resulting approximately 1.2T-token corpus is therefore an attempted reconstruction whose equivalence to the original data was not verified.

• The accompanying RedPajama-INCITE models evaluate the corpus at 3B and 7B parameters.
• The authors propose training precision and unresolved data differences as explanations for the 7B performance gap, without experimentally isolating their effects.

3.1 Data Processing Steps

The seven RedPajama-V1 source groups contain CommonCrawl 878B, C4 175B, GitHub 59B, Books 26B, ArXiv 28B, Wikipedia 24B, and StackExchange 20B tokens. Processing follows the available LLaMA descriptions while making concrete choices where those descriptions are incomplete.

• CommonCrawl uses snapshots 2019-30, 2020-05, 2021-04, 2022-5, and 2023-06. CCNet, a web-text processing pipeline, assigns head, middle, and tail buckets using Wikipedia-model perplexity; V1 retains head and middle.
• A fastText unigram classifier, a text classifier using individual-word features, is trained using crawled Wikipedia references. The authors extract 38M reference URLs, crawl 300K pages, and discard documents with classifier scores below 0.25; C4 uses the c4_en release.
• GitHub projects are restricted to Apache, BSD, and MIT licenses and filtered using file statistics and extensions. Wikipedia uses the 2023-03-20 dump; Gutenberg uses PG19, a book subset, with SimHash, a method for detecting near duplicates. Books3 was initially included and subsequently removed because of copyright issues.
• ArXiv processing retains LaTeX document sources, removes preambles, comments, and bibliographies, and expands macros. Stack Exchange processing retains the 28 largest sites, removes HTML web-formatting tags, groups question-answer pairs, and orders answers by score.

CommonCrawl contributes 878B tokens and C4 contributes 175B, making web data the dominant component of the approximately 1.2T-token reconstruction. The remaining sources add code, books, scientific text, encyclopedic text, and question-answer material; these reported construction counts do not imply that the subsequently removed Books3 component remains available.

3.2 The RedPajama-INCITE family of LLMs

3.2.1 Summit Training Setup describes training on Summit’s IBM Power9 nodes with six V100 graphics processing units (GPUs) per node. Because V100s lack bf16 support, a 16-bit numerical format used in the referenced stable training recipes, the authors use fp16, another 16-bit floating-point format, with loss scaling to stabilize numerical computation and reduced learning rates.

• The 3B model uses 256 nodes, 1536 GPUs, a learning rate of 1.6 × 10^-4, and 800B training tokens; the 7B model uses 512 nodes, 3072 GPUs, a learning rate of 1.2 × 10^-4, and 1.001T tokens. Both use a global batch of 4M tokens and linear learning-rate decay after warmup.
• The training stack required recompiling PyTorch, the model-training framework, for Power9. The authors use 6-way pipeline parallelism for 3B and 12-way for 7B, distributing model stages across devices, together with 2-way tensor parallelism, splitting individual layer computations, for both.

3.2.2 Evaluation reports that the 3B base model improves over similarly sized GPT-Neo and Pythia models on HELM classic, an evaluation suite averaged over 16 core scenarios, and selected lm-evaluation-harness tasks, evaluated through another language-model testing framework. HELM uses few-shot evaluation, with examples supplied in the prompt, while the selected harness comparison is zero-shot, without such examples. Training-token budgets differ: RedPajama uses 800B tokens, versus 420B for GPT-Neo and 300B for Pythia-2.8B.

• Table 7 gives the 3B model a selected-task average of 0.6662 and HELM average of 0.4060, compared with 0.6451 and 0.3770 for Pythia-2.8B, and 0.6197 and 0.3570 for GPT-Neo.
• The 7B base model scores 0.431 on HELM, compared with Falcon-7B’s 0.441 and Llama-7B’s 0.472. The authors locate the deficit in tasks scored through answer probabilities and describe comparable average performance on tasks that directly generate answers.
• Instruction tuning, additional training on tasks expressed through instructions, uses the P3 and Natural Instructions task collections. The 7B Instruct model reaches a HELM average of 0.492, although Table 9 shows that its lm-evaluation-harness average of 0.6858 does not improve over the base model’s 0.6882.

4 RedPajama-V2

RedPajama-V2 focuses exclusively on web data and provides a corpus for constructing different high-quality subsets. It retains lightly processed text and publishes accompanying metadata, allowing users to revise filtering recipes without repeating the initial collection process.

4.1 Data Processing Steps

4.1.1 Data Acquisition uses Common Crawl .wet files, files containing text extracted from web pages, from all 84 monthly snapshots between 2014 and April 2023. CCNet processing retains English, French, German, Italian, and Spanish text and keeps all three perplexity buckets.

• The collection is raw relative to subsequent quality filtering, but it has already undergone web-text extraction and CCNet processing.
• Retaining the tail bucket preserves material that the V1 bucket-selection rule would discard, leaving subsequent selection to dataset users.

4.1.2 Quality Signals groups annotations into natural-language properties, repetitiveness, content indicators, learned domain-similarity scores, and deduplication artifacts. These signals supply filtering criteria; they do not establish a document’s suitability for every training objective.

• Natural-language measures include uppercase fractions, unique-word fractions, punctuation, and line properties. Repetition measures count characters in the most frequent word n-grams, sequences of n consecutive words, for n ∈ {2, 3, 4}, and in duplicated n-grams for n ∈ {5, …, 10}, without double-counting overlapping characters.
• Content indicators count matches to LDNOOBW, an offensive-word blocklist, and identify domains listed by UT1, a categorized domain blocklist. These indicators provide limited proxies for harmful or offensive content.
• Learned signals use fastText classifiers and DSIR, Data Selection via Importance Resampling, weights measuring target-to-source log-likelihood ratios over individual words and word pairs. English classifier reference domains include Wikipedia, Wikipedia references, books, and OpenWebText, a web-text dataset; non-English classifiers use Wikipedia.
• Exact-duplicate identifiers use a Bloom filter, a probabilistic membership structure, with a 1% error rate on pre-CCNet .wet document hashes. MinHash signatures, compact representations for approximate document similarity, support fuzzy deduplication; exact deduplication proceeds from 2023-14 back to 2014-15.
• The paper acknowledges that learned quality filters can introduce bias or underrepresent minorities. Published quality signals cover the head+middle subset, approximately 50T tokens, rather than the entire 123.7T-token collection.

4.2 Dataset Statistics

Table 3 reports 113.3B documents and an estimated 123.7T tokens in RedPajama-V2, including 87.5B English documents and 90.5T English tokens. The head+middle partition contains 32.8B documents and 50.7T tokens; after the reported deduplication, it contains 20.8B documents and 30.4T tokens.

• The tail partition contains 80.5B documents and 73.0T tokens. Tail documents are typically shorter, approximately 850 tokens versus approximately 1500 tokens for head and middle documents.
• Token counts are estimates from an independent, identically distributed sample of 100M documents using the Mistral BPE tokenizer, which uses byte-pair encoding to represent text through merged subword units.

4.3 Dataset Ablations

4.3.1 Setup compares filtering recipes using decoder-only Llama-2 architectures, models that predict subsequent tokens from preceding text, with 468M and 1.6B parameters and a sequence length of 2048. Each 468M model trains on 100B tokens and each 1.6B model on 350B tokens, holding training-token budgets fixed within each model size.

• Both architectures have 24 layers, 16 attention heads that compute parallel attention operations, and a feed-forward expansion ratio of 4.0, specifying the intermediate layer width relative to the hidden dimension; hidden dimensions are 1024 and 2048. AdamW, the training optimizer, uses weight decay 0.1, maximum learning rates 5e−3 and 5e−4 respectively, cosine decay, and linear warmup over the first 1% of steps.
• Training uses the OLMo language-model training framework and FSDP, fully sharded data parallelism that partitions training state across devices, on up to 5 H100 nodes with InfiniBand interconnect.
• The 468M experiments use the 2023-14 crawl or nine crawls from 2021-49 to 2023-14. Fuzzy deduplication uses MinHash LSH, locality-sensitive hashing for finding similar documents, over word 13-grams with 128 hash functions, 9 bands, and 13 rows.
• For the 1.6B experiments, the authors filter the full corpus, sample roughly 1T tokens, and then deduplicate using the same MinHash settings.
• Evaluation reports average benchmark scores, normalized averages, and Rank-Score. It also measures perplexity on Pile and Paloma validation sets, which assess language-model fit across domains; lower perplexity indicates better fit.

The full 123.7T-token collection differs from the 50.7T-token head+middle partition associated with quality signals. The reported deduplication reduces head+middle tokens to 30.4T, distinguishing raw corpus scale from the size of a deduplicated candidate training pool.

4.3.2 Results shows that the 2023-14 subset with fuzzy deduplication and full Gopher filtering has the strongest aggregate results among tested RedPajama-V2 recipes at 468M parameters: average 37.6, normalized average 0.160, and Rank-Score 0.700. RefinedWeb scores 37.9, 0.165, and 0.650 respectively, so the preferred dataset depends on the aggregation metric.

• The unfiltered 2023-14 baseline scores 36.4 on average and 0.594 on Rank-Score. Its Paloma perplexity of 31.1 is lower than the Gopher-filtered model’s 34.5, showing that benchmark gains need not coincide with improved fit to a broad validation distribution.
• For otherwise matching nine-crawl recipes using the Palm-mix classifier, trained against a mixture of Wikipedia, OpenWebText, and books, replacing Gopher repetition filters with natural-language filters improves average scores from 35.9 to 36.7. Classifier-based and DSIR-based variants both score 35.9 on average in the reported comparison; this alone does not establish statistical equivalence.
• Replacing document-level C4 filtering with line-level C4 filtering in a matched configuration improves Pile perplexity from 41.8 to 32.4 and Paloma perplexity from 67.9 to 52.9, while the average benchmark score changes from 36.7 to 36.4.
• At 1.6B parameters, the full-Gopher plus Wikipedia-reference classifier configuration scores 50.0 on average, versus 47.9 for the C4 plus Gopher-natural-language plus Palm classifier configuration and 52.0 for RefinedWeb. Several filters differ between the two V2 recipes, so this comparison does not isolate the effect of full Gopher filtering.

5 Conclusion

The conclusion presents RedPajama as an open resource for studying corpus filtering, curation, and mixing, supported by the releases and the observed performance differences across filtering recipes. The authors limit the experimental evidence to relatively small models and note that thorough benchmark decontamination and analysis of personally identifiable information were not performed.

At 468M parameters and 100B training tokens, fuzzy deduplication plus full Gopher filtering on 2023-14 gives Rank-Score 0.700 versus RefinedWeb’s 0.650, but average score 37.6 versus 37.9. Lower Paloma perplexity for unfiltered 2023-14 data shows that the evaluation metric affects which recipe appears preferable.

The inclusion marks distinguish benchmarks contributing to the aggregate scores from additional reported tasks. The aggregate therefore summarizes a selected set of evaluations, rather than every task listed in the paper.

With 1.6B parameters and 350B training tokens, the stronger V2 configuration reaches average score 50.0, below RefinedWeb’s 52.0 and above the alternative V2 recipe’s 47.9. Multiple filtering components change between V2 rows, limiting causal attribution to any one filter.

Appendix

• A Intended Uses identifies language-model training as the primary purpose. V1 combines web text, encyclopedic and scientific material, and code; V2 provides web text with filtering signals.
• B Dataset Accessibility provides public Hugging Face datasets and HTTPS download lists. Sharded JSON Lines store text and metadata, while V2 supplies line-level and document-level quality annotations. Duplicate identifiers and MinHash signatures support exact and approximate deduplication; removing listed exact duplicates retains one representative per cluster.
• C RedPajama-V1 documents GitHub code filtering and task-level evaluations of the 3B and 7B models. It also records unresolved details of the original recipe and the authors’ choices for snapshots, classifiers, thresholds, Wikipedia dumps, and book deduplication.
• D RedPajama-V2 provides deduplication statistics, quality-signal distributions, embedding-based topic analysis, and detailed model evaluations. Under newest-first deduplication, older snapshots contain fewer unique documents; the authors hypothesize that crawler-seed changes explain the sharp decline in 2014–2017. The histograms cover only English data from the 2023-06 snapshot. Supplementary evaluations cover the 468M models and the 1.6B models trained on 350B tokens each.
• E Author Responsibility Statement identifies ODC-By-1.0 and applicable constituent licenses. It states that completeness and correctness cannot be guaranteed at this scale and disclaims liability for rights violations.
• F License specifies Apache 2.0 for repository code, Common Crawl terms for derived web data, and the original licenses or terms for other sources. The GitHub subset is restricted to MIT, BSD, or Apache licenses.

Brief Thoughts

RedPajama’s main contribution is the separation of a reusable corpus from filtering decisions: released annotations and deduplication artifacts make alternative recipes inspectable. The experiments support this design by showing differences across recipes under fixed training budgets, including disagreement between benchmark rankings and validation perplexity. The evidence supports using RedPajama-V2 for curation research. RefinedWeb retains higher average benchmark scores than the tested V2 configurations at both model sizes. V1’s model comparisons are less diagnostic of data quality because token budgets, hardware-constrained training, and corpus construction differ. V2’s experiments hold model size and training-token budgets fixed within each comparison, but larger-scale validation, estimates of variability across repeated runs, contamination checks, and privacy analysis remain open.