16  Appendix 1: LLM Data Schemas

Here, we describe many variants of LLM data. This will be relevant for when we extend the flywheel to include more types of data, and especially shift towards promoting the sharing (via opt-in flywheels, but also via new market mechanisms) of richer “content data”.

---

16.1 Open Web / Crawls

• WARC/WAT/WET

  • WARC (container for HTTP request/response records) — spec & overview: (International Internet Preservation Consortium 2017; Library of Congress, n.d.)
  • WAT (JSON metadata extracted from WARC) and WET (plain text extracted from HTML) — Common Crawl guides (Common Crawl, n.d.a, n.d.b)

• C4 (Colossal Clean Crawled Corpus) — TFDS catalog & generator code (TensorFlow Datasets, n.d.a, n.d.b)

• The Pile (22-source, mixed corpus) — paper (Gao et al. 2021)

---

16.2 Encyclopedic / Books

• Wikipedia XML dumps (page/revision XML; SQL tables for links) — (Wikimedia Meta-Wiki, n.d.; Wikipedia, n.d.)

• Project Gutenberg

  • Books: plain text/HTML master formats; ePub/MOBI derived (Project Gutenberg, n.d.a)
  • Catalog schema: daily RDF/XML (also CSV) for metadata; offline catalogs (Project Gutenberg, n.d.b)

---

16.3 Scientific / Legal

• arXiv (Atom/OAI-PMH metadata; bulk & API) — (arXiv.org, n.d.c, n.d.a, n.d.b)

• JATS XML (journal article tag suite) — (NISO 2024; NLM, n.d.)

---

16.4 Code

• BigCode — The Stack / The Stack v2 (source files + license/provenance metadata; dedup variants) — (BigCode Project, n.d.b, n.d.c, n.d.a; stack_paper?)

---

16.5 Forums / Q&A / Social

• Stack Exchange dumps (XML: Posts, Users, Comments, Votes, etc.) — (Meta Stack Exchange, n.d.; Stack Exchange, n.d.)

• Reddit

  • API JSON schema — official docs (Reddit, n.d.; Reddit Help, n.d.)
  • Pushshift (historical dumps; research dataset) — (Pushshift, n.d.; Baumgartner et al. 2020)

---

16.6 Instruction / Conversations (Post-training SFT)

• OpenAI-style chat schema (role-tagged: system|user|assistant, plus tool calls) — (OpenAI, n.d.c)

• Alpaca (JSON prompts/instructions/outputs) — (Stanford CRFM 2023; Tatsu Lab, n.d.; gururise, n.d.)

• Databricks Dolly-15k (human-written instruction/response pairs) — (Databricks, n.d.)

• OpenAssistant OASST1 (message-tree conversations with roles) — (OpenAssistant, n.d.)

---

16.7 Preference / Feedback (RLHF & DPO)

• HH-RLHF (Anthropic helpful/harmless, JSONL pairs: chosen vs rejected) — (Anthropic, n.d.)

• DPO format (prompt + preferred vs dispreferred response) — (Rafailov et al. 2024)

---

16.8 Multimodal (for VLMs/ASR)

• LAION-5B / Re-LAION-5B (image–text pairs with CLIP scores; links) — (LAION 2022a, 2022b)

• Whisper (weakly-supervised ASR; audio → text pairs) — (Radford et al. 2022; OpenAI 2022)

• HowTo100M (YouTube instructional video clips + narrations) — (École Normale Supérieure, n.d.; Miech et al. 2019)

---

16.9 Math Reasoning (often for post-training/eval)

• GSM8K (grade-school word problems; JSON) — (OpenAI, n.d.a, n.d.b)

• MATH (competition problems with step-by-step solutions) — (Hendrycks et al. 2021; Hendrycks, n.d.)

---

16.10 Common Storage Containers

• JSON Lines / NDJSON — (jsonlines.org, n.d.; ndjson, n.d.)

• TFRecord — (TensorFlow, n.d.)

• Apache Parquet — (Apache Software Foundation, n.d.)

Anthropic. n.d. “HH-RLHF Dataset.” https://github.com/anthropics/hh-rlhf.

Apache Software Foundation. n.d. “Apache Parquet Project.” https://parquet.apache.org/.

arXiv.org. n.d.a. “arXiv API User’s Manual.” https://info.arxiv.org/help/api/user-manual.html.

———. n.d.b. “arXiv Bulk Data Access.” https://info.arxiv.org/help/bulk_data.html.

———. n.d.c. “arXiv OAI-PMH Interface.” https://info.arxiv.org/help/oa/index.html.

Baumgartner, Jason, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. 2020. “The Pushshift Reddit Dataset” 14: 830–39.

BigCode Project. n.d.a. “BigCode Project Documentation.” https://www.bigcode-project.org/docs/about/the-stack/.

———. n.d.b. “The Stack Dataset on Hugging Face.” https://huggingface.co/datasets/bigcode/the-stack/tree/main.

———. n.d.c. “The Stack V2 Dataset on Hugging Face.” https://huggingface.co/datasets/bigcode/the-stack-v2.

Common Crawl. n.d.a. “Common Crawl – Get Started.” https://commoncrawl.org/get-started.

———. n.d.b. “Web Archiving File Formats Explained.” https://commoncrawl.org/blog/web-archiving-file-formats-explained.

Databricks. n.d. “Databricks Dolly Repository.” https://github.com/databrickslabs/dolly.

École Normale Supérieure. n.d. “HowTo100M Project.” https://www.di.ens.fr/willow/research/howto100m/.

Gao, Leo, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, et al. 2021. “The Pile: An 800GB Dataset of Diverse Text for Language Modeling.” CoRR abs/2101.00027. https://arxiv.org/abs/2101.00027.

gururise. n.d. “Alpaca Data Cleaned Repository.” https://github.com/gururise/AlpacaDataCleaned.

Hendrycks, Dan. n.d. “Competition Math Dataset on Hugging Face.” https://huggingface.co/datasets/hendrycks/competition_math.

Hendrycks, Dan, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. “Measuring Mathematical Problem Solving with the MATH Dataset.” https://arxiv.org/abs/2103.03874.

International Internet Preservation Consortium. 2017. “The WARC Format 1.1.” https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/.

jsonlines.org. n.d. “JSON Lines Specification.” https://jsonlines.org/.

LAION. 2022a. “LAION-5B: A New Era of Open Large-Scale Multi-Modal Datasets.” https://laion.ai/laion-5b-a-new-era-of-open-large-scale-multi-modal-datasets/.

———. 2022b. “Releasing Re-LAION-5B.” https://laion.ai/blog/relaion-5b/.

Library of Congress. n.d. “WARC, Web ARChive File Format.” https://www.loc.gov/preservation/digital/formats/fdd/fdd000236.shtml.

Meta Stack Exchange. n.d. “Why Is the Stack Exchange Data Dump Only Available in XML?” https://meta.stackexchange.com/questions/267329/why-is-the-stack-exchange-data-dump-only-available-in-xml-file-format.

Miech, Antoine, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. 2019. “HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips.” CoRR abs/1906.03327. http://arxiv.org/abs/1906.03327.

ndjson. n.d. “NDJSON Specification.” https://github.com/ndjson/ndjson-spec.

NISO. 2024. “ANSI/NISO Z39.96-2024, JATS: Journal Article Tag Suite.” https://www.niso.org/publications/z3996-2024-jats.

NLM. n.d. “Journal Article Tag Suite.” https://jats.nlm.nih.gov/.

OpenAI. 2022. “Introducing Whisper.” https://openai.com/index/whisper/.

———. n.d.a. “Grade-School Math (GSM8K) Repository.” https://github.com/openai/grade-school-math.

———. n.d.b. “GSM8K Hugging Face Dataset Card.” https://huggingface.co/datasets/openai/gsm8k.

———. n.d.c. “OpenAI API Reference – Chat.” https://platform.openai.com/docs/api-reference/chat.

OpenAssistant. n.d. “OpenAssistant OASST1 Dataset Card.” https://huggingface.co/datasets/OpenAssistant/oasst1.

Project Gutenberg. n.d.a. “Project Gutenberg File Formats.” https://www.gutenberg.org/help/file_formats.html.

———. n.d.b. “Project Gutenberg Offline Catalogs and Feeds.” https://www.gutenberg.org/ebooks/offline_catalogs.html.

Pushshift. n.d. “Pushshift.io.” https://pushshift.io/.

Radford, Alec, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022. “Robust Speech Recognition via Large-Scale Weak Supervision.” https://arxiv.org/abs/2212.04356.

Rafailov, Rafael, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2024. “Direct Preference Optimization: Your Language Model Is Secretly a Reward Model.” https://arxiv.org/abs/2305.18290.

Reddit. n.d. “Reddit API Documentation.” https://www.reddit.com/dev/api/.

Reddit Help. n.d. “Reddit Data API Wiki.” https://support.reddithelp.com/hc/en-us/articles/16160319875092-Reddit-Data-API-Wiki.

Stack Exchange. n.d. “Stack Exchange Data Explorer Help.” https://data.stackexchange.com/help.

Stanford CRFM. 2023. “Alpaca: A Strong, Replicable Instruction-Following Model.” https://crfm.stanford.edu/2023/03/13/alpaca.html.

Tatsu Lab. n.d. “Stanford Alpaca GitHub Repository.” https://github.com/tatsu-lab/stanford_alpaca.

TensorFlow. n.d. “TFRecord and Tf.train.example Tutorial.” https://www.tensorflow.org/tutorials/load_data/tfrecord.

TensorFlow Datasets. n.d.a. “C4 Dataset in TensorFlow Datasets.” https://www.tensorflow.org/datasets/catalog/c4.

———. n.d.b. “C4 Generator Code.” https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/text/c4.py.

Wikimedia Meta-Wiki. n.d. “Wikipedia Data Dumps – Dump Format.” https://meta.wikimedia.org/wiki/Data_dumps/Dump_format.

Wikipedia. n.d. “Wikipedia Database Download.” https://en.wikipedia.org/wiki/Wikipedia:Database_download.