CONFIRMEDDate confidence: CONFIRMEDStated as firmHow firmly the source words the date: confirmed, scheduled, estimated, delayed or unverified.Show all CONFIRMED pins84% CONFIDENCEOverall confidence: 84%How well the pin's source and references back up its dates.Weighted average of how firmly 6 references, the source included, support the pin's start and end times; a reference counts half as much for every 180 days older than the newestShow all pins at 75% confidence or better
Meta's[1] post 'Introducing Llama 3.1: Our most capable models to date' is dated July 23, 2024 and says the models are available to download that day; Zuckerberg's open letter and TechCrunch[6] carry the same date.- Start
- Jul 23, 202490% CONFIDENCEfrom the source
Llama 3.1 8B, 70B and 405B Released
Meta releases Llama 3.1 in 8B, 70B and 405B sizes with a 128K context window; the 405B, trained on over 15 trillion tokens, is billed as the first frontier-level open model and is free to download.
- Meta released Llama 3.1 on 23 July 2024 in three sizes, an updated 8B and 70B and a new 405B that it called the first frontier-level open source AI model, with weights free to download from llama.meta.com and Hugging Face[1][5]
- It arrived three months after Llama 3 (18 April 2024, 8B and 70B) and carries a 128K-token context window[1][3]
- The 405B was trained on more than 15 trillion tokens using over 16,000 H100 GPUs, as a standard dense decoder-only transformer rather than a mixture of experts; Meta's card counts 39.3M GPU hours across the three sizes, 30.84M of them for the 405B[1][2][6]
- Meta evaluated on more than 150 benchmark datasets plus human evaluations and called the 405B competitive with GPT-4, GPT-4o and Claude 3.5 Sonnet; TechCrunch's reading is on par with GPT-4 and mixed against GPT-4o and Claude 3.5 Sonnet, behind Claude on programming and general reasoning[1][6]
- The license changed to let developers use Llama outputs, including from the 405B, to improve other models (distillation and synthetic data), while apps with more than 700 million monthly users still need a special license[1][2][6]
- The 8B is the charted small model: it scores 69.4 on MMLU, 72.6 on HumanEval and 84.5 on GSM-8K as an instruct model, against 87.3, 89.0 and 96.8 for the 405B[2]
Notable features
- 128K context and eight supported languages, with tool use, steerability, math, translation and function calling built in[1][6]
- 405B is quantized from 16-bit BF16 to 8-bit FP8 so it can run within a single server node[1]
- Safety and tooling released alongside: Llama Guard 3 (multilingual), Prompt Guard against prompt injection, a reference system, and Llama Stack standard interfaces as a request for comment[1]
- More than 25 partners launched services on day one, among them AWS, NVIDIA, Databricks, Groq, Dell, Azure, Google Cloud and Snowflake; Meta said Llama had passed 300 million downloads, and the 405B also powered Meta AI in WhatsApp for US users[1][6]
- Zuckerberg's letter argues open source wins because developers can fine-tune, distill and run models themselves, and puts inference on the 405B at roughly half the cost of GPT-4o[5]
Benchmarks[2]
| Benchmark | Llama 3.1 8B Instruct | Llama 3.1 70B Instruct | Llama 3.1 405B Instruct |
|---|---|---|---|
| MMLU (5-shot) | 69.4 | 83.6 | 87.3 |
| MMLU (CoT, 0-shot) | 73.0 | 86.0 | 88.6 |
| MMLU-Pro (CoT, 5-shot) | 48.3 | 66.4 | 73.3 |
| IFEval | 80.4 | 87.5 | 88.6 |
| ARC-C (0-shot) | 83.4 | 94.8 | 96.9 |
| GPQA (0-shot) | 30.4 | 46.7 | 50.7 |
| HumanEval (pass@1) | 72.6 | 80.5 | 89.0 |
| MBPP++ base (pass@1) | 72.8 | 86.0 | 88.6 |
| MultiPL-E HumanEval (pass@1) | 50.8 | 65.5 | 75.2 |
| MultiPL-E MBPP (pass@1) | 52.4 | 62.0 | 65.7 |
| GSM-8K (CoT, 8-shot) | 84.5 | 95.1 | 96.8 |
| MATH (CoT, 0-shot) | 51.9 | 68.0 | 73.8 |
| API-Bank (0-shot) | 82.6 | 90.0 | 92.0 |
| BFCL (0-shot) | 76.1 | 84.8 | 88.5 |
| Gorilla API-Bench (0-shot) | 8.2 | 29.7 | 35.3 |
| Nexus (0-shot) | 38.5 | 56.7 | 58.7 |
| Multilingual MGSM (CoT, 0-shot) | 68.9 | 86.9 | 91.6 |
Buy onMeta AI