Estimate from the version's published specs, benchmarks and reviews, 0-5; scores are only comparable within one criteria year. Scored as launched; family pin, flagship Llama 3.1 405B named (8B and 70B ship alongside; 23 Jul 2024). Reasoning/coding null: no AA Intelligence Index or Terminal-Bench figure at launch; Meta's headline benchmarks (e.g. MMLU, GPQA) are not comparable to the anchors. Context 128k (Meta blog) -> 1. Price: Meta published no list price; hosted launch-week price for 405B was Together Turbo $5.00 per 1M in/out (Wayback 30 Jul 2024; Fireworks listed $3.00) -> 3.5. Speed null. Modalities 1: text only (TechCrunch). Reviews are launch coverage and commentary relaying Meta's benchmarks (Simon Willison, Interconnects, The Register, Tom's Guide); no hands-on test of the 405B opened.[1]
it really is benchmarking in that GPT-4+ class, competitive with both GPT-4o and Claude 3.5 Sonnet.
The 405 billion parameter model is directly in the ballpark of Anthropic's Claude 3 and OpenAI's GPT-4o.
The result is a model that, at least according to Meta's benchmarks, is ahead of larger, more proprietary systems from OpenAI and Anthropic on a variety of benchmarks.
quickly gained a reputation as one of the most powerful and useful models available
| Introducing Llama 3.1: Our most capable models to date · 2024. 7. 23. | 0.70 |
|---|---|
| Llama 3.1 405B, Meta's AI strategy, and the new, open frontier model ecosystem · 2024. 7. 23. | 0.50 |
| Meta says all new Llama 3.1 405B model bests OpenAI's GPT-4 · 2024. 7. 23. | 0.60 |
| Meta Llama 3.1 is out now — here's how to try it for free · 2024. 7. 24. | 0.80 |
독자들의 사진으로, 출처가 있는 핀의 미디어와 별도로 표시됩니다. 먹거나 본 것을 평가하고, 레스토랑 핀에서는 사진에 바로 요리를 태그해 보세요.
첫 번째 항목은 항상 핀의 출처입니다. 전체 신뢰도는 각 자료가 위의 시작·종료 시각을 얼마나 확실히 뒷받침하는지에 대한 가중 평균이며, 자료는 최신 자료보다 180일 오래될 때마다 가중치가 절반이 됩니다.
Meta의 게시글 'Introducing Llama 3.1: Our most capable models to date'은 2024년 7월 23일자이며 모델을 같은 날 다운로드할 수 있다고 밝혔다. 저커버그의 공개 서한과 TechCrunch[6] 기사도 같은 날짜다.
Meta's[1] model card: 15T+ pretraining tokens, 39.3M H100-80GB GPU hours (30.84M for the 405B), December 2023 knowledge cutoff, Llama 3.1 Community License and the instruction-tuned benchmark table.
Dates the line: Llama 3 on 18 April 2024 with 8B and 70B, Llama 3.1 on 23 July 2024 with 8B, 70B and 405B, Llama 4 in April 2025.[1]
The technical paper: dense 405B Transformer with a context window of up to 128K tokens, performance comparable to GPT-4 on many tasks, plus Llama Guard 3 and the not-yet-released multimodal work.[1]
Mark Zuckerberg's open letter published with the release: 405B is the first frontier-level open source model, inference at roughly 50% of the cost of closed models like GPT-4o, and the partners (Amazon, Databricks, NVIDIA, Groq, Dell, Scale.AI).[1]
빠지거나 잘못된 점이 있나요? 이 핀을 뒷받침하는 링크, 다른 시작일이나 종료일과 그 이유, 빠지거나 잘못된 사실을 자신의 말로 알려주세요. AI가 이 핀의 출처와 대조하고 더 나은 자료를 찾아, 뒷받침하는 페이지를 추가합니다. 핀 자체의 출처가 여전히 가장 중요하게 반영됩니다. AI는 이미지도 확인합니다. 다른 것을 보여주거나 제대로 보여주지 못하는 이미지는 뒤로 옮기거나 교체합니다.
원본 제목: Llama 3.1 8B, 70B and 405B Released
| 벤치마크 | Llama 3.1 8B Instruct | Llama 3.1 70B Instruct | Llama 3.1 405B Instruct |
|---|---|---|---|
| MMLU (5-shot) | 69.4 | 83.6 | 87.3 |
| MMLU (CoT, 0-shot) | 73.0 | 86.0 | 88.6 |
| MMLU-Pro (CoT, 5-shot) | 48.3 | 66.4 | 73.3 |
| IFEval | 80.4 | 87.5 | 88.6 |
| ARC-C (0-shot) | 83.4 | 94.8 | 96.9 |
| GPQA (0-shot) | 30.4 | 46.7 | 50.7 |
| HumanEval (pass@1) | 72.6 | 80.5 | 89.0 |
| MBPP++ base (pass@1) | 72.8 | 86.0 | 88.6 |
| MultiPL-E HumanEval (pass@1) | 50.8 | 65.5 | 75.2 |
| MultiPL-E MBPP (pass@1) | 52.4 | 62.0 | 65.7 |
| GSM-8K (CoT, 8-shot) | 84.5 | 95.1 | 96.8 |
| MATH (CoT, 0-shot) | 51.9 | 68.0 | 73.8 |
| API-Bank (0-shot) | 82.6 | 90.0 | 92.0 |
| BFCL (0-shot) | 76.1 | 84.8 | 88.5 |
| Gorilla API-Bench (0-shot) | 8.2 | 29.7 | 35.3 |
| Nexus (0-shot) | 38.5 | 56.7 | 58.7 |
| Multilingual MGSM (CoT, 0-shot) | 68.9 | 86.9 | 91.6 |