Estimate from the version's published specs, benchmarks and reviews, 0-5; scores are only comparable within one criteria year. Scored as launched (Claude 3.5 Sonnet, 20 Jun 2024): 200K-token context and $3 in / $15 out per 1M tokens = $6 blended at 3:1 (both from Anthropic's announcement); text and image input. Anthropic said it runs at twice the speed of Claude 3 Opus but gave no tokens/s, so speed null. Reasoning null: no launch-version Artificial Analysis Intelligence Index entry was opened (AA now shows re-based v4.1 scores); Anthropic's headline GPQA/MMLU/HumanEval figures have no anchor. Coding null (no Terminal-Bench 4.0). Reviews: Willison, Zvi, TechCrunch, The Decoder, Synthedia; these are launch-week analyses with some first-hand use (Willison, Zvi), not systematic tests.[1]
Anthropic released a new model this morning, and I think it's likely now the single best available LLM.
There is a new clear best (non-tiny) LLM. If you want to converse with an LLM, the correct answer is Claude Sonnet 3.5.
But it's more an incremental step than a monumental leap forward.
This makes Sonnet 3.5 potentially the most powerful model available today - at the same price as its predecessor.
Claude 3.5's Sonnet is "more intelligent" than Claude 3's Opus, and it delivers this quality at a far lower price point.
| Claude 3.5 Sonnet · 20 Jun 2024 | 0.90 |
|---|---|
| Anthropic claims its latest model is best-in-class · 20 Jun 2024 | 0.20 |
| Anthropic launches Claude 3.5, potentially the most capable AI model yet · 20 Jun 2024 | 0.80 |
| Anthropic Sonnet 3.5 Sets New Benchmark Standards · 21 Jun 2024 | 0.80 |
| On Claude 3.5 Sonnet · 24 Jun 2024 | 0.90 |
Foto daripada pembaca, berasingan daripada media bersumber pin. Nilaikan apa yang anda makan atau lihat, dan pada pin restoran, tag hidangan terus pada foto anda.
Entri pertama sentiasa sumber pin. Keyakinan keseluruhan ialah purata berpemberat bagi sejauh mana setiap rujukan menyokong masa mula dan tamat yang digunakan di atas; sesebuah rujukan diambil kira separuh kurang bagi setiap 180 hari lebih lama daripada yang terbaharu.
Anthropic[2]: "Today, we're launching Claude 3.5 Sonnet" dan ia "is now available for free on Claude.ai"; pengepala halaman kini menunjukkan Jun 21, 2024, tetapi TechCrunch[4] dan AWS kedua-duanya menyiarkan pelancaran itu pada 20 Jun 2024 dan ID modelnya ialah claude-3-5-sonnet-20240620.
Anthropic's[1] model card addendum: 59.4% on GPQA Diamond, 88.7% on MMLU (5-shot), 92.0% on HumanEval, 71.1% on MATH and 68.3% on MMMU, against Claude 3 Opus's 50.4%, 86.8%, 84.9%, 60.1% and 59.4%; 64% on the internal agentic coding evaluation against Opus's 38%.
"On June 20, 2024, Anthropic[1][2] released Claude 3.5 Sonnet, which, according to the company's own benchmarks, performed better than the larger Claude 3 Opus", alongside Artifacts, which previews code, SVG graphics or websites in a separate window.
TechCrunch, published 2024-06-20T14:00Z: Anthropic[1][2] "is releasing a powerful new generative AI model called Claude 3.5 Sonnet", about twice the speed of Claude 3 Opus, alongside Artifacts. It dates the launch 20 June, where Anthropic's own header now shows the 21 June UTC timestamp.
AWS News Blog, 20 JUN 2024: "Today, Anthropic[1][2] introduced Claude 3.5 Sonnet ... now available in Amazon Bedrock"; 80 percent cheaper than Opus and about 10 percent better than Claude 3 Opus on most vision benchmarks.
Ada yang tertinggal atau salah? Nyatakan dengan kata-kata anda sendiri: pautan yang menyokong pin ini, tarikh mula atau tamat yang berbeza beserta sebabnya, atau fakta yang tiada atau tersilap. AI menyemaknya berbanding sumber pin ini, mencari yang lebih baik, dan menambah mana-mana halaman yang menyokong anda. Sumber pin itu sendiri tetap paling diambil kira. Gambar yang menunjukkan sesuatu yang lain, atau menunjukkannya dengan buruk, turut dilihat, dan dialihkan ke bawah atau diganti.
Tajuk asal: Claude 3.5 Sonnet Released
| Penanda aras | Claude 3.5 Sonnet | Claude 3 Opus | GPT-4o | Gemini 1.5 Pro | Llama-400b (early snapshot) |
|---|---|---|---|---|---|
| Penaakulan peringkat pascasiswazah (GPQA, Diamond) | 59.4%* (0-shot CoT) | 50.4% (0-shot CoT) | 53.6% (0-shot CoT) | — | — |
| Pengetahuan peringkat sarjana muda (MMLU), 5-shot | 88.7%** (5-shot) | 86.8% (5-shot) | — | 85.9% (5-shot) | 86.1% (5-shot) |
| Pengetahuan peringkat sarjana muda (MMLU), 0-shot CoT | 88.3% (0-shot CoT) | 85.7% (0-shot CoT) | 88.7% (0-shot CoT) | — | — |
| Kod (HumanEval) | 92.0% (0-shot) | 84.9% (0-shot) | 90.2% (0-shot) | 84.1% (0-shot) | 84.1% (0-shot) |
| Matematik pelbagai bahasa (MGSM) | 91.6% (0-shot CoT) | 90.7% (0-shot CoT) | 90.5% (0-shot CoT) | 87.5% (8-shot) | — |
| Penaakulan merentas teks (DROP, F1 score) | 87.1 (3-shot) | 83.1 (3-shot) | 83.4 (3-shot) | 74.9 (variable shots) | 83.5 (3-shot, pre-trained model) |
| Penilaian campuran (BIG-Bench-Hard) | 93.1% (3-shot CoT) | 86.8% (3-shot CoT) | — | 89.2% (3-shot CoT) | 85.3% (3-shot CoT, pre-trained model) |
| Penyelesaian masalah matematik (MATH) | 71.1% (0-shot CoT) | 60.1% (0-shot CoT) | 76.6% (0-shot CoT) | 67.7% (4-shot) | 57.8% (4-shot CoT) |
| Matematik sekolah rendah (GSM8K) | 96.4% (0-shot CoT) | 95.0% (0-shot CoT) | — | 90.8% (11-shot) | 94.1% (8-shot CoT) |
* Claude 3.5 Sonnet mencatat 67.2% pada GPQA 5-shot CoT dengan maj@32. ** Claude 3.5 Sonnet mencatat 90.4% pada MMLU dengan gesaan 5-shot CoT.
Epoch AI model database (Claude 3.5 Sonnet); Epoch's confidence in the figures: Speculative. Blank tiles are not published.