Estimate from the version's published specs, benchmarks and reviews, 0-5; scores are only comparable within one criteria year. Scored as launched (Claude 3.5 Sonnet, 20 Jun 2024): 200K-token context and $3 in / $15 out per 1M tokens = $6 blended at 3:1 (both from Anthropic's announcement); text and image input. Anthropic said it runs at twice the speed of Claude 3 Opus but gave no tokens/s, so speed null. Reasoning null: no launch-version Artificial Analysis Intelligence Index entry was opened (AA now shows re-based v4.1 scores); Anthropic's headline GPQA/MMLU/HumanEval figures have no anchor. Coding null (no Terminal-Bench 4.0). Reviews: Willison, Zvi, TechCrunch, The Decoder, Synthedia; these are launch-week analyses with some first-hand use (Willison, Zvi), not systematic tests.[1]
Anthropic released a new model this morning, and I think it's likely now the single best available LLM.
There is a new clear best (non-tiny) LLM. If you want to converse with an LLM, the correct answer is Claude Sonnet 3.5.
But it's more an incremental step than a monumental leap forward.
This makes Sonnet 3.5 potentially the most powerful model available today - at the same price as its predecessor.
Claude 3.5's Sonnet is "more intelligent" than Claude 3's Opus, and it delivers this quality at a far lower price point.
| Claude 3.5 Sonnet · 20 Jun 2024 | 0.90 |
|---|---|
| Anthropic claims its latest model is best-in-class · 20 Jun 2024 | 0.20 |
| Anthropic launches Claude 3.5, potentially the most capable AI model yet · 20 Jun 2024 | 0.80 |
| Anthropic Sonnet 3.5 Sets New Benchmark Standards · 21 Jun 2024 | 0.80 |
| On Claude 3.5 Sonnet · 24 Jun 2024 | 0.90 |
Foto dari pembaca, terpisah dari media bersumber milik pin. Nilai apa yang Anda makan atau lihat, dan di pin restoran, tandai hidangan langsung pada foto Anda.
Entri pertama selalu merupakan sumber pin. Keyakinan keseluruhan adalah rata-rata tertimbang dari seberapa kuat setiap referensi mendukung waktu mulai dan berakhir yang digunakan di atas; bobot sebuah referensi berkurang setengahnya untuk setiap 180 hari lebih lama dari yang terbaru.
Anthropic[2]: “Hari ini, kami meluncurkan Claude 3.5 Sonnet” dan model ini “kini tersedia gratis di Claude.ai”; header halaman kini menunjukkan 21 Jun 2024, tetapi TechCrunch[4] dan AWS sama-sama menerbitkan peluncurannya pada 20 Juni 2024 dan ID modelnya adalah claude-3-5-sonnet-20240620.
Anthropic's[1] model card addendum: 59.4% on GPQA Diamond, 88.7% on MMLU (5-shot), 92.0% on HumanEval, 71.1% on MATH and 68.3% on MMMU, against Claude 3 Opus's 50.4%, 86.8%, 84.9%, 60.1% and 59.4%; 64% on the internal agentic coding evaluation against Opus's 38%.
"On June 20, 2024, Anthropic[1][2] released Claude 3.5 Sonnet, which, according to the company's own benchmarks, performed better than the larger Claude 3 Opus", alongside Artifacts, which previews code, SVG graphics or websites in a separate window.
TechCrunch, published 2024-06-20T14:00Z: Anthropic[1][2] "is releasing a powerful new generative AI model called Claude 3.5 Sonnet", about twice the speed of Claude 3 Opus, alongside Artifacts. It dates the launch 20 June, where Anthropic's own header now shows the 21 June UTC timestamp.
AWS News Blog, 20 JUN 2024: "Today, Anthropic[1][2] introduced Claude 3.5 Sonnet ... now available in Amazon Bedrock"; 80 percent cheaper than Opus and about 10 percent better than Claude 3 Opus on most vision benchmarks.
Ada yang kurang atau salah? Sampaikan dengan kata-kata Anda sendiri: tautan yang mendukung pin ini, tanggal mulai atau berakhir yang berbeda beserta alasannya, atau fakta yang kurang atau keliru. AI memeriksanya terhadap sumber pin ini, mencari sumber yang lebih baik, dan menambahkan halaman mana pun yang mendukung Anda. Sumber pin itu sendiri tetap paling diperhitungkan. Gambar yang menampilkan hal lain, atau menampilkannya dengan buruk, juga diperiksa, lalu diturunkan atau diganti.
Judul asli: Claude 3.5 Sonnet Released
| Tolok ukur | Claude 3.5 Sonnet | Claude 3 Opus | GPT-4o | Gemini 1.5 Pro | Llama-400b (early snapshot) |
|---|---|---|---|---|---|
| Penalaran tingkat pascasarjana (GPQA, Diamond) | 59.4%* (0-shot CoT) | 50.4% (0-shot CoT) | 53.6% (0-shot CoT) | — | — |
| Pengetahuan tingkat sarjana (MMLU), 5-shot | 88.7%** (5-shot) | 86.8% (5-shot) | — | 85.9% (5-shot) | 86.1% (5-shot) |
| Pengetahuan tingkat sarjana (MMLU), 0-shot CoT | 88.3% (0-shot CoT) | 85.7% (0-shot CoT) | 88.7% (0-shot CoT) | — | — |
| Kode (HumanEval) | 92.0% (0-shot) | 84.9% (0-shot) | 90.2% (0-shot) | 84.1% (0-shot) | 84.1% (0-shot) |
| Matematika multibahasa (MGSM) | 91.6% (0-shot CoT) | 90.7% (0-shot CoT) | 90.5% (0-shot CoT) | 87.5% (8-shot) | — |
| Penalaran atas teks (DROP, skor F1) | 87.1 (3-shot) | 83.1 (3-shot) | 83.4 (3-shot) | 74.9 (shot bervariasi) | 83.5 (3-shot, model pralatih) |
| Evaluasi campuran (BIG-Bench-Hard) | 93.1% (3-shot CoT) | 86.8% (3-shot CoT) | — | 89.2% (3-shot CoT) | 85.3% (3-shot CoT, model pralatih) |
| Pemecahan soal matematika (MATH) | 71.1% (0-shot CoT) | 60.1% (0-shot CoT) | 76.6% (0-shot CoT) | 67.7% (4-shot) | 57.8% (4-shot CoT) |
| Matematika sekolah dasar (GSM8K) | 96.4% (0-shot CoT) | 95.0% (0-shot CoT) | — | 90.8% (11-shot) | 94.1% (8-shot CoT) |
* Claude 3.5 Sonnet mendapat skor 67.2% pada GPQA 5-shot CoT dengan maj@32. ** Claude 3.5 Sonnet mendapat skor 90.4% pada MMLU dengan prompting 5-shot CoT.
Epoch AI model database (Claude 3.5 Sonnet); Epoch's confidence in the figures: Speculative. Blank tiles are not published.