Estimate from the version's published specs, benchmarks and reviews, 0-5; scores are only comparable within one criteria year. Scored as launched (upgraded Claude 3.5 Sonnet, 22 Oct 2024): 200K-token context (OpenRouter model page, released 22 Oct 2024; the Anthropic post restates no window); price unchanged at $3 in / $15 out = $6 blended at 3:1 (Willison, Zvi); text and image input plus computer-use beta. Anthropic reports SWE-bench Verified 49.0% (33.4% before) and OSWorld 14.9% screenshot-only; neither maps to an anchor. Reasoning null (no launch-version AA index opened), coding null (no Terminal-Bench 4.0), speed null (no tokens/s published). Reviews: Willison (hands-on with the computer-use demo), Zvi, TechCrunch, The Decoder.[1]
It worked exactly as advertised.
We only have very early feedback so far, so it's hard to tell how much what I will be calling Claude 3.5.1 improves performance in practice over Claude 3.5. It does seem like it is a clear improvement.
But don't fire your secretary just yet. In an evaluation designed to test an AI agent's ability to help with airline booking tasks, like modifying a flight reservation, the new 3.5 Sonnet managed to complete less than half of the tasks successfully.
The new Claude 3.5 Sonnet model shows improved performance, especially in logical reasoning, mathematical problem-solving and programming tasks.
| Initial explorations of Anthropic's new Computer Use capability · 2024年10月22日 | 0.60 |
|---|---|
| Anthropic's new AI model can control your PC · 2024年10月22日 | -0.20 |
| Anthropic releases updated, smarter Claude Haiku 3.5 and Sonnet 3.5 model · 2024年10月22日 | 0.70 |
| Claude Sonnet 3.5.1 and Haiku 3.5 · 2024年10月24日 | 0.30 |
读者上传的照片,与图钉带来源的媒体分开显示。你可以为吃到或看到的内容评分,在餐厅图钉上还能直接在照片上标记菜品。
第一项始终是图钉的来源。总体可信度是各份资料对上方所用开始和结束时间支持程度的加权平均;资料每比最新的一份旧 180 天,权重减半。
该文章标注日期为“2024 年 10 月 22 日”,并写道:“升级版 Claude 3.5 Sonnet 现已面向所有用户开放。从今天起,开发者可以使用计算机操作测试版进行构建”;TechCrunch[4] 和 CNBC[5] 均在同一天报道该发布“于周二”进行。
Anthropic's[1][3] model card addendum: the upgraded 3.5 Sonnet sets new state-of-the-art results in agentic coding (SWE-bench Verified), agentic tasks (TAU-bench) and computer use from screenshots (OSWorld), with a 14.9% average OSWorld success rate from screenshots alone.
Anthropic's[1][2] research post on the skill: Claude looks at screenshots and "counts how many pixels vertically or horizontally it needs to move a cursor in order to click in the correct place", and was trained on simple software such as a calculator and a text editor without internet access.
TechCrunch, 2024-10-22: "Anthropic[1][2][3] on Tuesday released an upgraded version of its Claude 3.5 Sonnet model that can understand and interact with any desktop app" through a Computer Use API in open beta on the API, Bedrock and Vertex AI.
CNBC, 2024-10-22: chief science officer Jared Kaplan says it can do tasks with "tens or even hundreds of steps"; Amazon had early access and beta testers included Asana, Canva and Notion; released Tuesday in public beta for developers.
有遗漏或错误吗?用你自己的话说明:能佐证此图钉的链接、不同的开始或结束日期及理由,或缺失、有误的信息。AI 会对照此图钉的来源进行核实,搜索更好的来源,并添加任何支持你说法的页面。图钉自身的来源仍然最重要。AI 也会查看图片:显示的是别的东西或显示效果差的图片会被移到后面或替换。
原标题: Claude 3.5 Sonnet Upgrade Released
| 基准测试 | Claude 3.5 Sonnet(新版) | Claude 3.5 Haiku | Claude 3.5 Sonnet | GPT-4o* | GPT-4o mini* | Gemini 1.5 Pro | Gemini 1.5 Flash |
|---|---|---|---|---|---|---|---|
| 研究生水平推理(GPQA Diamond) | 65.0%(0-shot CoT) | 41.6%(0-shot CoT) | 59.4%(0-shot CoT) | 53.6%(0-shot CoT) | 40.2%(0-shot CoT) | 59.1%(0-shot CoT) | 51.0%(0-shot CoT) |
| 本科水平知识(MMLU Pro) | 78.0%(0-shot CoT) | 65.0%(0-shot CoT) | 75.1%(0-shot CoT) | — | — | 75.8%(0-shot CoT) | 67.3%(0-shot CoT) |
| 代码(HumanEval) | 93.7%(0-shot) | 88.1%(0-shot) | 92.0%(0-shot) | 90.2%(0-shot) | 87.2%(0-shot) | — | — |
| 数学问题求解(MATH) | 78.3%(0-shot CoT) | 69.2%(0-shot CoT) | 71.1%(0-shot CoT) | 76.6%(0-shot CoT) | 70.2%(0-shot CoT) | 86.5%(4-shot CoT) | 77.9%(4-shot CoT) |
| 高中数学竞赛(AIME 2024) | 16.0%(0-shot CoT) | 5.3%(0-shot CoT) | 9.6%(0-shot CoT) | 9.3%(0-shot CoT) | — | — | — |
| 视觉问答(MMMU) | 70.4%(0-shot CoT) | — | 68.3%(0-shot CoT) | 69.1%(0-shot CoT) | 59.4%(0-shot CoT) | 65.9%(0-shot CoT) | 62.3%(0-shot CoT) |
| 智能体编程(SWE-bench Verified) | 49.0% | 40.6% | 33.4% | — | — | — | — |
| 智能体工具使用(TAU-bench),零售 | 69.2% | 51.0% | 62.6% | — | — | — | — |
| 智能体工具使用(TAU-bench),航空 | 46.0% | 22.8% | 36.0% | — | — | — | — |
* Anthropic 的评测表未纳入 OpenAI 的 o1 模型系列,因为该系列依赖大量的响应前计算时间,不同于典型模型。