Claude 3.5 Sonnet: Complete Benchmark Performance Guide

anthropic 3.5-sonnet 2024-06-20 Proprietary

In-depth analysis of Claude 3.5 Sonnet performance on MMLU, HumanEval, GSM8K, MATH, BBH benchmarks, including pricing and use cases.

Key Specifications

Vendoranthropic
Version3.5-sonnet
Release Date2024-06-20
Context Window200000 tokens
Input Modalitiestext, image
Output Modalitiestext
LicenseProprietary
Documentationhttps://docs.anthropic.com/claude/docs

Benchmark Performance

BenchmarkScoreUnitEvaluated AtNotesSource
MMLU88.7%2024-06-205-shotview
HUMANEVAL92pass@12024-06-20—view
GSM8K96.4%2024-06-200-shot CoTview
MATH71.1%2024-06-200-shot CoTview
BBH84.5%2024-06-203-shot CoTview

Pricing

TierPriceCurrency
Input$3 / MtokUSD
Output$15 / MtokUSD
Cache Read$0.3 / MtokUSD
Cache Write$3.75 / MtokUSD

Source: https://www.anthropic.com/pricing · as of 2024-08-01

Compliance

  • Data Residency: US
  • SOC2: ✓
  • HIPAA: ✗
  • GDPR: ✓
  • ISO 27001: ✗

Claude 3.5 Sonnet

Modellöversikt

Anthropic Claude 3.5 Sonnet 模型，200K 上下文窗口，在编码、视觉推理与长文本理解方面表现突出，平衡了智能与速度。

Kärnspecifikationer

LeverantörVersionReleasedatumKontextfönsterInmatningsmodaliteterUtmatningsmodaliteterLicens
Anthropic3.5-sonnet2024-06-20200Ktext, imagetextProprietary

Benchmarkprestanda

BenchmarkPoängEnhetAnteckningar
MMLU (Massive Multitask Language Understanding)88.7%5-shot
HumanEval92.0pass@1—
GSM8K (Grade School Math 8K)96.4%0-shot CoT
MATH71.1%0-shot CoT
BBH (BIG-Bench Hard)84.5%3-shot CoT

Priser

InmatningUtmatningCacheläsningCacheskrivning
————

per miljon tokens

Styrkor

  • MMLU score 88.7, strong knowledge reasoning.
  • HumanEval 92.0, excellent code generation.
  • GSM8K 96.4, robust math reasoning.
  • 上下文窗口 200K，支持长文本。

Svagheter

  • 闭源专有模型，不支持自托管。

Användningsområden

  • 代码生成与调试
  • 长文档摘要
  • Agent 工作流与工具调用

Referenser