Claude 3 Opus (2024-02-29): Complete Benchmark Performance Guide

anthropic 3-opus-20240229 2024-02-29 Proprietary

In-depth analysis of Claude 3 Opus (2024-02-29) performance on MMLU, HumanEval, GSM8K, MATH, BBH benchmarks, including pricing and use cases.

Key Specifications

Vendoranthropic
Version3-opus-20240229
Release Date2024-02-29
Context Window200000 tokens
Input Modalitiestext, image
Output Modalitiestext
LicenseProprietary
Documentationhttps://docs.anthropic.com/claude/docs

Benchmark Performance

BenchmarkScoreUnitEvaluated AtNotesSource
MMLU85.7%2024-02-295-shotview
HUMANEVAL79.7pass@12024-02-29—view
GSM8K82.3%2024-02-290-shot CoTview
MATH58.2%2024-02-290-shot CoTview
BBH79.5%2024-02-293-shot CoTview
GPQA38.6%2024-02-290-shotview
IFEVAL75.5%2024-02-29prompt_strictview
ARC93.4%2024-02-29challengeview
MUSR57.3%2024-02-290-shotview
WINOGRANDE87.8%2024-02-290-shotview

Pricing

TierPriceCurrency
Input$15 / MtokUSD
Output$75 / MtokUSD
Cache Read$0 / MtokUSD
Cache Write$0 / MtokUSD

Source: https://www.anthropic.com/pricing · as of 2024-02-29

Compliance

  • Data Residency: US
  • SOC2: ✓
  • HIPAA: ✗
  • GDPR: ✓
  • ISO 27001: ✗

Claude 3 Opus (2024-02-29)

Modellöversikt

Anthropic Claude 3 Opus 2024-02-29 快照版本, 200K 上下文, 旗舰级推理能力。

Kärnspecifikationer

LeverantörVersionReleasedatumKontextfönsterInmatningsmodaliteterUtmatningsmodaliteterLicens
Anthropic3-opus-202402292024-02-29200Ktext, imagetextProprietary

Benchmarkprestanda

BenchmarkPoängEnhetAnteckningar
MMLU (Massive Multitask Language Understanding)85.7%5-shot
HumanEval79.7pass@1—
GSM8K (Grade School Math 8K)82.3%0-shot CoT
MATH58.2%0-shot CoT
BBH (BIG-Bench Hard)79.5%3-shot CoT
GPQA38.6%0-shot
IFEval75.5%prompt_strict
ARC93.4%challenge
MUSR57.3%0-shot
WinoGrande87.8%0-shot

Priser

InmatningUtmatningCacheläsningCacheskrivning
————

per miljon tokens

Styrkor

  • MMLU score 85.7, strong knowledge reasoning.
  • 支持文本、图像、音频多模态输入。
  • 上下文窗口 200K，支持长文本。

Svagheter

  • 闭源专有模型，不支持自托管。

Användningsområden

  • 代码生成与调试
  • 长文档摘要
  • 视觉与图像理解
  • Agent 工作流与工具调用

Referenser