Yi 34B: Complete Benchmark Performance Guide

other yi-34b 2023-11-05 Apache 2.0

In-depth analysis of Yi 34B performance on MMLU, HumanEval, GSM8K, MATH, BBH benchmarks, including pricing and use cases.

Key Specifications

Vendorother
Versionyi-34b
Release Date2023-11-05
Context Window4096 tokens
Input Modalitiestext
Output Modalitiestext
LicenseApache 2.0
Documentationhttps://huggingface.co/models

Benchmark Performance

BenchmarkScoreUnitEvaluated AtNotesSource
MMLU55.5%2023-11-055-shotview
HUMANEVAL35.8pass@12023-11-05—view
GSM8K48.3%2023-11-050-shot CoTview
MATH20.5%2023-11-050-shot CoTview
BBH48.8%2023-11-053-shot CoTview
GPQA20.6%2023-11-050-shotview
IFEVAL48.2%2023-11-05prompt_strictview
ARC87.2%2023-11-05challengeview
MUSR36.8%2023-11-050-shotview
WINOGRANDE71.5%2023-11-050-shotview

Pricing

TierPriceCurrency
Input$0.4 / MtokUSD
Output$0.4 / MtokUSD
Cache Read$0 / MtokUSD
Cache Write$0 / MtokUSD

Source: https://huggingface.co/models · as of 2023-11-05

Compliance

  • Data Residency: self-host
  • SOC2: ✗
  • HIPAA: ✗
  • GDPR: ✗
  • ISO 27001: ✗

Yi 34B

Modellöversikt

01.AI Yi 34B 首代开源模型, 4K 上下文, 340 亿参数, 在开源模型中曾排名第一。

Kärnspecifikationer

LeverantörVersionReleasedatumKontextfönsterInmatningsmodaliteterUtmatningsmodaliteterLicens
Otheryi-34b2023-11-054KtexttextApache 2.0

Benchmarkprestanda

BenchmarkPoängEnhetAnteckningar
MMLU (Massive Multitask Language Understanding)55.5%5-shot
HumanEval35.8pass@1—
GSM8K (Grade School Math 8K)48.3%0-shot CoT
MATH20.5%0-shot CoT
BBH (BIG-Bench Hard)48.8%3-shot CoT
GPQA20.6%0-shot
IFEval48.2%prompt_strict
ARC87.2%challenge
MUSR36.8%0-shot
WinoGrande71.5%0-shot

Priser

InmatningUtmatningCacheläsningCacheskrivning
————

per miljon tokens

Styrkor

  • 可靠的通用模型。

Svagheter

  • MMLU 仅 55.5，知识推理偏弱。
  • HumanEval 35.8，代码能力较弱。
  • 闭源专有模型，不支持自托管。
  • 上下文窗口 4K 偏小。

Användningsområden

  • 通用对话与问答

Referenser