Skip to content

LLM Landscape Snapshot Scan Guide

Agent scan template — broad-horizon websearch for a current snapshot of the LLM ecosystem. Fill all fields via live websearch; no pre-populated data.

Last run: 2026-07-19

1. Models to Scan

For each model, search and fill: latest version, release date, context window, key differentiators.

1.1 Closed-Source / Frontier Models

Model Provider Latest Version Release Date Context Window Key Highlights
GPT series OpenAI
o-series (reasoning) OpenAI
Claude series Anthropic
Gemini series Google
Grok series xAI
DeepSeek (cloud API) DeepSeek

1.2 Open-Weight Models

Model Provider Latest Version Release Date Context Window License Key Highlights
Llama series Meta
Mistral series Mistral AI
Qwen series Alibaba (Qwen Team)
DeepSeek series (open) DeepSeek
Yi series 01.AI
Command R series Cohere

1.3 Reasoning / Thinking Models

Model Provider Base Model Thinking Token Pricing Key Differentiator
o3 / o4-mini OpenAI
Claude Opus Thinking Anthropic
DeepSeek-R1 DeepSeek
Gemini Thinking Google
Qwen-QwQ Alibaba
Grok 3 Think xAI

1.4 Notable Small / Efficient Models

Model Provider Size Release Date Use Case

2. Leaderboards & Benchmarks to Poll

For each source, search the URL and extract rankings, scores, and notable trends.

2.1 LMSys Chatbot Arena

  • URL: https://chat.lmsys.org / https://lmarena.ai
  • Search terms: "LMSys Chatbot Arena latest rankings [current quarter] [current year]"
  • Fields to capture: Overall Elo, Coding Elo, Reasoning Elo, top 10 models with scores, date of snapshot
Rank Model Overall Elo Coding Elo Reasoning Elo Trend (↑↓→)

2.2 LiveCodeBench

  • URL: https://livecodebench.github.io
  • Search terms: "LiveCodeBench latest results [current year]"
  • Fields to capture: Top models ranked by pass@1, coding score, date of snapshot
Rank Model Pass@1 Score Date

2.3 SWE-bench Verified

  • URL: https://www.swebench.com
  • Search terms: "SWE-bench Verified leaderboard latest [current year]"
  • Fields to capture: Top models, resolved rate (%), date of snapshot
Rank Model Resolved Rate (%) Date

2.4 AIME 2025 (Math Reasoning)

  • Search terms: "AIME 2025 LLM results leaderboard latest"
  • Fields to capture: Top models, score, pass@1
Model AIME Score Date

2.5 GPQA Diamond (Graduate-Level Q&A)

  • Search terms: "GPQA Diamond LLM benchmark latest results [current year]"
  • Fields to capture: Top models, score, date
Model GPQA Diamond Score Date

2.6 MMLU-Pro (Massive Multitask Language Understanding)

  • Search terms: "MMLU-Pro LLM benchmark latest results [current year]"
  • Fields to capture: Top models, score, date
Model MMLU-Pro Score Date

2.7 HumanEval+ / BigCodeBench

  • Search terms: "HumanEval+ LLM latest results [current year]" / "BigCodeBench leaderboard [current year]"
  • Fields to capture: Top models, pass@1 for code generation
Model HumanEval+ Score BigCodeBench Score Date

2.8 Human Evaluations (Scale AI / METR / SEAL)

  • Search terms: "Scale AI LLM evaluation [current year]" / "METR LLM evaluation latest" / "SEAL leaderboard latest"
  • Fields to capture: Organization, methodology, top-ranked models, key findings
Organization Top Model(s) Key Finding Date

3. API Pricing Snapshot

Search each provider's official pricing page. Capture current input/output/thinking token prices per million tokens. Note any cached prompt or batch discounts.

3.1 Pricing Table (per million tokens, USD)

Provider Model Input Price Output Price Thinking Price Cached Discount Batch Discount Max Context Date Sourced
OpenAI GPT-5
OpenAI o3
OpenAI o4-mini
Anthropic Claude Opus 4
Anthropic Claude Sonnet 4
Google Gemini 3 Pro
Google Gemini 3 Flash
xAI Grok 3
DeepSeek DeepSeek-V3
DeepSeek DeepSeek-R1

3.2 Recent Price Changes (since last scan)

Provider Model Old Price New Price Change (%) Date

4. Cost-per-Intelligence Ratio (for snapshot)

Calculate rough value ratio for top models: benchmark_score / cost_per_common_task. A quick heuristic, not rigorous.

Model Avg Benchmark Score (normalized) Est. Cost per 1M Output Tokens Value Ratio (score / cost)

5. Summary Output Template

Agent fills this after completing all scans above.

5.1 Top 10 Models by Arena Elo

Rank Model Elo Release Date

5.2 Top 5 Models by SWE-bench Verified

Rank Model Resolved Rate Date

5.3 New Entrants Since Last Quarter

Model Provider Release Date Significance
Trend Description Evidence

5.5 Best Value Models (Intelligence per Dollar)

Model Rationale

5.6 Open-Source vs. Closed-Source Gap Update

Dimension Closed-Source Leader Open-Source Leader Gap Date
Overall Quality (Arena Elo)
Coding (SWE-bench)
Math (AIME)
Long Context

6. Scan Instructions

  • Expected volume: ~20 web fetches (LMSys, LiveCodeBench, SWE-bench, and each provider's official pricing page).
  • Priority order: Benchmark leaderboards first, then pricing pages, then supplementary human evaluations.
  • Tolerance: Skip 404s; annotate stale data if the latest scan is older than 3 months. Prefer official sources over blog summaries.
  • Date stamp every data point. Without a date, a benchmark score or price is unreliable.
  • All content in English. Translate non-English sources.
  • Cost-per-intelligence ratio (section 4): Normalize benchmark scores across models on a 0–100 scale. Compute rough cost estimate using a standard task profile (e.g., 500 input + 2000 output tokens) for each model. Ratio = normalized_score / cost.
  • Output: This guide, filled with data, becomes the snapshot report. Alternatively, produce a separate companion file llm-landscape-snapshot-[yyyy-mm-dd].md with all findings.