LLM Landscape Snapshot Scan Guide

Agent scan template — broad-horizon websearch for a current snapshot of the LLM ecosystem. Fill all fields via live websearch; no pre-populated data.

Last run: 2026-07-19

1. Models to Scan

For each model, search and fill: latest version, release date, context window, key differentiators.

1.1 Closed-Source / Frontier Models

1.2 Open-Weight Models

1.3 Reasoning / Thinking Models

1.4 Notable Small / Efficient Models

2. Leaderboards & Benchmarks to Poll

For each source, search the URL and extract rankings, scores, and notable trends.

2.1 LMSys Chatbot Arena

• URL: https://chat.lmsys.org / https://lmarena.ai
• Search terms: “LMSys Chatbot Arena latest rankings [current quarter] [current year]”
• Fields to capture: Overall Elo, Coding Elo, Reasoning Elo, top 10 models with scores, date of snapshot

2.2 LiveCodeBench

• URL: https://livecodebench.github.io
• Search terms: “LiveCodeBench latest results [current year]”
• Fields to capture: Top models ranked by pass@1, coding score, date of snapshot

2.3 SWE-bench Verified

• URL: https://www.swebench.com
• Search terms: “SWE-bench Verified leaderboard latest [current year]”
• Fields to capture: Top models, resolved rate (%), date of snapshot

2.4 AIME 2025 (Math Reasoning)

• Search terms: “AIME 2025 LLM results leaderboard latest”
• Fields to capture: Top models, score, pass@1

2.5 GPQA Diamond (Graduate-Level Q&A)

• Search terms: “GPQA Diamond LLM benchmark latest results [current year]”
• Fields to capture: Top models, score, date

2.6 MMLU-Pro (Massive Multitask Language Understanding)

• Search terms: “MMLU-Pro LLM benchmark latest results [current year]”
• Fields to capture: Top models, score, date

2.7 HumanEval+ / BigCodeBench

• Search terms: “HumanEval+ LLM latest results [current year]” / “BigCodeBench leaderboard [current year]”
• Fields to capture: Top models, pass@1 for code generation

2.8 Human Evaluations (Scale AI / METR / SEAL)

• Search terms: “Scale AI LLM evaluation [current year]” / “METR LLM evaluation latest” / “SEAL leaderboard latest”
• Fields to capture: Organization, methodology, top-ranked models, key findings

3. API Pricing Snapshot

Search each provider’s official pricing page. Capture current input/output/thinking token prices per million tokens. Note any cached prompt or batch discounts.

3.1 Pricing Table (per million tokens, USD)

3.2 Recent Price Changes (since last scan)

4. Cost-per-Intelligence Ratio (for snapshot)

Calculate rough value ratio for top models: benchmark_score / cost_per_common_task. A quick heuristic, not rigorous.

5. Summary Output Template

Agent fills this after completing all scans above.

5.1 Top 10 Models by Arena Elo

5.2 Top 5 Models by SWE-bench Verified

5.3 New Entrants Since Last Quarter

5.4 Biggest Surprises / Trends

5.5 Best Value Models (Intelligence per Dollar)

5.6 Open-Source vs. Closed-Source Gap Update

6. Scan Instructions

• Expected volume: ~20 web fetches (LMSys, LiveCodeBench, SWE-bench, and each provider’s official pricing page).
• Priority order: Benchmark leaderboards first, then pricing pages, then supplementary human evaluations.
• Tolerance: Skip 404s; annotate stale data if the latest scan is older than 3 months. Prefer official sources over blog summaries.
• Date stamp every data point. Without a date, a benchmark score or price is unreliable.
• All content in English. Translate non-English sources.
• Cost-per-intelligence ratio (section 4): Normalize benchmark scores across models on a 0–100 scale. Compute rough cost estimate using a standard task profile (e.g., 500 input + 2000 output tokens) for each model. Ratio = normalized_score / cost.
• Output: This guide, filled with data, becomes the snapshot report. Alternatively, produce a separate companion file llm-landscape-snapshot-[yyyy-mm-dd].md with all findings.