Skip to content

LLM Model Intelligence, API Cost, and Cost-per-Task: Horizontal Review Plan

Planning document for llm-model-intelligence-api-cost-cost-per-task.md

Outline

1. Motivation & Core Problem

  • Raw API pricing is misleading — a cheaper model may consume far more tokens to complete the same task, making its real cost higher.
  • Need to quantify "intelligence efficiency": tokens consumed per task × price per token = real cost per task.
  • Horizontal review compares models side-by-side across all three axes: intelligence, raw API cost, and task-level cost.

2. Candidate Models (horizontal comparison targets)

  • GPT-5, Claude 4 / Opus 4, Gemini 3 Pro, DeepSeek V3 / R1, Grok 3, Qwen 3, Llama 4, Mistral Large, etc.
  • Distinguish between base/fast models and reasoning/thinking variants (e.g., o3, Claude Opus thinking mode, DeepSeek-R1).

3. Axis 1: Intelligence (accuracy / capability)

  • Proxy benchmarks: MMLU-Pro, HumanEval+, SWE-bench Verified, GPQA Diamond, AIME 2025, LiveCodeBench, BigCodeBench.
  • Reasoning depth: chain-of-thought quality, tool-calling accuracy, long-context retention (Needle-in-a-Haystack), instruction-following (IFEval).
  • Qualitative factors: multi-step planning, self-correction rate, hallucination frequency on structured outputs.

4. Axis 2: Raw API Cost

  • Input / output price per million tokens.
  • Cached prompt discount, batch API discount, reasoning token pricing (for thinking models).
  • Rate limits and quota tiers (RPM, TPM, RPD).
  • Hidden costs: tool-call overhead tokens (non-cached), system prompt spillover, streaming vs. non-streaming.

5. Axis 3: Cost per Task (combining Axis 1 and Axis 2)

  • Define a standardized task suite covering common use cases:
  • Code generation (function-level, multi-file refactor)
  • Summarization (short doc, long report)
  • Translation (short text, long document)
  • Data analysis (CSV processing, SQL generation)
  • Multi-turn conversation (customer support sim, tutoring)
  • Metrics per task:
  • Median token consumption (prompt + completion, including tool calls)
  • Total cost = tokens × price per token
  • First-try accuracy (no retries or self-corrections)
  • Number of turns / tool calls required to reach a correct answer
  • Efficiency ratio: accuracy ÷ cost ("dollars per correct answer").

6. Horizontal Comparison Matrix

  • A single table summarizing all models across:
  • Benchmark scores (MMLU-Pro, HumanEval+, SWE-bench, GPQA)
  • Input/output price per million tokens
  • Median tokens per task (for each task type)
  • Cost per task (for each task type)
  • Max context window
  • Sortable / filterable by metric to identify the Pareto frontier.

7. Key Findings & Recommendations

  • Best cost-to-intelligence ratio for each use case (coding, summarization, chat, agentic workflows).
  • When to choose a reasoning model vs. a fast model.
  • Regimes where a model is drastically overpriced or underpriced relative to its actual task-level cost.
  • Surprises and counter-intuitive results (e.g., more expensive model actually cheaper per task due to higher first-try accuracy).

Criteria for Horizontal Review

Category Specific Criteria Why It Matters
Intelligence Score MMLU-Pro, HumanEval+, SWE-bench Verified, GPQA Diamond, AIME, LiveCodeBench, IFEval Standardized benchmarks covering knowledge, code, math, reasoning, and instruction-following
Token Efficiency Average tokens consumed to solve each task More verbose or "chain-of-thought heavy" models cost more per task even at identical per-token prices
API Price $ per million input tokens, $ per million output tokens, cached prompt and batch discounts Raw market cost
Task Cost Median cost per standardized task (avg_tokens × price_per_token + overhead) Combines intelligence and cost into a single actionable metric
First-Try Accuracy Probability of correct answer without retries or self-corrections Retries multiply effective cost; a model that gets it right the first time may be cheaper even at a higher unit price
Rate Limits Requests per second (RPS), tokens per minute (TPM) Affects throughput for batch or production workloads
Context Window Maximum input tokens Critical for long-document tasks (codebase analysis, contract review, report summarization)
Tool-Call Overhead Hidden cost of tool-use and reasoning tokens For coding agents and agentic systems, the real cost can be 2-5× the chat cost due to function-calling token overhead
Thinking / Reasoning Mode Pricing Whether thinking tokens are charged at a premium, at output rate, or included in output Claude, OpenAI o-series, and DeepSeek-R1 charge differently for internal reasoning tokens
Multimodal Support Image, audio, video input pricing Separate pricing tiers for vision tasks; affects cost-per-task for multimodal benchmarks
Fine-Tuning & Hosting Cost Availability and cost of fine-tuning APIs, self-hosting requirements (for open-weight models) For production use, self-hosting Llama/Mistral/Qwen may be cheaper than closed APIs at scale (but adds infra cost)

Notes

  • All price data should be dated and sourced (official pricing pages, not third-party summaries).
  • Benchmark data should reference the latest published leaderboards (LMSys Chatbot Arena, LiveCodeBench, SWE-bench).
  • For evaluating cost-per-task, use a reproducible test harness that logs every API call with token counts and wall-clock time.
  • Revisit results quarterly — both model intelligence and pricing evolve rapidly.