Web Search for Agent CLIs: Tool Ecosystem, Rate-Limit Survival, and Free-Quota Optimization

A research article on how agent CLIs get web search, how to avoid being rate-limited (or captcha’d) out of it, and how to make free tiers go as far as possible. Research date: Aug 2, 2026. All URLs verified by direct fetch. Motivation from this workspace’s own experience: DDG/Bing/Mojeek/Brave all blocked automated searches (captcha/429) — this article maps the alternatives.

TL;DR

Web search in an agent CLI is a three-layer ecosystem:

1. Built into the CLI — Claude Code ships WebSearch/WebFetch tools (https://code.claude.com/docs/en/tools); OpenCode ships a websearch tool backed by Exa’s hosted MCP, no API key required (https://opencode.ai/docs/tools/); Gemini CLI has a browser_agent subagent (https://github.com/google-gemini/gemini-cli/blob/main/packages/core/src/prompts/snippets.ts); Codex CLI ships no web-search tool in its default GPT-5.1 prompt (verified in the raw prompt file) — it searches via shell (curl/rg).
2. MCP servers — the portable layer: Tavily (2.3k★, remote mcp.tavily.com/mcp), Exa (4.8k★), Perplexity (2.4k★, official), Brave (1.3k★, official, 8 tool types), Bright Data (2.6k★, hosted mcp.brightdata.com/mcp).
3. Direct search APIs — Perplexity ($5/1k requests), Tavily (1,000 free credits/month, no card), Brave ($5 free credits every month, $5/1k), Exa ($20 signup + $10/month free), Linkup (4,000 free queries), Bright Data ($1.5/1k), Google CSE (closed to new customers — sunsets Jan 1, 2027).

Avoiding rate limits is a spec-compliant skill: honor Retry-After/x-ratelimit-reset headers, exponential backoff + jitter, know that billing units ≠ rate-limit units (Perplexity multi-query: 5 queries = 1 billable request but 5 rate-limit units), batch and serialize (Wikimedia etiquette), and cache aggressively — Wikipedia’s official guidance is literally “take steps to cache it.”

Free quotas are real but small and recurring. Stacking them gives ~13,000+ free search calls/month across providers, and per-call cost dials (search depth, context size, result counts, multi-query batching) can stretch any of them 2-5x further. Free structured APIs (GitHub 5,000 req/hr, arXiv 1 req/3s, Wikimedia) can replace general web search entirely for code/repo/paper lookups.

Part 1 — The ecosystem, layer by layer

1.1 Built into the coding CLI (zero setup, zero API key)

The notable takeaway: the cheapest web search available to an agent is the one already built in — OpenCode + Exa’s hosted MCP is a free, keyless, always-available search path inside a coding agent.

1.2 MCP servers — the framework layer (portable across every CLI)

MCP (https://modelcontextprotocol.io) is now the standard extension point in Claude Code (https://code.claude.com/docs/en/mcp), OpenCode (https://opencode.ai/docs/mcp-servers/), Cursor, and others. The search MCPs:

1.3 Direct search APIs — the pay-as-you-go layer

DuckDuckGo — the engine that captcha’d this workspace’s research — has no official full-web API for agents; the community’s workarounds are wrappers of varying reliability. The reliable paths are the ones in the tables above.

Part 2 — How to avoid rate limits (evidence-backed playbook)

2.1 Know that billing units ≠ rate-limit units

Perplexity documents this explicitly: a multi-query request with 5 queries is one billable Search API request but consumes five rate-limit units (https://docs.perplexity.ai/guides/rate-limits). Batching saves money but still hits the limiter proportionally — design throttling around rate units, not billable requests.

2.2 Handle 429s by the headers, not by guessing

• Perplexity: 429s come with a Retry-After header; rejected 429s are not billed; docs recommend “exponential backoff with jitter” (https://docs.perplexity.ai/guides/rate-limits).
• Tavily: 429 body example includes retry-after: 60; docs say to implement retry logic “that respects the retry-after header value” (https://docs.tavily.com/documentation/rate-limits).
• GitHub: when limited, x-ratelimit-remaining = 0 — “You should not retry your request until after the time specified by the x-ratelimit-reset header”; for secondary limits, “wait for an exponentially increasing amount of time between retries”; and the warning every agent builder should memorize: “Continuing to make requests while you are rate limited may result in the banning of your integration.” (https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api)

2.3 Respect the algorithm you’re under

Perplexity’s limiter is a leaky bucket: 50 query-units/s with 50 burst capacity. Docs: take advantage of burst for batch operations, and don’t artificially spread requests when you have burst capacity — sustained rate is the constraint, not per-second pacing (https://docs.perplexity.ai/guides/rate-limits).

2.4 Serialize, batch, compress (Wikimedia’s official etiquette)

Wikimedia’s API etiquette page (https://www.mediawiki.org/wiki/API:Etiquette) is the canonical free-API manners guide:

• Make requests in series, not parallel; wait for one to finish.
• Batch with pipe-separated values (titles=PageA|PageB|PageC) and generators instead of N requests.
• Use Accept-Encoding: gzip; prefer GET (cacheable) over POST.
• Use maxlag for non-interactive tasks (defers when servers are loaded).
• Set a descriptive User-Agent — “or you may be IP-blocked without notice.”
• On the ratelimited error: retry with longer intervals, exponential backoff.
• Cache: “If your requests obtain data that can be cached for a while, you should take steps to cache it.”

2.5 Mind the secondary limits

GitHub: max 100 concurrent requests; ~900 points/min per endpoint; ~80 content-generating requests/min — checked via headers, never by polling /rate_limit (https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api).

2.6 Free keys vs. production keys

• Tavily: production-key rate limits (1,000 RPM) require an active paid plan or PAYGO (https://docs.tavily.com/documentation/rate-limits).
• Perplexity: rate tiers auto-upgrade with cumulative credit purchases ($50 → Tier 1 … $5,000 → Tier 5), not current balance (https://docs.perplexity.ai/guides/rate-limits).

2.7 If the problem is anti-bot (the reason this research exists)

Consumer search engines (DDG, Bing, Mojeek, Brave) gate automation with captchas and 429s. The ecosystem answer is to not scrape consumer engines:

• Use official APIs of the engines that have them (Brave’s whole pitch is an independent index “not a scraper that simply uses bots to query Google or Bing” — https://brave.com/search/api/).
• For maximum reliability, scraper-adjacent work is a solved market: Bright Data’s Web Search API includes proxy rotation, CAPTCHA solving, and 72M residential IPs, and charges only for successful requests (https://brightdata.com/products/data-feeds/web-search-api); its MCP server exposes a client-side RATE_LIMIT env knob (e.g. RATE_LIMIT=100/1h) and BASE_MAX_RETRIES (https://github.com/brightdata/brightdata-mcp).
• Providers themselves use anti-abuse gates: Brave requires a credit card on the free plan “as an anti-fraud measure” (https://brave.com/search/api/).

2.8 Cheat sheet

Part 3 — Making free quotas go further

3.1 What the free tiers actually are (recurring, per month)

Caveats: free credits don’t roll over (Bright Data README; Tavily monthly reset); free tiers carry tight rate limits (Tavily dev 100 RPM).

3.2 Per-call cost dials (documented, per provider)

• Tavily: search_depth: basic = 1 credit vs advanced = 2; extract charges only on success (https://docs.tavily.com/documentation/api-credits).
• Perplexity Search: search_context_size: low for lightweight previews (“minimize downstream token usage”); max_tokens/max_tokens_per_page for exact content budgets (up to 1M total); max_results 1–20 (default 10); domain filters to skip noise (https://docs.perplexity.ai/docs/search/quickstart). Legacy Sonar: request fee scales with context size — $5/$8/$12 per 1k for low/medium/high (https://docs.perplexity.ai/docs/getting-started/pricing).
• Exa: first 10 results are in the base price; extra results $1/1k; AI page summaries billed separately ($1/1k pages) — disable unless needed; /contents at $1/1k pages is the cheap path for known URLs (https://exa.ai/pricing).
• Brave: count (1–20) + freshness filters; the LLM-context endpoint bundles pre-extracted content with maximum_number_of_tokens (1,024–32,768) and per-URL budgets — one call replaces search + fetch (https://github.com/brave/brave-search-mcp-server).
• Perplexity multi-query: up to 5 queries in one request for one billable request — 5x value on the billing side (https://docs.perplexity.ai/docs/search/quickstart). Remember: still 5 rate-limit units.

3.3 Route structured lookups to free per-site APIs first

This workspace’s research sessions already run on this principle — after search engines blocked us, the work continued on official free APIs alone:

• GitHub REST API: 60 req/hr unauthenticated, 5,000 req/hr with a personal access token; enumeration, code search, repo metadata (https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api).
• arXiv API: free, open, keyless Atom API for paper metadata — terms: “make no more than one request every three seconds, and limit requests to a single connection at a time” (https://info.arxiv.org/help/api/tou.html, https://info.arxiv.org/help/api/basics.html).
• Wikimedia API: free with etiquette (User-Agent, serial, batched) — encyclopedia-grade coverage of facts and citations (https://www.mediawiki.org/wiki/API:Etiquette).

Pattern: official structured APIs > agent search APIs > scraping consumer engines. Each level down costs more and breaks more.

3.4 Cache — the single biggest free-quota multiplier

Dedupe repeated queries (the same CLI research session often re-asks near-identical questions) and cache results with a TTL. This is literally Wikimedia’s written guidance, and it converts rate limits from a wall into a mild scheduling constraint (https://www.mediawiki.org/wiki/API:Etiquette).

3.5 A concrete $0/mo stack for a research-style agent CLI

1. OpenCode’s built-in websearch (Exa hosted MCP, no key) for open-ended discovery (https://opencode.ai/docs/tools/).
2. Tavily free tier (1,000 credits, basic depth) when you need cleaned, LLM-ready content + extraction.
3. Brave free $5/month as the independent-index backup and for its LLM-context endpoint.
4. GitHub + arXiv + Wikimedia for code/paper/fact lookups (near-zero consumption of the above).
5. A local result cache (SQLite/JSON with TTL) in front of all of it.

≈15,000 free searches/month across providers, zero API spend, and a hard wall against consumer-engine captchas.

Conclusion

The web-search stack for agent CLIs has consolidated into a small set of credible layers: keyless built-ins (Claude Code, OpenCode/Exa), portable MCP servers (Exa 4.8k★, Bright Data 2.6k★, Perplexity 2.4k★, Tavily 2.3k★, Brave 1.3k★), and direct APIs with real, recurring free tiers (Tavily 1k credits, Brave $5, Exa $10, Linkup 4k queries, Bright Data 5k credits — while Google CSE is closing and Bing is effectively paywalled at ~$35/1k via Azure). Rate-limit survival is a matter of reading the response headers the providers document (Retry-After, x-ratelimit-reset), respecting the leaky-bucket semantics, batching and serializing per Wikimedia’s etiquette, and never scraping consumer engines — the market’s answer to that is proxy-backed SERP APIs. And free quotas stretch 2-5x with the documented cost dials: search depth, context size, result counts, multi-query billing units, per-site structured APIs, and caching. For a research CLI, $0/month plus etiquette is a fully workable stack.

References (all verified Aug 2, 2026 by direct fetch)

Built-in CLI search

• Claude Code tools reference (WebSearch, WebFetch, MCP): https://code.claude.com/docs/en/tools
• OpenCode tools (websearch via Exa hosted MCP, webfetch): https://opencode.ai/docs/tools/
• Gemini CLI prompt assembly (browser_agent): https://github.com/google-gemini/gemini-cli/blob/main/packages/core/src/prompts/snippets.ts
• Codex GPT-5.1 prompt (no web-search tool): https://raw.githubusercontent.com/openai/codex/main/codex-rs/core/gpt_5_1_prompt.md · registry: https://github.com/openai/codex/blob/main/codex-rs/core/src/tools/registry.rs

MCP servers

• Exa MCP (4.8k★): https://github.com/exa-labs/exa-mcp-server
• Bright Data MCP (2.6k★, free tier, RATE_LIMIT): https://github.com/brightdata/brightdata-mcp
• Perplexity MCP (2.4k★): https://github.com/perplexityai/modelcontextprotocol
• Tavily MCP (2.3k★, remote server, OAuth): https://github.com/tavily-ai/tavily-mcp
• Brave MCP (1.3k★): https://github.com/brave/brave-search-mcp-server

Search APIs — docs & pricing

• Perplexity Search API (multi-query, filters, budgets): https://docs.perplexity.ai/docs/search/quickstart · pricing: https://docs.perplexity.ai/docs/getting-started/pricing
• Perplexity rate limits & tiers (leaky bucket, 429/Retry-After, units): https://docs.perplexity.ai/guides/rate-limits
• Tavily credits & pricing (1,000 free/mo, cost per endpoint): https://docs.tavily.com/documentation/api-credits · rate limits: https://docs.tavily.com/documentation/rate-limits
• Brave Search API (free $5/mo credits, plans, FAQ): https://brave.com/search/api/
• Exa pricing ($20 + $10/mo free, per-endpoint rates): https://exa.ai/pricing
• Linkup pricing (4,000 free queries): https://www.linkup.so/pricing
• Google CSE (closed to new customers, 100 free/day legacy): https://developers.google.com/custom-search/v1/overview
• Bright Data Web Search API ($1.5/1k, anti-bot, engines): https://brightdata.com/products/data-feeds/web-search-api

Rate limits & etiquette

• GitHub REST rate limits (60/5,000 per hr, headers, secondary limits, banning warning): https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api
• arXiv API Terms of Use (1 request/3s, single connection): https://info.arxiv.org/help/api/tou.html · basics: https://info.arxiv.org/help/api/basics.html
• Wikimedia API etiquette (User-Agent, serial, batching, maxlag, caching): https://www.mediawiki.org/wiki/API:Etiquette