Web Search for Agent CLIs: Tool Ecosystem, Rate-Limit Survival, and Free-Quota Optimization
A research article on how agent CLIs get web search, how to avoid being rate-limited (or captcha'd) out of it, and how to make free tiers go as far as possible. Research date: Aug 2, 2026. All URLs verified by direct fetch. Motivation from this workspace's own experience: DDG/Bing/Mojeek/Brave all blocked automated searches (captcha/429) — this article maps the alternatives.
TL;DR
Web search in an agent CLI is a three-layer ecosystem:
- Built into the CLI — Claude Code ships
WebSearch/WebFetchtools (https://code.claude.com/docs/en/tools); OpenCode ships awebsearchtool backed by Exa's hosted MCP, no API key required (https://opencode.ai/docs/tools/); Gemini CLI has abrowser_agentsubagent (https://github.com/google-gemini/gemini-cli/blob/main/packages/core/src/prompts/snippets.ts); Codex CLI ships no web-search tool in its default GPT-5.1 prompt (verified in the raw prompt file) — it searches via shell (curl/rg). - MCP servers — the portable layer: Tavily (2.3k★, remote
mcp.tavily.com/mcp), Exa (4.8k★), Perplexity (2.4k★, official), Brave (1.3k★, official, 8 tool types), Bright Data (2.6k★, hostedmcp.brightdata.com/mcp). - Direct search APIs — Perplexity (\(5/1k requests), Tavily (1,000 free credits/month, no card), Brave (\)5 free credits every month, \(5/1k), Exa (\)20 signup + \(10/month free), Linkup (4,000 free queries), Bright Data (\)1.5/1k), Google CSE (closed to new customers — sunsets Jan 1, 2027).
Avoiding rate limits is a spec-compliant skill: honor Retry-After/x-ratelimit-reset headers, exponential backoff + jitter, know that billing units ≠ rate-limit units (Perplexity multi-query: 5 queries = 1 billable request but 5 rate-limit units), batch and serialize (Wikimedia etiquette), and cache aggressively — Wikipedia's official guidance is literally "take steps to cache it."
Free quotas are real but small and recurring. Stacking them gives ~13,000+ free search calls/month across providers, and per-call cost dials (search depth, context size, result counts, multi-query batching) can stretch any of them 2-5x further. Free structured APIs (GitHub 5,000 req/hr, arXiv 1 req/3s, Wikimedia) can replace general web search entirely for code/repo/paper lookups.
Part 1 — The ecosystem, layer by layer
1.1 Built into the coding CLI (zero setup, zero API key)
| CLI | Built-in search | Back-end / notes |
|---|---|---|
| Claude Code | WebSearch + WebFetch tools |
Built-in tool table: https://code.claude.com/docs/en/tools. Both permission-gated in default mode. |
| OpenCode | websearch + webfetch tools |
websearch uses Exa AI's hosted MCP service, "No API key is required" — enable via OPENCODE_ENABLE_EXA=1 or the OpenCode provider: https://opencode.ai/docs/tools/ |
| Gemini CLI | browser_agent subagent (search + browse tools) |
Defined in the prompt assembly, https://github.com/google-gemini/gemini-cli/blob/main/packages/core/src/prompts/snippets.ts |
| Codex CLI | none in the default prompt | GPT-5.1 prompt (raw file) contains no web-search tool — only rg guidance; web access is delegated to shell (curl). ToolSpec::WebSearch exists in the registry enum but is not a shipped tool: https://github.com/openai/codex/blob/main/codex-rs/core/src/tools/registry.rs |
The notable takeaway: the cheapest web search available to an agent is the one already built in — OpenCode + Exa's hosted MCP is a free, keyless, always-available search path inside a coding agent.
1.2 MCP servers — the framework layer (portable across every CLI)
MCP (https://modelcontextprotocol.io) is now the standard extension point in Claude Code (https://code.claude.com/docs/en/mcp), OpenCode (https://opencode.ai/docs/mcp-servers/), Cursor, and others. The search MCPs:
| MCP server | Stars (Aug 2, 2026) | Remote/hosted | Tools |
|---|---|---|---|
| Exa | 4.8k — https://github.com/exa-labs/exa-mcp-server | — | "web search and web crawling" (also backs OpenCode's built-in websearch) |
| Bright Data | 2.6k — https://github.com/brightdata/brightdata-mcp | https://mcp.brightdata.com/mcp | search, markdown scrape, discover + 60+ pro tools; free 5,000 credits/mo |
| Perplexity | 2.4k — https://github.com/perplexityai/modelcontextprotocol | — | official Perplexity API Platform MCP |
| Tavily | 2.3k — https://github.com/tavily-ai/tavily-mcp | https://mcp.tavily.com/mcp (OAuth supported) | search / extract / map / crawl |
| Brave | 1.3k — https://github.com/brave/brave-search-mcp-server | — | web, local, place, image, video, news, summarizer, LLM-context |
1.3 Direct search APIs — the pay-as-you-go layer
| Provider | Free tier | Paid price | Rate limits | Agent-friendly features |
|---|---|---|---|---|
| Perplexity Search | — (per-request pricing) | $5 / 1,000 requests (no token charges) | Search API: 50 query-units/s, burst 50 — independent of spend tier; Agent API: 1–33 QPS by tier (https://docs.perplexity.ai/docs/search/quickstart, https://docs.perplexity.ai/guides/rate-limits) | Multi-query (up to 5 queries = 1 billable request), domain allow/deny (≤20), language filters (≤10), search_context_size low/medium/high, max_tokens (≤1M) + max_tokens_per_page budgets |
| Tavily | 1,000 credits/month, no credit card | PAYG $0.008/credit; basic search = 1 credit, advanced = 2 | dev 100 RPM, prod 1,000 RPM; crawl 100 RPM; research 20 RPM; 429 + retry-after |
LLM-cleaned content; extract (5 URLs = 1 credit, free on failure), map, crawl, research endpoints (https://docs.tavily.com/documentation/api-credits, https://docs.tavily.com/documentation/rate-limits) |
| Brave | $5 free credits every month, auto-applied | Search $5/1k @ 50 QPS; Answers $4/1k + $5/1M tokens @ 2 QPS | 50 QPS (Search) | Independent 30B-page index; LLM-context endpoint (pre-extracted content, token-budget params); OpenAI-compatible chat endpoint; Goggles custom reranking; credit card required even on free plan (anti-fraud) (https://brave.com/search/api/) |
| Exa | $20 signup + $10 free credits/month (≈2,800 searches) | Search $7/1k (≤10 results incl.; extra results $1/1k); Contents $1/1k pages; no subscription | — | Neural search; token-efficient page contents; AI summaries billed separately ($1/1k pages); effort-metered Agent (https://exa.ai/pricing) |
| Linkup | 4,000 free queries | Search $0.005–0.006/request (1–3s); Fetch $0.001–0.005; Research $0.25–2.50 async | — | Agent-focused; SOC 2; async deep research (https://www.linkup.so/pricing) |
| Bright Data | Free trial | Web Search $1.5/1k results; SERP API $1/1k | — | 7 engines incl. regional (Google/Bing/DDG/Yandex); built-in proxy rotation + CAPTCHA solving; pay only for successful delivery (https://brightdata.com/products/data-feeds/web-search-api) |
| Google CSE | 100 free queries/day (existing customers only) | $5/1k, up to 10k/day | — | CLOSED to new customers; existing access ends Jan 1, 2027 → Vertex AI Search suggested (https://developers.google.com/custom-search/v1/overview) |
| Bing (consumer) | — | discontinued; Grounding with Bing via Azure ≈ $35/1k (community-reported) | — | Brave's FAQ independently confirms the pattern: Google/Bing have "more limited access" than independent-index APIs (https://brave.com/search/api/) |
DuckDuckGo — the engine that captcha'd this workspace's research — has no official full-web API for agents; the community's workarounds are wrappers of varying reliability. The reliable paths are the ones in the tables above.
Part 2 — How to avoid rate limits (evidence-backed playbook)
2.1 Know that billing units ≠ rate-limit units
Perplexity documents this explicitly: a multi-query request with 5 queries is one billable Search API request but consumes five rate-limit units (https://docs.perplexity.ai/guides/rate-limits). Batching saves money but still hits the limiter proportionally — design throttling around rate units, not billable requests.
2.2 Handle 429s by the headers, not by guessing
- Perplexity: 429s come with a
Retry-Afterheader; rejected 429s are not billed; docs recommend "exponential backoff with jitter" (https://docs.perplexity.ai/guides/rate-limits). - Tavily: 429 body example includes
retry-after: 60; docs say to implement retry logic "that respects the retry-after header value" (https://docs.tavily.com/documentation/rate-limits). - GitHub: when limited,
x-ratelimit-remaining= 0 — "You should not retry your request until after the time specified by the x-ratelimit-reset header"; for secondary limits, "wait for an exponentially increasing amount of time between retries"; and the warning every agent builder should memorize: "Continuing to make requests while you are rate limited may result in the banning of your integration." (https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api)
2.3 Respect the algorithm you're under
Perplexity's limiter is a leaky bucket: 50 query-units/s with 50 burst capacity. Docs: take advantage of burst for batch operations, and don't artificially spread requests when you have burst capacity — sustained rate is the constraint, not per-second pacing (https://docs.perplexity.ai/guides/rate-limits).
2.4 Serialize, batch, compress (Wikimedia's official etiquette)
Wikimedia's API etiquette page (https://www.mediawiki.org/wiki/API:Etiquette) is the canonical free-API manners guide:
- Make requests in series, not parallel; wait for one to finish.
- Batch with pipe-separated values (
titles=PageA|PageB|PageC) and generators instead of N requests. - Use
Accept-Encoding: gzip; prefer GET (cacheable) over POST. - Use
maxlagfor non-interactive tasks (defers when servers are loaded). - Set a descriptive User-Agent — "or you may be IP-blocked without notice."
- On the
ratelimitederror: retry with longer intervals, exponential backoff. - Cache: "If your requests obtain data that can be cached for a while, you should take steps to cache it."
2.5 Mind the secondary limits
GitHub: max 100 concurrent requests; ~900 points/min per endpoint; ~80 content-generating requests/min — checked via headers, never by polling /rate_limit (https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api).
2.6 Free keys vs. production keys
- Tavily: production-key rate limits (1,000 RPM) require an active paid plan or PAYGO (https://docs.tavily.com/documentation/rate-limits).
- Perplexity: rate tiers auto-upgrade with cumulative credit purchases ($50 → Tier 1 … $5,000 → Tier 5), not current balance (https://docs.perplexity.ai/guides/rate-limits).
2.7 If the problem is anti-bot (the reason this research exists)
Consumer search engines (DDG, Bing, Mojeek, Brave) gate automation with captchas and 429s. The ecosystem answer is to not scrape consumer engines:
- Use official APIs of the engines that have them (Brave's whole pitch is an independent index "not a scraper that simply uses bots to query Google or Bing" — https://brave.com/search/api/).
- For maximum reliability, scraper-adjacent work is a solved market: Bright Data's Web Search API includes proxy rotation, CAPTCHA solving, and 72M residential IPs, and charges only for successful requests (https://brightdata.com/products/data-feeds/web-search-api); its MCP server exposes a client-side
RATE_LIMITenv knob (e.g.RATE_LIMIT=100/1h) andBASE_MAX_RETRIES(https://github.com/brightdata/brightdata-mcp). - Providers themselves use anti-abuse gates: Brave requires a credit card on the free plan "as an anti-fraud measure" (https://brave.com/search/api/).
2.8 Cheat sheet
| Situation | Do |
|---|---|
429 with Retry-After |
Wait exactly that long (Tavily/Perplexity docs) |
403/429 with x-ratelimit-remaining: 0 |
Wait until x-ratelimit-reset epoch (GitHub docs) |
| Persistent 429s | Exponential backoff + jitter; then stop — further retries risk bans (GitHub/Perplexity docs) |
| Batch job | Use burst capacity in one burst; stay under sustained rate (Perplexity docs) |
| Many small reads | Serialize + pipe-batch + gzip + maxlag (Wikimedia etiquette) |
| Crawling consumers (Google/DDG) | Don't — use official APIs or proxy-backed SERP APIs (Brave/Bright Data) |
Part 3 — Making free quotas go further
3.1 What the free tiers actually are (recurring, per month)
| Provider | Monthly free quota | ≈ free searches/mo | Card required |
|---|---|---|---|
| Tavily | 1,000 credits (1 basic search = 1 credit) | 1,000 | No |
| Brave | $5 credits (auto-applied) | ~1,000 @ $5/1k | Yes (anti-fraud) |
| Exa | $10 (after $20 signup credit) | ~1,400 | No |
| Bright Data MCP | 5,000 credits (base tools = 1/request, ~$7.50 value) | 5,000 | No |
| Linkup | 4,000 queries | 4,000 | — |
| Google CSE | 100 queries/day (legacy customers only) | 3,000 | — |
| Stacked total | ~15,000+/mo |
Caveats: free credits don't roll over (Bright Data README; Tavily monthly reset); free tiers carry tight rate limits (Tavily dev 100 RPM).
3.2 Per-call cost dials (documented, per provider)
- Tavily:
search_depth: basic= 1 credit vsadvanced= 2; extract charges only on success (https://docs.tavily.com/documentation/api-credits). - Perplexity Search:
search_context_size: lowfor lightweight previews ("minimize downstream token usage");max_tokens/max_tokens_per_pagefor exact content budgets (up to 1M total);max_results1–20 (default 10); domain filters to skip noise (https://docs.perplexity.ai/docs/search/quickstart). Legacy Sonar: request fee scales with context size — \(5/\)8/$12 per 1k for low/medium/high (https://docs.perplexity.ai/docs/getting-started/pricing). - Exa: first 10 results are in the base price; extra results \(1/1k; AI page summaries billed separately (\)1/1k pages) — disable unless needed;
/contentsat $1/1k pages is the cheap path for known URLs (https://exa.ai/pricing). - Brave:
count(1–20) +freshnessfilters; the LLM-context endpoint bundles pre-extracted content withmaximum_number_of_tokens(1,024–32,768) and per-URL budgets — one call replaces search + fetch (https://github.com/brave/brave-search-mcp-server). - Perplexity multi-query: up to 5 queries in one request for one billable request — 5x value on the billing side (https://docs.perplexity.ai/docs/search/quickstart). Remember: still 5 rate-limit units.
3.3 Route structured lookups to free per-site APIs first
This workspace's research sessions already run on this principle — after search engines blocked us, the work continued on official free APIs alone:
- GitHub REST API: 60 req/hr unauthenticated, 5,000 req/hr with a personal access token; enumeration, code search, repo metadata (https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api).
- arXiv API: free, open, keyless Atom API for paper metadata — terms: "make no more than one request every three seconds, and limit requests to a single connection at a time" (https://info.arxiv.org/help/api/tou.html, https://info.arxiv.org/help/api/basics.html).
- Wikimedia API: free with etiquette (User-Agent, serial, batched) — encyclopedia-grade coverage of facts and citations (https://www.mediawiki.org/wiki/API:Etiquette).
Pattern: official structured APIs > agent search APIs > scraping consumer engines. Each level down costs more and breaks more.
3.4 Cache — the single biggest free-quota multiplier
Dedupe repeated queries (the same CLI research session often re-asks near-identical questions) and cache results with a TTL. This is literally Wikimedia's written guidance, and it converts rate limits from a wall into a mild scheduling constraint (https://www.mediawiki.org/wiki/API:Etiquette).
3.5 A concrete $0/mo stack for a research-style agent CLI
- OpenCode's built-in
websearch(Exa hosted MCP, no key) for open-ended discovery (https://opencode.ai/docs/tools/). - Tavily free tier (1,000 credits, basic depth) when you need cleaned, LLM-ready content + extraction.
- Brave free $5/month as the independent-index backup and for its LLM-context endpoint.
- GitHub + arXiv + Wikimedia for code/paper/fact lookups (near-zero consumption of the above).
- A local result cache (SQLite/JSON with TTL) in front of all of it.
≈15,000 free searches/month across providers, zero API spend, and a hard wall against consumer-engine captchas.
Conclusion
The web-search stack for agent CLIs has consolidated into a small set of credible layers: keyless built-ins (Claude Code, OpenCode/Exa), portable MCP servers (Exa 4.8k★, Bright Data 2.6k★, Perplexity 2.4k★, Tavily 2.3k★, Brave 1.3k★), and direct APIs with real, recurring free tiers (Tavily 1k credits, Brave $5, Exa \(10, Linkup 4k queries, Bright Data 5k credits — while Google CSE is closing and Bing is effectively paywalled at ~\)35/1k via Azure). Rate-limit survival is a matter of reading the response headers the providers document (Retry-After, x-ratelimit-reset), respecting the leaky-bucket semantics, batching and serializing per Wikimedia's etiquette, and never scraping consumer engines — the market's answer to that is proxy-backed SERP APIs. And free quotas stretch 2-5x with the documented cost dials: search depth, context size, result counts, multi-query billing units, per-site structured APIs, and caching. For a research CLI, $0/month plus etiquette is a fully workable stack.
References (all verified Aug 2, 2026 by direct fetch)
Built-in CLI search - Claude Code tools reference (WebSearch, WebFetch, MCP): https://code.claude.com/docs/en/tools - OpenCode tools (websearch via Exa hosted MCP, webfetch): https://opencode.ai/docs/tools/ - Gemini CLI prompt assembly (browser_agent): https://github.com/google-gemini/gemini-cli/blob/main/packages/core/src/prompts/snippets.ts - Codex GPT-5.1 prompt (no web-search tool): https://raw.githubusercontent.com/openai/codex/main/codex-rs/core/gpt_5_1_prompt.md · registry: https://github.com/openai/codex/blob/main/codex-rs/core/src/tools/registry.rs
MCP servers - Exa MCP (4.8k★): https://github.com/exa-labs/exa-mcp-server - Bright Data MCP (2.6k★, free tier, RATE_LIMIT): https://github.com/brightdata/brightdata-mcp - Perplexity MCP (2.4k★): https://github.com/perplexityai/modelcontextprotocol - Tavily MCP (2.3k★, remote server, OAuth): https://github.com/tavily-ai/tavily-mcp - Brave MCP (1.3k★): https://github.com/brave/brave-search-mcp-server
Search APIs — docs & pricing - Perplexity Search API (multi-query, filters, budgets): https://docs.perplexity.ai/docs/search/quickstart · pricing: https://docs.perplexity.ai/docs/getting-started/pricing - Perplexity rate limits & tiers (leaky bucket, 429/Retry-After, units): https://docs.perplexity.ai/guides/rate-limits - Tavily credits & pricing (1,000 free/mo, cost per endpoint): https://docs.tavily.com/documentation/api-credits · rate limits: https://docs.tavily.com/documentation/rate-limits - Brave Search API (free \(5/mo credits, plans, FAQ): https://brave.com/search/api/ - Exa pricing (\)20 + \(10/mo free, per-endpoint rates): https://exa.ai/pricing - Linkup pricing (4,000 free queries): https://www.linkup.so/pricing - Google CSE (closed to new customers, 100 free/day legacy): https://developers.google.com/custom-search/v1/overview - Bright Data Web Search API (\)1.5/1k, anti-bot, engines): https://brightdata.com/products/data-feeds/web-search-api
Rate limits & etiquette - GitHub REST rate limits (60/5,000 per hr, headers, secondary limits, banning warning): https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api - arXiv API Terms of Use (1 request/3s, single connection): https://info.arxiv.org/help/api/tou.html · basics: https://info.arxiv.org/help/api/basics.html - Wikimedia API etiquette (User-Agent, serial, batching, maxlag, caching): https://www.mediawiki.org/wiki/API:Etiquette