lagindicator

Polite Web Scraping: Rate-Limit Design Patterns, Whether You Need Proxies, and the AI-Agent-Friendly Open Source Stack

Agent Dev Tools Updated: September 5, 2026

The design pattern for scraping sites that allow robots: robots.txt etiquette, 429/Retry-After handling, jittered backoff, caching. When (if ever) you need proxies. Which open-source GitHub frameworks — Firecrawl, Crawl4AI, Crawlee, Scrapy — are AI-agent friendly.

A research article on how to scrape a website that explicitly allows robots: the politeness/rate-limit design pattern, when (if ever) a proxy is needed, and which open-source GitHub frameworks are worth using — including the new AI-agent-friendly generation. Research date: Aug 4, 2026. All URLs verified by direct fetch.

TL;DR

Scraping a robots-allowed site politely is a well-specified design pattern, mostly built on IETF RFCs, not folklore:

  1. Read and honor robots.txt — it’s now RFC 9309 (Sept 2022). Identify yourself in the User-Agent with a product token + contact URL. Note Google does not support crawl-delay; sites throttle you by serving 429/500/503 instead.
  2. Handle 429/Retry-After (RFC 6585 / RFC 9110): on a 429 or 5xx, back off — jittered exponential backoff is the standard (AWS’s own simulation cut call count by >half). Retry only idempotent requests.
  3. Cache aggressively — conditional requests (If-None-Match/ETag, If-Modified-Since) avoid re-downloading unchanged pages (RFC 9110). Wikimedia’s own etiquette literally says “take steps to cache it.”
  4. Cap concurrency and rate — concrete published numbers: Wikimedia allows website crawling at <10 concurrent / <20 req/s average, and unauthenticated API at ≤3 concurrent / <5 req/s. “Serial = safe” is the default politeness baseline.

Do you need a proxy? For polite scraping of a robots-allowed site: no. Wikimedia and framework best practices show a descriptive User-Agent + rate limits + delays suffice, and some sites explicitly forbid rotating identities to hide load. Proxies only become necessary when: per-IP limits bind at high volume, content is geo-restricted, or the site sits behind anti-bot systems (Cloudflare, DataDome). If you might need one, use an escalating proxy tier — Crawlee’s tieredProxyUrls starts at [null] (no proxy) and only moves up when blocked.

Open-source, agent-friendly stack on GitHub (stars Aug 4, 2026):

ProjectStarsLicenseAgent angle
Firecrawl160.8kAGPL-3.0 (SDKs MIT)self-host scrape API, LLM markdown/JSON, official MCP, “Agent ready”
Crawl4AI76kApache-2.0LLM-ready markdown, self-host Docker + MCP, natural-language extraction
ScrapeGraphAI29kMIT“extract X” via LLM, MCP
Crawlee (JS)25.2kApache-2.0polite autoscaling, tiered proxies, “data for LLMs”
Colly (Go)25.4kApache-2.0lightweight, per-domain delays
Scrapy (Python)63.6kBSD-3AutoThrottle + robots.txt middleware
Firecrawl MCP7.1kMIThosted keyless MCP server

For a project that “provides the service”: Firecrawl (open source + hosted cloud) and Crawl4AI (Docker API server with JWT auth) are the two that run a real scrape API you can self-host and point an agent at. Scrapy/Crawlee are libraries, not services; Apify and Bright Data platforms are closed source (their SDKs/MCPs are OSS).

Part 1 — The design pattern: polite, rate-limited scraping

The whole discipline is “be a good HTTP citizen.” Every piece maps to a standard:

1.1 robots.txt is a standard now (RFC 9309)

robots.txt became an IETF standard in Sept 2022 as RFC 9309, formalizing Koster’s 1994 original (whose motivation was robots that “swamped servers with rapid-fire requests” — https://www.robotstxt.org/orig.html).

The RFC makes the etiquette explicit:

Google’s framing matches: robots.txt “is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google” (https://developers.google.com/search/docs/crawling-indexing/robots/intro). Two implementation gotchas from Google’s spec:

So don’t expect crawl-delay to save you — sites throttle crawlers by HTTP status codes (500/503/429), which Google documents explicitly: “your site’s crawling rate” drops when the server returns “a significant number of URLs with 500, 503, or 429” (https://developers.google.com/crawling/docs/crawlers-fetchers/reduce-crawl-rate).

1.2 The HTTP rate-limit pattern: 429 → Retry-After → jittered backoff

1.3 Cache to cut load in half

The cheapest “rate limit” is not making the request twice.

  • Conditional requests: If-None-Match (ETag validation) exists “to enable efficient updates of cached information with a minimum amount of transaction overhead” — the server replies 304 Not Modified when the stored tag matches (RFC 9110 §13.1.2, §15.4.5). If-Modified-Since avoids transferring data when nothing changed (RFC 9110 §13.1.3) (https://www.rfc-editor.org/rfc/rfc9110.html).
  • Wikimedia’s etiquette for API clients is blunt: “If your requests obtain data that can be cached for a while, you should take steps to cache it, so you don’t request the same data over and over again” (https://www.mediawiki.org/wiki/API:Etiquette).

1.4 Concrete numbers: what “polite” actually means

The best published, site-verified numbers come from Wikimedia’s Robot policy (https://wikitech.wikimedia.org/wiki/Robot_policy):

TargetConcurrencyRate
Website (/wiki/Article, no query params)< 10 concurrentavg < 20 req/s
REST API (unauthenticated)≤ 3< 5 req/s
Action API (unauthenticated)1< 5 req/s
Media (upload.wikimedia.org)≤ 2≤ 25 Mbps
Other services (Gerrit, Phabricator, etc.)≤ 1≥ 1 s delay between requests

Plus: honor every robots.txt directive, use gzip, respect Retry-After on 429, and pause ≥15 minutes on a 5xx. Wikimedia also prefers you use dumps or CDN-cached endpoints over live requests — read https://wikitech.wikimedia.org/wiki/Robot_policy.

Framework-level politeness defaults:

  • Scrapy AutoThrottle: “spiders always start with a download delay of AUTOTHROTTLE_START_DELAY” (default 5s, max 60s), target concurrency ~1.0 in-flight per domain, and latencies of non-200 responses “are not allowed to decrease the delay.” Lowering target concurrency (e.g., 0.5) makes the crawler “more conservative and polite” (https://docs.scrapy.org/en/latest/topics/autothrottle.html).
  • MediaWiki’s baseline: making requests “in series rather than in parallel… should result in a safe request rate” (https://www.mediawiki.org/wiki/API:Etiquette).
  • Reddit’s (archived) API was a documented numeric example: 60 requests/min via OAuth, with a descriptive User-Agent and a rule to “NEVER lie about your user-agent” (https://github.com/reddit-archive/reddit/wiki/API).

The resulting pattern (in pseudocode):

identify via User-Agent (product + contact URL)
parse robots.txt (RFC 9309 semantics) before each host
schedule with a per-host politeness delay (start ~1s, tune up)
bound concurrency (start 1)
on 200: cache; store ETag/Last-Modified; revalidate with If-None-Match next time
on 429: sleep Retry-After, else jittered exp backoff
on 5xx: backoff; pause long on repeated failures
never retry non-idempotent requests

Part 2 — Is a proxy needed?

2.1 No — for polite, low-rate scraping of robots-allowed sites

The proxy is a tool for defeating IP-based blocking — Crawlee’s docs call IP blocking “one of the oldest and most effective ways of preventing access to a website” and the proxy “the most powerful weapon in our anti IP blocking arsenal” (https://crawlee.dev/docs/guides/proxy-management). If you’re not being blocked, there is nothing to defeat.

The evidence that compliant scrapers don’t need proxies:

2.2 When a proxy DOES become necessary

Three concrete triggers (proxy providers’ docs are the clearest sources):

  1. Anti-bot systems. Cloudflare’s bot docs state bots “can scrape content,” and Cloudflare products “detect this automated traffic and let you decide how to respond” — challenge or block (https://developers.cloudflare.com/bots/). DataDome installs its detection at the CDN/edge and tracks bot traffic in real time (https://docs.datadome.co/docs/getting-started). Once a site runs these, IP diversity matters.
  2. Geo-restricted content. “Residential proxies are, in fact, the way most scrapers bypass geo-restrictions,” while datacenter proxies “have a limited range of locations and are more easily detected” (https://www.scraperapi.com/blog/how-to-scrape-geo-restricted-data/). Residential networks route through real end-user IPs so “target sites see your requests as genuine local users” (https://docs.brightdata.com/proxy-networks/residential/introduction).
  3. Per-IP rate limits at scale. “As your requests increase, your target site will block your machine’s IP address” (https://www.scraperapi.com/blog/curl-with-proxy/).

2.3 If you need one: how the frameworks do it

Part 3 — The open-source landscape on GitHub

3.1 Traditional frameworks (libraries, not services)

ProjectStarsLicenseWhy it matters
Scrapy63.6k — https://github.com/scrapy/scrapyBSD-3-ClauseAutoThrottle (dynamic politeness), built-in RobotsTxtMiddleware (https://docs.scrapy.org/en/latest/topics/autothrottle.html, https://docs.scrapy.org/en/latest/topics/downloader-middleware.html), the canonical Python framework
Crawlee25.2k — https://github.com/apify/crawleeApache-2.0Ex-Apify SDK. Autoscaling, integrated proxy rotation, Cheerio/JSDOM/Playwright/Puppeteer crawlers, README markets “extract data for AI, LLMs, RAG, or GPTs” (https://github.com/apify/crawlee). Apify’s hosted platform is closed; Crawlee is the OSS core. Also has a Python port https://github.com/apify/crawlee-python
Colly25.4k — https://github.com/gocolly/collyApache-2.0Lightweight Go framework: “manages request delays and maximum concurrency per domain,” robots.txt support, caching — a great low-overhead polite crawler
ScrapyRT883 — https://github.com/scrapinghub/scrapyrtBSD-3-ClauseHTTP API wrapper: “You send a request… with spider name and URL… you get items collected by a spider” — turns Scrapy into a service

3.2 AI-agent-friendly scrapers (the new generation)

ProjectStarsLicenseAgent angle
Firecrawl160.8k — https://github.com/firecrawl/firecrawlAGPL-3.0 (SDKs/UI MIT)Self-hostable scrape API (SELF_HOST.md, docker-compose.yaml) that returns clean Markdown or LLM-structured JSON; “By default, Firecrawl respects robots.txt directives”; “Agent ready: connect to any AI agent or MCP client with a single command”
Crawl4AI76k — https://github.com/unclecode/crawl4aiApache-2.0“Crawl4AI turns the web into clean, LLM ready Markdown for RAG, agents, and data pipelines.” Clean/Fit markdown (BM25/pruning filters), LLM + CSS/XPath extraction, self-host Docker API server on port 11235 with JWT auth and MCP integration “for direct connection to AI tools like Claude Code” (https://github.com/unclecode/crawl4ai)
ScrapeGraphAI29k — https://github.com/ScrapeGraphAI/Scrapegraph-aiMIT“web scraping python library that uses LLM and direct graph logic… Just say which information you want to extract.” MIT-licensed self-host with your own LLM; managed cloud API separately; ships an MCP server via Smithery (https://smithery.ai/server/@ScrapeGraphAI/scrapegraph-mcp)

Firecrawl and Crawl4AI are the two that “provide the service”: both run a real HTTP scrape API you can self-host. (Firecrawl’s core is AGPL-3.0 — the copyleft outlier in this list; its SDKs and MCP server are MIT. Crawl4AI and ScrapeGraphAI are permissively licensed.)

3.3 MCP servers — the agent plumbing

3.4 HTML → LLM-ready Markdown helpers (pair with any crawler)

ProjectStarsLicenseNotes
Jina Reader11.8k — https://github.com/jina-ai/readerApache-2.0Open-source core of r.jina.ai (URL→Markdown) and s.jina.ai (search→Markdown); self-host via Docker ghcr.io/jina-ai/reader:oss
Mozilla Readability11.4k — https://github.com/mozilla/readabilityApache-2.0Main-content extraction used by Firefox Reader View; basis of many scrapers
Trafilatura6.4k — https://github.com/adbar/trafilaturaApache-2.0HTML→TXT/MD/JSON; “efficient and polite processing of download queues”

Part 4 — Which stack, when? (decision guide)

Polite scraping of one robots-allowed site, low volume (the case in the question): No proxy. A single-threaded client with a descriptive User-Agent, robots.txt honoring, a 1s+ per-host delay, caching with ETag revalidation, and 429/Retry-After + jittered backoff is the entire design. In practice: requests + a robots parser, or Scrapy with AutoThrottle + RobotsTxtMiddleware, or Crawlee with defaults. This is ~20 lines of logic you can also get for free from any framework’s politeness defaults.

Whole-site crawl at moderate volume: Scrapy or Crawlee, both of which give you rate/scheduling for free. Crawlee’s autoscaling + session pool is the most “reactive” — it scales up politely and (via tiered proxies) only escalates to a proxy when the site actually blocks you.

An agent / LLM needs to extract data (structured JSON, clean markdown) from many sites: The agent-friendly layer wins. Self-host Firecrawl (LLM extraction + official MCP + skill for Claude Code/OpenCode, AGPL caveat) or Crawl4AI (Apache-2.0, Docker API + MCP). For “just tell me what to extract,” ScrapeGraphAI. All three self-host, so you keep control of robots.txt respect and rate limits on your own infrastructure rather than a hosted scraper’s.

The site is protected (Cloudflare/DataDome), geo-restricted, or needs millions of pages: Now proxies matter. Use Crawlee’s tiered proxy configuration (escalate only when blocked), and only then consider residential proxies from Bright Data/Oxylabs/ScraperAPI — or pay a hosted service (Apify, Firecrawl Cloud) that manages rotation for you. If the site doesn’t want robots, this is no longer “polite scraping,” it’s anti-bot evasion — a different (riskier) game entirely.

Sources