Firecrawl
The web, translated for machines
What it is
An engine to search, scrape, crawl, and interact with the web for AI use: converts URLs — including JavaScript-heavy pages — into clean markdown or structured JSON, handles site crawling and batch scraping, and offers an autonomous agent mode for data gathering.
Why it's interesting
The de facto web layer for agents and RAG pipelines. Its AGPL core is genuine OSI open source with a deliberate open-core twist: self-host under copyleft, or pay for the cloud.
Use cases
- Feeding clean web content to RAG pipelines
- Site crawling and structured extraction for agents
- Search-and-scrape as an agent tool
Who it's for
AI engineers, data teams, agent builders
Setup
Moderate. Cloud: an API key. Self-host: a Docker Compose stack with Redis and headless browser workers
Limitations & cautions
Self-hosted lags cloud on stealth features, scraping carries its own legal and terms-of-service weather, and agent mode consumes LLM credits.
Editorial takeaway
Every agent needs eyes on the web; most teams stopped building their own the day this got good.