One TypeScript client for scraping URLs to markdown. Pick Firecrawl, TinyFish, Jina, Tavily, Spider, Browserbase, or local Cheerio, then fail over without rewriting callers.
- Adapters against live vendor APIs: Firecrawl v2, TinyFish Fetch/Search/Agent, Jina, Tavily, Spider.cloud, Browserbase Fetch, and Cheerio
scrape(url)is the verb.map(),crawl(),extract(),search(),agent(), andscrapeMany()when a provider can do them- Abortable timeouts, retries on retryable errors, and automatic failover
- Site/docs roots try
/llms.txtbefore HTML fromEnv()builds the client from the keys you already have- CLI, Vercel AI SDK tools, and an MCP server for full-page markdown
npm install scrape-sdkWorks on Node 20+ and Bun. Keep provider API keys out of client code.
import { scrape } from "scrape-sdk";
const page = await scrape("https://stripe.com");
console.log(page.markdown);
console.log(`via ${page.provider} in ${page.latencyMs}ms`);No API key required — Jina + local are always in the chain. Add keys and it fails over:
import { fromEnv } from "scrape-sdk";
const scraper = fromEnv();
const page = await scraper.scrape("https://stripe.com");Firecrawl Keyless is opt-in so a no-key upgrade does not silently change your network or free-quota usage:
const scraper = fromEnv({ firecrawlKeyless: true });Or pick the order yourself:
import { createScrapeClient } from "scrape-sdk";
import { firecrawl } from "scrape-sdk/firecrawl";
import { jina } from "scrape-sdk/jina";
import { local } from "scrape-sdk/local";
const scraper = createScrapeClient({
providers: [
firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY! }),
jina(),
local(),
],
});Providers that cannot perform an operation are skipped. If none can, you get a CapabilityError instead of a fake result.
- Client timeouts are cumulative across the
/llms.txtprobe, retries, and fallback providers. maxChars: 0is a hard zero-character limit; non-empty output is markedtruncated.- Unsupported adapter options now fail explicitly instead of silently degrading. Target-page
headersandformat: "html"can raiseUnsupportedOptionErroron adapters that cannot honor them.
| Method | Use when |
|---|---|
scrape(url) |
You already have a URL |
search(query) |
You need to find URLs |
map(url) |
You need a site's URL list, not bodies |
crawl(url) |
You need many page bodies from one site |
extract(url, { schema }) |
You need structured JSON |
scrapeMany(urls) |
You have a list of URLs |
agent(url, { goal }) |
You need an interactive, multi-step web task |
| Provider | Import | Key | scrape | search | map | crawl | extract | JS |
|---|---|---|---|---|---|---|---|---|
| Firecrawl v2 / Keyless | scrape-sdk/firecrawl |
optional | yes | yes | yes | yes | yes | yes |
| TinyFish Fetch/Search + Agent* | scrape-sdk/tinyfish |
yes | yes | yes | — | — | Agent* | yes |
| Jina | scrape-sdk/jina |
optional | yes | yes | — | — | — | yes |
| Tavily | scrape-sdk/tavily |
yes | yes | yes | — | — | — | — |
| Spider.cloud | scrape-sdk/spider |
yes | yes | — | — | yes | — | yes |
| Browserbase | scrape-sdk/browserbase |
yes | yes | — | — | — | yes | — |
| Local Cheerio | scrape-sdk/local |
no | yes | — | — | — | — | no |
Agent* is available only when TinyFish is configured with enableAgent: true.
TinyFish Fetch/Search are free within the provider's account limits. TinyFish Agent is opt-in because it is a metered goal-based browser run; configure tinyfish({ apiKey, enableAgent: true }) or fromEnv({ tinyfishAgent: true }). TinyFish does not provide native map() or crawl() in this adapter.
Firecrawl Keyless supports the no-key free surface; use FIRECRAWL_KEYLESS=1 or fromEnv({ firecrawlKeyless: true }). Firecrawl crawl returns a job id; the adapter polls until it finishes. Browserbase is POST /v1/fetch, not a Playwright session, so it does not run page JavaScript.
npx scrape-sdk https://stripe.com
npx scrape-sdk search "vercel ai sdk tools"
npx scrape-sdk map https://docs.firecrawl.dev
npx scrape-sdk crawl https://docs.firecrawl.dev --limit 5 --json
npx scrape-sdk scrape https://example.com --provider local{
"mcpServers": {
"scrape-sdk": {
"command": "npx",
"args": ["-y", "scrape-sdk-mcp"],
"env": {
"FIRECRAWL_API_KEY": "fc-...",
"FIRECRAWL_KEYLESS": "1",
"TINYFISH_API_KEY": "sk-tinyfish-...",
"TINYFISH_AGENT": "1",
"TAVILY_API_KEY": "tvly-..."
}
}
}
}scrape_url returns the full page as markdown — not a summary. Host WebFetch often summarizes; this does not. map_site, crawl_site, and extract_json register when the configured providers support them.
Full docs live at scrape-sdk.com/docs. Good places to start:
MIT © Arush Wadhawan