open source · mit · v0.1.0

The web toolkit for AI agents.

Search, scrape, crawl, and index the web — one static binary, zero API bills.

localhost:8080

POST /scrape 200 OK · 996ms

go.dev/doc/effective_go

{
  "data": {
    "url": "https://go.dev/doc/effective_go",
    "title": "Effective Go",
    "markdown": "# Effective Go\n\nGo is a new language…",
    "tier_used": "http",
    "extractor": "trafilatura",
    "est_tokens": 6118,
    "deduped": 2,
    "cached": false,
    "links": ["/doc/", "/doc/effective_go#concurrency", "/pkg/"],
    "link_count": 214,
    "warnings": []
  }
}

preview

Effective Go

Go is a new language. Although it borrows ideas from existing languages, it has unusual properties that make effective Go programs different in character.

Concurrency
Channels and goroutines
Interfaces

extractor: trafilatura · −84% tokens

webxd · ready 15 providers · p50 996ms · fts5

15/15 js corpus

2.8× tighter md

996ms p50 scrape

mit license

works inside claude code · cursor · devin · windsurf · codex · any mcp client

js-corpus · go run ./bench · 2026-10-03

spec

scrape: page → clean markdown, 3-tier fetch, --fit token saver
search: 15-provider metasearch, weighted rrf, --scrape --highlights
index: sqlite fts5 + embeddings — your corpus, your box
query: exa-style semantic search over your index
crawl: goal-directed adaptive crawl, concurrency, robots-aware
extract: css, schema.org, or llm via your endpoint (ollama ok)
serve: http api, async jobs, webhooks, tenancy
mcp: 17-tool mcp server, stdio + sse

from url to answer, in-process.

Four hops, one binary — every stage writes to the same sqlite file it reads from.

pipeline — single process

url any site

scrape 3-tier fetch

md trafilatura

index fts5+vec

query bm25+cos

diff

app = FirecrawlApp(

− api_key="fc-…"

+ api_url="http://localhost:8080"

)

drop-in for firecrawl.

Repaint your base URL — the SDK keeps working.

routes

/v2/scrape
/v2/crawl
/v2/search
/v2/map
/v2/extract
/v2/agent

migration guide →

pricing

one binary, zero bills.

scraping api $83/mosearch api $7/1kbrowser infra $0.10/hr webx $0

community for the self-hoster

$0 forever

includes:

all 16 commands
3 api keys
community support
self-hosted

go install

pro for team deployments

$49 /mo

everything in community, plus:

unlimited api keys
rbac
audit log
priority support

get pro

enterprise for scaled orgs

contact

everything in pro, plus:

per-deployment license
dedicated support
air-gap ok

talk to us

the free tier is a product, not a demo.