Web Search Plus

Stop choosing search providers. Let the skill do it for you.

This skill now connects you to 10 search providers and adds a companion extraction flow for pulling content from URLs. Broad web query? → Brave or Serper. Research question? → Tavily or Exa. Need citations and grounding? → Linkup. Want scrape-ready content? → Firecrawl. Prefer privacy? → SearXNG.

✨ What Makes This Different?

• Just search — no need to think about which provider to use

• Smart routing — query analysis picks the best provider automatically

• 10 providers, 1 interface — general web, research, semantic discovery, direct answers, privacy-first, and extraction-capable providers together

• URL extraction included — pull markdown/HTML content with fallback across five providers

• Works with just 1 credential — start with any single provider, add more later

• Free/self-hosted options available — SearXNG can run at $0 API cost

🚀 Quick Start

# Interactive setup (recommended for first run)
python3 scripts/setup.py

# Or manually
cp .env.example .env
python3 scripts/search.py -q "latest OpenClaw release"
python3 scripts/extract.py --url https://example.com

The wizard explains providers, collects keys, and sets defaults.

🔑 Providers

Search providers

• Serper — shopping, prices, local, and general Google-style results; fast broad fallback

• Brave — independent web index and generic current-web queries; strong non-Google complement

• Tavily — research, explanations, and synthesis; strong research routing

• Querit — multilingual and international updates; good for cross-language recency

• Linkup — source-grounded/citation-heavy search; evidence-first queries

• Exa — semantic discovery, similar sites, and deep research; supports deep + deep-reasoning

• Firecrawl — search with scrape-ready metadata; also strong extraction provider

• Perplexity — direct answers with citations; via PERPLEXITY_API_KEY or KILOCODE_API_KEY

• You.com — current-web / RAG-friendly snippets; also supports extraction

• SearXNG — private/self-hosted search; no API key, just instance URL

Extraction providers

scripts/extract.py auto-falls back across:

1. Firecrawl

2. Linkup

3. Tavily

4. Exa

5. You.com

🧠 Routing at a Glance

Default priority:

tavily → linkup → querit → exa → firecrawl → perplexity → brave → serper → you → searxng

Examples:

python3 scripts/search.py -q "weather in Vienna today"

# generic current-web intent → Brave or Serper

python3 scripts/search.py -q "find credible sources for AI tutoring outcomes"

# citation/evidence intent → Linkup

python3 scripts/search.py -q "latest AI policy updates in Germany"

# multilingual + recency → Querit or Tavily

python3 scripts/search.py -p exa --exa-depth deep -q "LLM scaling laws research"
python3 scripts/search.py -p firecrawl -q "YC startups web scraping"

Debug routing:

python3 scripts/search.py --explain-routing -q "your query"

📖 Extraction Examples

python3 scripts/extract.py --url https://example.com
python3 scripts/extract.py --url https://docs.linkup.so --provider linkup
python3 scripts/extract.py --url https://example.com --url https://example.org --include-images
python3 scripts/extract.py --url https://example.com --format html --include-raw-html

⚙️ Configuration Notes

• .env.example documents supported env vars

• config.example.json includes provider priority and provider-specific defaults

• config.json is your local runtime config

• SearXNG still supports explicit URL config and docker-aware auto-detection

🔒 Security

SearXNG SSRF protection:

• Enforces http / https only

• Blocks common cloud metadata endpoints

• Blocks private/internal IP resolution unless SEARXNG_ALLOW_PRIVATE=1

• Uses operator-controlled config/env only for the instance URL

✅ Verification

python3 -m unittest discover -s tests -p 'test_*.py'
python3 scripts/search.py --explain-routing -q "find credible sources for climate change impacts"
python3 scripts/extract.py --url https://example.com --provider auto --compact