Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.crawlstack.dev/llms.txt

Use this file to discover all available pages before exploring further.

Crawlstack is designed to be Agent-Native. By connecting Gemini CLI to your Relay Server via the Model Context Protocol (MCP), you can delegate the entire lifecycle of developing, testing, and debugging crawlers to an AI agent.

​
1. Setup the Connection

First, ensure your Relay Server is running and accessible. By default, it exposes an MCP endpoint at http://localhost:3002/mcp. Run the following command in your terminal to automatically configure the server: 
gemini mcp add --transport http crawlstack http://localhost:3002/mcp

​
Manual Configuration

Alternatively, you can manually add the server to your ~/.gemini/settings.json file: 
{
  "mcpServers": {
    "crawlstack": {
      "url": "http://localhost:3002/mcp"
    }
  }
}
Once configured, verify the connection by running /mcp list inside Gemini CLI.

​
2. Available Agent Tools

Once connected, the agent has access to a specialized toolkit for browser automation:
  • list_nodes: Find connected browser instances.
  • extension_get_cluster_state: Get a global view of all node states (running tasks, metrics) in a specific tenant cluster.
  • extension_list_crawlers: See existing crawlers on a specific node.
  • extension_upsert_crawler: Create or edit extraction scripts.
  • extension_trigger_run: Execute a crawler.
  • extension_get_run_logs: Read execution logs to fix bugs.
  • extension_preview_script: Run a script instantly. Supports keep_alive: true to leave the tab open for inspection.
  • extension_take_tab_screenshot: Capture any tab by ID (useful during previews).
  • extension_close_tab: Manually cleanup kept-alive tabs.

​
3. Interactive Debugging Pattern

Agents can iterate faster by keeping tabs alive between requests. This allows for a “Playground” workflow:
  1. Start Preview: Call extension_preview_script with keep_alive: true and some initial code.
  2. Inspect: If it fails, call extension_take_tab_screenshot with the returned tabId.
  3. Refine: Call extension_preview_script again on the same URL (or a different one) without waiting for a full reload if needed.
  4. Cleanup: Call extension_close_tab when finished.

​
4. Advanced Data Extraction

Agents can handle binary files and complex streams without leaving the browser environment:
  • XLSX/PDF Parsing: Use await import() or script injection to load libraries like SheetJS directly in the tab.
  • Local File Access: Use standard `await fetch(url)` to read intercepted downloads from the extension’s protected storage (mapped to `https://opfs-local.internal/\`).
  • Dynamic Waiting: Always prefer await runner.waitFor() over hard sleeps to ensure robustness across different network speeds.