> ## Documentation Index
> Fetch the complete documentation index at: https://docs.crawlstack.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Autonomous Development

> How to use Gemini CLI to develop and debug crawlers.

Crawlstack is designed to be **Agent-Native**. By connecting Gemini CLI to your Relay Server via the Model Context Protocol (MCP), you can delegate the entire lifecycle of developing, testing, and debugging crawlers to an AI agent.

## 1. Setup the Connection

First, ensure your Relay Server is running and accessible. By default, it exposes an MCP endpoint at `http://localhost:3002/mcp`.

### Using the CLI (Recommended)

Run the following command in your terminal to automatically configure the server:

```bash theme={null}
gemini mcp add --transport http crawlstack http://localhost:3002/mcp
```

### Manual Configuration

Alternatively, you can manually add the server to your `~/.gemini/settings.json` file:

```json theme={null}
{
  "mcpServers": {
    "crawlstack": {
      "url": "http://localhost:3002/mcp"
    }
  }
}
```

Once configured, verify the connection by running `/mcp list` inside Gemini CLI.

## 2. Available Agent Tools

Once connected, the agent has access to a specialized toolkit for browser automation:

* `list_nodes`: Find connected browser instances.
* `extension_get_cluster_state`: Get a global view of all node states (running tasks, metrics) in a specific tenant cluster.
* `extension_list_crawlers`: See existing crawlers on a specific node.
* `extension_upsert_crawler`: Create or edit extraction scripts.
* `extension_trigger_run`: Execute a crawler.
* `extension_get_run_logs`: Read execution logs to fix bugs.
* `extension_preview_script`: Run a script instantly. Supports `keep_alive: true` to leave the tab open for inspection.
* `extension_take_tab_screenshot`: Capture any tab by ID (useful during previews).
* `extension_close_tab`: Manually cleanup kept-alive tabs.

## 3. Interactive Debugging Pattern

Agents can iterate faster by keeping tabs alive between requests. This allows for a "Playground" workflow:

1. **Start Preview**: Call `extension_preview_script` with `keep_alive: true` and some initial code.
2. **Inspect**: If it fails, call `extension_take_tab_screenshot` with the returned `tabId`.
3. **Refine**: Call `extension_preview_script` again on the *same URL* (or a different one) without waiting for a full reload if needed.
4. **Cleanup**: Call `extension_close_tab` when finished.

## 4. Advanced Data Extraction

Agents can handle binary files and complex streams without leaving the browser environment:

* **XLSX/PDF Parsing**: Use `await import()` or script injection to load libraries like SheetJS directly in the tab.
* **Local File Access**: Use standard \`await fetch(url)\` to read intercepted downloads from the extension's protected storage (mapped to \`[https://opfs-local.internal/\\\`](https://opfs-local.internal/\\`)).
* **Dynamic Waiting**: Always prefer `await runner.waitFor()` over hard sleeps to ensure robustness across different network speeds.