> ## Documentation Index
> Fetch the complete documentation index at: https://docs.crawlstack.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Architecture & Storage

> How Crawlstack handles infrastructure and data

Crawlstack is designed to be highly portable and scalable. The core philosophy is that the browser handles the entire scraping lifecycle.

## Databases

Crawlstack gives you the flexibility to choose the database backend that fits your deployment needs.

### Built-in SQLite

By default, Crawlstack uses a built-in SQLite database (running via OPFS in the browser). This is perfect for single-node setups and local development. It requires zero configuration and provides the lowest possible latency.

### libSQL (Self-hosted)

For multi-node setups where you want to share a crawl queue and extracted items across multiple browsers, you can link an external libSQL instance. This allows you to run multiple Crawlstack nodes (e.g., in Docker) pointing to a central self-hosted database.

### Turso (Cloud)

If you prefer a managed solution, Crawlstack can connect directly to [Turso](https://turso.tech). This provides a distributed, cloud-native database backend that is easy to manage and scale globally.

## File & Blob Storage

When you capture file downloads or intercept large binary blobs, Crawlstack stores them separately from your SQLite database.

### OPFS (Local)

By default, all intercepted files are saved locally using the **Origin Private File System (OPFS)**. This is a secure, high-performance storage layer built into Chromium.

* **Zero Config**: Works out of the box without external dependencies.
* **Virtual Access**: Files are served to your crawler scripts via a virtual HTTPS domain: `https://opfs-local.internal/`.

### S3 / R2 / MinIO (Cloud)

For production environments and multi-node clusters, you can configure an **S3-compatible bucket**.

* **Streaming**: Large files (50MB+) are streamed directly to the bucket via presigned URLs, keeping the extension's memory footprint low.
* **Persistence**: Files remain available even if the browser instance is destroyed or the extension is reloaded.
* **Compatibility**: Supports AWS S3, Cloudflare R2, Google Cloud Storage, and MinIO.

## Deployment Modes

<CardGroup cols={2}>
  <Card title="Browser Extension" icon="chrome">
    Install the extension in your local Chrome browser. Ideal for manual oversight, debugging, and using your existing authenticated sessions.
  </Card>

  <Card title="Headless Docker" icon="docker">
    Deploy as a headless container for production scale. Uses the exact same engine and scripts as the extension but optimized for automation.
  </Card>
</CardGroup>

## The "Direct Connection"

Unlike traditional scrapers that rely on complex API layers or proxies between the runner and the storage, Crawlstack establishes a direct connection from the browser to the database. This eliminates unnecessary network hops and allows for extraction speeds of up to **10,000 items per second**.