> ## Documentation Index
> Fetch the complete documentation index at: https://deepline.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Cloudflare: Crawl

> Crawling websites with browser rendering and extracting content as markdown, HTML, or structured JSON. Includes SDK V2 examples, schemas, and cost.

Crawling websites with browser rendering and extracting content as markdown, HTML, or structured JSON.

> [!NOTE]
>
> Tool ID: `cloudflare_crawl`

## Run this action

Use the TypeScript SDK for one call. Use `ctx.tools.execute(...)` inside a Play for durable calls, scheduled calls, or CSV processing.

```ts theme={"languages":{"custom":["/languages/deepline-prompt.json"]}}
import { Deepline } from 'deepline';

const deepline = await Deepline.connect();
const result = await deepline.tools.execute(
  'cloudflare_crawl',
  {
    "url": "https://example.com"
  },
);

console.log(result.toolResponse.raw);
```

### CLI

```bash theme={"languages":{"custom":["/languages/deepline-prompt.json"]}}
deepline tools execute cloudflare_crawl --input '{
  "url": "https://example.com"
}' --json
```

## Example response

The SDK returns this shape at `result.toolResponse.raw`. The example values illustrate the response.

```json theme={"languages":{"custom":["/languages/deepline-prompt.json"]}}
{
  "data": {
    "jobId": "job_123",
    "status": "example",
    "browserSecondsUsed": 123,
    "total": 123,
    "finished": 123,
    "records": [
      {
        "url": "https://example.com"
      }
    ],
    "cursor": 123,
    "timedOut": true
  }
}
```

Use `deepline describe cloudflare_crawl --json` for the latest machine-readable contract.

## Input reference

Cloudflare Browser Rendering crawl input.

| Name | Type | Required | Default | Details |
| - | - | - | - | - |
| `payload.url` | `string` | Yes | — | Starting URL to crawl |
| `payload.accountId` | `string` | No | — | Cloudflare account ID. Falls back to CLOUDFLARE\_ACCOUNT\_ID env var |
| `payload.limit` | `integer` | No | `10` | Max pages to crawl (default 10, max 100,000) |
| `payload.depth` | `integer` | No | — | Max link depth from starting URL (default/max 100,000) |
| `payload.source` | `"all" \| "sitemaps" \| "links"` | No | — | URL discovery source: all (default), sitemaps, or links Allowed: `all`, `sitemaps`, `links`. |
| `payload.formats` | `array` | No | `["markdown"]` | Content formats to return (default \["markdown"]). JSON uses Workers AI for extraction |
| `payload.render` | `boolean` | No | `true` | Execute JavaScript via headless browser (default true). false = fast HTML fetch |
| `payload.jsonOptions` | `object` | No | — | AI extraction config when formats includes "json" |
| `payload.maxAge` | `integer` | No | — | Cache TTL in seconds (default 86400, max 604800) |
| `payload.modifiedSince` | `integer` | No | — | Unix timestamp — only crawl pages modified since this time |
| `payload.options` | `object` | No | — | URL filtering options. excludePatterns takes priority over includePatterns |
| `payload.timeoutMs` | `integer` | No | `300000` | Client-side poll timeout in ms (default 300000). Not sent to Cloudflare API |
| `payload.overallTimeoutMs` | `integer` | No | — | Optional total wall-clock budget including submission, polling, retries and backoff. An ambiguous submission timeout does not prove the provider created no job. Minimum: 1. |

<details>
  <summary>Show raw input schema</summary>

  ### Input JSON Schema

  ```json theme={"languages":{"custom":["/languages/deepline-prompt.json"]}}
  {
    "type": "object",
    "description": "Cloudflare Browser Rendering crawl input.",
    "properties": {
      "url": {
        "type": "string",
        "description": "Starting URL to crawl"
      },
      "accountId": {
        "type": "string",
        "description": "Cloudflare account ID. Falls back to CLOUDFLARE_ACCOUNT_ID env var"
      },
      "limit": {
        "type": "integer",
        "description": "Max pages to crawl (default 10, max 100,000)",
        "default": 10
      },
      "depth": {
        "type": "integer",
        "description": "Max link depth from starting URL (default/max 100,000)"
      },
      "source": {
        "type": "string",
        "description": "URL discovery source: all (default), sitemaps, or links",
        "enum": [
          "all",
          "sitemaps",
          "links"
        ]
      },
      "formats": {
        "type": "array",
        "description": "Content formats to return (default [\"markdown\"]). JSON uses Workers AI for extraction",
        "default": [
          "markdown"
        ],
        "items": {
          "type": "string",
          "enum": [
            "html",
            "markdown",
            "json"
          ]
        }
      },
      "render": {
        "type": "boolean",
        "description": "Execute JavaScript via headless browser (default true). false = fast HTML fetch",
        "default": true
      },
      "jsonOptions": {
        "type": "object",
        "description": "AI extraction config when formats includes \"json\"",
        "properties": {
          "prompt": {
            "type": "string"
          },
          "response_format": {},
          "custom_ai": {}
        },
        "additionalProperties": false
      },
      "maxAge": {
        "type": "integer",
        "description": "Cache TTL in seconds (default 86400, max 604800)"
      },
      "modifiedSince": {
        "type": "integer",
        "description": "Unix timestamp — only crawl pages modified since this time"
      },
      "options": {
        "type": "object",
        "description": "URL filtering options. excludePatterns takes priority over includePatterns",
        "properties": {
          "includePatterns": {
            "type": "array",
            "items": {
              "type": "string"
            }
          },
          "excludePatterns": {
            "type": "array",
            "items": {
              "type": "string"
            }
          },
          "includeExternalLinks": {
            "type": "boolean"
          },
          "includeSubdomains": {
            "type": "boolean"
          }
        },
        "additionalProperties": false
      },
      "timeoutMs": {
        "type": "integer",
        "description": "Client-side poll timeout in ms (default 300000). Not sent to Cloudflare API",
        "default": 300000
      },
      "overallTimeoutMs": {
        "type": "integer",
        "description": "Optional total wall-clock budget including submission, polling, retries and backoff. An ambiguous submission timeout does not prove the provider created no job.",
        "minimum": 1
      }
    },
    "required": [
      "url"
    ],
    "additionalProperties": false
  }
  ```
</details>

## Output reference

Cloudflare Browser Rendering crawl result.

| Name | Type | Required | Default | Details |
| - | - | - | - | - |
| `result.data` | `object` | Yes | — | Provider response payload. |
| `result.data.jobId` | `string` | Yes | — | — |
| `result.data.status` | `string` | Yes | — | — |
| `result.data.browserSecondsUsed` | `number` | Yes | — | — |
| `result.data.total` | `number` | Yes | — | — |
| `result.data.finished` | `number` | Yes | — | — |
| `result.data.records` | `array` | Yes | — | — |
| `result.data.records[].url` | `string` | Yes | — | — |
| `result.data.records[].status` | `string` | No | — | — |
| `result.data.records[].markdown` | `string` | No | — | — |
| `result.data.records[].html` | `string` | No | — | — |
| `result.data.records[].json` | `unknown` | No | — | — |
| `result.data.records[].metadata` | `record` | No | — | — |
| `result.data.cursor` | `number \| string \| unknown` | Yes | — | — |
| `result.data.timedOut` | `boolean` | Yes | — | — |
| `result.meta` | `record` | No | — | Additional response metadata (status, paging). |

<details>
  <summary>Show raw output schema</summary>

  ### Output JSON Schema

  ```json theme={"languages":{"custom":["/languages/deepline-prompt.json"]}}
  {
    "type": "object",
    "description": "Cloudflare Browser Rendering crawl result.",
    "properties": {
      "data": {
        "type": "object",
        "description": "Provider response payload.",
        "properties": {
          "jobId": {
            "type": "string"
          },
          "status": {
            "type": "string"
          },
          "browserSecondsUsed": {
            "type": "number"
          },
          "total": {
            "type": "number"
          },
          "finished": {
            "type": "number"
          },
          "records": {
            "type": "array",
            "items": {
              "type": "object",
              "properties": {
                "url": {
                  "type": "string"
                },
                "status": {
                  "type": "string"
                },
                "markdown": {
                  "type": "string"
                },
                "html": {
                  "type": "string"
                },
                "json": {},
                "metadata": {
                  "type": "object",
                  "additionalProperties": {}
                }
              },
              "required": [
                "url"
              ],
              "additionalProperties": false
            }
          },
          "cursor": {
            "anyOf": [
              {
                "type": "number"
              },
              {
                "type": "string"
              },
              {}
            ]
          },
          "timedOut": {
            "type": "boolean"
          }
        },
        "required": [
          "jobId",
          "status",
          "browserSecondsUsed",
          "total",
          "finished",
          "records",
          "cursor",
          "timedOut"
        ],
        "additionalProperties": false
      },
      "meta": {
        "type": "object",
        "description": "Additional response metadata (status, paging).",
        "additionalProperties": {}
      }
    },
    "required": [
      "data"
    ],
    "additionalProperties": false
  }
  ```
</details>

## Deepline cost

* Pricing model: `provider_usage` (provider usage).
* Estimated Deepline credits: `0.01` per pricing unit.

## Related documentation

* [Cloudflare provider guide](https://deepline.com/docs/providers/cloudflare/guide.md)
* [SDK V2 quickstart](https://deepline.com/docs/sdk-v2/quickstart.md)
* [SDK reference](https://deepline.com/docs/sdk-v2/sdk-reference.md)
* [Run tools across a CSV](https://deepline.com/docs/sdk-v2/batch-csv.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.