Cloudflare: Crawl

Crawling websites with browser rendering and extracting content as markdown, HTML, or structured JSON.

Tool ID: cloudflare_crawl

Run this action

Use the TypeScript SDK for one call. Use ctx.tools.execute(...) inside a Play for durable calls, scheduled calls, or CSV processing.

import { Deepline } from 'deepline';

const deepline = await Deepline.connect();
const result = await deepline.tools.execute(
  'cloudflare_crawl',
  {
    "url": "https://example.com"
  },
);

console.log(result.toolResponse.raw);

CLI

deepline tools execute cloudflare_crawl --input '{
  "url": "https://example.com"
}' --json

Example response

The SDK returns this shape at result.toolResponse.raw. The example values illustrate the response.

{
  "data": {
    "jobId": "job_123",
    "status": "example",
    "browserSecondsUsed": 123,
    "total": 123,
    "finished": 123,
    "records": [
      {
        "url": "https://example.com"
      }
    ],
    "cursor": 123,
    "timedOut": true
  }
}

Use deepline describe cloudflare_crawl --json for the latest machine-readable contract.

Input reference

Cloudflare Browser Rendering crawl input.

Name	Type	Required	Default	Details
`payload.url`	`string`	Yes	—	Starting URL to crawl
`payload.accountId`	`string`	No	—	Cloudflare account ID. Falls back to CLOUDFLARE_ACCOUNT_ID env var
`payload.limit`	`integer`	No	`10`	Max pages to crawl (default 10, max 100,000)
`payload.depth`	`integer`	No	—	Max link depth from starting URL (default/max 100,000)
`payload.source`	`"all" \| "sitemaps" \| "links"`	No	—	URL discovery source: all (default), sitemaps, or links Allowed: `all`, `sitemaps`, `links`.
`payload.formats`	`array`	No	`["markdown"]`	Content formats to return (default [“markdown”]). JSON uses Workers AI for extraction
`payload.render`	`boolean`	No	`true`	Execute JavaScript via headless browser (default true). false = fast HTML fetch
`payload.jsonOptions`	`object`	No	—	AI extraction config when formats includes “json”
`payload.maxAge`	`integer`	No	—	Cache TTL in seconds (default 86400, max 604800)
`payload.modifiedSince`	`integer`	No	—	Unix timestamp — only crawl pages modified since this time
`payload.options`	`object`	No	—	URL filtering options. excludePatterns takes priority over includePatterns
`payload.timeoutMs`	`integer`	No	`300000`	Client-side poll timeout in ms (default 300000). Not sent to Cloudflare API
`payload.overallTimeoutMs`	`integer`	No	—	Optional total wall-clock budget including submission, polling, retries and backoff. An ambiguous submission timeout does not prove the provider created no job. Minimum: 1.

Output reference

Cloudflare Browser Rendering crawl result.

Name	Type	Required	Default	Details
`result.data`	`object`	Yes	—	Provider response payload.
`result.data.jobId`	`string`	Yes	—	—
`result.data.status`	`string`	Yes	—	—
`result.data.browserSecondsUsed`	`number`	Yes	—	—
`result.data.total`	`number`	Yes	—	—
`result.data.finished`	`number`	Yes	—	—
`result.data.records`	`array`	Yes	—	—
`result.data.records[].url`	`string`	Yes	—	—
`result.data.records[].status`	`string`	No	—	—
`result.data.records[].markdown`	`string`	No	—	—
`result.data.records[].html`	`string`	No	—	—
`result.data.records[].json`	`unknown`	No	—	—
`result.data.records[].metadata`	`record`	No	—	—
`result.data.cursor`	`number \| string \| unknown`	Yes	—	—
`result.data.timedOut`	`boolean`	Yes	—	—
`result.meta`	`record`	No	—	Additional response metadata (status, paging).

Deepline cost

Pricing model: provider_usage (provider usage).
Estimated Deepline credits: 0.01 per pricing unit.

​Run this action

​CLI

​Example response

​Input reference

​Output reference

​Deepline cost

​Related documentation

Run this action

CLI

Example response

Input reference

Output reference

Deepline cost

Related documentation