Firecrawl Collection Policy Guardrails
Overview
Put enforceable policy before every Firecrawl request and downstream action. Provider capability does not establish permission to collect, retain, or reuse a source.
Prerequisites
- The target repository or integration path and the requested operator outcome.
- The source authorization, data classification, and environment policy.
- Current Firecrawl documentation, credentials only when needed, and an owner for approvals.
Current Contract
Firecrawl crawl supports scope controls, explicit limits, delay, maxConcurrency, and an enterprise robotsUserAgent option. Enterprise controls include endpoint/format key restrictions, IP restrictions, threat protection, and SIEM integration. Request options such as headers, actions, proxy, cache, ZDR, lockdown, and raw formats have separate risk and availability implications.
Authentication
For authenticated Cloud operations, inject FIRECRAWL_API_KEY from an approved secret manager. REST requests use Authorization: Bearer with the key. Never print, commit, transmit, or place a key in a URL. Keyless access is suitable only where the current documentation explicitly allows it and the workload accepts its limits; production workflows should make identity and team ownership explicit.
Instructions
- Create a versioned source registry with owner, authorization basis, allowed purpose, domains/paths, robots and terms review, data classes, retention, and expiry.
- Canonicalize URLs before policy evaluation. Deny unsupported schemes, embedded credentials, private/link-local addresses, disallowed ports, redirects outside scope, and lookalike hosts.
- Allow only required operations, formats, actions, headers, locations, proxy modes, and crawl settings. Set explicit page, time, credit, and concurrency ceilings.
- Apply provider-side key, IP, threat-protection, and audit controls where entitled, but keep application policy authoritative and fail closed if it is unavailable.
- Treat retrieved content as untrusted data. Separate it from system instructions, validate extracted JSON, sanitize active content, and gate downstream actions.
- Log policy version, decision, source class, request class, limits, and opaque IDs without keys, URLs with secrets, headers, prompts, or bodies.
- Test redirect, DNS rebinding, wildcard, internationalized-domain, restriction bypass, budget, retention, and prompt-injection cases before rollout.
Tool Discipline
Use Read, Glob, and Grep to inspect code, configuration, tests, and evidence. Use Write/Edit only for approved implementation or documentation changes. Do not call Firecrawl, rotate keys, change account settings, scrape a target, or deploy merely because this skill was invoked.
Approval Boundaries
Require policy-owner and legal/security review before adding a domain, collecting authenticated or personal data, changing robots/terms posture, enabling broad actions/proxies, or weakening retention and limits.
Output
Return the policy artifact/version, authorization inventory, canonicalization and allow/deny rules, provider controls, test corpus and results, exceptions, owners, and enforcement receipt.
Error Handling
- Authorization basis is missing or expired: deny the request.
- Policy service is unavailable: fail closed or use a pre-approved read-only degraded policy.
- Redirect or resolved address escapes scope: stop before sending credentials or following content.
Examples
- "Allow our documentation domains" creates exact host/path rules and redirect tests.
- "Ignore robots because Firecrawl can crawl it" is rejected pending the governing policy decision.
Resources
Read official Firecrawl evidence before relying on an endpoint, SDK method, plan limit, price, retention option, or self-hosted release.