Bright Data Worker Deployment
Overview
Deploy collection as bounded workers instead of one privileged process. Separate synchronous proxy or Browser API requests from scraper triggers, snapshot polling, downloads, and destination delivery so each stage has its own authority, queue, resource ceiling, and rollback.
Prerequisites
- An approved workload manifest naming targets, fields, products, and destinations
- Managed secret references and environment-specific Bright Data resources
- Queue, object-storage, quarantine, and observability facilities
Instructions
Step 1: Inventory runtime responsibilities
Read the application and Grep for proxy requests, Browser API sessions, /datasets/v3/trigger, progress checks, snapshot downloads, and delivery calls. Assign each responsibility to the smallest worker role.
Step 2: Define admission and isolation
Write or Edit deployment manifests with separate queues and identities for request, browser, trigger, poll, download, validate, and delivery workers. Add target allowlists, byte and record ceilings, timeouts, concurrency bounds, and dead-letter routing.
Step 3: Protect data movement
Stream large snapshots to bounded storage, validate schema before downstream use, quarantine unexpected content, and require an approved destination before invoking /datasets/v3/deliver/SNAPSHOT_ID.
Step 4: Make rollout reversible
Specify health signals, queue-depth and error-class alerts, canary capacity, drain behavior, rollback version, secret revocation, and replay rules. Produce manifests and a deployment plan; do not initiate production traffic from this skill.
Tool Discipline
Use Read and Grep for topology and configuration discovery. Use Write and Edit for deployment manifests, queue contracts, policies, tests, and runbooks. This skill does not deploy, rotate credentials, create zones, or send provider requests.
Output
- Worker and queue topology with explicit trust boundaries
- Resource, data, and destination controls per stage
- Canary, drain, rollback, quarantine, and replay plan
Examples
A trigger worker creates one approved scraper job, a poller records state transitions, a downloader streams the completed snapshot to quarantine, a validator enforces the schema, and a delivery worker releases only accepted records to an approved destination.
Error Handling
| Failure | Meaning | Response |
|---|---|---|
| One worker holds every credential | Blast radius is too large | Split identities and secret scopes |
| Download memory grows with snapshot size | Transfer is unbounded | Stream with byte ceilings and backpressure |
| Delivery destination is runtime input | Exfiltration boundary is open | Resolve destinations from an approved manifest |