Bright Data Governed Collection Architecture
Overview
Design collection as a policy-governed data pipeline, not a direct application call. Separate human approval and configuration from collection workers, then quarantine and validate all results before any trusted consumer or external destination receives them.
Prerequisites
- Approved purpose, targets, fields, products, retention, recipients, and owners
- Current proxy, Browser API, scraper, snapshot, and delivery requirements
- Platform identity, queue, storage, policy, and observability capabilities
Instructions
Step 1: Discover the existing planes
Read design and runtime files and Grep for Bright Data credentials, zones, API endpoints, browser sessions, queues, snapshot storage, and downstream destinations. Mark every trust transition and uncontrolled shortcut.
Step 2: Define the topology
Write or Edit an architecture with these responsibilities:
Approval + workload registry -> admission controller -> bounded collection workers
-> quarantine storage
Quarantine -> schema/policy validation -> approved internal consumer
\-> approved snapshot delivery destination
Telemetry <- redacted events, error classes, usage units, and decision receipts
Keep proxy and Browser API credentials in worker-specific secret bindings and REST API keys in authorized control-plane identities.
Step 3: Apply cross-cutting controls
Specify target and destination allowlists, least privilege, environment isolation, job and byte ceilings, backpressure, idempotency, schema validation, retention, deletion, redaction, audit evidence, and independent emergency stop.
Step 4: Challenge the design
Test policy denial, revoked credentials, 429, provider 5xx, browser timeout, snapshot failure or expiry, schema drift, duplicate delivery, storage exhaustion, and downstream outage. Require an owner and recovery decision for each path.
Tool Discipline
Use Read and Grep for topology discovery and evidence gathering. Use Write and Edit for diagrams, threat models, contracts, tests, and architecture decisions. This skill does not provision Bright Data resources or initiate collection.
Output
- Control-plane and data-plane topology with trust boundaries
- Responsibility, secret, data, and destination matrix
- Failure analysis, rollback path, and decision record
Examples
An admission controller accepts only signed workload manifests. Product-specific workers emit raw results to quarantine, validators release an approved schema, and delivery resolves only named destinations while redacted telemetry preserves an audit trail.
Error Handling
| Failure | Meaning | Response |
|---|---|---|
| Application code can choose arbitrary targets | Admission boundary is missing | Route requests through the workload registry |
| Raw collection reaches analytics directly | Validation boundary is bypassed | Require quarantine and schema promotion |
| Control and collection share one broad API key | Identity blast radius is excessive | Split identities and scopes |