Bright Data Adaptive Capacity Control
Overview
Bright Data does not publish one global proxy request limit. Build separate controllers for proxy traffic and dataset jobs, honor current provider signals, and keep target, cost, and policy ceilings stricter than capacity.
Prerequisites
- A workload profile with target, product, and idempotency classes
- Observed latency, error, and provider-code samples
- Owner-approved concurrency, byte, cost, and wall-time ceilings
Instructions
Step 1: Measure the lane
Read recent receipts and Grep for 429, Retry-After, provider error codes, active dataset jobs, per-IP concentration, and queue depth. Do not infer capacity from successful bursts alone.
Step 2: Separate controllers
Write independent limits for proxy requests, Browser API sessions, async triggers, progress polling, downloads, and delivery workers. One product's success must not raise another product's ceiling.
Step 3: Apply evidence-based backoff
Retry only idempotent transient work. Honor provider guidance, use bounded jitter, cap attempts and elapsed time, and open the circuit on policy, authentication, or data errors.
Step 4: Tune gradually
Edit one limit at a time, run an approved canary, compare throughput, error, and cost, and retain the lower limit unless the evidence and owner approve promotion.
Tool Discipline
Use Read and Grep to analyze redacted metrics and documented codes. Use Write and Edit for controller configuration, tests, and runbooks. This skill does not run live traffic or change Bright Data account limits.
Output
- Per-lane concurrency and queue budgets
- Retry and circuit matrix by documented failure class
- Canary comparison with rollback threshold
Examples
If dataset triggers return a tenant-specific parallel-job error, stop launching new jobs and continue bounded progress polling for owned snapshots. If proxy traffic returns a per-IP 429, reduce pressure and review distribution rather than guessing a universal rate.
Error Handling
| Failure | Meaning | Response |
|---|---|---|
| No provider code is retained | 429 cause is ambiguous | Improve redacted instrumentation before tuning |
| Retries amplify queue depth | Backoff lacks admission control | Open the circuit and drain owned work |
| Throughput improves but cost or error ceiling fails | Optimization violates the budget | Roll back the changed limit |