Bright Data Evidence-Based Performance Tuning
Overview
Improve the slowest measured phase while preserving authorization, correctness, and cost ceilings. Choose the Bright Data product that matches the interaction, separate provider time from local queue and processing time, change one variable, and retain rollback evidence.
Prerequisites
- Approved targets and representative request or snapshot fixtures
- Baseline traces for queue, connect, provider, transfer, parse, and downstream phases
- Explicit latency, success, concurrency, byte, and cost thresholds
Instructions
Step 1: Build the baseline
Read traces and Grep for serialized work, unbounded concurrency, full-body buffering, repeated browser startup, hot polling, and retry amplification. Segment by product, target class, response size, and error class.
Step 2: Verify product fit
Use proxy requests for simple HTTP collection, Browser API when a browser session is actually required, and asynchronous scraper snapshots for batch workloads. Do not hide a product mismatch with more concurrency.
Step 3: Run one-variable experiments
Write or Edit a canary plan that changes only batch size, worker concurrency, connection reuse, browser-session reuse, polling cadence, streaming boundary, or downstream parallelism. Keep admission, byte, and cost ceilings fixed.
Step 4: Decide from evidence
Compare median and tail latency, success, 429, provider errors, bytes, queue time, and unit cost. Retain changes only when the target metric improves without violating safety, correctness, or budget constraints.
Tool Discipline
Use Read and Grep for trace and implementation analysis. Use Write and Edit for benchmarks, fixtures, canary configuration, and the decision record. This skill does not generate production load or alter live Bright Data resources.
Output
- Phase-level baseline and identified bottleneck
- One-variable experiment matrix with ceilings
- Keep or rollback decision supported by metrics
Examples
A snapshot workload spends most time parsing after download. Streamed NDJSON parsing lowers memory and tail latency in a fixed-size canary while provider concurrency and collection scope remain unchanged.
Error Handling
| Failure | Meaning | Response |
|---|---|---|
| Baseline mixes unlike products | Comparison is invalid | Segment proxy, Browser API, and snapshot paths |
| Throughput rises with more 429 responses | Concurrency exceeds an effective boundary | Back off and lower admission |
| Faster output loses records | Optimization broke correctness | Roll back and add integrity assertions |