Data Streaming

Modern Best Practices: choose the event backbone and stream processor separately, treat schemas and replay as product interfaces, default to event-time processing for stateful analytics, and verify managed-service behavior from primary docs before making vendor-specific recommendations.

Primary sources live in data/sources.json. Refresh time-sensitive claims against official docs before making definitive recommendations about managed services, version-specific features, limits, or pricing.

This skill covers the data platform side of streaming: event backbones, CDC, stateful processing, schema governance, and real-time delivery into lakes, warehouses, search, or serving systems.

When to Use

• Choose between Kafka, Redpanda, Pulsar, Kinesis, or managed Kafka offerings
• Design topic strategy, partitioning, retention, replay, and ordering guarantees
• Build or fix CDC pipelines with Debezium, Flink CDC, or managed database-streaming tools
• Choose between Flink, Kafka Streams, Spark Structured Streaming, or lighter transformation paths
• Define schema registry, compatibility, contract, and tombstone handling rules
• Deliver streams into Iceberg, Hudi, Delta, ClickHouse, warehouses, caches, or search systems
• Review streaming SLOs, lag, checkpointing, reprocessing, and operational failure modes

When NOT to Use

• Lakehouse storage formats, catalogs, or medallion architecture -> Use data-lake-platform
• OLTP schema tuning or transactional query optimization -> Use data-sql-optimization
• Event-driven application architecture, CQRS, or domain event design -> Use software-architecture-design
• BI dashboard automation and Metabase APIs -> Use data-metabase
• Product instrumentation and attribution strategy -> Use marketing-product-analytics

Triage Questions

1. What is the real requirement: operational events, CDC, analytical enrichment, or customer-facing low-latency delivery?
2. What matters most: portability, managed simplicity, geo-replication, cost, or end-to-end latency?
3. Where must ordering hold: globally, per key, or only within a local processing step?
4. What is the replay model: full retention, compacted snapshots, time-bounded backfills, or one-shot delivery?
5. Which guarantees are required: at-most-once, at-least-once, or business-level exactly-once with idempotent sinks?
6. Which downstream systems consume the stream: lakehouse tables, warehouses, search, caches, APIs, or ML features?
7. What is the operational baseline: small team, platform team, managed service, or self-hosted multi-region cluster?

Default Workflow (Use Unless User Overrides)

1. Choose the backbone first with references/platform-selection.md.
2. Define topic, key, retention, replay, and schema strategy before discussing processors. Use assets/topic-contract-template.md.
3. Choose the processing model with references/stream-processing-patterns.md: pass-through, enrich, aggregate, join, dedupe, or CDC normalization.
4. Lock CDC and schema-governance rules with references/cdc-and-schema-governance.md and assets/cdc-rollout-checklist.md.
5. Define delivery and sink behavior: upserts, deletes, late data, watermarking, and reprocessing boundaries.
6. Add SLOs, lag monitoring, checkpoint and savepoint policy, and incident drills with references/operations-and-slos.md.
7. Score tradeoffs explicitly with assets/streaming-platform-scorecard.md when the user asks for the "best" platform.

ASCII Flow

streaming data request
  -> classify need: events, CDC, enrichment, analytics, or low-latency delivery
  -> choose backbone: Kafka-compatible, Pulsar, Kinesis, or managed service
  -> define topics: owner, key, partitions, retention, replay, schema
  -> choose processor: pass-through, enrich, join, aggregate, dedupe, CDC normalize
  -> define sink semantics: idempotency, deletes, late data, reprocessing
  -> add SLOs, lag alerts, checkpoints, DLQ/retry, and incident drills
  -> document tradeoffs and verify managed-service behavior

Default Baseline

• Backbone: Kafka-compatible event log unless a clear managed-service or multi-tenant requirement pushes elsewhere
• Stream processing: Flink for stateful event-time pipelines; Kafka Streams for lighter in-app processing
• CDC: log-based CDC first; avoid trigger-based CDC unless constraints force it
• Contracts: registry-backed schemas for shared or long-lived topics
• Reprocessing: plan for replay before launch; do not treat backfills as exceptional
• Sinks: design sink idempotency explicitly; "exactly-once" claims are incomplete without sink behavior

Backbone Decision Table

WarpStream context (July 2026): Confluent acquired WarpStream (Sept 2024); IBM completed its $11B acquisition of Confluent on March 17, 2026 — WarpStream is now part of IBM's streaming portfolio. Production customers: Grafana Labs, Cursor, Robinhood. AutoMQ production: JD.com (13T msgs/day), Grab, HubSpot.

Exactly-Once Decision Path

Need exactly-once?
  -> Is the sink idempotent or transactional?
       No  -> Add sink-level deduplication key or upsert semantics first
       Yes -> Enable broker/processor exactly-once:
                Kafka: enable.idempotence=true + transactional.id
                Flink: CheckpointingMode.EXACTLY_ONCE + two-phase commit sink
  -> Does the sink support two-phase commit?
       No  -> Business-level idempotency (dedupe key + conditional write)
       Yes -> End-to-end exactly-once boundary confirmed
  -> Document the exact guarantee boundary — broker, processor, AND sink

Kafka Version Quick-Ref (July 2026)

KRaft migration note: All 4.x clusters are KRaft-only. Controller quorum sizing differs from ZooKeeper ensemble sizing — do not map 1:1. Migrate non-production first; use the provided migration tool; verify controller quorum before cutting over.

KIP-932 (Queues for Kafka): GA as of 4.2. Share groups allow multiple consumers to cooperate on the same partition with per-record acknowledgment — enables queue semantics without strict partition-per-consumer assignment. Evaluate for fan-out and task-queue workloads that previously required topic-per-consumer workarounds.

Flink Version Quick-Ref (July 2026)

Upgrade rule: Audit savepoints before upgrading from 1.x — checkpoint format changed. Review connector compatibility. Validate SQL Gateway behavior for complex queries before migrating Table API jobs.

Iceberg Streaming Ingestion Pattern

Standard production stack: Kafka -> Flink (Dynamic Iceberg Sink) -> Iceberg table -> compaction job.

# Flink Dynamic Iceberg Sink — baseline config
checkpoint.interval = 5 min          # drives commit frequency
write.target-file-size-bytes = 128MB  # reduces small-file problem
write.fanout.enabled = true           # unordered writes across partitions
schema-registry.compatibility = FULL_TRANSITIVE
# Compaction: run on cold partitions; skip hot (current) partition

• Every streaming approach produces small files — pair ingestion with a scheduled compaction job.
• With one commit opportunity per checkpoint and all else equal, a 5-minute interval creates one-fifth as many commit opportunities as a 1-minute interval (80% fewer). That arithmetic does not predict file count: partitions, writer parallelism, rollover, traffic shape, and compaction also matter. Measure file count, size distribution, recovery time, and latency on the real workload before tuning.
• Iceberg v3 (ratified): deletion vectors, row lineage (_row_id), variant data, default column values, geometry/geography types, nanosecond timestamps, encryption foundations. Snowflake announced GA on May 7, 2026; Databricks' April 9, 2026 announcement still labels Runtime 18.0+ support Public Preview, not GA. Verify each engine's current read, write, and maintenance support before committing a multi-engine stack to v3 features.

Replay Acceptance Gate

A replay plan is complete only when it answers four separate questions:

1. Source position: Which offsets, timestamps, snapshot, or CDC log position define the replay boundary?
2. State reset: Which processor state and checkpoints are restored, discarded, or rebuilt?
3. Sink behavior: Which idempotency key, upsert, transaction, or dedupe window prevents duplicate business effects?
4. Consumer isolation: How are replay records kept from triggering emails, payments, alerts, or other irreversible side effects twice?

Rehearse one bounded replay through the real sink before launch. Pass only if row/event counts reconcile at source and sink, deletes and late events behave as contracted, consumer lag recovers inside the SLO, and a second replay produces the same business state.

Quick Reference

Operating Principles

1. Ordering Is Scoped, Not Global

• Promise ordering only where the platform can really preserve it, usually per partition and key.
• If the business process needs entity-level sequencing, make the key choice explicit.

2. Schemas Are Contracts

• Shared topics need governed evolution rules, owners, compatibility mode, and deprecation windows.
• Plain JSON is acceptable for prototyping, not for durable shared interfaces.

3. Replay Is A First-Class Operation

• Retention, compaction, checkpoints, and sink idempotency define whether replay is safe.
• Do not ship a pipeline that cannot be re-run after bad code or bad data.

4. "Exactly-Once" Is End-To-End, Not A Checkbox

• Broker or processor guarantees are insufficient if the sink can duplicate writes or mishandle deletes.
• State the exact boundary where deduplication or transactional guarantees end.

5. CDC Needs Delete And Snapshot Strategy

• Decide how snapshots, schema changes, tombstones, and source failover behave before launch.
• Downstream consumers must know whether deletes arrive as tombstones, hard deletes, or soft-delete flags.

Templates

• assets/topic-contract-template.md
• assets/cdc-rollout-checklist.md
• assets/streaming-platform-scorecard.md

Known Traps

• Designing the event backbone around broker features before defining domain ownership, event contracts, and replay expectations.
• Treating topic retention as a substitute for a durable system of record, replay plan, or downstream recovery workflow.
• Claiming exactly-once behavior without specifying the guarantee boundary across broker, processor, sink, and side effects.
• Mixing operational events, analytical CDC, and integration commands into the same topics without independent retention, schema, and consumer-SLA rules.
• Shipping CDC streams without idempotency keys, snapshot semantics, tombstone handling, and late-arrival rules agreed by consumers.
• Scaling partitions, consumer groups, and stateful processors independently and then discovering the keying model breaks ordering or hotspot behavior.

Kafka 4.x KRaft-only

Kafka 4.0+ is KRaft-only — no ZooKeeper path exists. Controller quorum sizing differs from ZooKeeper ensemble sizing; do not map 1:1. Migrate non-production first; validate quorum before cutting over production. Latest stable: 4.3.1 (Jun 25, 2026) — a patch release; upgrade past 4.3.0 promptly if running Kafka Streams, since 4.3.0 shipped a RocksDB native memory leak. KIP-932 share groups (queue semantics) are GA as of 4.2.

Flink 2.x

Flink 2.0 changed checkpoint format and removed deprecated 1.x APIs. Savepoints from 1.x require explicit migration before restoring on 2.x state backends. Audit savepoints and connector compatibility before upgrading. Latest stable: 2.3.0 (Jun 25, 2026), which adds changelog-conversion SQL operators and a redesigned native S3 filesystem. Disaggregated state (remote primary storage) and Materialized Tables are production features since the 2.0.x line.

Common Anti-Patterns

• Using the stream platform as a generic dumping ground for every event rather than curating contracts by domain and use case.
• Putting business-critical enrichment or policy decisions in opaque stream jobs with no replay procedure, lineage, or owner.
• Letting producers evolve schemas opportunistically while expecting consumers to absorb breaking changes.
• Building low-latency pipelines on top of unstable event keys, non-deterministic joins, or external side-effect calls inside hot-path processors.
• Choosing real-time processing because it sounds strategic when batch or micro-batch would meet the product and cost requirements.
• Treating DLQs as the main error-handling strategy instead of fixing classifier logic, validation, backpressure, and recovery paths upstream.

Navigation

• references/platform-selection.md — Load when choosing between Kafka, Redpanda, Pulsar, Kinesis, or managed offerings; includes decision axes and recommendation protocol.
• references/stream-processing-patterns.md — Load when designing topology, windows, joins, deduplication, or delivery semantics.
• references/cdc-and-schema-governance.md — Load when building CDC pipelines, handling deletes/tombstones, or setting schema registry policy.
• references/operations-and-slos.md — Load when defining SLOs, dashboards, incident patterns, or consumer commit/DLQ policy.
• references/engine-table-format-state.md — Load when reasoning about Iceberg v3, Paimon, or Kafka KIP-1150 diskless status.
• references/control-theory-applied.md — Load when designing lag-aware autoscalers, producer flow control, or watermark tuning.
• references/queueing-theory-applied.md — Load when sizing partitions, modeling lag SLOs, or scaling coordinator throughput.
• references/distributed-systems-applied.md — Load when reasoning about ISR quorum, exactly-once via idempotency, leader-epoch fencing, or consumer-group rebalance correctness.
• Formal methods — Load only when a replay, offset/side-effect, rebalance, or fencing protocol has an explicit invariant to check. Return the state/transition boundary and counterexample or bounded safety result; keep delivery guarantees conditional on runtime semantics. Skip routine connector setup.
• data/sources.json — Primary-source URLs for all platforms, processors, CDC tools, and table formats.

Current-Source Policy

• Prefer trust_tier: primary entries in data/sources.json for platform capabilities, service limits, compatibility, and release-sensitive behavior.
• For recommendation questions, verify current managed-service behavior, connector support, quotas, and pricing from official docs instead of relying on frozen comparisons.
• Separate verified facts from judgment calls when comparing Kafka, Redpanda, Pulsar, Kinesis, Flink, and managed offerings.
• If web access is unavailable, say the recommendation is partially unverified.

Related Skills

• data-lake-platform for storage formats, catalogs, serving layers, and lakehouse architecture
• data-analytics-engineering for marts, semantic layers, and metric governance on top of streaming data
• data-sql-optimization for database-side performance and transactional operations
• software-architecture-design for application eventing, CQRS, and domain architecture
• ops-devops-platform for deployment, infra automation, and runbook operations

Fact-Checking

• Use web search or web fetch to verify current external facts, versions, managed-service behavior, quotas, pricing, and release-specific capabilities before final answers.
• Prefer primary sources; include source links and dates for volatile recommendations.

Learnings Loop

When prior decisions or pitfalls are relevant, consult learnings.consolidated.md if present; use learnings.md only for needed history or as the available fallback. Otherwise skip both.

After applying it, if you encountered a pattern worth remembering, a mistake worth preventing, or a domain fact that surprised you, append one dated bullet to learnings.md via agents-skills-feedback-loop/scripts/append_learning.py. Do not modify SKILL.md itself.