System Design (High-Level Design) Module 12 – Messaging and stream processing
Stream processing: windows, event time, watermarks
Why stream jobs aggregate by event time, how tumbling, sliding and session windows group events, and how watermarks and allowed lateness handle late data.
What you will learn
- Distinguish event time from processing time, and choose event time for business numbers
- Choose tumbling, sliding or session windows, and estimate the state each one keeps
- Choose a watermark bound from measured lateness, and predict which events arrive late
- Handle late data with allowed lateness, side outputs and upserts
- Diagnose an idle partition that holds the watermark back, and make results exactly once
Before you start
What you’ll learn · Before you start
What you will learn
- Distinguish event time from processing time, and choose event time for business numbers
- Choose tumbling, sliding or session windows, and estimate the state each one keeps
- Choose a watermark bound from measured lateness, and predict which events arrive late
- Handle late data with allowed lateness, side outputs and upserts
- Diagnose an idle partition that holds the watermark back, and make results exactly once
Before you start
On this page
Ten clicks from four phones, counted per minute at the moment the job sees them, give a wrong count for every minute, because two of the phones were offline and sent their clicks late. Stream processing computes results continuously over data that never ends, such as clicks, payments and sensor readings, and its hard part is time. Events arrive late and out of order, so a job must say which time a result is about: event time: The time an event happened, recorded by the device or service that produced it. A stream job that counts by event time puts a late event into the window it belongs to., when the event happened, or processing time: The time on the clock of the machine running a stream job when it handles an event. Counting by it puts a delayed event into the window it arrived in., when the job saw it. Business numbers almost always need event time.
A window cuts the endless stream into pieces that can be counted or summed. A watermark (stream processing): A stream job's running estimate of how far event time has progressed. When it passes the end of a window, the window's result is emitted, and an event for that window that comes afterwards is late. is the job’s running estimate of how far event time has progressed: when it passes the end of a window, the window’s result is emitted, and an event for that window that arrives afterwards is late.
One stream, two clocks
The Apache Flink documentation defines processing time as the clock of the machine running the operation and event time as the time each event occurred on the device that produced it. Here are the ten clicks counted per minute both ways, the same program in 14 languages:
Python 3.14 · main.py
# Ten clicks from four phones: when each happened and when it reached the job, in seconds
# after 12:00:00. Phones C and D were offline for a while and sent their clicks late.
CLICKS = [
(5, 6, "A"),
(20, 21, "B"),
(31, 150, "C"),
(42, 150, "C"),
(55, 56, "A"),
(70, 71, "B"),
(88, 205, "D"),
(101, 102, "A"),
(130, 131, "B"),
(170, 171, "A"),
]
MINUTES = 4
def per_minute(times: list[int]) -> list[int]:
counts = [0] * MINUTES # one-minute tumbling windows, 12:00 to 12:03
for t in times:
counts[t // 60] += 1
return counts
by_event = per_minute([happened for happened, _, _ in CLICKS])
by_arrival = per_minute([arrived for _, arrived, _ in CLICKS])
print("minute by event time by processing time")
for m in range(MINUTES):
print(f"12:0{m} {by_event[m]:>13} {by_arrival[m]:>18}")
late = [phone for happened, arrived, phone in CLICKS if arrived // 60 != happened // 60]
phones = ", ".join(late)
print(
f"{len(late)} of {len(CLICKS)} clicks were counted in a later minute, from phones {phones}"
)
latest = max(arrived - happened for happened, arrived, _ in CLICKS)
print(f"the latest arrived {latest} s after it happened") In Python: t // 60 is the minute of a time in seconds, and a list of four counts stands for the one-minute windows from 12:00 to 12:03.
Recorded with CPython 3.14.8 on macOS 26 arm64. It runs in your browser with Pyodide 314.0.7 (CPython 3.14.2). To run it yourself: python3 main.py
JavaScript Node 24 · main.mjs
// Ten clicks from four phones: when each happened and when it reached the job, in seconds
// after 12:00:00. Phones C and D were offline for a while and sent their clicks late.
const CLICKS = [
[5, 6, "A"],
[20, 21, "B"],
[31, 150, "C"],
[42, 150, "C"],
[55, 56, "A"],
[70, 71, "B"],
[88, 205, "D"],
[101, 102, "A"],
[130, 131, "B"],
[170, 171, "A"],
];
const MINUTES = 4;
function perMinute(times) {
// one-minute tumbling windows, 12:00 to 12:03
const counts = new Array(MINUTES).fill(0);
for (const t of times) {
counts[Math.floor(t / 60)] += 1;
}
return counts;
}
const byEvent = perMinute(CLICKS.map(([happened]) => happened));
const byArrival = perMinute(CLICKS.map(([, arrived]) => arrived));
console.log("minute by event time by processing time");
for (let m = 0; m < MINUTES; m++) {
console.log(
`12:0${m} ${String(byEvent[m]).padStart(13)} ${String(byArrival[m]).padStart(18)}`,
);
}
const late = CLICKS.filter(([h, a]) => Math.floor(a / 60) !== Math.floor(h / 60)).map((c) => c[2]);
const phones = late.join(", ");
console.log(
`${late.length} of ${CLICKS.length} clicks were counted in a later minute, from phones ${phones}`,
);
const latest = Math.max(...CLICKS.map(([happened, arrived]) => arrived - happened));
console.log(`the latest arrived ${latest} s after it happened`); In JavaScript: JavaScript has no integer division, so Math.floor(t / 60) gives the minute, and padStart lines up the columns.
Recorded with Node.js 24.21.0 LTS (V8 13.6) on macOS 26 arm64. It runs in your browser with QuickJS (quickjs-emscripten 0.32.0). To run it yourself: node main.mjs
Java 25 LTS · Main.java
import java.util.ArrayList;
import java.util.List;
import java.util.Locale;
public class Main {
// Ten clicks from four phones: when each happened and when it reached the job, in seconds
// after 12:00:00. Phones C and D were offline for a while and sent their clicks late.
record Click(int happened, int arrived, String phone) {}
static final List<Click> CLICKS =
List.of(
new Click(5, 6, "A"),
new Click(20, 21, "B"),
new Click(31, 150, "C"),
new Click(42, 150, "C"),
new Click(55, 56, "A"),
new Click(70, 71, "B"),
new Click(88, 205, "D"),
new Click(101, 102, "A"),
new Click(130, 131, "B"),
new Click(170, 171, "A"));
static final int MINUTES = 4;
public static void main(String[] args) {
Locale.setDefault(Locale.ROOT);
int[] byEvent = new int[MINUTES]; // one-minute tumbling windows, 12:00 to 12:03
int[] byArrival = new int[MINUTES];
List<String> late = new ArrayList<>();
int latest = 0;
for (Click c : CLICKS) {
byEvent[c.happened() / 60]++;
byArrival[c.arrived() / 60]++;
if (c.arrived() / 60 != c.happened() / 60) {
late.add(c.phone());
}
latest = Math.max(latest, c.arrived() - c.happened());
}
System.out.println("minute by event time by processing time");
for (int m = 0; m < MINUTES; m++) {
System.out.printf("12:0%d %13d %18d%n", m, byEvent[m], byArrival[m]);
}
System.out.printf(
"%d of %d clicks were counted in a later minute, from phones %s%n",
late.size(), CLICKS.size(), String.join(", ", late));
System.out.printf("the latest arrived %d s after it happened%n", latest);
}
} In Java: Integer division drops the remainder, so t / 60 is the minute. printf's %13d pads each count to its column, and main sets Locale.ROOT first, so that every locale prints the same digits.
Recorded with OpenJDK 25.0.4.1 LTS (Eclipse Temurin) on macOS 26 arm64. It runs in your browser with a TeaVM 0.16.0 build in QuickJS (quickjs-emscripten 0.32.0). To run it yourself: java Main.java
C++23 · main.cpp
#include <algorithm>
#include <array>
#include <print>
#include <string>
#include <vector>
// Ten clicks from four phones: when each happened and when it reached the job,
// in seconds after 12:00:00. Phones C and D were offline for a while and sent
// their clicks late.
struct Click {
int happened, arrived;
char phone;
};
constexpr std::array<Click, 10> clicks{{
{5, 6, 'A'},
{20, 21, 'B'},
{31, 150, 'C'},
{42, 150, 'C'},
{55, 56, 'A'},
{70, 71, 'B'},
{88, 205, 'D'},
{101, 102, 'A'},
{130, 131, 'B'},
{170, 171, 'A'},
}};
constexpr int minutes = 4;
int main() {
// one-minute tumbling windows, 12:00 to 12:03
std::array<int, minutes> by_event{};
std::array<int, minutes> by_arrival{};
std::vector<std::string> late;
int latest = 0;
for (const auto &[happened, arrived, phone] : clicks) {
by_event[happened / 60]++;
by_arrival[arrived / 60]++;
if (arrived / 60 != happened / 60) {
late.emplace_back(1, phone);
}
latest = std::max(latest, arrived - happened);
}
std::println("minute by event time by processing time");
for (int m = 0; m < minutes; m++) {
std::println("12:0{} {:>13} {:>18}", m, by_event[m], by_arrival[m]);
}
std::string phones;
for (const auto &phone : late) {
phones += (phones.empty() ? "" : ", ") + phone;
}
std::println("{} of {} clicks were counted in a later minute, from phones {}",
late.size(), clicks.size(), phones);
std::println("the latest arrived {} s after it happened", latest);
} In C++: Structured bindings unpack each click in the range-for loop, and std::println's {:>13} right-aligns the counts.
Recorded with Apple clang 21.0.0 (C++23) and GCC 16.2.0 on macOS 26 arm64. It runs in your browser with Clang in WebAssembly (YoWASP) 22.0.0-git20542-10. To run it yourself: clang++ -std=c++23 -O2 -Wall -Wextra -Werror -pedantic -fsanitize=address,undefined main.cpp -o main && ./main
C# 14 · .NET 10 · app.cs
// Ten clicks from four phones: when each happened and when it reached the job, in seconds
// after 12:00:00. Phones C and D were offline for a while and sent their clicks late.
(int Happened, int Arrived, string Phone)[] clicks =
[
(5, 6, "A"), (20, 21, "B"), (31, 150, "C"), (42, 150, "C"), (55, 56, "A"),
(70, 71, "B"), (88, 205, "D"), (101, 102, "A"), (130, 131, "B"), (170, 171, "A"),
];
const int Minutes = 4;
int[] byEvent = PerMinute(clicks.Select(c => c.Happened));
int[] byArrival = PerMinute(clicks.Select(c => c.Arrived));
Console.WriteLine("minute by event time by processing time");
for (int m = 0; m < Minutes; m++)
{
Console.WriteLine($"12:0{m} {byEvent[m],13} {byArrival[m],18}");
}
var late = clicks.Where(c => c.Arrived / 60 != c.Happened / 60).Select(c => c.Phone).ToList();
string phones = string.Join(", ", late);
Console.WriteLine(
$"{late.Count} of {clicks.Length} clicks were counted in a later minute, from phones {phones}");
int latest = clicks.Max(c => c.Arrived - c.Happened);
Console.WriteLine($"the latest arrived {latest} s after it happened");
static int[] PerMinute(IEnumerable<int> times)
{
var counts = new int[Minutes]; // one-minute tumbling windows, 12:00 to 12:03
foreach (int t in times)
{
counts[t / 60]++;
}
return counts;
} In C#: The clicks are named tuples that LINQ's Select, Where and Max read, and {byEvent[m],13} pads a count to 13 characters.
Recorded with C# 14 on .NET 10.0.12 (SDK 10.0.401) on macOS 26 arm64. It runs in your browser with the .NET runtime for WebAssembly 10.0.12. To run it yourself: dotnet run app.cs
C23 · main.c
#include <stddef.h>
#include <stdio.h>
// Ten clicks from four phones: when each happened and when it reached the job,
// in seconds after 12:00:00. Phones C and D were offline for a while and sent
// their clicks late.
typedef struct {
int happened, arrived;
char phone;
} Click;
static const Click CLICKS[] = {
{5, 6, 'A'}, {20, 21, 'B'}, {31, 150, 'C'}, {42, 150, 'C'},
{55, 56, 'A'}, {70, 71, 'B'}, {88, 205, 'D'}, {101, 102, 'A'},
{130, 131, 'B'}, {170, 171, 'A'},
};
enum { COUNT = sizeof CLICKS / sizeof CLICKS[0], MINUTES = 4 };
int main(void) {
int by_event[MINUTES] = {0}; // one-minute tumbling windows, 12:00 to 12:03
int by_arrival[MINUTES] = {0};
char late[COUNT * 2] = {0};
int late_count = 0, latest = 0;
for (size_t i = 0; i < COUNT; i++) {
const Click *c = &CLICKS[i];
by_event[c->happened / 60]++;
by_arrival[c->arrived / 60]++;
if (c->arrived / 60 != c->happened / 60) {
late[late_count++] = c->phone;
}
if (c->arrived - c->happened > latest) {
latest = c->arrived - c->happened;
}
}
puts("minute by event time by processing time");
for (int m = 0; m < MINUTES; m++) {
printf("12:0%d %13d %18d\n", m, by_event[m], by_arrival[m]);
}
printf("%d of %d clicks were counted in a later minute, from phones",
late_count, (int)COUNT);
for (int i = 0; i < late_count; i++) {
printf("%s %c", i > 0 ? "," : "", late[i]);
}
printf("\nthe latest arrived %d s after it happened\n", latest);
return 0;
} In C: C has no join, so the late phones, single characters, are printed one by one with a comma before every one after the first.
Recorded with Apple clang 21.0.0 (C23) and GCC 16.2.0 on macOS 26 arm64. It runs in your browser with Clang in WebAssembly (YoWASP) 22.0.0-git20542-10. To run it yourself: clang -std=c23 -O2 -Wall -Wextra -Werror -pedantic -fsanitize=address,undefined main.c -o main && ./main
PHP 8.5 · main.php
<?php
declare(strict_types=1);
// Ten clicks from four phones: when each happened and when it reached the job, in seconds
// after 12:00:00. Phones C and D were offline for a while and sent their clicks late.
const CLICKS = [
[5, 6, 'A'], [20, 21, 'B'], [31, 150, 'C'], [42, 150, 'C'], [55, 56, 'A'],
[70, 71, 'B'], [88, 205, 'D'], [101, 102, 'A'], [130, 131, 'B'], [170, 171, 'A'],
];
const MINUTES = 4;
/** @param list<int> $times @return list<int> */
function perMinute(array $times): array
{
$counts = array_fill(0, MINUTES, 0); // one-minute tumbling windows, 12:00 to 12:03
foreach ($times as $t) {
$counts[intdiv($t, 60)]++;
}
return $counts;
}
$byEvent = perMinute(array_column(CLICKS, 0));
$byArrival = perMinute(array_column(CLICKS, 1));
echo "minute by event time by processing time\n";
for ($m = 0; $m < MINUTES; $m++) {
printf("12:0%d %13d %18d\n", $m, $byEvent[$m], $byArrival[$m]);
}
$late = [];
$latest = 0;
foreach (CLICKS as [$happened, $arrived, $phone]) {
if (intdiv($arrived, 60) !== intdiv($happened, 60)) {
$late[] = $phone;
}
$latest = max($latest, $arrived - $happened);
}
printf(
"%d of %d clicks were counted in a later minute, from phones %s\n",
count($late),
count(CLICKS),
implode(', ', $late),
);
printf("the latest arrived %d s after it happened\n", $latest); In PHP: array_column picks the happened and the arrived times out of the rows, and intdiv divides whole numbers.
Recorded with PHP 8.5.11 (CLI) on macOS 26 arm64. It runs in your browser with php-wasm 0.2.0 (PHP 8.5.2). To run it yourself: php main.php
Rust 1.99 · main.rs
// Ten clicks from four phones: when each happened and when it reached the job, in seconds
// after 12:00:00. Phones C and D were offline for a while and sent their clicks late.
const CLICKS: [(usize, usize, &str); 10] = [
(5, 6, "A"),
(20, 21, "B"),
(31, 150, "C"),
(42, 150, "C"),
(55, 56, "A"),
(70, 71, "B"),
(88, 205, "D"),
(101, 102, "A"),
(130, 131, "B"),
(170, 171, "A"),
];
const MINUTES: usize = 4;
fn per_minute(times: impl Iterator<Item = usize>) -> [usize; MINUTES] {
let mut counts = [0; MINUTES]; // one-minute tumbling windows, 12:00 to 12:03
for t in times {
counts[t / 60] += 1;
}
counts
}
fn main() {
let by_event = per_minute(CLICKS.iter().map(|&(happened, _, _)| happened));
let by_arrival = per_minute(CLICKS.iter().map(|&(_, arrived, _)| arrived));
println!("minute by event time by processing time");
for m in 0..MINUTES {
println!("12:0{m} {:>13} {:>18}", by_event[m], by_arrival[m]);
}
let late: Vec<&str> = CLICKS
.iter()
.filter(|&&(happened, arrived, _)| arrived / 60 != happened / 60)
.map(|&(_, _, phone)| phone)
.collect();
println!(
"{} of {} clicks were counted in a later minute, from phones {}",
late.len(),
CLICKS.len(),
late.join(", ")
);
let latest = CLICKS.iter().map(|&(h, a, _)| a - h).max().unwrap_or(0);
println!("the latest arrived {latest} s after it happened");
} In Rust: per_minute takes any iterator of times, so both calls pass a map over the same array without building a list.
Recorded with rustc 1.99.0 (edition 2024) on macOS 26 arm64. It runs in your browser with a WebAssembly build on the WASI runner (browser_wasi_shim 0.4.2). To run it yourself: rustc --edition 2024 -O main.rs && ./main
TypeScript 6.0 · main.ts
// Ten clicks from four phones: when each happened and when it reached the job, in seconds
// after 12:00:00. Phones C and D were offline for a while and sent their clicks late.
const CLICKS: [number, number, string][] = [
[5, 6, "A"],
[20, 21, "B"],
[31, 150, "C"],
[42, 150, "C"],
[55, 56, "A"],
[70, 71, "B"],
[88, 205, "D"],
[101, 102, "A"],
[130, 131, "B"],
[170, 171, "A"],
];
const MINUTES = 4;
function perMinute(times: number[]): number[] {
// one-minute tumbling windows, 12:00 to 12:03
const counts: number[] = new Array(MINUTES).fill(0);
for (const t of times) {
counts[Math.floor(t / 60)] += 1;
}
return counts;
}
const byEvent = perMinute(CLICKS.map(([happened]) => happened));
const byArrival = perMinute(CLICKS.map(([, arrived]) => arrived));
console.log("minute by event time by processing time");
for (let m = 0; m < MINUTES; m++) {
console.log(
`12:0${m} ${String(byEvent[m]).padStart(13)} ${String(byArrival[m]).padStart(18)}`,
);
}
const late = CLICKS.filter(([h, a]) => Math.floor(a / 60) !== Math.floor(h / 60)).map((c) => c[2]);
const phones = late.join(", ");
console.log(
`${late.length} of ${CLICKS.length} clicks were counted in a later minute, from phones ${phones}`,
);
const latest = Math.max(...CLICKS.map(([happened, arrived]) => arrived - happened));
console.log(`the latest arrived ${latest} s after it happened`); In TypeScript: Each click is typed as a tuple of two numbers and a string, so a row of the wrong shape does not compile.
Recorded with TypeScript 6.0.3 (strict), run by Node.js 24.21.0 on macOS 26 arm64. It runs in your browser with QuickJS (quickjs-emscripten 0.32.0) after Sucrase 3.35.1. To run it yourself: npx -p typescript@6.0.3 tsc --noEmit --strict main.ts && node main.ts
Go 1.27 · main.go
package main
import (
"fmt"
"strings"
)
// Click is one of ten clicks from four phones: when it happened and when it reached the job,
// in seconds after 12:00:00. Phones C and D were offline for a while and sent their clicks late.
type Click struct {
Happened, Arrived int
Phone string
}
var clicks = []Click{
{5, 6, "A"}, {20, 21, "B"}, {31, 150, "C"}, {42, 150, "C"}, {55, 56, "A"},
{70, 71, "B"}, {88, 205, "D"}, {101, 102, "A"}, {130, 131, "B"}, {170, 171, "A"},
}
const minutes = 4
func main() {
var byEvent, byArrival [minutes]int // one-minute tumbling windows, 12:00 to 12:03
var late []string
latest := 0
for _, c := range clicks {
byEvent[c.Happened/60]++
byArrival[c.Arrived/60]++
if c.Arrived/60 != c.Happened/60 {
late = append(late, c.Phone)
}
latest = max(latest, c.Arrived-c.Happened)
}
fmt.Println("minute by event time by processing time")
for m := range minutes {
fmt.Printf("12:0%d %13d %18d\n", m, byEvent[m], byArrival[m])
}
fmt.Printf("%d of %d clicks were counted in a later minute, from phones %s\n",
len(late), len(clicks), strings.Join(late, ", "))
fmt.Printf("the latest arrived %d s after it happened\n", latest)
} In Go: The counts are arrays of fixed size, [minutes]int, that start at zero, and range over the constant counts from 0 to 3.
Recorded with Go 1.27.1 on macOS 26 arm64. It runs in your browser with the Go compiler in WebAssembly 1.27.1. To run it yourself: go run main.go
Swift 6.4 · main.swift
// Ten clicks from four phones: when each happened and when it reached the job, in seconds
// after 12:00:00. Phones C and D were offline for a while and sent their clicks late.
let clicks: [(happened: Int, arrived: Int, phone: String)] = [
(5, 6, "A"), (20, 21, "B"), (31, 150, "C"), (42, 150, "C"), (55, 56, "A"),
(70, 71, "B"), (88, 205, "D"), (101, 102, "A"), (130, 131, "B"), (170, 171, "A"),
]
let minutes = 4
func perMinute(_ times: [Int]) -> [Int] {
var counts = [Int](repeating: 0, count: minutes) // one-minute tumbling windows, 12:00 to 12:03
for t in times {
counts[t / 60] += 1
}
return counts
}
func pad(_ n: Int, to width: Int) -> String {
let digits = String(n)
return String(repeating: " ", count: max(0, width - digits.count)) + digits
}
let byEvent = perMinute(clicks.map { $0.happened })
let byArrival = perMinute(clicks.map { $0.arrived })
print("minute by event time by processing time")
for m in 0..<minutes {
print("12:0\(m) \(pad(byEvent[m], to: 13)) \(pad(byArrival[m], to: 18))")
}
let late = clicks.filter { $0.arrived / 60 != $0.happened / 60 }.map { $0.phone }
print(
"\(late.count) of \(clicks.count) clicks were counted in a later minute,"
+ " from phones \(late.joined(separator: ", "))")
let latest = clicks.map { $0.arrived - $0.happened }.max() ?? 0
print("the latest arrived \(latest) s after it happened") In Swift: The clicks are tuples with labels, so $0.happened reads a field, and pad lines up the columns without Foundation.
Recorded with Apple Swift 6.4 on macOS 26 arm64. It runs in your browser with a WebAssembly build on the WASI runner (browser_wasi_shim 0.4.2). To run it yourself: swiftc -O main.swift -o main && ./main
Ruby 4.0 · main.rb
# frozen_string_literal: true
# Ten clicks from four phones: when each happened and when it reached the job, in seconds
# after 12:00:00. Phones C and D were offline for a while and sent their clicks late.
CLICKS = [
[5, 6, 'A'], [20, 21, 'B'], [31, 150, 'C'], [42, 150, 'C'], [55, 56, 'A'],
[70, 71, 'B'], [88, 205, 'D'], [101, 102, 'A'], [130, 131, 'B'], [170, 171, 'A']
].freeze
MINUTES = 4
def per_minute(times)
counts = Array.new(MINUTES, 0) # one-minute tumbling windows, 12:00 to 12:03
times.each { |time| counts[time / 60] += 1 }
counts
end
by_event = per_minute(CLICKS.map(&:first))
by_arrival = per_minute(CLICKS.map { |click| click[1] })
puts 'minute by event time by processing time'
MINUTES.times do |minute|
puts "12:0#{minute} #{by_event[minute].to_s.rjust(13)} " \
"#{by_arrival[minute].to_s.rjust(18)}"
end
late = CLICKS.reject { |happened, arrived, _| arrived / 60 == happened / 60 }.map(&:last)
puts "#{late.size} of #{CLICKS.size} clicks were counted in a later minute, " \
"from phones #{late.join(', ')}"
latest = CLICKS.map { |happened, arrived, _| arrived - happened }.max
puts "the latest arrived #{latest} s after it happened" In Ruby: map(&:first) takes the first field of each row, and reject keeps the clicks whose minute changed on the way.
Recorded with CRuby 4.0.7 on macOS 26 arm64. It runs in your browser with ruby.wasm 2.10.1 (Ruby 4.0.0). To run it yourself: ruby main.rb
Kotlin 2.4 · Main.kt
// Ten clicks from four phones: when each happened and when it reached the job, in seconds
// after 12:00:00. Phones C and D were offline for a while and sent their clicks late.
data class Click(
val happened: Int,
val arrived: Int,
val phone: String,
)
val clicks =
listOf(
Click(5, 6, "A"),
Click(20, 21, "B"),
Click(31, 150, "C"),
Click(42, 150, "C"),
Click(55, 56, "A"),
Click(70, 71, "B"),
Click(88, 205, "D"),
Click(101, 102, "A"),
Click(130, 131, "B"),
Click(170, 171, "A"),
)
const val MINUTES = 4
fun perMinute(times: List<Int>): IntArray {
val counts = IntArray(MINUTES) // one-minute tumbling windows, 12:00 to 12:03
for (t in times) counts[t / 60]++
return counts
}
fun main() {
val byEvent = perMinute(clicks.map { it.happened })
val byArrival = perMinute(clicks.map { it.arrived })
println("minute by event time by processing time")
for (m in 0 until MINUTES) {
val event = byEvent[m].toString().padStart(13)
val arrival = byArrival[m].toString().padStart(18)
println("12:0$m $event $arrival")
}
val late = clicks.filter { it.arrived / 60 != it.happened / 60 }.map { it.phone }
println(
"${late.size} of ${clicks.size} clicks were counted in a later minute, " +
"from phones ${late.joinToString(", ")}",
)
println("the latest arrived ${clicks.maxOf { it.arrived - it.happened }} s after it happened")
} In Kotlin: IntArray(MINUTES) starts at zero, and maxOf with a lambda finds the longest delay in one call.
Recorded with Kotlin 2.4.20 on Java 25.0.4.1 on macOS 26 arm64. It runs in your browser with a WebAssembly build on the WASI runner (browser_wasi_shim 0.4.2). To run it yourself: kotlinc Main.kt -include-runtime -d main.jar && java -jar main.jar
Dart 3.13 · main.dart
/// Ten clicks from four phones: when each happened and when it reached the job, in seconds
/// after 12:00:00. Phones C and D were offline for a while and sent their clicks late.
const clicks = [
(happened: 5, arrived: 6, phone: 'A'),
(happened: 20, arrived: 21, phone: 'B'),
(happened: 31, arrived: 150, phone: 'C'),
(happened: 42, arrived: 150, phone: 'C'),
(happened: 55, arrived: 56, phone: 'A'),
(happened: 70, arrived: 71, phone: 'B'),
(happened: 88, arrived: 205, phone: 'D'),
(happened: 101, arrived: 102, phone: 'A'),
(happened: 130, arrived: 131, phone: 'B'),
(happened: 170, arrived: 171, phone: 'A'),
];
const minutes = 4;
List<int> perMinute(Iterable<int> times) {
// one-minute tumbling windows, 12:00 to 12:03
final counts = List.filled(minutes, 0);
for (final t in times) {
counts[t ~/ 60]++;
}
return counts;
}
void main() {
final byEvent = perMinute(clicks.map((c) => c.happened));
final byArrival = perMinute(clicks.map((c) => c.arrived));
print('minute by event time by processing time');
for (var m = 0; m < minutes; m++) {
print(
'12:0$m ${byEvent[m].toString().padLeft(13)} '
'${byArrival[m].toString().padLeft(18)}',
);
}
final late = [
for (final c in clicks)
if (c.arrived ~/ 60 != c.happened ~/ 60) c.phone,
];
print(
'${late.length} of ${clicks.length} clicks were counted in a later minute, '
'from phones ${late.join(', ')}',
);
final latest = clicks
.map((c) => c.arrived - c.happened)
.reduce((a, b) => a > b ? a : b);
print('the latest arrived $latest s after it happened');
} In Dart: Each click is a record with named fields, and a collection for with an if gathers the late phones.
Recorded with Dart 3.13.5 on macOS 26 arm64. It runs in your browser with a WebAssembly build by dart2wasm 3.13.5. To run it yourself: dart run main.dart
What every language printed
minute by event time by processing time 12:00 5 3 12:01 3 2 12:02 2 4 12:03 0 1 3 of 10 clicks were counted in a later minute, from phones C, C, D the latest arrived 119 s after it happened
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs. Runs on this device, in your browser. The first run downloads JavaScript (about 0.6 MB), which is kept for the next runs. Runs on this device: a JavaScript build of this program (about 66 KB), made when the page was built. The first run also downloads the JavaScript runner (about 0.6 MB), which is kept for the next runs. To change the code, copy it and run it with the command below. Runs on this device: a WebAssembly build of this program (about 152 KB), made when the page was built. Edit runs your changes in the browser: the first edited run downloads the C and C++ compiler (about 29.1 MB), which is kept for the next runs. It is best on a computer. Runs on this device, in your browser. The first run downloads the C and C++ compiler (about 29.1 MB), which is kept for the next runs. It is best on a computer. Runs on this device: a .NET build of this program (about 3 KB), made when the page was built. The first run also downloads .NET (about 3.6 MB), which is kept for the next runs. Edit runs your changes in the browser: the first edited run downloads .NET (about 8.9 MB), which is kept for the next runs. Runs on this device, in your browser. The first run downloads .NET (about 8.9 MB), which is kept for the next runs. Runs on this device: a WebAssembly build of this program (about 8 KB), made when the page was built. Edit runs your changes in the browser: the first edited run downloads the C and C++ compiler (about 29.1 MB), which is kept for the next runs. It is best on a computer. Runs on this device, in your browser. The first run downloads the C and C++ compiler (about 29.1 MB), which is kept for the next runs. It is best on a computer. Runs on this device, in your browser. The first run downloads PHP (about 4.3 MB), which is kept for the next runs. Runs on this device: a WebAssembly build of this program (about 20 KB), made when the page was built. To change the code, copy it and run it with the command below. Runs on this device, in your browser. The first run downloads the JavaScript runner (about 0.9 MB), which is kept for the next runs. Runs on this device: a WebAssembly build of this program (about 719 KB), made when the page was built. Edit runs your changes in the browser: the first edited run downloads the Go compiler (about 18.7 MB), which is kept for the next runs. Runs on this device, in your browser. The first run downloads the Go compiler (about 18.7 MB), which is kept for the next runs. Runs on this device: a WebAssembly build of this program (about 1.7 MB), made when the page was built. To change the code, copy it and run it with the command below. Runs on this device, in your browser. The first run downloads Ruby (about 10.2 MB), which is kept for the next runs. Runs on this device: a WebAssembly build of this program (about 182 KB), made when the page was built. To change the code, copy it and run it with the command below. Runs on this device: a WebAssembly build of this program (about 14 KB), made when the page was built. To change the code, copy it and run it with the command below.Your run, in this browser
By processing time every minute is wrong. The clicks phone C made at 12:00 were counted at 12:02, so 12:00 shows 3 clicks instead of 5 and 12:02 shows 4 instead of 2, and a fourth minute appears that had no clicks at all. A dashboard built on processing time shows network delays as changes in what users did, and a replay of the same clicks gives other numbers. Counting by event time gives the right answer, at a price: the job must wait, because a click from 12:00 may still be on its way.
The figure sorts nine other clicks into the three kinds of window:
One set of clicks, three kinds of window
Text description of the diagram
A time axis of event time runs from 0 to 6 minutes, and each click is a short mark above it. Nine clicks happen at 10, 25, 70, 100 and 130 seconds, then at 250, 260 and 275 seconds, and finally at 340 seconds.
- Tumbling windows of 2 minutes cut the axis into 0 to 2, 2 to 4 and 4 to 6 minutes, with no overlap. They hold 4, 1 and 4 clicks; every click is in exactly one window.
- Sliding windows are 2 minutes long and one starts every minute, so they overlap: 0 to 2 holds 4 clicks, 1 to 3 holds 3, 2 to 4 holds 1, 3 to 5 holds 3 and 4 to 6 holds 4. Each click falls in two windows, counting the windows that start before 0 or end after 6 minutes, which the drawing leaves out.
- Session windows end after a minute without a click. The first five clicks are never more than a minute apart, so they form one session from 10 to 130 seconds, which closes a minute after its last click. The next three form a second session, and the click at 340 seconds, 65 seconds after the one before it, starts a third, which runs past the end of the drawing.
The three window types answer different questions. tumbling window: A window of fixed length that follows the previous one without overlap, such as one per minute, so that every event belongs to exactly one window. windows are equal, back to back and never overlap: per-minute counts. sliding window: A window of fixed length that starts at a fixed interval shorter than its length, so that windows overlap and an event belongs to several, as in a 10-minute average updated every minute. windows have a fixed length and start at a fixed interval, so they overlap: a 10-minute moving average updated every minute. session window: A window per key that groups events until a gap with no events, so that each result covers one burst of activity, such as a visit. windows group a key’s events until a gap with no events: one result per visit.
The rest of the lesson covers when a window is complete and what to do about stragglers:
- Bounded out-of-orderness: the watermark trails the largest event time seen by a fixed bound.
- allowed lateness: How long a stream window keeps its state after the watermark has passed its end, so that late events can still update its result.: a window keeps its state for a while after it closes, so late events can update it.
- side output: A separate stream that receives the events too late for any window, so that they can be counted, stored or reconciled instead of disappearing.: events too late for any window go to a separate stream instead of disappearing.
- idle input: An input of a stream operator, such as a partition, that has stopped sending, and so holds back every watermark that depends on it.: a partition that stops sending holds back every watermark that depends on it; watermark alignment handles the opposite, a source far ahead of the rest.
- checkpoint (stream processing): A consistent snapshot of a stream job's state and of the positions it has read its inputs up to, so that after a failure the job can restart from it.: consistent snapshots of state and input positions; with a transactional or idempotent sink, they make results exactly once after a failure.
Check the two clocks on clicks of your own.
Checkpoint
Check: Event time, processing time and windows
Three windows on one stream, run Pro: opens with a Pro, Premium or Ultimate pass
In this part:
- Watermarks: completeness against latency, measured
- Late data: drop it, divert it or update the result
- Parallel inputs: the slowest partition sets the clock
- Exactly-once results need checkpoints and a safe sink
- Practise: tumbling windows behind a watermark
- Count tumbling windows behind a watermark
- Interview questions
- What is the difference between event time and processing time, and which should a per-minute sales dashboard use?
- A dashboard counts app events per minute, and the numbers for the past hour keep changing for 20 minutes because phones send events late. Product wants fast numbers that are also right. How do you design the pipeline?
- A streaming job reads a topic with 48 partitions. Its windowed results stopped coming out an hour ago, although events keep arriving and the job looks healthy. What might be wrong, and how do you find and fix it?
- A streaming job counts payments per merchant per minute and writes the counts to a database. After a crash and restart, some minutes show counts that are too high. Why, and how do you make the results exactly once?
- Key takeaways
3 diagrams · 3 runnable examples (JavaScript, Python) · 1 exercise with sample tests · 3 interview answers · about 46 minutes
Opening this part with your pass…
This part opens with any MySmartCoPilot pass.
- Pro pass · 1 week₹79$2.99
- 30 days₹179$5.99
- 1 year₹1,699$45
One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.
Passes can’t be bought right now, so this part can’t be opened at the moment. The free part above stands on its own. If you have a pass, enter your licence key.
It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).
Terms in this lesson
- allowed lateness
- How long a stream window keeps its state after the watermark has passed its end, so that late events can still update its result.
- checkpoint (stream processing)
- A consistent snapshot of a stream job's state and of the positions it has read its inputs up to, so that after a failure the job can restart from it.
- event time
- The time an event happened, recorded by the device or service that produced it. A stream job that counts by event time puts a late event into the window it belongs to.
- idle input
- An input of a stream operator, such as a partition, that has stopped sending, and so holds back every watermark that depends on it.
- processing time
- The time on the clock of the machine running a stream job when it handles an event. Counting by it puts a delayed event into the window it arrived in.
- session window
- A window per key that groups events until a gap with no events, so that each result covers one burst of activity, such as a visit.
- side output
- A separate stream that receives the events too late for any window, so that they can be counted, stored or reconciled instead of disappearing.
- sliding window
- A window of fixed length that starts at a fixed interval shorter than its length, so that windows overlap and an event belongs to several, as in a 10-minute average updated every minute.
- tumbling window
- A window of fixed length that follows the previous one without overlap, such as one per minute, so that every event belongs to exactly one window.
- watermark (stream processing)
- A stream job's running estimate of how far event time has progressed. When it passes the end of a window, the window's result is emitted, and an event for that window that comes afterwards is late.
References
- Apache Flink 2.3 documentation: Timely stream processing (The Apache Software Foundation)
- Apache Flink 2.3 documentation: Windows (The Apache Software Foundation)
- Apache Flink 2.3 documentation: Generating watermarks (The Apache Software Foundation)
- Apache Flink 2.3 documentation: Stateful stream processing (The Apache Software Foundation)
- The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing (Proceedings of the VLDB Endowment (Akidau and others))
- Apache Kafka 4.3 documentation: Design (The Apache Software Foundation)
Related tools
Report a problem with this lesson
Kept only in this browser. Your Learn progress