Your country

Tools that support it use your country for local currency, number formats, units and paper size. Your choice is saved only in this browser.

Type a name or a two-letter code. Use the up and down arrow keys to move through the countries, Enter to choose one and Escape to close.

System Design (High-Level Design)  Module 12 – Messaging and stream processing

Stream processing: windows, event time, watermarks

Why stream jobs aggregate by event time, how tumbling, sliding and session windows group events, and how watermarks and allowed lateness handle late data.

  • Advanced
  • 20 minutes
  • Examples run with Python 3.14.8, Pyodide 314.0.7, Node.js 24.21.0 and quickjs 0.32.0
  • By MySmartCoPilot
Free intro, Pro: 5 of 20 minutes free
Free: One stream, two clocks. With a pass: the other 8 sections and 3 interview answers, about 15 minutes.

What you will learn

  • Distinguish event time from processing time, and choose event time for business numbers
  • Choose tumbling, sliding or session windows, and estimate the state each one keeps
  • Choose a watermark bound from measured lateness, and predict which events arrive late
  • Handle late data with allowed lateness, side outputs and upserts
  • Diagnose an idle partition that holds the watermark back, and make results exactly once

Before you start

What you’ll learn · Before you start

What you will learn

  • Distinguish event time from processing time, and choose event time for business numbers
  • Choose tumbling, sliding or session windows, and estimate the state each one keeps
  • Choose a watermark bound from measured lateness, and predict which events arrive late
  • Handle late data with allowed lateness, side outputs and upserts
  • Diagnose an idle partition that holds the watermark back, and make results exactly once

Before you start

On this page

Ten clicks from four phones, counted per minute at the moment the job sees them, give a wrong count for every minute, because two of the phones were offline and sent their clicks late. Stream processing computes results continuously over data that never ends, such as clicks, payments and sensor readings, and its hard part is time. Events arrive late and out of order, so a job must say which time a result is about: event time: The time an event happened, recorded by the device or service that produced it. A stream job that counts by event time puts a late event into the window it belongs to., when the event happened, or processing time: The time on the clock of the machine running a stream job when it handles an event. Counting by it puts a delayed event into the window it arrived in., when the job saw it. Business numbers almost always need event time.

A window cuts the endless stream into pieces that can be counted or summed. A watermark (stream processing): A stream job's running estimate of how far event time has progressed. When it passes the end of a window, the window's result is emitted, and an event for that window that comes afterwards is late. is the job’s running estimate of how far event time has progressed: when it passes the end of a window, the window’s result is emitted, and an event for that window that arrives afterwards is late.

One stream, two clocks

The Apache Flink documentation defines processing time as the clock of the machine running the operation and event time as the time each event occurred on the device that produced it. Here are the ten clicks counted per minute both ways, the same program in 14 languages:

One program · 14 languages Ten clicks counted by event time and by processing time

Python 3.14 · main.py

# Ten clicks from four phones: when each happened and when it reached the job, in seconds
# after 12:00:00. Phones C and D were offline for a while and sent their clicks late.
CLICKS = [
    (5, 6, "A"),
    (20, 21, "B"),
    (31, 150, "C"),
    (42, 150, "C"),
    (55, 56, "A"),
    (70, 71, "B"),
    (88, 205, "D"),
    (101, 102, "A"),
    (130, 131, "B"),
    (170, 171, "A"),
]
MINUTES = 4


def per_minute(times: list[int]) -> list[int]:
    counts = [0] * MINUTES  # one-minute tumbling windows, 12:00 to 12:03
    for t in times:
        counts[t // 60] += 1
    return counts


by_event = per_minute([happened for happened, _, _ in CLICKS])
by_arrival = per_minute([arrived for _, arrived, _ in CLICKS])
print("minute    by event time   by processing time")
for m in range(MINUTES):
    print(f"12:0{m}     {by_event[m]:>13}   {by_arrival[m]:>18}")
late = [phone for happened, arrived, phone in CLICKS if arrived // 60 != happened // 60]
phones = ", ".join(late)
print(
    f"{len(late)} of {len(CLICKS)} clicks were counted in a later minute, from phones {phones}"
)
latest = max(arrived - happened for happened, arrived, _ in CLICKS)
print(f"the latest arrived {latest} s after it happened")

In Python: t // 60 is the minute of a time in seconds, and a list of four counts stands for the one-minute windows from 12:00 to 12:03.

Recorded with CPython 3.14.8 on macOS 26 arm64. It runs in your browser with Pyodide 314.0.7 (CPython 3.14.2). To run it yourself: python3 main.py

JavaScript Node 24 · main.mjs

// Ten clicks from four phones: when each happened and when it reached the job, in seconds
// after 12:00:00. Phones C and D were offline for a while and sent their clicks late.
const CLICKS = [
  [5, 6, "A"],
  [20, 21, "B"],
  [31, 150, "C"],
  [42, 150, "C"],
  [55, 56, "A"],
  [70, 71, "B"],
  [88, 205, "D"],
  [101, 102, "A"],
  [130, 131, "B"],
  [170, 171, "A"],
];
const MINUTES = 4;

function perMinute(times) {
  // one-minute tumbling windows, 12:00 to 12:03
  const counts = new Array(MINUTES).fill(0);
  for (const t of times) {
    counts[Math.floor(t / 60)] += 1;
  }
  return counts;
}

const byEvent = perMinute(CLICKS.map(([happened]) => happened));
const byArrival = perMinute(CLICKS.map(([, arrived]) => arrived));
console.log("minute    by event time   by processing time");
for (let m = 0; m < MINUTES; m++) {
  console.log(
    `12:0${m}     ${String(byEvent[m]).padStart(13)}   ${String(byArrival[m]).padStart(18)}`,
  );
}
const late = CLICKS.filter(([h, a]) => Math.floor(a / 60) !== Math.floor(h / 60)).map((c) => c[2]);
const phones = late.join(", ");
console.log(
  `${late.length} of ${CLICKS.length} clicks were counted in a later minute, from phones ${phones}`,
);
const latest = Math.max(...CLICKS.map(([happened, arrived]) => arrived - happened));
console.log(`the latest arrived ${latest} s after it happened`);

In JavaScript: JavaScript has no integer division, so Math.floor(t / 60) gives the minute, and padStart lines up the columns.

Recorded with Node.js 24.21.0 LTS (V8 13.6) on macOS 26 arm64. It runs in your browser with QuickJS (quickjs-emscripten 0.32.0). To run it yourself: node main.mjs

Java 25 LTS · Main.java

import java.util.ArrayList;
import java.util.List;
import java.util.Locale;

public class Main {
  // Ten clicks from four phones: when each happened and when it reached the job, in seconds
  // after 12:00:00. Phones C and D were offline for a while and sent their clicks late.
  record Click(int happened, int arrived, String phone) {}

  static final List<Click> CLICKS =
      List.of(
          new Click(5, 6, "A"),
          new Click(20, 21, "B"),
          new Click(31, 150, "C"),
          new Click(42, 150, "C"),
          new Click(55, 56, "A"),
          new Click(70, 71, "B"),
          new Click(88, 205, "D"),
          new Click(101, 102, "A"),
          new Click(130, 131, "B"),
          new Click(170, 171, "A"));
  static final int MINUTES = 4;

  public static void main(String[] args) {
    Locale.setDefault(Locale.ROOT);
    int[] byEvent = new int[MINUTES]; // one-minute tumbling windows, 12:00 to 12:03
    int[] byArrival = new int[MINUTES];
    List<String> late = new ArrayList<>();
    int latest = 0;
    for (Click c : CLICKS) {
      byEvent[c.happened() / 60]++;
      byArrival[c.arrived() / 60]++;
      if (c.arrived() / 60 != c.happened() / 60) {
        late.add(c.phone());
      }
      latest = Math.max(latest, c.arrived() - c.happened());
    }
    System.out.println("minute    by event time   by processing time");
    for (int m = 0; m < MINUTES; m++) {
      System.out.printf("12:0%d     %13d   %18d%n", m, byEvent[m], byArrival[m]);
    }
    System.out.printf(
        "%d of %d clicks were counted in a later minute, from phones %s%n",
        late.size(), CLICKS.size(), String.join(", ", late));
    System.out.printf("the latest arrived %d s after it happened%n", latest);
  }
}

In Java: Integer division drops the remainder, so t / 60 is the minute. printf's %13d pads each count to its column, and main sets Locale.ROOT first, so that every locale prints the same digits.

Recorded with OpenJDK 25.0.4.1 LTS (Eclipse Temurin) on macOS 26 arm64. It runs in your browser with a TeaVM 0.16.0 build in QuickJS (quickjs-emscripten 0.32.0). To run it yourself: java Main.java

C++23 · main.cpp

#include <algorithm>
#include <array>
#include <print>
#include <string>
#include <vector>

// Ten clicks from four phones: when each happened and when it reached the job,
// in seconds after 12:00:00. Phones C and D were offline for a while and sent
// their clicks late.
struct Click {
  int happened, arrived;
  char phone;
};

constexpr std::array<Click, 10> clicks{{
    {5, 6, 'A'},
    {20, 21, 'B'},
    {31, 150, 'C'},
    {42, 150, 'C'},
    {55, 56, 'A'},
    {70, 71, 'B'},
    {88, 205, 'D'},
    {101, 102, 'A'},
    {130, 131, 'B'},
    {170, 171, 'A'},
}};
constexpr int minutes = 4;

int main() {
  // one-minute tumbling windows, 12:00 to 12:03
  std::array<int, minutes> by_event{};
  std::array<int, minutes> by_arrival{};
  std::vector<std::string> late;
  int latest = 0;
  for (const auto &[happened, arrived, phone] : clicks) {
    by_event[happened / 60]++;
    by_arrival[arrived / 60]++;
    if (arrived / 60 != happened / 60) {
      late.emplace_back(1, phone);
    }
    latest = std::max(latest, arrived - happened);
  }
  std::println("minute    by event time   by processing time");
  for (int m = 0; m < minutes; m++) {
    std::println("12:0{}     {:>13}   {:>18}", m, by_event[m], by_arrival[m]);
  }
  std::string phones;
  for (const auto &phone : late) {
    phones += (phones.empty() ? "" : ", ") + phone;
  }
  std::println("{} of {} clicks were counted in a later minute, from phones {}",
               late.size(), clicks.size(), phones);
  std::println("the latest arrived {} s after it happened", latest);
}

In C++: Structured bindings unpack each click in the range-for loop, and std::println's {:>13} right-aligns the counts.

Recorded with Apple clang 21.0.0 (C++23) and GCC 16.2.0 on macOS 26 arm64. It runs in your browser with Clang in WebAssembly (YoWASP) 22.0.0-git20542-10. To run it yourself: clang++ -std=c++23 -O2 -Wall -Wextra -Werror -pedantic -fsanitize=address,undefined main.cpp -o main && ./main

C# 14 · .NET 10 · app.cs

// Ten clicks from four phones: when each happened and when it reached the job, in seconds
// after 12:00:00. Phones C and D were offline for a while and sent their clicks late.
(int Happened, int Arrived, string Phone)[] clicks =
[
    (5, 6, "A"), (20, 21, "B"), (31, 150, "C"), (42, 150, "C"), (55, 56, "A"),
    (70, 71, "B"), (88, 205, "D"), (101, 102, "A"), (130, 131, "B"), (170, 171, "A"),
];
const int Minutes = 4;

int[] byEvent = PerMinute(clicks.Select(c => c.Happened));
int[] byArrival = PerMinute(clicks.Select(c => c.Arrived));
Console.WriteLine("minute    by event time   by processing time");
for (int m = 0; m < Minutes; m++)
{
    Console.WriteLine($"12:0{m}     {byEvent[m],13}   {byArrival[m],18}");
}
var late = clicks.Where(c => c.Arrived / 60 != c.Happened / 60).Select(c => c.Phone).ToList();
string phones = string.Join(", ", late);
Console.WriteLine(
    $"{late.Count} of {clicks.Length} clicks were counted in a later minute, from phones {phones}");
int latest = clicks.Max(c => c.Arrived - c.Happened);
Console.WriteLine($"the latest arrived {latest} s after it happened");

static int[] PerMinute(IEnumerable<int> times)
{
    var counts = new int[Minutes]; // one-minute tumbling windows, 12:00 to 12:03
    foreach (int t in times)
    {
        counts[t / 60]++;
    }
    return counts;
}

In C#: The clicks are named tuples that LINQ's Select, Where and Max read, and {byEvent[m],13} pads a count to 13 characters.

Recorded with C# 14 on .NET 10.0.12 (SDK 10.0.401) on macOS 26 arm64. It runs in your browser with the .NET runtime for WebAssembly 10.0.12. To run it yourself: dotnet run app.cs

C23 · main.c

#include <stddef.h>
#include <stdio.h>

// Ten clicks from four phones: when each happened and when it reached the job,
// in seconds after 12:00:00. Phones C and D were offline for a while and sent
// their clicks late.
typedef struct {
  int happened, arrived;
  char phone;
} Click;

static const Click CLICKS[] = {
    {5, 6, 'A'},     {20, 21, 'B'},   {31, 150, 'C'}, {42, 150, 'C'},
    {55, 56, 'A'},   {70, 71, 'B'},   {88, 205, 'D'}, {101, 102, 'A'},
    {130, 131, 'B'}, {170, 171, 'A'},
};
enum { COUNT = sizeof CLICKS / sizeof CLICKS[0], MINUTES = 4 };

int main(void) {
  int by_event[MINUTES] = {0}; // one-minute tumbling windows, 12:00 to 12:03
  int by_arrival[MINUTES] = {0};
  char late[COUNT * 2] = {0};
  int late_count = 0, latest = 0;
  for (size_t i = 0; i < COUNT; i++) {
    const Click *c = &CLICKS[i];
    by_event[c->happened / 60]++;
    by_arrival[c->arrived / 60]++;
    if (c->arrived / 60 != c->happened / 60) {
      late[late_count++] = c->phone;
    }
    if (c->arrived - c->happened > latest) {
      latest = c->arrived - c->happened;
    }
  }
  puts("minute    by event time   by processing time");
  for (int m = 0; m < MINUTES; m++) {
    printf("12:0%d     %13d   %18d\n", m, by_event[m], by_arrival[m]);
  }
  printf("%d of %d clicks were counted in a later minute, from phones",
         late_count, (int)COUNT);
  for (int i = 0; i < late_count; i++) {
    printf("%s %c", i > 0 ? "," : "", late[i]);
  }
  printf("\nthe latest arrived %d s after it happened\n", latest);
  return 0;
}

In C: C has no join, so the late phones, single characters, are printed one by one with a comma before every one after the first.

Recorded with Apple clang 21.0.0 (C23) and GCC 16.2.0 on macOS 26 arm64. It runs in your browser with Clang in WebAssembly (YoWASP) 22.0.0-git20542-10. To run it yourself: clang -std=c23 -O2 -Wall -Wextra -Werror -pedantic -fsanitize=address,undefined main.c -o main && ./main

PHP 8.5 · main.php

<?php

declare(strict_types=1);

// Ten clicks from four phones: when each happened and when it reached the job, in seconds
// after 12:00:00. Phones C and D were offline for a while and sent their clicks late.
const CLICKS = [
    [5, 6, 'A'], [20, 21, 'B'], [31, 150, 'C'], [42, 150, 'C'], [55, 56, 'A'],
    [70, 71, 'B'], [88, 205, 'D'], [101, 102, 'A'], [130, 131, 'B'], [170, 171, 'A'],
];
const MINUTES = 4;

/** @param list<int> $times @return list<int> */
function perMinute(array $times): array
{
    $counts = array_fill(0, MINUTES, 0); // one-minute tumbling windows, 12:00 to 12:03
    foreach ($times as $t) {
        $counts[intdiv($t, 60)]++;
    }
    return $counts;
}

$byEvent = perMinute(array_column(CLICKS, 0));
$byArrival = perMinute(array_column(CLICKS, 1));
echo "minute    by event time   by processing time\n";
for ($m = 0; $m < MINUTES; $m++) {
    printf("12:0%d     %13d   %18d\n", $m, $byEvent[$m], $byArrival[$m]);
}
$late = [];
$latest = 0;
foreach (CLICKS as [$happened, $arrived, $phone]) {
    if (intdiv($arrived, 60) !== intdiv($happened, 60)) {
        $late[] = $phone;
    }
    $latest = max($latest, $arrived - $happened);
}
printf(
    "%d of %d clicks were counted in a later minute, from phones %s\n",
    count($late),
    count(CLICKS),
    implode(', ', $late),
);
printf("the latest arrived %d s after it happened\n", $latest);

In PHP: array_column picks the happened and the arrived times out of the rows, and intdiv divides whole numbers.

Recorded with PHP 8.5.11 (CLI) on macOS 26 arm64. It runs in your browser with php-wasm 0.2.0 (PHP 8.5.2). To run it yourself: php main.php

Rust 1.99 · main.rs

// Ten clicks from four phones: when each happened and when it reached the job, in seconds
// after 12:00:00. Phones C and D were offline for a while and sent their clicks late.
const CLICKS: [(usize, usize, &str); 10] = [
    (5, 6, "A"),
    (20, 21, "B"),
    (31, 150, "C"),
    (42, 150, "C"),
    (55, 56, "A"),
    (70, 71, "B"),
    (88, 205, "D"),
    (101, 102, "A"),
    (130, 131, "B"),
    (170, 171, "A"),
];
const MINUTES: usize = 4;

fn per_minute(times: impl Iterator<Item = usize>) -> [usize; MINUTES] {
    let mut counts = [0; MINUTES]; // one-minute tumbling windows, 12:00 to 12:03
    for t in times {
        counts[t / 60] += 1;
    }
    counts
}

fn main() {
    let by_event = per_minute(CLICKS.iter().map(|&(happened, _, _)| happened));
    let by_arrival = per_minute(CLICKS.iter().map(|&(_, arrived, _)| arrived));
    println!("minute    by event time   by processing time");
    for m in 0..MINUTES {
        println!("12:0{m}     {:>13}   {:>18}", by_event[m], by_arrival[m]);
    }
    let late: Vec<&str> = CLICKS
        .iter()
        .filter(|&&(happened, arrived, _)| arrived / 60 != happened / 60)
        .map(|&(_, _, phone)| phone)
        .collect();
    println!(
        "{} of {} clicks were counted in a later minute, from phones {}",
        late.len(),
        CLICKS.len(),
        late.join(", ")
    );
    let latest = CLICKS.iter().map(|&(h, a, _)| a - h).max().unwrap_or(0);
    println!("the latest arrived {latest} s after it happened");
}

In Rust: per_minute takes any iterator of times, so both calls pass a map over the same array without building a list.

Recorded with rustc 1.99.0 (edition 2024) on macOS 26 arm64. It runs in your browser with a WebAssembly build on the WASI runner (browser_wasi_shim 0.4.2). To run it yourself: rustc --edition 2024 -O main.rs && ./main

TypeScript 6.0 · main.ts

// Ten clicks from four phones: when each happened and when it reached the job, in seconds
// after 12:00:00. Phones C and D were offline for a while and sent their clicks late.
const CLICKS: [number, number, string][] = [
  [5, 6, "A"],
  [20, 21, "B"],
  [31, 150, "C"],
  [42, 150, "C"],
  [55, 56, "A"],
  [70, 71, "B"],
  [88, 205, "D"],
  [101, 102, "A"],
  [130, 131, "B"],
  [170, 171, "A"],
];
const MINUTES = 4;

function perMinute(times: number[]): number[] {
  // one-minute tumbling windows, 12:00 to 12:03
  const counts: number[] = new Array(MINUTES).fill(0);
  for (const t of times) {
    counts[Math.floor(t / 60)] += 1;
  }
  return counts;
}

const byEvent = perMinute(CLICKS.map(([happened]) => happened));
const byArrival = perMinute(CLICKS.map(([, arrived]) => arrived));
console.log("minute    by event time   by processing time");
for (let m = 0; m < MINUTES; m++) {
  console.log(
    `12:0${m}     ${String(byEvent[m]).padStart(13)}   ${String(byArrival[m]).padStart(18)}`,
  );
}
const late = CLICKS.filter(([h, a]) => Math.floor(a / 60) !== Math.floor(h / 60)).map((c) => c[2]);
const phones = late.join(", ");
console.log(
  `${late.length} of ${CLICKS.length} clicks were counted in a later minute, from phones ${phones}`,
);
const latest = Math.max(...CLICKS.map(([happened, arrived]) => arrived - happened));
console.log(`the latest arrived ${latest} s after it happened`);

In TypeScript: Each click is typed as a tuple of two numbers and a string, so a row of the wrong shape does not compile.

Recorded with TypeScript 6.0.3 (strict), run by Node.js 24.21.0 on macOS 26 arm64. It runs in your browser with QuickJS (quickjs-emscripten 0.32.0) after Sucrase 3.35.1. To run it yourself: npx -p typescript@6.0.3 tsc --noEmit --strict main.ts && node main.ts

Go 1.27 · main.go

package main

import (
	"fmt"
	"strings"
)

// Click is one of ten clicks from four phones: when it happened and when it reached the job,
// in seconds after 12:00:00. Phones C and D were offline for a while and sent their clicks late.
type Click struct {
	Happened, Arrived int
	Phone             string
}

var clicks = []Click{
	{5, 6, "A"}, {20, 21, "B"}, {31, 150, "C"}, {42, 150, "C"}, {55, 56, "A"},
	{70, 71, "B"}, {88, 205, "D"}, {101, 102, "A"}, {130, 131, "B"}, {170, 171, "A"},
}

const minutes = 4

func main() {
	var byEvent, byArrival [minutes]int // one-minute tumbling windows, 12:00 to 12:03
	var late []string
	latest := 0
	for _, c := range clicks {
		byEvent[c.Happened/60]++
		byArrival[c.Arrived/60]++
		if c.Arrived/60 != c.Happened/60 {
			late = append(late, c.Phone)
		}
		latest = max(latest, c.Arrived-c.Happened)
	}
	fmt.Println("minute    by event time   by processing time")
	for m := range minutes {
		fmt.Printf("12:0%d     %13d   %18d\n", m, byEvent[m], byArrival[m])
	}
	fmt.Printf("%d of %d clicks were counted in a later minute, from phones %s\n",
		len(late), len(clicks), strings.Join(late, ", "))
	fmt.Printf("the latest arrived %d s after it happened\n", latest)
}

In Go: The counts are arrays of fixed size, [minutes]int, that start at zero, and range over the constant counts from 0 to 3.

Recorded with Go 1.27.1 on macOS 26 arm64. It runs in your browser with the Go compiler in WebAssembly 1.27.1. To run it yourself: go run main.go

Swift 6.4 · main.swift

// Ten clicks from four phones: when each happened and when it reached the job, in seconds
// after 12:00:00. Phones C and D were offline for a while and sent their clicks late.
let clicks: [(happened: Int, arrived: Int, phone: String)] = [
  (5, 6, "A"), (20, 21, "B"), (31, 150, "C"), (42, 150, "C"), (55, 56, "A"),
  (70, 71, "B"), (88, 205, "D"), (101, 102, "A"), (130, 131, "B"), (170, 171, "A"),
]
let minutes = 4

func perMinute(_ times: [Int]) -> [Int] {
  var counts = [Int](repeating: 0, count: minutes)  // one-minute tumbling windows, 12:00 to 12:03
  for t in times {
    counts[t / 60] += 1
  }
  return counts
}

func pad(_ n: Int, to width: Int) -> String {
  let digits = String(n)
  return String(repeating: " ", count: max(0, width - digits.count)) + digits
}

let byEvent = perMinute(clicks.map { $0.happened })
let byArrival = perMinute(clicks.map { $0.arrived })
print("minute    by event time   by processing time")
for m in 0..<minutes {
  print("12:0\(m)     \(pad(byEvent[m], to: 13))   \(pad(byArrival[m], to: 18))")
}
let late = clicks.filter { $0.arrived / 60 != $0.happened / 60 }.map { $0.phone }
print(
  "\(late.count) of \(clicks.count) clicks were counted in a later minute,"
    + " from phones \(late.joined(separator: ", "))")
let latest = clicks.map { $0.arrived - $0.happened }.max() ?? 0
print("the latest arrived \(latest) s after it happened")

In Swift: The clicks are tuples with labels, so $0.happened reads a field, and pad lines up the columns without Foundation.

Recorded with Apple Swift 6.4 on macOS 26 arm64. It runs in your browser with a WebAssembly build on the WASI runner (browser_wasi_shim 0.4.2). To run it yourself: swiftc -O main.swift -o main && ./main

Ruby 4.0 · main.rb

# frozen_string_literal: true

# Ten clicks from four phones: when each happened and when it reached the job, in seconds
# after 12:00:00. Phones C and D were offline for a while and sent their clicks late.
CLICKS = [
  [5, 6, 'A'], [20, 21, 'B'], [31, 150, 'C'], [42, 150, 'C'], [55, 56, 'A'],
  [70, 71, 'B'], [88, 205, 'D'], [101, 102, 'A'], [130, 131, 'B'], [170, 171, 'A']
].freeze
MINUTES = 4

def per_minute(times)
  counts = Array.new(MINUTES, 0) # one-minute tumbling windows, 12:00 to 12:03
  times.each { |time| counts[time / 60] += 1 }
  counts
end

by_event = per_minute(CLICKS.map(&:first))
by_arrival = per_minute(CLICKS.map { |click| click[1] })
puts 'minute    by event time   by processing time'
MINUTES.times do |minute|
  puts "12:0#{minute}     #{by_event[minute].to_s.rjust(13)}   " \
       "#{by_arrival[minute].to_s.rjust(18)}"
end
late = CLICKS.reject { |happened, arrived, _| arrived / 60 == happened / 60 }.map(&:last)
puts "#{late.size} of #{CLICKS.size} clicks were counted in a later minute, " \
     "from phones #{late.join(', ')}"
latest = CLICKS.map { |happened, arrived, _| arrived - happened }.max
puts "the latest arrived #{latest} s after it happened"

In Ruby: map(&:first) takes the first field of each row, and reject keeps the clicks whose minute changed on the way.

Recorded with CRuby 4.0.7 on macOS 26 arm64. It runs in your browser with ruby.wasm 2.10.1 (Ruby 4.0.0). To run it yourself: ruby main.rb

Kotlin 2.4 · Main.kt

// Ten clicks from four phones: when each happened and when it reached the job, in seconds
// after 12:00:00. Phones C and D were offline for a while and sent their clicks late.
data class Click(
    val happened: Int,
    val arrived: Int,
    val phone: String,
)

val clicks =
    listOf(
        Click(5, 6, "A"),
        Click(20, 21, "B"),
        Click(31, 150, "C"),
        Click(42, 150, "C"),
        Click(55, 56, "A"),
        Click(70, 71, "B"),
        Click(88, 205, "D"),
        Click(101, 102, "A"),
        Click(130, 131, "B"),
        Click(170, 171, "A"),
    )
const val MINUTES = 4

fun perMinute(times: List<Int>): IntArray {
    val counts = IntArray(MINUTES) // one-minute tumbling windows, 12:00 to 12:03
    for (t in times) counts[t / 60]++
    return counts
}

fun main() {
    val byEvent = perMinute(clicks.map { it.happened })
    val byArrival = perMinute(clicks.map { it.arrived })
    println("minute    by event time   by processing time")
    for (m in 0 until MINUTES) {
        val event = byEvent[m].toString().padStart(13)
        val arrival = byArrival[m].toString().padStart(18)
        println("12:0$m     $event   $arrival")
    }
    val late = clicks.filter { it.arrived / 60 != it.happened / 60 }.map { it.phone }
    println(
        "${late.size} of ${clicks.size} clicks were counted in a later minute, " +
            "from phones ${late.joinToString(", ")}",
    )
    println("the latest arrived ${clicks.maxOf { it.arrived - it.happened }} s after it happened")
}

In Kotlin: IntArray(MINUTES) starts at zero, and maxOf with a lambda finds the longest delay in one call.

Recorded with Kotlin 2.4.20 on Java 25.0.4.1 on macOS 26 arm64. It runs in your browser with a WebAssembly build on the WASI runner (browser_wasi_shim 0.4.2). To run it yourself: kotlinc Main.kt -include-runtime -d main.jar && java -jar main.jar

Dart 3.13 · main.dart

/// Ten clicks from four phones: when each happened and when it reached the job, in seconds
/// after 12:00:00. Phones C and D were offline for a while and sent their clicks late.
const clicks = [
  (happened: 5, arrived: 6, phone: 'A'),
  (happened: 20, arrived: 21, phone: 'B'),
  (happened: 31, arrived: 150, phone: 'C'),
  (happened: 42, arrived: 150, phone: 'C'),
  (happened: 55, arrived: 56, phone: 'A'),
  (happened: 70, arrived: 71, phone: 'B'),
  (happened: 88, arrived: 205, phone: 'D'),
  (happened: 101, arrived: 102, phone: 'A'),
  (happened: 130, arrived: 131, phone: 'B'),
  (happened: 170, arrived: 171, phone: 'A'),
];
const minutes = 4;

List<int> perMinute(Iterable<int> times) {
  // one-minute tumbling windows, 12:00 to 12:03
  final counts = List.filled(minutes, 0);
  for (final t in times) {
    counts[t ~/ 60]++;
  }
  return counts;
}

void main() {
  final byEvent = perMinute(clicks.map((c) => c.happened));
  final byArrival = perMinute(clicks.map((c) => c.arrived));
  print('minute    by event time   by processing time');
  for (var m = 0; m < minutes; m++) {
    print(
      '12:0$m     ${byEvent[m].toString().padLeft(13)}   '
      '${byArrival[m].toString().padLeft(18)}',
    );
  }
  final late = [
    for (final c in clicks)
      if (c.arrived ~/ 60 != c.happened ~/ 60) c.phone,
  ];
  print(
    '${late.length} of ${clicks.length} clicks were counted in a later minute, '
    'from phones ${late.join(', ')}',
  );
  final latest = clicks
      .map((c) => c.arrived - c.happened)
      .reduce((a, b) => a > b ? a : b);
  print('the latest arrived $latest s after it happened');
}

In Dart: Each click is a record with named fields, and a collection for with an if gathers the late phones.

Recorded with Dart 3.13.5 on macOS 26 arm64. It runs in your browser with a WebAssembly build by dart2wasm 3.13.5. To run it yourself: dart run main.dart

What every language printed

minute    by event time   by processing time
12:00                 5                    3
12:01                 3                    2
12:02                 2                    4
12:03                 0                    1
3 of 10 clicks were counted in a later minute, from phones C, C, D
the latest arrived 119 s after it happened

By processing time every minute is wrong. The clicks phone C made at 12:00 were counted at 12:02, so 12:00 shows 3 clicks instead of 5 and 12:02 shows 4 instead of 2, and a fourth minute appears that had no clicks at all. A dashboard built on processing time shows network delays as changes in what users did, and a replay of the same clicks gives other numbers. Counting by event time gives the right answer, at a price: the job must wait, because a click from 12:00 may still be on its way.

The figure sorts nine other clicks into the three kinds of window:

Nine clicks over six minutes grouped three ways: tumbling 2-minute windows, sliding 2-minute windows and sessions.Nine clicks in event timeTumbling, 2 minutes4 clicks1 click4 clicksSliding, 2 minutes, every minute0-2: 42-4: 14-6: 41-3: 33-5: 3Sessions, ended by a 1-minute gap5 clicks3 clicks10123456event time, in minutes

One set of clicks, three kinds of window

Text description of the diagram

A time axis of event time runs from 0 to 6 minutes, and each click is a short mark above it. Nine clicks happen at 10, 25, 70, 100 and 130 seconds, then at 250, 260 and 275 seconds, and finally at 340 seconds.

  • Tumbling windows of 2 minutes cut the axis into 0 to 2, 2 to 4 and 4 to 6 minutes, with no overlap. They hold 4, 1 and 4 clicks; every click is in exactly one window.
  • Sliding windows are 2 minutes long and one starts every minute, so they overlap: 0 to 2 holds 4 clicks, 1 to 3 holds 3, 2 to 4 holds 1, 3 to 5 holds 3 and 4 to 6 holds 4. Each click falls in two windows, counting the windows that start before 0 or end after 6 minutes, which the drawing leaves out.
  • Session windows end after a minute without a click. The first five clicks are never more than a minute apart, so they form one session from 10 to 130 seconds, which closes a minute after its last click. The next three form a second session, and the click at 340 seconds, 65 seconds after the one before it, starts a third, which runs past the end of the drawing.

The three window types answer different questions. tumbling window: A window of fixed length that follows the previous one without overlap, such as one per minute, so that every event belongs to exactly one window. windows are equal, back to back and never overlap: per-minute counts. sliding window: A window of fixed length that starts at a fixed interval shorter than its length, so that windows overlap and an event belongs to several, as in a 10-minute average updated every minute. windows have a fixed length and start at a fixed interval, so they overlap: a 10-minute moving average updated every minute. session window: A window per key that groups events until a gap with no events, so that each result covers one burst of activity, such as a visit. windows group a key’s events until a gap with no events: one result per visit.

The rest of the lesson covers when a window is complete and what to do about stragglers:

  • Bounded out-of-orderness: the watermark trails the largest event time seen by a fixed bound.
  • allowed lateness: How long a stream window keeps its state after the watermark has passed its end, so that late events can still update its result.: a window keeps its state for a while after it closes, so late events can update it.
  • side output: A separate stream that receives the events too late for any window, so that they can be counted, stored or reconciled instead of disappearing.: events too late for any window go to a separate stream instead of disappearing.
  • idle input: An input of a stream operator, such as a partition, that has stopped sending, and so holds back every watermark that depends on it.: a partition that stops sending holds back every watermark that depends on it; watermark alignment handles the opposite, a source far ahead of the rest.
  • checkpoint (stream processing): A consistent snapshot of a stream job's state and of the positions it has read its inputs up to, so that after a failure the job can restart from it.: consistent snapshots of state and input positions; with a transactional or idempotent sink, they make results exactly once after a failure.

Check the two clocks on clicks of your own.

Checkpoint

Check: Event time, processing time and windows

  1. Question 1 of 3 A click happened at 12:00:50 and reached the job at 12:02:10. In which minute does a per-minute count put it?

    Choose one answer.

    Show the answer: Question 1 of 3

    Answer: 12:00 by event time, 12:02 by processing time

    By event time the click counts in 12:00, the minute it happened. By processing time it counts in 12:02, the minute it arrived, which is how phone C's clicks moved two minutes later in the program.

  2. Question 2 of 3 When do the event time and the processing time of an event differ?

    Choose one answer.

    Show the answer: Question 2 of 3

    Answer: Whenever the event is delayed or reordered on its way to the job

    Event time is when the event happened, processing time when the job sees it. Any delay or reordering on the way separates them, which is why business numbers count by event time.

  3. Question 3 of 3 Which kind of window can put one event into several windows at once?

    Choose one answer.

    Show the answer: Question 3 of 3

    Answer: Sliding windows, which have a fixed length and start at a fixed interval

    A 10-minute window that starts every minute overlaps its neighbours, so each event is in several windows. Tumbling windows split time into back-to-back pieces, and sessions end after a gap.

Three windows on one stream, run Pro: opens with a Pro, Premium or Ultimate pass

In this part:

  • Watermarks: completeness against latency, measured
  • Late data: drop it, divert it or update the result
  • Parallel inputs: the slowest partition sets the clock
  • Exactly-once results need checkpoints and a safe sink
  • Practise: tumbling windows behind a watermark
    • Count tumbling windows behind a watermark
  • Interview questions
    • What is the difference between event time and processing time, and which should a per-minute sales dashboard use?
    • A dashboard counts app events per minute, and the numbers for the past hour keep changing for 20 minutes because phones send events late. Product wants fast numbers that are also right. How do you design the pipeline?
    • A streaming job reads a topic with 48 partitions. Its windowed results stopped coming out an hour ago, although events keep arriving and the job looks healthy. What might be wrong, and how do you find and fix it?
    • A streaming job counts payments per merchant per minute and writes the counts to a database. After a crash and restart, some minutes show counts that are too high. Why, and how do you make the results exactly once?
  • Key takeaways

3 diagrams · 3 runnable examples (JavaScript, Python) · 1 exercise with sample tests · 3 interview answers · about 46 minutes

Opening this part with your pass…

This part opens with any MySmartCoPilot pass.

  • Pro pass · 1 week₹79$2.99
  • 30 days₹179$5.99
  • 1 year₹1,699$45

One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.

It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).

Read a complete sample first: Design a unique ID generator

Terms in this lesson

allowed lateness
How long a stream window keeps its state after the watermark has passed its end, so that late events can still update its result.
checkpoint (stream processing)
A consistent snapshot of a stream job's state and of the positions it has read its inputs up to, so that after a failure the job can restart from it.
event time
The time an event happened, recorded by the device or service that produced it. A stream job that counts by event time puts a late event into the window it belongs to.
idle input
An input of a stream operator, such as a partition, that has stopped sending, and so holds back every watermark that depends on it.
processing time
The time on the clock of the machine running a stream job when it handles an event. Counting by it puts a delayed event into the window it arrived in.
session window
A window per key that groups events until a gap with no events, so that each result covers one burst of activity, such as a visit.
side output
A separate stream that receives the events too late for any window, so that they can be counted, stored or reconciled instead of disappearing.
sliding window
A window of fixed length that starts at a fixed interval shorter than its length, so that windows overlap and an event belongs to several, as in a 10-minute average updated every minute.
tumbling window
A window of fixed length that follows the previous one without overlap, such as one per minute, so that every event belongs to exactly one window.
watermark (stream processing)
A stream job's running estimate of how far event time has progressed. When it passes the end of a window, the window's result is emitted, and an event for that window that comes afterwards is late.

References

Related tools

Report a problem with this lesson

Quick answers and tool search

Type to search tools or to get a quick answer, for example 18% of 2500. Use the up and down arrow keys to move through the results, Enter to choose, and Escape to close.