Design activity tracking like Strava: GPS ingestion

On a busy Sunday an activity tracker takes in about 670 uploads a second, each a run or ride from a watch, a bike computer or a phone, and keeps about 570 TB of new files and samples a year. Every upload must become a saved workout within seconds: a map, distance, moving time, climb and charts for followers.

The design rests on four ideas. Devices upload the finished file, which the service stores unchanged and processes later from a queue, so a retry or a second copy of the same run is recognised instead of saved twice. Processing is a pipeline over a time series: drop impossible jumps, smooth the track, then derive the statistics. Results are stored as columns of samples at several resolutions, with a short encoded line for each feed card’s map. And privacy is applied on the server before anything is shared, because a hidden start that is always the same circle still points at a home.

About 670 uploads a second, and 570 TB a year to keep

Settle these first, with the answers this design assumes:

• Which devices? Watches and bike computers upload files; phone apps record and upload themselves. Strava’s API accepts FIT, TCX and GPX files, gzipped or not, and processes each upload asynchronously (Strava).
• How big? 10 million uploads a day; a Sunday brings twice that, 12 % of it in the busiest hour.
• How fast? A new activity appears within seconds; Strava’s API documentation gives a mean processing time under 2 seconds.
• Live sharing? A recording can share its position with chosen contacts; Strava Beacon updates about every 15 seconds and needs mobile signal (Strava).
• Privacy? Each activity has a visibility, and starts and ends near home are hidden; Strava hides the first and last 200 m of new activity maps by default (Strava).

So the service accepts uploads that may repeat, computes the statistics, keeps the original so that activities can be recomputed when the algorithms improve, serves maps and charts, and never shows a hidden part to anyone but the owner. Uploads may arrive late and in bursts after a race; nothing may be lost once accepted.

Storage, not traffic, is the big number: about 570 TB a year that is read rarely and must never be lost. The upload rate itself is modest, but each upload starts several seconds of processing.

HOUR_SECONDS = 3_600
DAY_SECONDS = 86_400


def peak_uploads_per_second(daily_uploads, busiest_day_factor, busiest_hour_share):
    """Uploads a second in the busiest hour of the busiest day."""
    # Replace this line with your code.
    return 0


def yearly_file_storage_tb(daily_uploads, kb_per_file):
    """TB for a year of original files."""
    # Replace this line with your code.
    return 0


def yearly_stream_storage_tb(daily_uploads, samples_per_activity, bytes_per_sample, resolutions_factor):
    """TB for a year of processed streams at every resolution."""
    # Replace this line with your code.
    return 0


def thumbnail_gbit_per_second(daily_views, kb_per_image, peak_factor):
    """Gbit/s for feed map thumbnails at the busiest hour."""
    # Replace this line with your code.
    return 0

import math

from sizing import peak_uploads_per_second, thumbnail_gbit_per_second, yearly_file_storage_tb, yearly_stream_storage_tb


def near(actual, expected):
    """Within 10%, as an estimate should be."""
    return math.isclose(actual, expected, rel_tol=0.10)


def test_peak_uploads():
    """the busiest day's uploads, the busiest hour's share of them, per second"""
    assert near(peak_uploads_per_second(10_000_000, 2, 0.12), 667)
    assert near(peak_uploads_per_second(1_400_000, 1.5, 0.1), 58)


def test_file_storage():
    """a year of files at their uploaded size"""
    assert near(yearly_file_storage_tb(10_000_000, 100), 365)
    assert near(yearly_file_storage_tb(2_000_000, 60), 43.8)


def test_stream_storage():
    """samples times bytes, plus the smaller resolutions"""
    assert near(yearly_stream_storage_tb(10_000_000, 3_600, 12, 1.3), 205)
    assert near(yearly_stream_storage_tb(1_000_000, 1_800, 10, 1.2), 7.9)


def test_thumbnail_bandwidth():
    """the day's image bits per second, at the peak"""
    assert near(thumbnail_gbit_per_second(300_000_000, 20, 3), 1.67)
    assert near(thumbnail_gbit_per_second(50_000_000, 30, 2.5), 0.35)

The first design: store first, process from a queue

The device sends the finished file to the upload API, which checks for repeats, saves the original in the file store and queues an upload id. Workers parse, clean and measure the track, hide the private parts, and write the activity’s statistics, its streams and an event that the feed, segment matching and map thumbnails consume. Live positions take a short path of their own. Follow one run through it:

The questions this design has to answer:

• Repeat uploads: a retry sends the same bytes, a re-export different bytes, and a second device a second recording of the same run.
• Cleaning a track: drop fixes that imply an impossible speed, then smooth what remains before measuring.
• Moving time: pauses recorded by the device win; otherwise a speed threshold decides when the athlete rested.
• Climb: a barometer’s readings, smoothed, or a terrain model; raw GPS altitude counts every wobble as climbing.
• Streams and polylines: columns of samples at several resolutions, and an encoded line of points for maps.
• Privacy zones: hide the start and end, with a hidden distance that varies, so repeated runs do not outline a home.
• Reprocessing: keep every original and compute old activities again as a background job.

A content hash catches only retries; overlapping times catch a second device Pro: opens with a Pro, Premium or Ultimate pass