MySmartCoPilot
Ask Passes Your pass 7 days left 1 day left Last day

Your country

Tools that support it use your country for local currency, number formats, units and paper size. Your choice is saved only in this browser.

System Design (High-Level Design)  Module 26 – Case studies: booking, commerce and contention

Design a price tracker: collecting prices

Design the collection side of a price tracker: sources each shop allows, safe link checks, a fetch budget per site, polite fetching and parsing.

  • Advanced
  • 25 minutes
  • Examples run with Python 3.14.8, Pyodide 314.0.7, Node.js 24.21.0 and quickjs 0.32.0
  • By MySmartCoPilot
Free: Every shop's allowance is a budget, and the first design spends it. With a pass: the other 12 sections and 3 interview answers, about 19 minutes.

What you will learn

  • Estimate fetches, per-site budgets and bandwidth for a price tracker
  • Choose price sources that each site's terms and APIs allow
  • Resolve pasted links to one product, seller and region without fetching internal addresses
  • Plan each site's fetch budget so that watchers see current prices
  • Design fetchers and parsers that respect limits and catch wrong prices

Before you start

What you’ll learn · Before you start

What you will learn

  • Estimate fetches, per-site budgets and bandwidth for a price tracker
  • Choose price sources that each site's terms and APIs allow
  • Resolve pasted links to one product, seller and region without fetching internal addresses
  • Plan each site's fetch budget so that watchers see current prices
  • Design fetchers and parsers that respect limits and catch wrong prices

Before you start

On this page

The collection side of a price tracker turns the links people paste into tracked products and keeps each product’s price current, within what every shop allows, though one shop may allow only 5 requests a second. Each shop’s terms decide the sources: an official feed or API first, data from an affiliate programme only where its policies allow tracking, and pages only where robots rules and terms allow, never by getting around a block.

A pasted link is checked so that the server never fetches an internal address, then keyed by product, seller and region, because prices differ by seller and by delivery area. Each site gets a fetch budget, and a scheduler spends it where watchers will notice a change. Fetchers make conditional requests and slow down when a site asks, parsers prefer structured data, and a change detector emits an event only when a price or the stock has changed.

Not affiliated

Based on public information about Amazon and Flipkart. MySmartCoPilot is not affiliated with Amazon and Flipkart, and Amazon and Flipkart’s real policies and systems may differ from what this lesson describes.

Every shop’s allowance is a budget, and the first design spends it

What makes collecting prices different

The data belongs to someone else. Shops publish terms, feeds and robots rules, and the tracker’s design starts from what they allow. Every site’s capacity is a budget, so freshness cannot simply be bought with more fetchers. The input is hostile: a pasted link can point anywhere, including at the tracker’s own network. And parsing is fragile, because one template change can turn every price on a site into a wrong number overnight.

What is different What the design does
The data belongs to the shops A source rule for each site, from its terms
Each site’s capacity is a budget A scheduler that spends a fixed number of fetches
A pasted link can point anywhere Address checks before any fetch
One template change breaks every price Checks before a price is published

Requirements

People paste a product link or search for a product, and the tracker follows it on the sites it supports. It records the price and stock of each product, seller and region, keeps watched products as current as the site’s allowance lets it, and hands every change to the history and the alerts. The qualities: no fetch beyond what a site allows; no request to a private or internal address; a wrong price is caught before anyone is alerted; and every shown price says when it was checked.

Estimates

Assume 2 million tracked items checked 4 times a day, 30 % of them on the largest site, one site that allows 5 requests a second for its 600,000 items, and pages of 120 KB. Work each figure out, then check yourself.

Checkpoint

Check the estimate

  1. Question 1 of 3 What makes collecting prices from shops different from reading a catalogue of your own?

    Choose one answer.

    Show the answer

    Answer: The data belongs to the shops, so their terms, feeds and robots rules decide what may be fetched

    The data belongs to someone else: shops publish terms, feeds and robots rules, and each site's capacity is a budget. The input is hostile, because a pasted link can point anywhere, and parsing is fragile.

  2. Question 2 of 3 A tracker follows 2 million items (a product, a seller and a region each) and wants to check each one 4 times a day. How many fetches a second is that?

    Type a number.

    A hint

    Work out the fetches of a day first, then divide by the 86,400 seconds of a day.

    Show the answer

    Answer: 92.6 fetches a second (anything from 83.3 to 101.9 counts)

    2,000,000 × 4 = 8,000,000 fetches a day, and 8,000,000 ÷ 86,400 ≈ 92.6 a second across every site. The total is small; what limits the design is how much each site allows.

  3. Question 3 of 3 One site allows 5 requests a second and holds 600,000 tracked items. How many checks a day can each item get on average?

    Type a number.

    A hint

    Find the requests the site allows in a day first.

    Show the answer

    Answer: 0.72 checks per item a day (anything from 0.648 to 0.792 counts)

    5 × 86,400 = 432,000 requests a day, and 432,000 ÷ 600,000 = 0.72, under one check a day on average. The site's allowance, not the tracker, decides how fresh its items can be, so where the fetches go is a design decision.

Quantity Working Result
Fetches for 4 checks a day 2 million items × 4 ÷ 86,400 s about 93 a second
The largest site, 30 % of items 93 × 0.30 about 28 a second
One site allowing 5 a second 432,000 a day ÷ 600,000 items 0.72 checks per item a day
Pages downloaded 93 a second × 120 KB about 89 Mbit/s

The total is modest; one site’s allowance is the binding limit.

The first design

Price collection: checked links become tracked items; a scheduler spends each site budget; fetchers and parsers emit price changes.Userspaste a link or searchLink intakeaddress checks, product keyTracked itemsproduct, seller, regionSource rules per sitefeed, API or allowed pagesRefresh schedulera fetch budget per sitePer-site queuestoken bucketsFetchersconditional requests,back off when toldParsersstructured data firstChange detectorprice or stock changed?Price eventsto history and alertschange rates

The first design of a price tracker's collection side

Text description of the diagram

The diagram shows how a price is collected, from top to bottom.

  1. Users paste a link or search. Link intake checks the addresses the link leads to and turns it into a product key.
  2. Tracked items are kept as a product, a seller and a region.
  3. The refresh scheduler reads the tracked items and the source rules of each site (a feed, an API or the pages that may be fetched) and spends a fetch budget per site.
  4. Fetches wait in per-site queues with token buckets, and fetchers send conditional requests and back off when a site tells them to.
  5. Parsers read structured data first, and a change detector checks whether the price or the stock changed.
  6. Changes become price events for the history and the alerts. A dashed arrow takes the observed change rates back to the scheduler.

Link intake checks where a pasted link leads and turns it into a tracked item. The scheduler reads every item and each site’s source rules, and decides what to fetch next within the site’s budget. Fetches wait in per-site queues, the fetchers make them, the parsers read the price and stock, and the change detector passes on only what changed, with change rates going back to the scheduler.

Step through one pasted link, from the paste to a price event:

Interactive · step by step One pasted link through the first design Step through one link from the paste to a price event, and see where each site's allowance is spent.
One pasted link through the first design, step 7 of 7. Showing User, Link intake, Tracked items, Source rules per site, Refresh scheduler, Per-site queues, Fetchers, Parsers, Change detector, Price events. In focus: Change detector.Userpastes a linkLink intakeaddress checks, product keyTracked itemsproduct, seller, regionSource rules per sitefeed, API or allowed pagesRefresh schedulera fetch budget per sitePer-site queuestoken bucketsFetchersconditional requestsParsersstructured data firstChange detectorprice or stock changed?Price eventsto history and alertsprice event

The drawing shows the whole scene; its 7 steps are listed below.

Every step as text
  1. A link is pasted. A user pastes a product link. Link intake checks where it leads, so the server never fetches an internal address, and turns it into one key for the product.
  2. One tracked item. The tracked item is the product, the seller and the region, because one product can have several sellers and prices can depend on the delivery area.
  3. The site's rules. The scheduler reads every tracked item and the site's source rule, which says whether a feed, an API or the pages may be used.
  4. Spent from the budget. The site's allowance is a budget, so the scheduler decides what to fetch next within it, where watchers will notice a change.
  5. Fetched politely. A fetcher takes the next fetch from the site's queue, sends a conditional request and slows down when the site asks it to.
  6. Read the price. The parser reads the price and the stock, from the page's structured data first.
  7. Only changes go on. The change detector passes on an event only when the price or the stock changed, to the history and the alerts, and tells the scheduler how often this item changes.

Checkpoint

Check the first design

  1. Question 1 of 2 Which of these are on the path from a scheduled fetch to a price event? Choose every one.

    Choose every answer that is right.

    Show the answer

    Answer:

    • The fetcher
    • The per-site queue with its token bucket
    • The parser
    • The change detector

    A fetch waits in its site's queue, is made by a fetcher, read by a parser and compared by the change detector, which emits the event. Alerts and notifications act on that event later, outside the collection side.

  2. Question 2 of 2 Watched products on the largest site are stale, and a teammate proposes doubling the fetchers. What does that change?

    Choose one answer.

    Show the answer

    Answer: Little, because the site's allowance is the limit; the budget has to be spent better instead

    The total is modest and one site's allowance is the binding limit, so freshness cannot simply be bought with more fetchers. The scheduler decides what to fetch next within the site's budget.

The deep-dive questions

  1. Which sources may a tracker use for each site, and what do their terms require?
  2. How does a pasted link become one product, seller and region without the server fetching internal addresses?
  3. How is a site’s fetch budget spent so that watchers see current prices?
  4. How do fetchers stay within each site’s limits and back off when asked?
  5. How are prices parsed reliably, and how are wrong prices caught before anyone sees them?
  6. What changes ahead of an announced sale?
  7. What breaks at 10× and 100×, and which designs are rejected?

Each site’s terms choose the source, in an order of preference Pro

In this part:

  • A pasted link is checked at every address and every redirect
  • A site’s budget goes where watchers look, measured
  • Each site has one polite gate: a bucket, conditional requests and back-off
  • A parsed price is only a candidate until it passes two alarms
  • At 10× each site’s budget gives way, and at 100× link intake
  • Failure drills: five faults, each with a safe way out
  • Options considered and rejected
  • What each level covers, and a 45-minute self-mock
    • What each level is expected to cover
    • A 45-minute self-mock
  • Practise: the polite gate in code, then the whole design
    • Keep fetches within a site's limits and back off when told
  • Interview questions
    • Your tracker follows 600,000 products on one shop that allows 5 requests a second. Users say sale prices show up hours late, yet most fetches find nothing changed. The scheduler refreshes products in proportion to how often their prices change. What is wrong, and how do you decide what to fetch next?
    • Users paste product links and your server fetches them to identify the product. A security test shows that a pasted short link made the server fetch an internal admin page. How did that happen, and what does the server check before and during every fetch?
    • Overnight your tracker recorded a 90 % price drop on 3,000 products of one shop, and alerts went out. The shop had changed its page template, and the parser read a different number. How do you stop wrong prices from reaching users, and how do you clean up?
  • Key takeaways

4 diagrams · 3 runnable examples (Python) · 1 exercise with sample tests · 3 interview answers · about 47 minutes

Opening this part with your pass…

This part opens with any MySmartCoPilot pass.

  • Pro pass · 1 week₹79$2.99
  • 30 days₹179$5.99
  • 1 year₹1,699$45

One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.

Choose a passI have a licence key

It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).

Read a complete sample first: Design movie seat booking like BookMyShow

References

Related tools

Robots.txt URL Tester See which crawlers may fetch a URL — and the exact rule that decides it. URL Expander Find out where a short link really goes — without clicking it. Schema Markup Validator Find syntax errors, schema.org mistakes and missing Google properties in your markup. IP Address Converter Any IP address in every notation — and what the registries say about it.

Report a problem with this lesson