This page needs a newer browser to run its tools. You can still read everything here.

Orchestration: running pipelines on time interview questions

Interview questions on “Orchestration: running pipelines on time”: warm-ups, then what each harder question tests and how a good answer starts.

16 questions: 5 warm-ups and 11 with model answers · Levels: Fresher to staff

Warm-up questions

Short questions with short answers, each answered in full.

Fresher

How would you explain a DAG to a developer who has only used cron?

Answered in the lesson Orchestration concepts: DAGs, schedules, backfills
Fresher

What is a data interval, and why does an orchestrator need it?

Answered in the lesson Orchestration concepts: DAGs, schedules, backfills
Fresher

What does it mean for a pipeline task to be idempotent, and why does an orchestrator need its tasks to be?

Answered in the lesson Idempotent tasks, retries and backfills
Fresher to mid

What makes four orchestrators the same, and where do they differ?

Answered in the lesson Dagster, Prefect, Kestra and cron: how they compare
Fresher

Why does Airflow 3 import things from airflow.sdk instead of airflow?

Answered in the lesson Airflow 3: TaskFlow, assets and schedules

Questions with model answers

What each question tests, the first step of a good answer and the follow-ups to expect.

Fresher to mid

The canteen team runs its seven tasks as seven crontab lines that start between 02:00 and 02:50, and it wants one Airflow DAG instead without losing or doubling a single day. Plan the move: what changes in the scripts, how you prove the DAG gives the same results, and how you switch over.
What it tests

Moving cron jobs into a DAG safely: scripts that take their interval instead of the clock, a parallel run that keeps side effects apart, a switch-over that skips and repeats no day, and numbers that prove it.

First step

List the seven crontab lines and, for each script, write down what it reads, what it writes and what it changes outside its own files, such as the email to ops. Those answers decide which tasks may run in both systems during a parallel week and which must be kept apart.
Follow-ups
- Which tasks must not run in both systems during the parallel week?
- How do you switch over so that no day is processed twice or skipped?
- What would you compare before you switch cron off?
From the lesson Orchestration concepts: DAGs, schedules, backfills
Model answer, Pro. Model answers open with any pass; prices are in the notice on this page.
Opening the model answers with your pass…
Model answers open with any MySmartCoPilot pass.
Pro pass · 1 week₹79$2.99
30 days₹179$5.99
1 year₹1,699$45
One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.
Passes can’t be bought right now, so the model answers can’t be opened at the moment. If you have a pass, enter your licence key.
Choose a pass I have a licence key
It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 6 minutes
Mid to senior

Ops in Pune wants the canteen summary at 10:00 India time on weekdays only, each one covering "the previous day" in India time. Today the pipeline runs on a daily 00:00 UTC schedule. Design the schedule and the interval each run processes, Monday's included.
What it tests

Turning a business calendar into schedules and intervals: time zones, half-open days, what a weekday-only schedule does to Monday, and keeping processing and delivery apart so backfills stay simple.

First step

Write the intervals down before any code. For the Tuesday email, which orders count, from which moment to which, in India time and in UTC? Then do the same for Monday's email, and ask which email carries Friday's, Saturday's and Sunday's orders.
Follow-ups
- What interval would a data-interval timetable on the weekday schedule give Monday's run?
- Which email carries the weekend's orders?
- How does a backfill of last month read the same days?
From the lesson Orchestration concepts: DAGs, schedules, backfills
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 4 minutes
Fresher to senior

For six weeks the canteen's publish task appended to each day's partition file, and every run succeeded. A dashboard shows 1,480 orders on a day the till counted 740. The task now writes a temporary file and renames it into place. How do you find every damaged day, repair each one, and prove the repair?
What it tests

Finding every damaged day without trusting run states, repairing days by rebuilding them from their source, choosing what to rebuild, and proving the repair with numbers.

First step

Stop the damage spreading first: tell the owners of the dashboards which weeks are suspect and pause what reads the partitions. Then list the 42 days and, for each one, set what its partition holds beside what its source export holds; a successful run says nothing about whether a day is right. Only when you know which days are wrong do you decide how to rebuild them.
Follow-ups
- Why not delete the extra lines from each damaged file?
- Do you rebuild only the six damaged days, or all 42?
- Why does one day hold 1,150 rows, and what stops that happening again?
- How do you prove the repair, and that the other 36 days are untouched?
From the lesson Idempotent tasks, retries and backfills
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 7 minutes
Mid to senior

The canteen DAG must be rerun for three past days after a fix to its source data, and one of those days had failed while the other two succeeded. Which Airflow 3 command do you run, what does it do with each of the three days, and how do you show afterwards that no day was duplicated or skipped?
What it tests

Knowing what airflow backfill create does, choosing between the reprocess behaviours, and checking a backfill's result day by day instead of trusting its success.

First step

Start from what changed: the source data of all three days was fixed, so every day must be rebuilt, including the two whose runs succeeded. Before creating anything, confirm that the tasks are idempotent and read their day from the run's data interval, then decide which of the existing runs the backfill must run again, and how many runs the source can bear at once.
Follow-ups
- What would the other two reprocess behaviours have done here?
- Does airflow backfill create run the tasks itself?
- What if a run for one of the days is still running when you start?
- How do you check the result?
From the lesson Idempotent tasks, retries and backfills
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 4 minutes
Mid to staff

A daily task with retries=5 has not failed in months, yet a weekly audit finds wrong totals on some days. Its logs show that most runs succeeded on their third or fourth try. What were the retries hiding, and how would you set retries, timeouts and alerts so this cannot stay hidden again?
What it tests

Telling transient errors from bugs, sizing retries against time limits, failing fast on errors a retry cannot fix, and alerting on retries as well as on failures.

First step

Treat the success on the third or fourth try as the symptom, not the cure. Before you change any setting, read the error of every failed try over a few weeks and set those errors beside the days the audit found wrong; then ask why months of near-failures never reached anyone.
Follow-ups
- Which errors deserve a retry, and which should fail at once?
- How long can this task hold up its run, and what does that mean for dagrun_timeout?
- Where do retry alerts go, and how do you keep them from being ignored?
From the lesson Idempotent tasks, retries and backfills
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 4 minutes
Mid to staff

A team runs the canteen job as an Airflow 3 DAG of three tasks, and the business now asks only whether the canteen revenue table is fresh. Would you port the job to Dagster assets or declare an asset inside Airflow 3? Say what changes in the code, in the run record and on the on-call's first screen.
What it tests

Contrasts dependencies from task calls with dependencies from asset parameters, names where each tool records a run and its outcome, and weighs a port against an Airflow 3 asset with one concrete cost of each.

First step

Start from what the business asks: the freshness of one table. Compare the two smallest changes that answer it: in Airflow 3 the load task declares the table as an asset; in Dagster each step becomes an asset whose parameters name what it reads. Then say what each change costs the team.
Follow-ups
- What does the Dagster version of the job look like, and where does its dependency graph come from?
- Where does each tool record that last night's run succeeded, and what does the on-call open first?
- What would the team have to relearn in the first month after a port to Dagster?
- When is declaring an asset in Airflow 3 enough, and when is it not?
From the lesson Dagster, Prefect, Kestra and cron: how they compare
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 5 minutes
Mid to staff

How would you set up a laptop to develop and test a daily pipeline on Airflow 3, Dagster and Prefect without starting a web server or a scheduler, and where do the three paths differ?
What it tests

Names the one laptop command for each tool, names the folder each tool keeps state under, and names one thing a laptop run cannot cover for each.

First step

List each tool's one laptop command and the HOME variable that holds its state, then say which parts of a production run do not appear in that path.
Follow-ups
- What state does each tool leave behind between runs, and how do you keep laptop runs independent?
- How would you prove a scheduled run before you ship, if the local path has no scheduler?
- Where does testing stop and deployment start for each tool?
- Which tool's local path is closest to its production path, and does that help?
From the lesson Dagster, Prefect, Kestra and cron: how they compare
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 6 minutes
Mid to staff

A team of three analysts and one engineer runs 15 cron jobs on one Linux box they look after themselves, and their on-call reads a failed job's output first. Which of Airflow 3, Dagster and Prefect would you put on that box, and what must keep running for it?
What it tests

Rules cron out with the reason, lists what each engine keeps running on one box, uses where each tool reports a run's outcome, and ends with one pick and a first step for this week.

First step

Rule cron out first: at 15 jobs, retries, reruns of past days and one view of every run would all be scripts the team writes and keeps. Then list the processes each engine keeps up on the box, and say where each one tells the on-call that a run failed.
Follow-ups
- What has to stay running on the box for each tool, and what stops if one of those processes dies?
- Where does each tool tell the on-call that last night's run failed?
- Which two of the 15 jobs would you move first, and what would you check after a week?
- How does the pick change if the team can no longer look after the box?
From the lesson Dagster, Prefect, Kestra and cron: how they compare
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 5 minutes
Fresher to staff

A canteen DAG's transform task returns a 400 MB pandas DataFrame so load can read it. In production the metadata database starts filling up and the scheduler slows down. What is going wrong, and how do you fix the DAG so the two tasks still hand data from one to the next?
What it tests

Explaining where Airflow 3 keeps a value a task returns and why its size matters there, choosing where the frame should live instead, and sketching the hand-off between the two tasks.

First step

Start from what Airflow does with a value a task returns: where it is kept, which way it travels in Airflow 3, and what else depends on that place. Then decide where 400 MB belongs, and which small value can still travel from transform to load.
Follow-ups
- Where should the frame live, and what does transform return instead?
- How small should each XCom stay, and how would you check it on a real run?
- How would you show that the fix helped, before and after?
- When is an object storage XCom backend the better fix, and when is it not?
From the lesson Airflow 3: TaskFlow, assets and schedules
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 5 minutes
Fresher to staff

A canteen DAG has CronDataIntervalTimetable("0 0 * * *", timezone="Asia/Kolkata"), so each run covers one Kolkata day. To redo a day that failed, an engineer runs airflow dags test with that day's date and no time, and the totals belong to another day. What happened, and how do you rerun the right day?
What it tests

Knowing how airflow dags test reads a date with no time, which interval a manual run of a data-interval timetable gets, how to name the day you mean, and what a change of timetable does to existing runs.

First step

Before rerunning anything, print what the run was handed: data_interval_start, data_interval_end and logical_date from the task's context, in UTC and in Kolkata time. Then work out which instant a date with no time stands for, and set it beside the interval the run printed.
Follow-ups
- How would you print what each run was handed, and what should this DAG print for the day it covers?
- Which value should the engineer give airflow dags test to cover the day they meant?
- What is this run's logical date, and how would it differ on a scheduled run?
- The DAG had a bare cron string until last week. What did the switch to this timetable change for its scheduled runs?
From the lesson Airflow 3: TaskFlow, assets and schedules
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 5 minutes
Fresher to staff

An orders file lands in object storage several times a day at unpredictable moments. The team wants a cleanup DAG to run every time a new file arrives, and never when the file is unchanged. How would you schedule the DAG, and what has to be true for the trigger to fire?
What it tests

Choosing an asset schedule (not a cron), matching the producer and consumer Asset by name and URI, and naming the role of AssetWatcher for events outside Airflow.

First step

Say first why a clock is the wrong trigger here: a cron runs whether or not a file arrived, so it can run twice on one file and never see another that was replaced between two runs. Then name the kind of schedule that lets the arrival itself start the DAG, and what the writer of the file has to declare.
Follow-ups
- What breaks if the producer's Asset URI differs from the consumer's by one character?
- When would you reach for AssetWatcher instead of a producer DAG?
- How would you test the wiring without a scheduler, and what would you still have to prove with the scheduler?
- Where should the Asset be defined so that the two DAGs cannot drift apart?
From the lesson Airflow 3: TaskFlow, assets and schedules
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 5 minutes

The lessons of this module

Orchestration concepts: DAGs, schedules, backfills Free lesson, Pro answers: 25 minutes
Airflow 3: TaskFlow, assets and schedules Free core, Pro depth: 6 of 15 minutes free
Idempotent tasks, retries and backfills Free core, Pro depth: 7 of 20 minutes free
Dagster, Prefect, Kestra and cron: how they compare Free core, Pro depth: 7 of 20 minutes free

All Data Engineering interview questions · The Data Engineering track

Your country

Orchestration: running pipelines on time interview questions

Warm-up questions

How would you explain a DAG to a developer who has only used cron?

What is a data interval, and why does an orchestrator need it?

What does it mean for a pipeline task to be idempotent, and why does an orchestrator need its tasks to be?

What makes four orchestrators the same, and where do they differ?

Why does Airflow 3 import things from `airflow.sdk` instead of `airflow`?

Questions with model answers

The canteen team runs its seven tasks as seven crontab lines that start between 02:00 and 02:50, and it wants one Airflow DAG instead without losing or doubling a single day. Plan the move: what changes in the scripts, how you prove the DAG gives the same results, and how you switch over.

Ops in Pune wants the canteen summary at 10:00 India time on weekdays only, each one covering "the previous day" in India time. Today the pipeline runs on a daily 00:00 UTC schedule. Design the schedule and the interval each run processes, Monday's included.

The canteen DAG must be rerun for three past days after a fix to its source data, and one of those days had failed while the other two succeeded. Which Airflow 3 command do you run, what does it do with each of the three days, and how do you show afterwards that no day was duplicated or skipped?

A daily task with retries=5 has not failed in months, yet a weekly audit finds wrong totals on some days. Its logs show that most runs succeeded on their third or fourth try. What were the retries hiding, and how would you set retries, timeouts and alerts so this cannot stay hidden again?

A team runs the canteen job as an Airflow 3 DAG of three tasks, and the business now asks only whether the canteen revenue table is fresh. Would you port the job to Dagster assets or declare an asset inside Airflow 3? Say what changes in the code, in the run record and on the on-call's first screen.

How would you set up a laptop to develop and test a daily pipeline on Airflow 3, Dagster and Prefect without starting a web server or a scheduler, and where do the three paths differ?

A team of three analysts and one engineer runs 15 cron jobs on one Linux box they look after themselves, and their on-call reads a failed job's output first. Which of Airflow 3, Dagster and Prefect would you put on that box, and what must keep running for it?

A canteen DAG's `transform` task returns a 400 MB pandas DataFrame so `load` can read it. In production the metadata database starts filling up and the scheduler slows down. What is going wrong, and how do you fix the DAG so the two tasks still hand data from one to the next?

An orders file lands in object storage several times a day at unpredictable moments. The team wants a cleanup DAG to run every time a new file arrives, and never when the file is unchanged. How would you schedule the DAG, and what has to be true for the trigger to fire?

Warm-up questions

How would you explain a DAG to a developer who has only used cron?

What is a data interval, and why does an orchestrator need it?

What does it mean for a pipeline task to be idempotent, and why does an orchestrator need its tasks to be?

What makes four orchestrators the same, and where do they differ?

Why does Airflow 3 import things from airflow.sdk instead of airflow?

Questions with model answers

The canteen team runs its seven tasks as seven crontab lines that start between 02:00 and 02:50, and it wants one Airflow DAG instead without losing or doubling a single day. Plan the move: what changes in the scripts, how you prove the DAG gives the same results, and how you switch over.

Ops in Pune wants the canteen summary at 10:00 India time on weekdays only, each one covering "the previous day" in India time. Today the pipeline runs on a daily 00:00 UTC schedule. Design the schedule and the interval each run processes, Monday's included.

The canteen DAG must be rerun for three past days after a fix to its source data, and one of those days had failed while the other two succeeded. Which Airflow 3 command do you run, what does it do with each of the three days, and how do you show afterwards that no day was duplicated or skipped?

A daily task with retries=5 has not failed in months, yet a weekly audit finds wrong totals on some days. Its logs show that most runs succeeded on their third or fourth try. What were the retries hiding, and how would you set retries, timeouts and alerts so this cannot stay hidden again?

How would you set up a laptop to develop and test a daily pipeline on Airflow 3, Dagster and Prefect without starting a web server or a scheduler, and where do the three paths differ?

A team of three analysts and one engineer runs 15 cron jobs on one Linux box they look after themselves, and their on-call reads a failed job's output first. Which of Airflow 3, Dagster and Prefect would you put on that box, and what must keep running for it?

A canteen DAG's transform task returns a 400 MB pandas DataFrame so load can read it. In production the metadata database starts filling up and the scheduler slows down. What is going wrong, and how do you fix the DAG so the two tasks still hand data from one to the next?

An orders file lands in object storage several times a day at unpredictable moments. The team wants a cleanup DAG to run every time a new file arrives, and never when the file is unchanged. How would you schedule the DAG, and what has to be true for the trigger to fire?

Why does Airflow 3 import things from `airflow.sdk` instead of `airflow`?

A canteen DAG's `transform` task returns a 400 MB pandas DataFrame so `load` can read it. In production the metadata database starts filling up and the scheduler slows down. What is going wrong, and how do you fix the DAG so the two tasks still hand data from one to the next?