A first look at any dataset: shape, types, surprises

A first look is the few minutes you spend learning what a dataset really contains before you compute anything from it: how big it is, which type each column has, what one row stands for, and which values cannot be right. pandas has a handful of commands for it, and their output shows most problems at once, if you know where to read.

This lesson takes a first look at a bakery’s export of online orders: 2,025 rows that hide seven problems common in real exports. It finds every one with those commands, checks the table’s key, and writes the findings into a data log, the record of what you found and what you decided to do about it.

The file

orders.csv has one row per order and nine columns: order_id, order_date, customer_id, city, item, quantity, unit_price, delivery_minutes and distance_km. It is synthetic, made for this lesson by a script with a fixed random seed, and every problem in it is one that real exports from order and payment systems show. Try to spot them in the output before you read the explanation under each example.

Size, types and a few rows

Four commands come first. shape gives the number of rows and columns, dtypes the type pandas chose for each column, head() the first rows and sample() rows drawn at random:

# The first four commands on a new file: its size, its column types, its first rows and a random sample.
import pandas as pd

pd.set_option("display.width", 120)  # print wide tables in full (this changes only the printing)
pd.set_option("display.max_columns", 20)

orders = pd.read_csv("orders.csv")
print(orders.shape, "\n")
print(orders.dtypes, "\n")
print(orders.head(3), "\n")
print(orders.sample(3, random_state=7))

(2025, 9) 

order_id                str
order_date              str
customer_id         float64
city                    str
item                    str
quantity              int64
unit_price              str
delivery_minutes      int64
distance_km         float64
dtype: object 

  order_id  order_date  customer_id     city          item  quantity unit_price  delivery_minutes  distance_km
0   A10001  2026-03-03       1239.0   Mumbai  Banana bread         5        240                69          1.8
1   A10002  2026-03-26       1422.0  Chennai    Cookie box         3        320                34         11.3
2   A10003  2026-03-18       1246.0   Mumbai    Cookie box         2        320                85         13.2 

     order_id  order_date  customer_id        city            item  quantity unit_price  delivery_minutes  distance_km
1028   A11015  2026-03-06       1032.0   bengaluru  Sourdough loaf         1        180                23          7.3
140    A10139  2026-03-23       1017.0   Bengaluru       Croissant         5         90                20         13.7
110    A10109  2026-03-12       1158.0        Pune  Sourdough loaf         1        180                30          5.1

The types already point at three problems:

• customer_id is float64 and prints as 1239.0. An identifier made of digits would read as int64, so the column holds missing values: a missing value is NaN, which is a float, and it takes the whole column with it.
• unit_price is str, text rather than numbers, so at least one price is written in a way pandas could not read as a number. Adding up text fails, and sorting it puts “90” after “320”.
• order_date is str too. read_csv leaves dates as text unless you ask it to parse them, so date arithmetic and sorting by date need a conversion first.

head() shows the first rows, and exports often start with their tidiest ones. sample(3, random_state=7) draws three rows from the whole file, and the fixed random_state makes it the same three rows on every run, so a colleague sees what you saw. Here the sample already turns up a city written as bengaluru, with a leading space and a small letter. The two set_option lines only make pandas print all nine columns side by side (options and settings).

info() and describe()

info() summarises the structure: every column’s type and how many values are not missing, and with memory_usage="deep" the real memory the table takes, text included (DataFrame.info). describe() summarises the values. By default it covers only the numeric columns; include="str" describes the text columns, and include="all" does both at once (DataFrame.describe):

# info() for types, gaps and memory; describe() for the spread of every column.
import pandas as pd

pd.set_option("display.width", 120)
pd.set_option("display.max_columns", 20)

orders = pd.read_csv("orders.csv")
orders.info(memory_usage="deep")
print()
print(orders.describe().round(1))
print()
print(orders.describe(include="str"))

<class 'pandas.DataFrame'>
RangeIndex: 2025 entries, 0 to 2024
Data columns (total 9 columns):
 #   Column            Non-Null Count  Dtype  
---  ------            --------------  -----  
 0   order_id          2025 non-null   str    
 1   order_date        2025 non-null   str    
 2   customer_id       1900 non-null   float64
 3   city              2025 non-null   str    
 4   item              2025 non-null   str    
 5   quantity          2025 non-null   int64  
 6   unit_price        2025 non-null   str    
 7   delivery_minutes  2025 non-null   int64  
 8   distance_km       2025 non-null   float64
dtypes: float64(2), int64(2), str(5)
memory usage: 217.4 KB

       customer_id  quantity  delivery_minutes  distance_km
count       1900.0    2025.0            2025.0       2025.0
mean        1296.5       2.3             320.4        241.1
std          171.3       1.5            1602.5       1449.8
min         1001.0      -3.0              20.0          0.5
25%         1150.0       1.0              38.0          4.1
50%         1296.0       2.0              56.0          8.0
75%         1440.0       3.0              74.0         11.8
max         1600.0       6.0            9999.0      14600.0

       order_id  order_date       city            item unit_price
count      2025        2025       2025            2025       2025
unique     2000          28          7               6          6
top      A10055  2026-03-26  Hyderabad  Sourdough loaf        180
freq          2          85        444             615        615

Read the numbers as questions:

• customer_id has 1,900 non-null values in 2,025 rows, so 125 orders have no customer. Its mean and quartiles mean nothing: an identifier is a label, not a quantity.
• quantity has a minimum of −3. This shop records returns elsewhere, so a negative quantity is impossible here.
• delivery_minutes has a mean of 320.4, a median (the 50 % row) of 56 and a maximum of 9999. Deliveries take about an hour; 9999 is a code someone typed to mean “unknown”, and those few values drag the mean up to more than five hours.
• distance_km reaches 14,600. A bakery does not deliver 14,600 km away, and the 75 % row sits at 11.8, so the largest values look like metres recorded in a column of kilometres.
• order_id has 2,000 distinct values in 2,025 rows, and its most frequent value occurs twice, so some ids repeat.
• city has 7 distinct values for a bakery that delivers to five cities.
• unit_price has 6 distinct values, one per item, so a few prices are written differently, not wrongly.

The grain and the key

Before going further, write down the table’s grain and its key. The grain of orders.csv is one row per order, so its key is order_id: no two rows should share one. duplicated(subset=key) marks every row whose key appeared in an earlier row, so .sum() counts the repeats, and duplicated() without a subset marks rows that copy an earlier row in every column (DataFrame.duplicated). The next example runs both. When the two counts are equal, every repeated key belongs to an exact copy, and the export simply wrote some rows twice. When they differ, two different orders share an id, or the grain is not what you thought, and that needs an answer from whoever owns the data before you change anything.

The usual surprises

Each kind of surprise has a command that shows it:

This program runs a targeted check for each one and turns the results into a data log:

# Targeted checks for the usual surprises, and a data log that records each one with the decision to take.
import pandas as pd

pd.set_option("display.width", 120)
pd.set_option("display.max_colwidth", 60)

orders = pd.read_csv("orders.csv")
key = ["order_id"]  # the grain: one row per order, so order_id should be unique

print("missing values:", orders.isna().sum()[lambda s: s > 0].to_dict())
print("exact duplicate rows:", orders.duplicated().sum())
print("rows whose order_id repeats:", orders.duplicated(subset=key).sum())
print("city spellings:", orders["city"].value_counts().to_dict())
print("unit_price values that are not plain digits:", sorted(set(orders.loc[~orders["unit_price"].str.isdigit(), "unit_price"])))
print("delivery_minutes above 600:", orders.loc[orders["delivery_minutes"] > 600, "delivery_minutes"].unique().tolist())
print("distance_km above 100:", (orders["distance_km"] > 100).sum(), "rows, for example", orders.loc[orders["distance_km"] > 100, "distance_km"].head(3).tolist())
print("negative quantities:", (orders["quantity"] < 0).sum())

log = pd.DataFrame(
    [
        ("(all)", "exact duplicate rows", orders.duplicated().sum(), "drop the second copy"),
        ("customer_id", "missing, so the column became float64", orders["customer_id"].isna().sum(), "keep the rows; read as Int64"),
        ("unit_price", "thousands separator, so the column is text", (~orders["unit_price"].str.isdigit()).sum(), "remove the commas, convert"),
        ("delivery_minutes", "9999 stands for unknown", (orders["delivery_minutes"] == 9999).sum(), "treat as missing"),
        ("distance_km", "some values are metres", (orders["distance_km"] > 100).sum(), "divide those by 1,000"),
        ("quantity", "negative, impossible here", (orders["quantity"] < 0).sum(), "leave out; ask the shop"),
        ("city", "three spellings of Bengaluru", orders["city"].isin(["Bangalore", " bengaluru"]).sum(), "map to Bengaluru"),
    ],
    columns=["column", "issue", "rows", "decision"],
)
print()
print(log.to_string(index=False))

missing values: {'customer_id': 125}
exact duplicate rows: 25
rows whose order_id repeats: 25
city spellings: {'Hyderabad': 444, 'Chennai': 405, 'Mumbai': 404, 'Pune': 396, 'Bengaluru': 275, 'Bangalore': 68, ' bengaluru': 33}
unit_price values that are not plain digits: ['1,250', '2,400']
delivery_minutes above 600: [9999]
distance_km above 100: 70 rows, for example [11300.0, 14400.0, 14500.0]
negative quantities: 3

          column                                      issue  rows                     decision
           (all)                       exact duplicate rows    25         drop the second copy
     customer_id      missing, so the column became float64   125 keep the rows; read as Int64
      unit_price thousands separator, so the column is text   267   remove the commas, convert
delivery_minutes                    9999 stands for unknown    54             treat as missing
     distance_km                     some values are metres    70        divide those by 1,000
        quantity                  negative, impossible here     3      leave out; ask the shop
            city               three spellings of Bengaluru   101             map to Bengaluru

All seven problems are confirmed. The 25 repeated order ids are exactly the 25 duplicate rows, so dropping the second copy loses nothing. Only two prices are written differently, 1,250 and 2,400, with a thousands separator that turns the whole column into text. And the city column needs a mapping, not a delete: 101 orders were placed in Bengaluru under another spelling.

Keep a data log

The last table is the data log, and it is as much a result of the first look as any number. It records, for each problem, the column, what is wrong, how many rows of the file as received it touches (duplicates included) and what you decided to do. Nothing has been changed yet: the decisions are applied in one cleaning step later, in the order the log lists them, so that anyone can repeat them and check them.

A log turns judgement calls into something people can review. “Leave out the 3 orders with a negative quantity” is a choice someone may disagree with, and it is far better to have that discussion before a number is reported than after. Keep the log next to the analysis, add a row whenever you find something new, and report the counts with the result: “2,000 orders after removing 25 duplicate rows; 125 have no customer id”.

Exercise

def gap_counts(rows, sentinels):
    """Per column, how many values are empty, only spaces, or a sentinel such as "NA" or "9999".

    rows is a list of dictionaries from csv.DictReader. Returns {column: count} for columns with a gap,
    sorted by column name.
    """
    # Replace this line with your code.
    return {}


def repeated_keys(rows, key):
    """The key values (tuples) that occur in more than one row, sorted; key is a list of column names."""
    # Replace this line with your code.
    return []

import csv
import io

from checks import gap_counts, repeated_keys

ORDERS = """order_id,item,quantity,delivery_minutes,city
A1,Croissant,2,35,Pune
A2,Cookie box,NA,9999,Mumbai
A3,Croissant,1,,
A3,Croissant,1,,
A4,Banana bread,  ,42,Pune
A5,Croissant,3,9999, - 
A5,Cookie box,1,51,Chennai
"""


def rows():
    return list(csv.DictReader(io.StringIO(ORDERS)))


def test_gap_counts():
    """counts empty values, spaces and sentinels per column"""
    assert gap_counts(rows(), ["NA", "-", "9999"]) == {"city": 3, "delivery_minutes": 4, "quantity": 2}


def test_gap_counts_without_sentinels():
    """counts only empty values and spaces when there are no sentinels"""
    assert gap_counts(rows(), []) == {"city": 2, "delivery_minutes": 2, "quantity": 1}


def test_repeated_one_column_key():
    """finds order ids used by more than one row"""
    assert repeated_keys(rows(), ["order_id"]) == [("A3",), ("A5",)]


def test_repeated_two_column_key():
    """treats rows as repeats only when every key column matches"""
    assert repeated_keys(rows(), ["order_id", "item"]) == [("A3", "Croissant")]
    assert repeated_keys(rows(), ["order_id", "item", "quantity", "city"]) == [("A3", "Croissant", "1", "")]

Check yourself

Interview questions

Fresher · What does DataFrame.info() tell you that describe() does not?

info() describes the structure of a table: how many rows it has, each column’s name and dtype, how many values in each column are not missing, and how much memory the table uses (with memory_usage="deep", the text included). describe() describes the values: for numbers the count, mean, standard deviation, minimum, quartiles and maximum, and for text the number of distinct values and the most frequent one. So info() is where you notice that a column of prices is text or that a column has gaps, and describe() is where you notice impossible values, such as a minimum of −3 or a maximum of 9999. By default describe() leaves out text columns; include="all" shows both kinds.

Key takeaways

• Start every dataset with shape, dtypes, head(), a seeded sample(), info() and describe(), and read their output as a list of questions.
• Types tell a story: whole numbers shown as floats mean missing values, and numbers stored as text mean a value pandas could not read.
• Minimums, maximums and the gap between mean and median expose sentinels, mixed units and impossible values.
• Write the grain and the key, and check the key with duplicated(subset=key) before anything else.
• Record every problem, its row count and your decision in a data log, and change nothing until the log is written.