Data Preprocessing Tool (Impute, Outliers, Scale, Encode)

About the Data Preprocessing Tool (Impute, Outliers, Scale, Encode)

Raw tables are rarely ready for analysis or machine learning: values are missing, a few extreme ones distort averages, columns sit on very different scales, and categories are words a model cannot use. This tool prepares a table in steps you can see and change: fill missing values (mean, median, most frequent value, a value you type, or the value above or below), find outliers with IQR fences, the z-score or the modified z-score and flag, cap or remove them, scale or transform (min–max, z-score, robust, max-abs, logarithms, Box–Cox and Yeo–Johnson), bin numbers into ranges, and encode categories as one-hot, label or ordinal codes.

Every step shows what it learned from your data — the median it filled with, the fences, the mean and standard deviation, the λ of a power transform, the bin edges, the categories. With a Pro pass, download the prepared table as CSV or Excel, and keep the recipe as a parameters file (which the tool can apply again to new data) or as pandas code that reapplies the same values to tomorrow’s file. Everything runs in your browser.

Common uses

• Preparing a CSV for a model in scikit-learn, R, Excel or an AutoML service.
• Cleaning survey or sensor data before charts and statistics.
• Teaching and learning what imputation, scaling and encoding do, with every fitted value in view.
• Making a recipe once and applying exactly the same transformation to new data every month.

Missing values

Empty cells and the marks pandas reads as missing (NA, N/A, NaN, null, None, #N/A and a few more) count as missing. Mean and median fill numbers; most frequent value works for numbers and text (ties go to the smallest value, as in scikit-learn’s SimpleImputer); a value you type fills any column; the value above or below copies the nearest earlier or later value (time series), and Drop the rows removes rows with a gap in the chosen columns instead. Tick Add a 0/1 column marking the filled cells to keep the information that a value was missing, which can itself predict the outcome. The median resists outliers better than the mean.

Outliers

• IQR fences (Tukey): values below Q1 − k·IQR or above Q3 + k·IQR, with k = 1.5 for outliers and 3 for far-out values, the inner and outer fences of the NIST/SEMATECH e-Handbook of Statistical Methods.
• z-score: values more than k standard deviations from the mean (k = 3). The mean and standard deviation are themselves pulled by outliers, so this misses some in small samples.
• Modified z-score: 0.6745·(x − median)/MAD above k = 3.5, as recommended by Iglewicz and Hoaglin; robust, because the median and the median absolute deviation ignore extremes.

Then flag them in a new 0/1 column, cap them at the limits (winsorising), or remove their rows. Quantiles use linear interpolation, as NumPy and pandas compute them by default.

Scaling and transforms

• z-score (standardise): (x − mean)/sd, with the population standard deviation as scikit-learn’s StandardScaler.
• Min–max: (x − min)/(max − min), stretched to the range you choose (0 to 1 by default).
• Robust: (x − median)/IQR, as RobustScaler, for data with outliers.
• Max-abs: x/max|x|, which keeps zeros and signs.
• Logarithms (natural, base 10, log(1 + x)) for right-skewed values such as incomes or counts.
• Box–Cox (Box and Cox) for positive values and Yeo–Johnson (Yeo and Johnson) for any values: power transforms whose λ is chosen by maximum likelihood to make the data as close to normal as possible.

A column without spread is scaled by 1, as scikit-learn does, so it becomes a column of zeros instead of an error. The fitted centre, scale or λ is shown for each column. See scikit-learn’s preprocessing guide for when each one helps.

Bins and encodings

Bins cut numbers into ranges of equal width, into quantiles (about the same number of rows in each), or at edges you type. As in scikit-learn’s KBinsDiscretizer, a value on an edge goes into the upper bin and values beyond the outer edges into the end bins; each bin is written as its range, such as [18, 30), or as its number.

One-hot encoding turns a category column into one 0/1 column per category, named like pandas’ get_dummies (city_Delhi); tick Drop the first category’s column for linear models, which would otherwise see the columns always add up to 1. Label codes number the categories 0, 1, 2… in sorted order; ordinal codes follow the order you type, for categories with a natural order such as sizes or ratings. Values a recipe has not seen get 0 in every one-hot column or become missing, and the page lists them.

Reapplying a recipe

A transformation must be fitted once, on the training data, and then applied unchanged to new data; fitting it again on each new file would scale the same value differently. The parameters file keeps your steps with every fitted value; choose Apply saved parameters to use it on another table in this tool. The pandas code does the same in Python: it reads your file with the page’s column names and every cell as text, recognises missing values and numbers as this page does, and applies the fitted values written into it, so a CSV file gives the same values as the page’s download (pandas may write a whole number as 3.0). Steps that could not run on the page are left out of the code, with a comment.

Limitations

• Up to 200,000 rows and 200 columns; very large files can make a phone slow.
• Dates are treated as text; split them into numbers (year, month) first if a model needs them.
• For Excel files the pandas code uses read_excel, which reads the values stored in the cells rather than the text Excel shows: percentages (12% is 0.12 there, 12 here), dates and other formatted numbers can differ. Save the sheet as CSV for an exact match.
• Target encoding and other encodings that use the outcome column are not offered.

Privacy

On this device: nothing is sent

Everything happens in your browser. What you enter or open here is not uploaded or stored by MySmartCoPilot.