sodcalculator

10 Tech How-To

Handling Missing Data in Python: A Practical Guide

Guide 10-47580 · Updated · Step by step

Handling missing data in Python means identifying and dealing with NaN (Not a Number) or None values in your dataset, typically using the pandas library. You can either remove the rows or columns containing missing values, or fill them with a computed value like the mean, median, or a forward fill. The right approach depends on your data, the amount missing, and what you plan to do with the analysis.

What Causes Missing Data?

Missing values rarely appear randomly. Understanding why they are missing helps you choose the correct handling strategy. Common causes include:

  • Data entry errors: A sensor fails, a human skips a field, or a system glitch drops a record.
  • Non-response: In surveys, people may refuse to answer certain questions, like income or age.
  • Merging datasets: When you join two tables, rows that don’t match on the key will have NaN in the non-matching columns.
  • Feature creation: New columns derived from existing ones may be undefined for some rows (e.g., log of a negative number).

Statisticians classify missingness into three types: Missing Completely at Random (MCAR), where the missingness has no relationship to any variable; Missing at Random (MAR), where it depends on observed variables (e.g., older people more likely to skip a question); and Missing Not at Random (MNAR), where the missingness is related to the value itself (e.g., people with very high income don’t report it). Your handling method should be more careful for MNAR data, as simple deletion or imputation can introduce bias.

How to Detect Missing Values in pandas

Before you can handle missing data, you need to find it. In pandas, the primary functions are isna() and isnull() (they are identical). Here is a typical workflow:

  1. Load your data: df = pd.read_csv('data.csv')
  2. Get a quick overview: df.info() shows the count of non-null values per column.
  3. Count missing per column: df.isna().sum()
  4. Visualize missingness: import missingno as msno; msno.matrix(df) gives a quick visual of where gaps are.
  5. Check the percentage: df.isna().mean() * 100

Also be aware that missing values can be represented by placeholders like 0, -999, or an empty string ''. You may need to replace those with np.nan first: df.replace({'': np.nan, -999: np.nan}, inplace=True).

Removing Missing Data: dropna()

The simplest approach is to delete rows or columns that contain missing values. Use df.dropna() to drop rows with any NaN. You can refine this:

  • df.dropna(axis=1) drops columns with any missing values.
  • df.dropna(how='all') drops rows where all values are missing.
  • df.dropna(thresh=5) keeps rows with at least 5 non-null values.
  • df.dropna(subset=['age', 'income']) only considers those columns when deciding to drop.

Dropping is fast and keeps your data unchanged, but it can be wasteful. If you drop 30% of your rows, you lose information that might be valuable. It is generally acceptable when the missing proportion is small (e.g., under 5%) and the missingness is MCAR. For time series, dropping rows can break the temporal order, so be cautious.

Filling Missing Data: fillna() and Interpolation

Instead of dropping, you can fill missing values with a computed statistic. The most common methods in pandas are:

  • Constant fill: df.fillna(0) or df.fillna('Unknown') – useful for categorical data.
  • Mean/median/mode: df['age'].fillna(df['age'].mean()) – simple but reduces variance. Median is better for skewed data.
  • Forward fill: df.fillna(method='ffill') – propagates the last valid value forward. Great for time series where a sensor reading stays valid until the next one.
  • Backward fill: df.fillna(method='bfill') – uses the next valid value.
  • Interpolation: df.interpolate() – estimates missing values by linear interpolation between surrounding points. For time series, use df.interpolate(method='time').

For example, if you have a column of temperatures with a gap, df['temp'].interpolate() will fill the gap with a straight line between the last known and next known values. This is more sophisticated than a constant fill and often works well for smooth trends.

Choosing the Right Strategy

There is no one-size-fits-all answer. Here is a decision guide:

  1. Check the proportion: If less than 1-2% of a column is missing, dropping those rows is usually safe. If more than 50% of a column is missing, consider dropping the column entirely.
  2. Check the type of data: For numerical data, mean/median or interpolation. For categorical data, use the mode or a new category like ‘Missing’.
  3. Check the purpose: If you are building a machine learning model, many algorithms (e.g., scikit-learn’s) cannot handle NaN. You must impute. For a simple report, dropping may be fine.
  4. Check the pattern: If missing values are clustered in a specific time period, forward fill may be misleading. If they are random, mean imputation is less harmful.

A common misconception is that you should always fill with the mean. That is not true. Mean imputation reduces the variance of the column, which can distort correlations and make p-values unreliable. It is a quick fix, but for serious analysis, consider more advanced methods like SimpleImputer or KNNImputer from scikit-learn, which use other features to predict missing values.

Advanced Imputation with scikit-learn

When you need a more robust imputation, scikit-learn provides several transformers. They must be used with care: fit them on the training set only, then transform the test set to avoid data leakage.

  • SimpleImputer – similar to pandas fillna but with a consistent API. You can set strategy to ‘mean’, ‘median’, ‘most_frequent’, or ‘constant’.
  • KNNImputer – finds the k nearest rows (based on other columns) and averages their values to fill the gap. This works well when features are correlated.
  • IterativeImputer – models each column with missing values as a function of other columns, iteratively. It is more powerful but slower.

Example using KNNImputer:

from sklearn.impute import KNNImputer
imputer = KNNImputer(n_neighbors=3)
df_imputed = imputer.fit_transform(df[['age', 'income']])

Remember that imputation is not a silver bullet. If the missingness is MNAR, no imputation method can recover the true values. In such cases, you may need to model the missingness itself or collect more data.

Common Pitfalls and Mistakes

Here are frequent errors people make when handling missing data:

  • Ignoring the dtype: Sometimes a column with missing numbers is read as object (string) because of a stray character. Convert it to numeric first: pd.to_numeric(df['col'], errors='coerce').
  • Using fillna(0) blindly: Zero is not a neutral value. It can bias statistics and models. Only use 0 if it is a meaningful value (e.g., count of purchases).
  • Not checking the result: After filling, verify that no NaN remains: df.isna().sum().sum().
  • Filling before splitting: If you compute the mean on the whole dataset and then split into train/test, you leak information from the test set into the training set. Always fit imputers on the training set only.
  • Forgetting that None and NaN are different: In pandas, None is often converted to NaN automatically, but in object columns, you may need to handle both.

Frequently Asked Questions

What is the difference between isna() and isnull()?

There is no difference. They are aliases in pandas. Both return a boolean DataFrame where True indicates a missing value. You can use either; isna() is more consistent with the pandas API naming.

Should I drop rows or fill missing values?

It depends on the amount and pattern. If the missing values are few (under 5%) and random, dropping is simple and safe. If they are more substantial, filling is better to retain data. For time series, filling with interpolation or forward fill is usually preferred because dropping breaks the sequence.

How do I fill missing values with the mean of a group?

Use groupby() with transform(). For example, to fill missing age with the mean age per gender: df['age'] = df['age'].fillna(df.groupby('gender')['age'].transform('mean')). This preserves group-specific differences.

Can I use machine learning to predict missing values?

Yes, that is what IterativeImputer or KNNImputer do. You can also build your own regression model for each column with missing data. However, this is only useful if other columns are good predictors of the missing column. It adds complexity, so start with simpler methods unless you have a strong reason.

What does inplace=True do?

It modifies the DataFrame in place without returning a new object. For example, df.dropna(inplace=True) changes df directly. Modern pandas versions prefer not using inplace and instead reassigning: df = df.dropna(). The latter is clearer and avoids the common mistake of forgetting to assign the result.

How do I handle missing data in a time series?

For time series, use df.interpolate(method='time') to fill gaps based on the time index. Forward fill (ffill) is also common when the value is constant until updated. Avoid mean imputation because it ignores the temporal order and can create unrealistic jumps.

Related

General steps for current mainstream versions; menus can differ by device and version. See the disclaimer or report an error.

More in this bed

The whole Tech How-To bed →