Python Pandas - 缺失数据

Pandas - 处理缺失数据

缺失数据在现实世界数据集中是一个普遍存在的挑战。它可能来源于各种原因：数据录入错误、设备故障、调查中的无应答或数据合并问题。有效处理缺失值对于准确的数据分析、可视化和机器学习模型训练至关重要。

Pandas 主要使用特殊的浮点值 NaN (Not a Number，非数字) 来表示缺失数据。对于 object 类型的数据（如字符串），也可能使用 Python 的 None 对象，对于 datetime 类型，NaT (Not a Time，非时间) 表示缺失值。Pandas 提供了旨在跨这些表示形式一致工作的工具。

让我们使用 .reindex 创建一个包含一些缺失值的 DataFrame：

# Import libraries
import pandas as pd
import numpy as np

# Create a base DataFrame
df_base = pd.DataFrame(np.random.randn(5, 3),
                     index=['a', 'c', 'e', 'f', 'h'],
                     columns=['one', 'two', 'three'])

# Reindex to introduce missing rows ('b', 'd', 'g')
df_missing = df_base.reindex(['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h'])

print("DataFrame with Missing Values (NaN):")
print(df_missing)

输出（随机值会不同）：

DataFrame with Missing Values (NaN):
        one       two     three
a  0.077988  0.476149  0.965836
b       NaN       NaN       NaN
c -0.390208 -0.551605 -2.301950
d       NaN       NaN       NaN
e -2.000303 -0.788201  1.510072
f -0.930230 -0.670473  1.146615
g       NaN       NaN       NaN
h  0.085100  0.532791  0.887415

行 ‘b’、‘d’ 和 ‘g’ 不在原始索引中，因此重新索引时用 NaN 填充了它们。

检测缺失值

Pandas 提供了易于检测缺失值的方法。这些方法返回布尔对象（与输入具有相同的形状），在数据缺失的地方指示 True。

• .isna(): 推荐的方法，对于 NaN、None 或 NaT 返回 True。
• .isnull(): .isna() 的别名。
• .notna(): .isna() 的反向操作，对于非缺失值返回 True。
• .notnull(): .notna() 的别名。

示例：使用 .isna()

# (Using df_missing from the previous step)

# Check for missing values in column 'one'
print("Missing values in column 'one' (.isna()):")
print(df_missing['one'].isna())

# Check for non-missing values in the entire DataFrame
print("\nNon-missing values in DataFrame (.notna()):")
print(df_missing.notna())

输出：

Missing values in column 'one' (.isna()):
a    False
b     True
c    False
d     True
e    False
f    False
g     True
h    False
Name: one, dtype: bool

Non-missing values in DataFrame (.notna()):
     one    two  three
a   True   True   True
b  False  False  False
c   True   True   True
d  False  False  False
e   True   True   True
f   True   True   True
g  False  False  False
h   True   True   True

缺失数据的计算

默认情况下，Pandas 中的大多数描述性统计和计算（如 .sum()、.mean()、.std()）会自动排除缺失值（skipna=True 是默认设置）。

# (Using df_missing from the previous step)

# Calculate the sum of column 'one' (NaNs are skipped)
sum_one = df_missing['one'].sum()
print(f"Sum of column 'one' (skipping NaN): {sum_one:.4f}")

# Calculate the mean of column 'two'
mean_two = df_missing['two'].mean()
print(f"Mean of column 'two' (skipping NaN): {mean_two:.4f}")

# Create a column containing only NaNs
df_missing['all_nan'] = np.nan
sum_all_nan = df_missing['all_nan'].sum()
print(f"Sum of a column with only NaNs: {sum_all_nan}")

# Calculate sum including NaNs (treats NaN as 0 if possible, else result is NaN)
# sum_one_with_nan = df_missing['one'].sum(skipna=False)
# print(f"Sum of column 'one' (skipna=False): {sum_one_with_nan}") # This will result in NaN

输出（总和/平均值取决于初始随机数据）：

Sum of column 'one' (skipping NaN): -2.9674
Mean of column 'two' (skipping NaN): -0.1991
Sum of a column with only NaNs: 0.0

注意：如果某个操作的所有数据点（不包括 NaN）都缺失，结果通常是 NaN（例如，全 NaN 列的平均值）。然而，全 NaN 列的 .sum() 结果为 0。设置 skipna=False 会强制包含 NaN，这通常会将 NaN 值传播到计算结果中。

填充缺失数据 (.fillna())

除了忽略缺失值，您通常希望将它们替换为特定值或估计值。.fillna() 方法是实现此功能的主要工具。

用标量值替换 NaN

将 Series 或 DataFrame 中的所有缺失值替换为一个常量值（例如 0、-1，或该列的平均值/中位数）。

import pandas as pd
import numpy as np

df = pd.DataFrame(np.random.randn(4, 3), index=['a', 'c', 'e', 'g'], columns=['one', 'two', 'three'])
df = df.reindex(['a', 'b', 'c', 'd', 'e', 'f', 'g'])

print("Original DataFrame with NaNs:")
print(df)

# Fill all NaNs with 0
df_filled_zero = df.fillna(0)
print("\nDataFrame with NaNs replaced by 0:")
print(df_filled_zero)

# Fill NaNs in column 'one' with the mean of that column
mean_one = df['one'].mean()
df_filled_mean = df.copy() # Work on a copy to keep original df
df_filled_mean['one'] = df_filled_mean['one'].fillna(mean_one)
print(f"\nFilling column 'one' NaNs with its mean ({mean_one:.2f}):")
print(df_filled_mean)

输出（随机值会不同）：

Original DataFrame with NaNs:
        one       two     three
a -0.576991 -0.741695  0.553172
b       NaN       NaN       NaN
c  0.744328 -1.735166  1.749580
d       NaN       NaN       NaN
e -1.234567  0.987654 -0.123456
f       NaN       NaN       NaN
g  0.456789 -0.321098  1.112233

DataFrame with NaNs replaced by 0:
        one       two     three
a -0.576991 -0.741695  0.553172
b  0.000000  0.000000  0.000000
c  0.744328 -1.735166  1.749580
d  0.000000  0.000000  0.000000
e -1.234567  0.987654 -0.123456
f  0.000000  0.000000  0.000000
g  0.456789 -0.321098  1.112233

Filling column 'one' NaNs with its mean (-0.15):
        one       two     three
a -0.576991 -0.741695  0.553172
b -0.152610       NaN       NaN
c  0.744328 -1.735166  1.749580
d -0.152610       NaN       NaN
e -1.234567  0.987654 -0.123456
f -0.152610       NaN       NaN
g  0.456789 -0.321098  1.112233

向前填充 (ffill) 和向后填充 (bfill) NA

这些方法将非缺失值传播以填充空缺。这在时间序列数据中尤其常见。

方法	操作
method='ffill' 或 .ffill()	向前填充：将上一个有效观测值向前传播。
method='bfill' 或 .bfill()	向后填充：将下一个有效观测值向后传播。

# (Using df from the previous example)

# Forward fill NaNs
df_ffill = df.fillna(method='ffill') # or df.ffill()
print("\nDataFrame after forward fill (ffill):")
print(df_ffill)

# Backward fill NaNs
df_bfill = df.fillna(method='bfill') # or df.bfill()
print("\nDataFrame after backward fill (bfill):")
print(df_bfill)

输出：

DataFrame after forward fill (ffill):
        one       two     three
a -0.576991 -0.741695  0.553172
b -0.576991 -0.741695  0.553172 # Filled from 'a'
c  0.744328 -1.735166  1.749580
d  0.744328 -1.735166  1.749580 # Filled from 'c'
e -1.234567  0.987654 -0.123456
f -1.234567  0.987654 -0.123456 # Filled from 'e'
g  0.456789 -0.321098  1.112233

DataFrame after backward fill (bfill):
        one       two     three
a -0.576991 -0.741695  0.553172
b  0.744328 -1.735166  1.749580 # Filled from 'c'
c  0.744328 -1.735166  1.749580
d -1.234567  0.987654 -0.123456 # Filled from 'e'
e -1.234567  0.987654 -0.123456
f  0.456789 -0.321098  1.112233 # Filled from 'g'
g  0.456789 -0.321098  1.112233

删除缺失值 (.dropna())

如果填充缺失值不合适，您可能选择删除包含它们的行或列，使用 .dropna() 方法。

dropna() 的关键参数：

• axis：0 或 'index' 用于删除行（默认值），1 或 'columns' 用于删除列。
• how：'any'（默认值）表示如果存在至少一个 NA 则删除，'all' 表示仅在所有值都是 NA 时才删除。
• thresh：整数值，指定保留行/列所需的非 NA 值的最小数量。
• subset：查找 NaN 时要考虑的列/索引标签列表。

示例：删除包含任何 NaN 的行

# (Using df_missing created earlier)
print("Original DataFrame with NaNs:")
print(df_missing)

# Drop rows containing any NaN values (default behavior)
df_dropped_rows = df_missing.dropna() # axis=0, how='any' are defaults

print("\nDataFrame after dropping rows with any NaNs:")
print(df_dropped_rows)

输出：

Original DataFrame with NaNs:
        one       two     three  all_nan
a  0.077988  0.476149  0.965836      NaN
b       NaN       NaN       NaN      NaN
c -0.390208 -0.551605 -2.301950      NaN
d       NaN       NaN       NaN      NaN
e -2.000303 -0.788201  1.510072      NaN
f -0.930230 -0.670473  1.146615      NaN
g       NaN       NaN       NaN      NaN
h  0.085100  0.532791  0.887415      NaN

DataFrame after dropping rows with any NaNs:
Empty DataFrame
Columns: [one, two, three, all_nan]
Index: []

由于 df_missing 中的每一行都包含至少一个 NaN（特别是我们添加的 all_nan 列），使用默认设置的 dropna() 删除了所有行。

示例：删除包含任何 NaN 的列

# (Using df_missing created earlier)

# Drop columns containing any NaN values
df_dropped_cols = df_missing.dropna(axis=1, how='any')

print("\nDataFrame after dropping columns with any NaNs:")
print(df_dropped_cols)

输出：

DataFrame after dropping columns with any NaNs:
Empty DataFrame
Columns: []
Index: [a, b, c, d, e, f, g, h]

同样，由于所有列都包含至少一个 NaN，所有列都被删除了。

示例：使用 thresh

# Use the original df_missing before adding 'all_nan'
df_original_missing = df_base.reindex(['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h'])
print("Original df_missing (before adding all_nan column):")
print(df_original_missing)

# Keep rows that have at least 2 non-NaN values
df_thresh = df_original_missing.dropna(thresh=2)
print("\nDataFrame keeping rows with at least 2 non-NaN values:")
print(df_thresh)

输出：

Original df_missing (before adding all_nan column):
        one       two     three
a  0.077988  0.476149  0.965836
b       NaN       NaN       NaN
c -0.390208 -0.551605 -2.301950
d       NaN       NaN       NaN
e -2.000303 -0.788201  1.510072
f -0.930230 -0.670473  1.146615
g       NaN       NaN       NaN
h  0.085100  0.532791  0.887415

DataFrame keeping rows with at least 2 non-NaN values:
        one       two     three
a  0.077988  0.476149  0.965836
c -0.390208 -0.551605 -2.301950
e -2.000303 -0.788201  1.510072
f -0.930230 -0.670473  1.146615
h  0.085100  0.532791  0.887415

行 ‘b’、‘d’、‘g’ 被删除了，因为它们包含 0 个非 NaN 值，小于设定的阈值 2。

替换通用值 (.replace())

.replace() 方法比 fillna() 更通用。它允许将任何指定的值（不仅是 NaN）替换为另一个值。这对于更正数据录入错误、映射代码或替换特定占位符很有用。

使用 replace(np.nan, value) 将 NaN 替换为标量值与 fillna(value) 类似。

示例：替换特定值

import pandas as pd
import numpy as np

df = pd.DataFrame({'one': [10, 20, 30, 40, 50, 999], # 999 might be a placeholder for missing
                   'two': [1000, 0, 30, 40, 50, 60]})

print("Original DataFrame:")
print(df)

# Replace the placeholder 999 with NaN
df_cleaned = df.replace({999: np.nan})

# Replace 1000 with 10, and 40 with 400 across the DataFrame
df_replaced = df_cleaned.replace({1000: 10, 40: 400})

print("\nDataFrame after replacing 999 with NaN:")
print(df_cleaned)

print("\nDataFrame after replacing 1000->10 and 40->400:")
print(df_replaced)

输出：

Original DataFrame:
   one   two
0   10  1000
1   20     0
2   30    30
3   40    40
4   50    50
5  999    60

DataFrame after replacing 999 with NaN:
    one   two
0  10.0  1000
1  20.0     0
2  30.0    30
3  40.0    40
4  50.0    50
5   NaN    60

DataFrame after replacing 1000->10 and 40->400:
    one   two
0  10.0    10 # 1000 replaced
1  20.0     0
2  30.0    30
3 400.0   400 # 40 replaced in both columns
4  50.0    50
5   NaN    60

您可以提供字典一次替换多个值，或者使用更复杂的参数仅替换特定列中的值。有关高级用法，请参阅 .replace() 文档。