1,November,2023

Handling Anomalies and Missing Data in Datasets

---

1. Anomalies (Outliers):

Definition: Data points that differ significantly from other observations in the dataset.

Detection:

• Visual Inspection: Scatter plots, Box plots.
• Statistical Tests: Z-score, IQR.

Handling Techniques: a. Deletion: Remove outlier data points.

• Pros: Quick and simple.
• Cons: May lose valuable information.

b. Transformation: Apply log or square root transformations to reduce variance.

c. Capping: Cap the outlier to a maximum/minimum value.

d. Imputation: Replace outliers with statistical measures such as mean, median, or mode.

e. Binning: Convert numerical variable into categorical bins.

---

2. Missing Data:

Types of Missingness: a. MCAR (Missing Completely At Random): Missingness is not related to any other variable. b. MAR (Missing At Random): Missingness is related to some other observed variable. c. MNAR (Missing Not At Random): Missingness is related to the missing data itself.

Detection:

• Use libraries like pandas (e.g., dataframe.isnull().sum()) or visualization tools like missingno.

Handling Techniques: a. Listwise Deletion: Remove any row with a missing value.

• Pros: Simple.
• Cons: Risk of losing a lot of data.

b. Pairwise Deletion: Use available data for statistical analysis.

c. Mean/Median/Mode Imputation: Fill missing values with mean, median, or mode of the column.

• Good for MCAR.

d. Forward/Backward Fill: Use the previous or next data point to fill missing values. Useful for time series data.

e. Model-Based Imputation: Use regression, KNN, or other models to estimate and impute missing values.

f. Multiple Imputation: Generate multiple predictions for every missing value.

g. Use Algorithms Robust to Missing Values: Some algorithms (like XGBoost) can handle missing values.

---

General Recommendations:

1. Understand Your Data: Always explore and visualize your data before handling anomalies or missing values.
2. Consider Data’s Context: Understand the potential real-world implications of removing or imputing data.
3. Validate: After handling anomalies and missing values, validate the results using appropriate statistical tests or performance me