Solved – Which is better, replacement by mean and replacement by median

data-imputationmeanmedian

I'm doing a project that involves replacing missing values in a set of data (first time doing this). This involves using two methods replacement by mean and replacement by median to fill in the missing values. There is not a lot of difference between the results of the minimum, median, maximum, mean and standard deviation of the data using both methods and I was wondering which method is better and how can I make a decision to which one is better using the results produced?

Best Answer

It always depends on your data and your task.

If there is a dataset that have great outliers, I'll prefer median. E.x.: 99% of household income is below 100, and 1% is above 500.

On the other hand, if we work with wear of clothes that customers give to dry-cleaner (assuming that dry-cleaners' operators fill this field intuitively), I'll fill missings with mean value of wear.

It is better to start from data understanding and then this article will be helpful starting point.

Related Solutions

Solved – What are the pros and cons of using median imputation to handle missing value

These are not appropriate for computing missing data - consider the case of heteroskedasticity in the data - neither of these approaches would work if their were 'weird' or idiosyncratic values in your data. In fact it would be more damaging (ie less accurate) to use mean or median replacement in this case

if youre familiar with R, you could check out the MI package (my fave) or mice. This essentially runs a series of chained (ie bayesian) regressions on the data until some convergence criteria

other options are expectation maximization (subject to overfitting problems IMO) and Hotdeck imputation

check out these resources for more explanation about why mean/median replacement is generally a bad idea

Rubin, D. B. (1996). Multiple imputation after 18+ years. Journal of the American Statistical Association, 91(434):473–489.

Schafer, J. L. (1999). Multiple imputation: a primer. Statistical Methods in Medical Research, 8:3–15.

Solved – How to calculate mean and standard deviation from median and quartiles

You can check Wan et al. (2014)^*. They build on Bland (2014) to estimate these parameters according to the data summaries available. See scenario C₃ in their paper :

$$ \bar{X} ≈ \frac {q_{1} + m + q_{3}}{3}$$

$$ S ≈ \frac {q_{3} - q_{1}}{1.35}$$

or, if you have the sample size :

$$ S ≈ \frac {q_{3} - q_{1}}{2 \Phi^{-1}(\frac{0.75n-0.125}{n+0.25}) }$$

where $q_{1}$ is the first quartile, $m$ the median, $q_{3}$ is the 3^rd quartile and $\Phi^{-1}(z)$ the upper z^th percentile of the standard normal distribution.

So, in R :

q1 <- 0.02
q3 <- 0.04
n <- 100

(s <- (q3 - q1) / (2 * (qnorm((0.75 * n - 0.125) / (n + 0.25)))))
#[1] 0.0150441

^{* Wan, Xiang, Wenqian Wang, Jiming Liu, and Tiejun Tong. 2014. “Estimating the Sample Mean and Standard Deviation from the Sample Size, Median, Range And/or Interquartile Range.” BMC Medical Research Methodology 14 (135). doi:10.1186/1471-2288-14-135.}

Best Answer

Related Solutions

Solved – What are the pros and cons of using median imputation to handle missing value

Solved – How to calculate mean and standard deviation from median and quartiles

Related Question