Solved – How to combine variances from sensors where each observation has its own variance

estimationnormal distributionsamplevariance

I have a set of measurements $x_1$ … $x_n$. These measurements are normally distributed, measuring the same value. However due to the way the data is measured, each $x$ has its own standard deviation: $s_1$ … $s_n$. In other words I have a sensor which returns pairs (x,s).
Now, I wan to estimate parameters of distribution of $x$ using $\text{N}$ samples. The common mean would be just a sample mean. What about variance? I could not apply "pooled variance" formula because I do not know how many samples were used to estimate each $s$.
Can I just use a mean of $s$ as sample variance?

Update: I can not make any assumptions on how sensor produced (x,s) values. They might be based on some hidden iterations or perhaps something else. However it is safe to assume that each (x,s) pair returned to me is independent from others and measuring the same true value of 'x'.

Best Answer

You can use the mean of s^2 as sample variance. For sample SD you'd take the square root of that. You cannot average the standard deviations together.

While it's nice to be able to have a weighted mean by the number of samples that went into each individual variance measurement, if those aren't available then this is your next best estimate method.

Related Solutions

Solved – How to compare observed distribution to sampled distributions, using cumulative distibution function

You can use the ecdf, quantile and rank functions to do this.

# Sample data
n <- 10
m <- 50
observed <- rnorm(m)
set1 <- matrix(rnorm(n*m),nc=n)
set2 <- matrix(rnorm(n*m),nc=n)

# Average cumulated distribution function of the first set of samples
ecdfs <- apply(set1, 2, ecdf)
acdf <- function(u) mean(sapply(ecdfs, function(f) f(u)))
acdf <- Vectorize(acdf)

# 95% envelope
alpha <- .025
x <- unique(sort(set1))
y <- sapply(ecdfs, function(f) f(x))
y <- apply(y, 1, quantile, c(alpha, 1-alpha))
lower <- approxfun( x, y[1,], method="constant", rule=2 )
upper <- approxfun( x, y[2,], method="constant", rule=2 )
curve(upper(x), xlim=c(-3,3))
curve(lower(x), add=TRUE)
curve(acdf(x), add=TRUE, lty=3)

# Rank of the observed sample 
# among the maximum differences
difference <- function(u) {
  max( abs( ecdf(u)(c(x,u)) - acdf(c(x,u)) ) )
}
sdi <- apply(set2, 2, difference)
sdi <- c( difference(observed), sdi )
sdi <- rank(sdi)[1]

# If you were to write a function to do this,
# it would return the following
list(
  acdf  = acdf,
  upper = upper,
  lower = lower, 
  sdi = sdi
)

Variance – What Does Pooled Variance Actually Mean?

Put simply, the pooled variance is an (unbiased) estimate of the variance within each sample, under the assumption/constraint that those variances are equal.

This is explained, motivated, and analyzed in some detail in the Wikipedia entry for pooled variance.

It does not estimate the variance of a new "meta-sample" formed by concatenating the two individual samples, like you supposed. As you have already discovered, estimating that requires a completely different formula.

Best Answer

Related Solutions

Solved – How to compare observed distribution to sampled distributions, using cumulative distibution function

Variance – What Does Pooled Variance Actually Mean?

Related Question