Solved – Modify standard deviation with fixed mean

meanoutliersstandard deviationvariance

I have a set of pipes that are supposed to receive balanced load from a source. For instance, with 100 of inputs if I have 5 pipes each is supposed to handle 20. But that's not the case in reality and normally it's like:

[23, 26, 19, 18, 14]

I'm working on a system to trigger an alarm if one of the pipes are underloaded or overloaded. So clearly I'm looking for outliers. I found that standard deviation would be good fit for detecting that but I wonder that the mean of random numbers might skew the expected value (here 20).

So the question is, if instead of the mean I use 20 and calculate the SD based on that, how correct the final result will be? Or am I completely confused?

Few cases that might trigger the alarm:

[20, 20, 20, 3, 37]
[20, 0, 20, 20, 40]

Best Answer

To answer the question itself:

So the question is, if instead of the mean I use 20 and calculate the SD based on that, how correct the final result will be?

Since the total flow is always 100, the sample mean will always be 20. As a result, if you do the same standard deviation calculation it will make no difference whether you use the known mean or the sample mean.

What might change is the denominator (if you cared about an unbiased-for-variance estimate for some reason), but if you're just comparing with some yet-to-be-determined cut-off there's no particular reason to bother about that. Even if you did divide by $n=5$ rather than $n-1=4$ to make the variance unbiased, all you'd need to do is adjust the cut-off point the same way and performance would again be identical either way. The numerator of the variance is the thing that is putting a partial order on sets of pipe-loads, the rest is really just scaling. Indeed since the mean is fixed one could just as well simply use the sum of squares of the loads as a criterion (that statistic has been used in similar contexts where uniformity is an issue).

I think this particular issue is about as not-really-an-issue as they come. If the standard deviation really does what you want, just think of it as whichever of the two calculations is easiest for you to think about, since it makes no difference. Once you're sure sd does what you want, the next trick is to figure out a suitable cutoff value for your alarm.

Related Solutions

Solved – Best way to display data with outliers

The standard traditional tool is a histogram. You can do this with the analysis tool pack in Excel, but I'd recommend using a stats package instead.

An extension of the histogram is a line plot showing the density - this is basically your idea of shwoing the bell curve, and it is probably the right one. From here there are various options such as drawing vertical lines to show the mean, median, 95th percentile, etc. To do this you will definitely want a stats package.

Some examples are below, including the code in R (which is free) that generated the data and drew the plots. You can see it that's not necessarily that hard to do this sort of thing in a stats package, if you're prepared to move beyond Excel.

# generate data
times <- rgamma(1000,1,1)

# draw histogram, showing counts
hist(times, col="grey")

# draw a density line plot
plot(density(times), bty="l")

# add vertical lines for the median and 95th percentile
abline(v=quantile(times, c(0.5, 0.95)), lty=2:3)

enter image description here

Solved – Creating alert system based on standard deviation and mean: How to deal with small sample size, trends and seasonality

You seem a bit confused about the differences between type I and type II errors.

First off, a critical value of +/- 2 stdevs means that (for normal data), you will be sending false positive alerts with a 1/20 chance. Isn't that going to be annoying considering that, over a year, you'd expect to get an average of 2.6 incorrect weekly warnings per year when the data are perfectly regular? I'd consider specifying a more conservative false positive error rate.

When you take too few observations, your estimate of the variance is not "too high", it's just unstable (unless, you mean to say that you're using the standard error of the mean of the data and not the standard deviation). "Unstable" means it [the SD estimate] could be too high, it could be too low. Even if the estimated standard deviation (or standard error) were too large, you would fire alerts not often enough (conservative) and not too often (anticonservative).

The real question here is whether or not "unusual" in this circumstance means "data are consistent with what the data usually are" or "data are consistent with some pre-identified target value". The latter warrants use of a control chart which is an effective graphical tool for displaying temporal variability and trends in a controlled data process. The former is a time when one can use an ARMA (autoregressive moving average) model to roll forward previously observed data and infer whether present values either are or are not consistent with the historical values. The asymptotic theory of these models easily extend a decision rule about sending alerts based on the normal distributions of model parameters.

If certain days are heterogeneous, you will want to turn off your alert system because it makes sense to do so.

With small sample sizes, you should consider parametric probability models for website hits. Instead of estimating an independent mean/variance process, you should consider some counting process models like poisson or negative binomial for the mean/variance process. That way, you would need fewer observations to infer whether a specific day was abnormally high/low in hit rate. These, of course, have a lot more assumptions embedded in them.

Best Answer

Related Solutions

Solved – Best way to display data with outliers

Solved – Creating alert system based on standard deviation and mean: How to deal with small sample size, trends and seasonality

Related Question