Solved – Explain Root Mean Square Error to non-technical audience

intuitionrmsteaching

My company is in the process of switching equipment from one vendor to another. We measured several metrics from the existing and new equipment and compared the time series. The ideal is to have no difference between the old and new equipment. For that, we use Root Mean Square Error (RMSE) to measure the difference.

We are to present the findings to upper management and employees at the plant, many of whom have no background in statistics. How should I explain the concept of RMSE to those audiences? If series X and Y have an RMSE of 5, how should I explain that 5 in layman's terms?

Best Answer

Starting with the most general, and then with increasing details:

RMSE is a measure of error (or in your case, difference between two equipments.)
The unit of RMSE is the same as the unit of the metric used. More details in The unit of Root Mean Square Error (RMSE)
In estimation problems, RMSE corresponds to the standard error of an estimator.
You didn't give details of your comparison procedure, but the explanation of RMSE should be tied to the details of the comparison. So in the following points some possibilities.
Say you have measurements with one equipment, when you compare to a gold standard, say specimenes with very precise measurements taken as the truth (and the lab didn't know the thruth.) Then the RMSE is $\sqrt{ \frac1{n}\sum_{i=1}^n (x_i-\text{gold}_i)^2 }$ so is the standard deviation of the measurements errors. So, if the measurement errors $x_i-\text{gold}_i$ is normalishly distributed, around 95% of the measurement errors will be in $\pm 2\cdot\text{RMSE}$. You should also show some plots with your actual data. Say the equipment measures some length in mm, then your example value of 5 means a typical error of around 5 mm. (With a gold standard, your measurements can also be used to see if the measurements are typically to low or too high, or the bias. The RMSE does not measure bias alone, but bias is a component of it. See Bias-Variance decomposition derivation.)
Without a gold standard, say you have two series of measurements, using the two equipments on the same specimenes. Then the RMSE is $ \sqrt{ \frac1{n}\sum_{i=1}^n (x_i-y_i)^2 } $ is a measure of the typical difference between the measurement results. It should be given together with a plot of the differences. Again, the squaring forgets about the sign of the difference, which could be important too.

Searching the web for information, in this case can be difficult, as RMSE also is an important concept in electricity, and most of the hits will reflect that! Some other posts here that can be useful is Forecast accuracy calculation and What is the difference between Mean Squared Deviation and Variance?, see also Wikipedia.

Related Solutions

Solved – Average of root mean square error

I actually wasn't sure about this either so I tested it out with a short example:

## Create simple function to calcualte the error
rmse <- function(error){sqrt(mean(error^2))}

## Define two example error vectors
error1 <- c(0.4, 0.2, 0.01)
error2 <- c(0.1, 0.3, 0.79)

## Find the RMSE of each error vector
rmse1 <- rmse(error1)
rmse2 <- rmse(error2)

## Compare the RMSE variants
print(rmse_all <- rmse(c(error1, error2)))
[1] 0.3924708
print(rmse_avg <- mean(rmse1, rmse2))
[1] 0.2582634

So we can se that they are not equal.

## As described by @whuber in the comments:

a <- rmse1^2*length(rmse1)                 # - square each rmse & multiply
b <- rmse2^2*length(rmse2)                 #   it by its associated count
c <- sum(a, b)                             # - sum that stuff up,                  
d <- c/sum(length(rmse1), length(rmse2))   # - divide by the total count,          
print(total_rmse <- sqrt(d))               # - take the square root.
[1] 0.3924708

Solved – Weighted Root Mean Square Error

As already noticed by whuber in a comment, it is not clear if your procedure of setting weights is valid. Notice that in non-weighted RMSE larger areas already have greater weight on the estimate since they are larger, so they appear more often in your data. That is why, as suggested, people rather down-weight such subpopulations, so that the final estimate treats all the subpopulations more evenly.

However if you wanted to use weighted RMSE, then recall that RMSE is by design pretty close to standard deviation, so why not look at how weighted variance is calculated?

$$ \sigma^2 = \sum_{i=1}^n w_i (x_i - \bar x)^2 $$

where weights are non-negative and $\sum_{i=1}^n w_i = 1$. The same you can take weighted RMSE as

$$ \text{RMSE} = \sqrt{\sum_{i=1}^n w_i (\hat x_i - x_i)^2} $$

Notice that we take sum of weighted differences, not the mean. Unweighted mean is the same as weighted mean with weights that are all equal to $w_i = 1/n$, so if you took arithmetic mean, it would be like dividing RMSE by $n$ second time.

Check also:
Weighted Variance, one more time

Best Answer

Related Solutions

Solved – Average of root mean square error

Solved – Weighted Root Mean Square Error

Related Question