Data Analysis – Determining Significant Figures in Least Squares Analyses

Question

I'm tasked with fitting a set of data to a line using Excel's $\text{LINEST}$ least-squares fitting routine. The data are $l$ and $m$ values, assumed to follow the relationship:

$$l = l_0 + {g\over k}m$$

…where $g$, $k$, and $l_0$ are some constants

Using $\text{LINEST}$, I obtain estimated values for $g/k$ (the slope) and $l_0$ (the $y$-intercept), reported to some absurd number of significant figures, in addition to uncertainties in both of those values as well as the $m$ values.

Let's suppose that $l_0$ is found to be $3.6857142… \pm 0.222953…$, $g/k$ to be $0.00607143… \pm 0.003742…$, and the uncertainty in the $m$ values to be $\pm 0.24250267…$, where the $m$ values are reported to $1$ significant figure (e.g. $500$).

To how many significant figures to I round each of these values?

---

Across undergraduate physics courses, I've been told many contradictory things about rounding and to how many significant figures one should report a measurement.

As I understand it, one would round the uncertainty in $l_0$ to just one significant figure (i.e. to $0.2$), then round $l_0$ to the same decimal place as this uncertainty (i.e. to $3.7$), and likewise for $g/k$. I'm not sure about the uncertainty in $m$, however. Should it be rounded to one significant figure, as well?

I've also been told, however, that the reverse is true (i.e. that one should first round the measurement/estimated value to a single significant figure, then round the uncertainty accordingly).

Which, if any, is most appropriate? Moreover, why?

Answer 1

The most authoritative source for how to estimate and report measurement uncertainty is the BIPM’s publication: Evaluation of measurement data — Guide to the expression of uncertainty in measurement. In section 7.2.6 it says

The numerical values of the estimate y and its standard uncertainty uc(y) or expanded uncertainty U should not be given with an excessive number of digits. It usually suffices to quote uc(y) and U [as well as the standard uncertainties u(xi) of the input estimates xi] to at most two significant digits, although in some cases it may be necessary to retain additional digits to avoid round-off errors in subsequent calculations.

(emphasis added)

So the recommendation is to typically use two digits to report your uncertainty. Then, it doesn’t make any sense to report your measurement to any higher precision. Thus your $l_0$ would be reported as 3.69(22)

A single digit for the uncertainty would be a little too imprecise. If you reported it as 3.7(2) then you would have to change your precision by 50% to have it change your report. That is too coarse.

Anyway you definitely decide on the number of digits to report for your error first, then that determines how many digits to keep for the measurement.