Solved – Fitting a smoothed curve to a noisy data

curve fittingloessmoving averagesmoothingstata

I have a variable with sales data over time. It is very noisy at a disaggregate level but if you look at it as a whole, you can see a smoothing curve that follows a polynomial pattern. Is there a way to fit such curve and actually generate values of it over time in the dataset?

I tried many different options here including a moving average (which leaves a lot of noise still); I tried lowess smoother in Stata, which is on the opposite side – too neutral and leaves out important curve patterns. I also tried fitting a nonlinear function nl but to no avail.

The original goal of this task is to determine any infection points that the curve can have. In my mind, I am looking for a curve that would run in the middle of all the noise.

Best Answer

Since you mention the "polynomial pattern" in your question, try to fit your data using polynomial least squares fitting.

I tried to reproduce your data (more or less) and plotted a third degree least squares fit on the data. The result is in the graph below.

Actually, I used two goniometric functions to generate the data. The period and amplitude of the base cycle can be estimated by the fitted polynomial: half the period is the distance between the two extremes. The amplitude is half of the difference between the two extremes.

The advantage of this approach is that is it very easy to understand, calculate and apply (maybe at the cost of some mathematical inaccuracy).

There are two disadvantages of using a moving average over any fitting approach:

the moving average is always lagging. This results in shift in the direction of the past data (in the example below: the moving average is above the polynomial fit line; the trend is going down, so the moving average is lagging upwards in this case).
the moving average moves along with any spike in the data. This results in a more "wobbely" line compared to the fitting approach.

These effects can be clearly seen in the picture below (red = polynomial fitting; black = 20 period moving average).

Related Solutions

Solved – Smoothing 2D data

You need a to specify a model that separates the signal from the noise.

There is the component of noise at the measurement level that you assume gaussian. The other components, dependent across measurements:

"This noise, for a fixed frequency, can probably be modelled as random with gaussian distribution". Needs clarification — is the noise component common to all timepoints, given the frequency? Is the standard deviation same for all frequencies? Etc.
"At a fixed time, however, the data shows a different kind of noise, with large spurious spikes and fast oscillations" How do you separate that from the signal, for assumably you are interested on variation of the intensity across the frequency. Is the interesting variation somehow different from the uninteresting variation, and if so, how?

Spurious oscillatioins or non-gaussian noise in general is not a big problem, if you have a realistic idea of its characteristics. It can be modeled by transforming the data (and then using a gaussian model) or by explicitly using a non-gaussian error distribution. Modeling noise that is correlated over measurements is more challenging.

Depending on how your noise and data model are, you might be able to model the data with a general-purpose tool like the GAMs in the mgcv package, or you may need a more flexible tool, which easily leads to a quite customized bayesian setup. There are tools for such models, but if you are not a statistician, learning to use them will take a while.

I guess either a solution specific to spectral analysis or the mgcv package are your best bets.

R – Finding Inflection Points from Smoothed Data

From the perspective of using R to find the inflections in the smoothed curve, you just need to find those places in the smoothed y values where the change in y switches sign.

infl <- c(FALSE, diff(diff(out)>0)!=0)

Then you can add points to the graph where these inflections occur.

points(xl[infl ], out[infl ], col="blue")

From the perspective of finding statistically meaningful inflection points, I agree with @nico that you should look into change-point analysis, sometimes also referred to as segmented regression.

Best Answer

Related Solutions

Solved – Smoothing 2D data

R – Finding Inflection Points from Smoothed Data

Related Question