I have a noisy time-series data (Figure 1). As you can see the variance in this data set is very high and the "Gaussian noise" needs to be removed for me to analyze this signal. Normally we apply a median filter (I have also tried moving average and Savitsky Golay) to this dataset but that only removes some of the noise. None of these filters are able to provide me with just a smoothed-out shape of this curve (which is what I want in order to calculate parameters such as velocity and determine steps).
So I thought one thing I could try is to run a sliding window through this dataset, determine the variance in each window, and then remove the known variance. The point where I'm getting stuck is – how do you subtract variance from these points.
Quick note: I have also tried fitting this data set with a smoothing spline but since each dataset has different variance (gaussian noise), the smoothing spline has to identify the variance and then smooth over the subsequent number of points.
I greatly appreciate your help.
Best Answer
One doesn't subtract variance from a time series .. one adjusts ...http://docplayer.net/12080848-Outliers-level-shifts-and-variance-changes-in-time-series.html . The idea of time series analysis is to separate signal from noise ...Identify an appropriate arima model with any and all necessary deterministic structures e,g, pulses/seasonal pulse / time trends and determine if there is any remaining structure suggesting the need to stabilize the error variance.