Calculate average on set of data points where recent data has more weight

averagedata analysis

Take for example this dataset:

[1,1,1,1,1,1,1,1,1,200,1,1,1,1,....,1]

I want to calculate a running average on this with a certain window size for avg, let's say 10 most recent data point.

A regular average with a window of 10 (or whatever) would look something like this:

The bump is when the 200 enters the calculation and the cliff is when it leaves the window.

I want it to look like this:

I believe this means I need to give more weight to recent data to reduce the impact of the 200 point bump over time.

What is a good way to do this? I'm looking for pointers in the right direction.

Thanks!

Best Answer

What you might want to use as your weight given your graph is using a three-part piecewise function: $$ z=\begin{cases} 0 & x<0\\ 1-0.08x & 0\leq x\leq10 \\ 0 & 10\leq x \end{cases} $$ where $x$ denotes days past and $z$ denotes the weight. The first part prevents returning a value to weigh anything that happens, the second makes the linear regression and the third creates the cutoff at the end of the graph. After finding the weight, you calculate the value of a certain point as you would for any arithmetic mean. For example, for $2$ days past the $200$ value, you would find $z$ for the $200$ point as $1-0.08\cdot2=0.84$, and the weights for the $1$s being $1,0.92,0.76,0.68$ e.t.c.

Related Solutions

[Math] Given sample data with three data points, how to predict or guess the 3rd data point when only given two data points

Having copied data from your post, 3D plot suggest that data-points are on a hyper-plane, which suggests a fitting model. I am using Mathematica:

enter image description here

Added: Per OP's additional question, the model does change a little with 7-th record removed:

In[37]:= x3 == Exp[c] x1^a x2^b /. 
 FindFit[Log[data], a logx1 + b logx2 + c, {a, b, c}, {logx1, logx2}]

Out[37]= x3 == (1444.8 x1^0.800888)/x2^1.29128

In[38]:= x3 == Exp[c] x1^a x2^b /. 
 FindFit[Delete[Log[data], 7], 
  a logx1 + b logx2 + c, {a, b, c}, {logx1, logx2}]

Out[38]= x3 == (1581.69 x1^0.797488)/x2^1.29123

Statistics – Most Scientific Way to Assign Weights to Historical Data

There are several ways to do that, depending on your goals. You can pick a few examples from any website that reports economic data, for example stock quotes history. Here are most popular methods:

Method 1: Perhaps the most well-behaved function would be exponential, as Rahul suggested. That means that you pick some number $a<1$, and use geometric progression $1,a,a^2,a^3,…$ to assign weights to each year. The total sum would be $1\over{1-a}$, so you if you pick $a$ you normalize weight to $\frac{1}{1-a}, \frac{a}{1-a}, \frac{a^2}{1-a}, …$ if you want all weights to add up to $1$.

Method 2: If you want to give equal weights to the most recent $n$ years and $0$ to earlier ones you can do what economists call "running average", weights $\frac{1}{n}$ to most recent $n$ years. This function is popular as well, but not as well-behaved as exponential. I guess this answers the 2nd question as well.

Method 3: Some engineers prefer sigmoid. This is an analytic function that pretends to assign equal weights to most recent data, then quickly recedes to $0$, but without the discontinuous step of the running average.

Other methods: In some circumstances physics or probability dictate other distributions. For example, if signal propagation is Gaussian (which happens often enough in physics) then the only relevant choice is erfc.

About justification: yes, there are many reasons to put more emphasis on more recent data and assign equal weight $0$ to older one. For example, older economic data may be irrelevant to projections. Or you are modeling some other effects that naturally decay in time.

Best Answer

Related Solutions

[Math] Given sample data with three data points, how to predict or guess the 3rd data point when only given two data points

Statistics – Most Scientific Way to Assign Weights to Historical Data

Related Question