In simple linear regression model between RVs $(X,Y)$, the slope $\hat\beta_1$ is given as
$$
\hat\beta_1 = \dfrac{\sum_i^N(x-\overline{x})(y – \overline{y})}{\sum_i^N(x – \overline{x})^2} \tag{1}
$$
This is then interpreted quickly in relation to Covariance and Varaince in many text books 1, as
$$
\hat\beta_1 = \dfrac{Cov(x,y)}{Var(x)} \tag{2}
$$
Question:
But couldn't this only be true if we assume uniform distribution of both joint pmf in covariance and pmf in varaince? That is, it is like assuming as below, and cancelling out $\dfrac{1}{N}$?
$$
\hat\beta_1 = \dfrac{\dfrac{1}{N}\sum_i^N(x-\overline{x})(y – \overline{y})}{\dfrac{1}{N}\sum_i^N(x – \overline{x})^2} \tag{3}
$$
In case both pmfs not uniform,
$$
\dfrac{Cov(x,y)}{Var(x)} = \dfrac{\sum\limits_{x}\sum\limits_{y}(x-\overline{x})(y – \overline{y})p(x,y)}{\sum\limits_{x}(x – \overline{x})^2p(x)} \tag{4}
$$
which is not same as (1), so (2) cant be true, right?
Best Answer
The OP's Eq. $(1)$ is the slope of the regression line if we have $N$ pairs $(x_i, y_i)$ of real numbers. and ask what is the "best" straight line that fits these $N$ data points. In general, it is not the slope of the regression line when we have a pair of random variables $(X, Y)$ and ask what is the random variable $\hat{Y} = \alpha + \beta X$ such that $E[(Y-\hat{Y})^2]$ is as small as possible. The answer to the latter question is indeed that $\beta$ must have value $\frac{\operatorname{cov}(X,Y)}{\operatorname{var}(X)}$ as the OP states in $(2)$ but this result applies to all random variables with finite variances, not just discrete random variables. Indeed, if $X$ and $Y$ are discrete random variables taking on values $x_1, x_2, \ldots, x_M$ and $y_1,y_2,\ldots, y_N$ respectively, then the covariance $\operatorname{cov}(X,Y)$ is given by \begin{align}\operatorname{cov}(X,Y) &= \sum_{m=1}^M \sum_{n=1}^N P(X=x_m, Y = y_n)(x_m-\bar{x})(y_n-\bar{y})\\&= \sum_{m=1}^M \sum_{n=1}^N p_{X,Y}(x_m, y_n)(x_m-\bar{x})(y_n-\bar{y})\end{align} where $\bar{x}$ and $\bar{y}$ are the means $E[X]$ and $E[Y]$ respectively and $p_{X,Y}(x_m, y_n)$ is the joint probability mass function (joint pmf) of $(X,Y)$. This is a slightly more general version of the numerator of $(4)$ in the OP's question. As the OP correctly asserts, if $M=N$ and the joint pmf has value $\frac 1N$ for exactly $N$ points $(x_i,y_i)$, then it is indeed the case that $\operatorname{cov}(X,Y)$ is (proportional to) the numerator of $(1)$ in the OP's question.