The sample correlation of vectors $(X_1, \dots, X_n)$ and
$(Y_1, \dots, Y_n)$ is
$$\rho_{(X,Y)} = \frac{\frac{1}{n-1}\sum_{i=1}^n (X_i - \bar X)(Y_i - \bar Y) }{S_XS_Y},$$
where $\bar X, \bar Y$ are the respective sample means and $S_XS_Y$ are the respective sample standard deviations.
Roughly speaking, the sample autocorrelation of lag $\ell$ of a vector $(X_1, \dots X_n)$ is the sample correlation of the
vector $(X_1, \dots, X_{n-\ell})$ and and the lagged vector
$(X_\ell, X_{\ell + 1}, \dots, X_n).$
Various refinements are used in specific applications.
Perhaps the one you are looking for is of the following
form:
$$\rho_\ell = \frac{\sum_{i=1}^{n-\ell} (X_1 - \bar X)(X_\ell - \bar X) }{(n-1)S_X^2},$$
Notice that $\bar X$ and $S_X^2$ are based on the
entire sequence. Also, when $\ell=0,$ we have $\rho_\ell = 1.$
See Wikipedia
at the last bullet under Estimation.
As I recall, this is used in the R function acf:
set.seed(1115)
x = round(rnorm(10,200,15))-20*(1:10); x
[1] 176 169 127 99 96 92 45 70 10 12
acf(x)
acf(x, plot=F)
Autocorrelations of series ‘x’, by lag
0 1 2 3 4 5 6 7 8 9
1.000 0.625 0.299 0.138 -0.060 -0.177 -0.304 -0.363 -0.434 -0.223
For the case of $\frac{\sqrt{2}}{2}$, we look to find $k\in\mathbb{N}$ such that $\frac{\sqrt{2}}{2}k\pmod 1$ is as large as possible.
First, consider the sequence of "best possible" rational over-estimates of $\frac{\sqrt{2}}{2}$. We can do this using finite iterations of the continued fraction $[0;1,\bar 2]$ evaluated with the last term as $0$, giving the sequence:
$$S_n=\frac{1}{1},\frac{3}{4},\frac{5}{7},\frac{17}{24},\frac{29}{41}...\space\space\space\space$$
The (strictly increasing) sequence of denominators is then:
$$D_n=1,4,7,24,41...$$
Since we know $\frac{\sqrt{2}}{2}<S_n$ for all $n$, we have that:
$$\frac{\sqrt{2}}{2}D_n<S_nD_n=k,\space k\in\mathbb{N}$$
This implies:
$$\left(\frac{\sqrt{2}}{2}-S_n\right)D_n\equiv\frac{\sqrt{2}}{2}D_n\pmod 1\tag{1}$$
Since $S_n$ is a "best possible" approximation for $\frac{\sqrt{2}}{2}$, we have that:
$$\left|\frac{\sqrt{2}}{2}-\frac{a}{b}\right|<\left|\frac{\sqrt{2}}{2}-S_n\right|\implies b>D_n$$
Hence:
$$\left|\frac{\sqrt{2}}{2}-S_n\right|D_n=\left(S_n-\frac{\sqrt{2}}{2}\right)D_n$$
Is minimal. Multiplying both sides of $(1)$ by $-1$, we find that:
$$\left(S_n-\frac{\sqrt{2}}{2}\right)D_n\equiv -\frac{\sqrt{2}}{2}D_n\equiv 1-\frac{\sqrt{2}}{2}D_n \pmod 1$$
Must also be minimal. Therefore:
$$\frac{\sqrt{2}}{2}k\pmod 1$$
Is maximal for some $k\in\mathbb{N}\iff k=D_n$
Note:
On further inspection, I realised that (provided my proof is correct), substituting any irrational number $\alpha$ (perhaps rational too?) for $\frac{\sqrt{2}}{2}$ would lead to the same result:
$\alpha D_n\pmod 1>\alpha k\pmod 1$ for all $k\in\mathbb{N}, k<D_n \iff D_n$ represents the sequence of over-estimate convergent denominators from the continued fraction of $\alpha$.
Best Answer
Look at the closed subgroup $G\subset \mathbb T^2$ (using the notation that $\mathbb T=\mathbb R / \mathbb Z$) generated by $(b_1,b_2)$; it will be of the form $G=\{ (ax,bx):x\in\mathbb T\} $ for certain integers $a$ and $b$. (In your case, $(a,b)=(5,4)$.) Your orbit $\{(nb_ 1,nb_2):n=1,2,\ldots\}$ is uniformly distributed in $G$. Your correlation is obtained by integrating with respect to the uniform (Haar) measure on $G$, namely by $\int_0^1 (ax\bmod1)(bx\bmod1)dx$.
It might help in general to develop the Fourier series for the functions $ x\mapsto ax\bmod1$ and $bx\mapsto x\bmod1$ on $\mathbb T$.
In your specific case, however, the integral $I= \int_0^1 (5x\bmod1)(4x\bmod1)dx$ is given by the following edge-of-tedious expression $$I=\int_0^{1/5}(5x)(4x)dx+ \int_{1/5}^{1/4}(5x-1)(4x)dx+ \int_{1/4}^{2/5}(5x-1)(4x-1)dx+ \int_{2/5}^{1/2}(5x-2)(4x-1)dx+ \int_{1/2}^{3/5}(5x-2)(4x-2)dx+ \int_{3/5}^{3/4}(5x-3)(4x-2)dx+ \int_{3/4}^{4/5}(5x-3)(4x-3)dx+ \int_{4/5}^{1}(5x-4)(4x-3)dx. $$ The correlation coefficient is $\frac{I-1/4}{1/12}$.