I learned about modular exponentiation from this website and at fast modular exponentiation they calculate the modulo of the number to the power of two and then they repeat this step. Why not calculate to the power of three ?
https://www.khanacademy.org/computing/computer-science/cryptography/modarithmetic/a/fast-modular-exponentiation
[Math] Why is it more efficient to compute the modular exponentiation by calculating to the power of two and not three for example
exponentiationmodular arithmetic
Best Answer
If we want to calculate $x^N$, the monomials: $$x, x^2, x^4, x^8, \ldots, x^{2^{\lfloor \log_2 N \rfloor}}$$ are the only initial powers we need to calculate (this is $\lfloor \log_2 N \rfloor - 1$ multiplications, because we obtain each term by squaring the previous term). Then we write $N$ in binary as $$N = \sum_{i=0}^{\lfloor \log_2 N\rfloor} a_i2^i,$$ $a_i \in \left\{0,1\right\}$. If there are $k$ $1$s among the $a_i$, there are $k$ nonzero summands of $N$, and it takes $k$ multiplications to get $x^N$ by multiplying the corresponding monomials $x^{2^i}$. In the worst case, we perform $2 \log_2 N$ total multiplications of large numbers. In this analysis, we ignore the cost to write $N$ in binary, because we assume $N$ is small compared to the powers of $x$ we will be dealing with ($N \ll x^N$).
If we try to use base 3 instead of base 2, we now have $\log_3 N$ monomials to calculate, but each one takes 2 multiplications to obtain (2 multiplications to get $x^3$ from $x$, another 2 to get $x^9$ from $x^3$, etc.), so in total $2 \log_3 N$ multiplications to get the monomials. In analogy with the above, now the expansion of $N$ has coefficients in $\{0,1,2\}$, and each $2$ introduces an extra multiplication in the final step: $$x^{24} = x^{2(9) + 2(3)} = x^9x^9x^3x^3,$$ so in the worst case again we have $2 \log_3 N$ multiplications to get the final result.
Since $$\frac{2 \log_3 x}{\log_2 x} = \frac{2\ln 2}{\ln 3} \approx 1.26,$$ this is less efficient in the worst case. More than that, the fact that the coefficients of $N$ in ternary have 3 cases instead of 2 makes the algorithm more complicated.
This is not the end of the story, because there are ways to avoid some of the extra multiplications introduced by modular exponentiation in base 3, by combining terms that have coefficients of 2: $$x^{24} = (x^9x^3)^2,$$ a technique that can be broadly applied (see this question on the CS StackExchange).
So I have not answered why (or whether) base 2 is used in practice in modern algorithms, but I think I have shown why base 2 is optimal for the naive algorithm, and why there is no immediate improvement from moving to a larger base.