While going through this answer I found an interesting but slightly complicated relation between Rogers-Ramanujan continued fraction and the j-invariant. I would like to know an elementary proof of the same.
Before proceeding let me define all the necessary terms and symbolism to set the proper context. Let $\tau$ be a complex number with positive imaginary part and $q=\exp(2\pi i\tau) $ so that $|q|<1$. Below I define functions and the relations which I am aware of. The Rogers-Ramanujan continued fraction is given by $$R(q) =\cfrac{q^{1/5}}{1+\cfrac{q}{1+\cfrac{q^2}{1+\cfrac{q^3}{1+\dots}}}}\tag{1}$$ Ramanujan studied this function in great detail and obtained the following fundamental identities $$\frac{1}{R(q)}-1-R(q)=\frac{\eta(q^{1/5})} {\eta(q^5)}\tag{2}$$ and $$\frac{1}{R^5(q)}-11-R^5(q)=\left(\frac{\eta(q)}{\eta(q^5)}\right)^6\tag{3}$$ where $\eta(q) $ is Dedekind eta function defined by $$\eta(q) =q^{1/24}\prod_{n=1}^{\infty} (1-q^n)\tag{4}$$ To define the $j$-invariant we need to introduce Ramanujan's version of Eisenstein series denoted by $L, M, N$ (symbols $P, Q, R$ are typically used but we want to avoid conflict with Rogers-Ramanujan continued fraction $R(q) $)
\begin{align}
L(q) &= 1-24\sum_{n=1}^{\infty}\frac{nq^n}{1-q^n}\tag{5a}\\
M(q)&=1+240\sum_{n=1}^{\infty} \frac{n^3q^n}{1-q^n}\tag{5b}\\
N(q) &=1-504\sum_{n=1}^{\infty} \frac{n^5q^n}{1-q^n}\tag{5c}
\end{align}
It should be observed that $L$ is related to $\eta$ via $$L(q) =24q\frac{d}{dq}(\log\eta(q))\tag{6}$$ The $j$-invariant is defined as $$j(q) =\frac{1728M^3(q)}{M^3(q)-N^2(q)}\tag{7}$$
Ramanujan obtained a system of differential equations connecting $L, M, N$:
\begin{align}
q\frac{dL(q) } {dq} &=\frac{L^2(q)-M(q)}{12}\tag{8a}\\
q\frac{dM(q)}{dq}&=\frac{L(q)M(q)-N(q)}{3}\tag{8b}\\
q\frac{dN(q)} {dq} &=\frac{L(q) N(q) – M^2(q)}{2}\tag{8c}
\end{align}
Using $(6)$ and $(8a)$ it is evident that $M(q) $ can also be expressed in terms of $\eta(q) $. On the other hand the above differential equations allow us to prove that $$M^3(q)-N^2(q)=1728\eta^{24}(q)\tag{9}$$ and thus we have some expression for $j(q) $ in terms of $\eta(q) $.
The following complicated relation holds between Rogers Ramanujan continued fraction $R(q)$ and $j(q) $ : $$ R^5 (R^{10}+11 R^5-1)^5j+(R^{20}-228 R^{15}+494 R^{10}+228 R^5+1)^3 = 0\tag{10}$$ I checked Wikipedia and found that this is derived from another identity $$j(q) =\frac{(x^2+10x+5)^3} {x} \tag{11}$$ where $$x=125\left(\frac {\eta(q^5)}{\eta(q)}\right)^6\tag{12}$$ Using $(11),(12)$ and $(3)$ we can deduce $(10)$ with a little algebra.
Thus the problem boils down to a proof of equation $(11)$. I don't know if this can be derived using algebraic manipulation of the identities given above or does it need some specific modular equation.
Any proofs or suggestions for proof are welcome. I don't understand the machinery of modular forms properly and would prefer an approach more in the spirit of Ramanujan. The question is however tagged "modular-forms" to get the attention of experts from that tag.
Update: I have finally managed to give a proof based on modular equation of degree $5$ given by Ramanujan and posted it as an answer. The proof is more of a verification and a more natural proof utilizing some transformation formula of eta function is desired.
There is a related question which assumes $(10),(11)$ and proves $(3)$, but the approach uses Mathematica.
Best Answer
An approach based on modular equations of degree $5$ works out fine with some tedious algebra. The identity $(11) $ in question will be proved by replacing $q$ with $q^2$.
Let $k, l$ denote the elliptic modulus and $K, L$ the complete elliptic integrals of first kind corresponding to nomes $q, q^5$ respectively and further let $\alpha=k^2,\beta =l^2$. Also let $m=K/L$ denote the multiplier.
Then we have the standard formulas \begin{align} M(q^2)&=\left(\frac{2K}{\pi}\right)^4(1-\alpha(1-\alpha))\tag{1}\\ \eta(q^2)&=2^{-1/3}\sqrt {\frac {2K}{\pi}}(\alpha(1-\alpha))^{1/12}\tag{2}\\ \eta(q^{10})&=2^{-1/3}\sqrt{\frac{2L}{\pi}}(\beta(1-\beta))^{1/12}\tag{3} \end{align} Dr. Bruce C. Berndt and his collaborators analyzed Ramanujan's modular equations of degree $5$ and gave the following relations between $\alpha, \beta, m$: \begin{align} (\alpha^3\beta)^{1/8}&=\frac{\rho+3m-5}{4m}\tag{4}\\ (\alpha\beta^3)^{1/8}&=\frac{\rho+m^2-3m}{4m}\tag{5}\\ \rho&=\sqrt{m^3-2m^2+5m}\tag{6} \end{align} From the above equations we get $$\alpha=\frac{(\alpha^3\beta)^{3/8}}{(\alpha\beta^3)^{1/8}}=\frac{(\rho+3m-5)^3}{16m^2(\rho+m^2-3m)}$$ Multiplying numerator and denominator by $\rho-m^2+3m$ and using some algebra we get $$\alpha=\frac{(\rho+3m-5)^2(2m+\rho)} {16m^3(m-1)} $$ Next we replace $\alpha, \beta, m$ by $1-\beta,1-\alpha,5/m$ to get $$((1-\alpha)(1-\beta)^3)^{1/8}=\frac{\rho+3m-m^2}{4m}$$ and $$((1-\alpha)^3(1-\beta))^{1/8}=\frac{\rho+5-3m}{4m}$$ and thus $$1-\alpha=\frac{(\rho+5-3m)^2(2m-\rho)} {16m^3(m-1)} $$ and hence $$\alpha(1-\alpha)=\frac{(m-1)(5-m)^5}{256m^5} \tag{7}$$ A similar calculation leads to the relation $$\left(\frac{\beta(1-\beta)}{\alpha(1-\alpha)} \right)^{1/2}=\frac{m^2(m-1)^2}{(5-m)^2}\tag{8}$$ Let us now observe that if $$x=125\left(\frac{\eta (q^{10})}{\eta(q^2)}\right)^6$$ then by the identities $(2),(3),(8)$ we get $$x=\frac{125(m-1)^2}{m(5-m)^2}$$ On the other hand we have $$j(q^2)=\frac{M^3(q^2)}{\eta^{24}(q^2)}=\frac{(1-\alpha(1-\alpha)) ^3}{2^{-8}(\alpha(1-\alpha))^2}=\frac{(256m^5-(m-1)(5-m)^5)^3}{m^5(m-1)^2(5-m)^{10}}$$ And the expression $(x^2+10x+5)^3/x$ equals $$\frac{(3125(m-1)^4+250m(m-1)^2(5-m)^2+m^2(5-m)^4)^3}{m^5(m-1)^2(5-m)^{10}}$$ Our job is thus complete if we can show that $$3125(m-1)^4+250m(m-1)^2(5-m)^2+m^2(5-m)^4=256m^5-(m-1)(5-m)^5$$ Both sides are polynomials of degree $6$ and hence we need to verify the identity for at most $7$ values of $m$. The verification is trivial for $m=0,1,5$ and one may try to verify it for small values like $m=-1,-2, 2,3$.
There is another approach to rewrite the desired identity so that both sides can be factorized easily. Thus the desired identity is equivalent to $$250m(m-1)^2(5-m)^2+3125(m-1)^4=256m^5+(m-1)(m-5)^5-m^2(m-5)^4$$ or $$250m(m-1)^2(m-5)^2+3125(m-1)^4=3125-6250m+3750m^2-1000m^3+125m^4+250m^5$$ or $$(m-1)^2(2m(m-5)^2+25(m-1)^2)=25-50m+30m^2-8m^3+m^4+2m^5$$ It can be checked now that each side factors as $$(m-1)^2(2m^3+5m^2+25)$$