Look at $$5x^2 - 4xy + 9y^2 +z(Ax +B y + C z + D) - 64=0$$ which has the required intersection with $z=0.$
Then we have the condition $576A^2+256AB+320B^2-41D^2-10496C=0$ for it to be a cone (the determinant of the $4\times 4$ matrix).
$$5x^2 - 4xy + 9y^2 -64\\-(41D^2-320B^2-256AB-576A^2)\,z^2/10496 +z\,(Ax +B y + D)=0$$
We find the apex, the singular point, by solving the systems of partials: $$(x,y,z)=(-(128B+576A)/(41D), -(320B+128A)/(41D), 128/D)\tag1$$ and check that it indeed lies on the surface.
It remains to make it a right circular cone with aperture $\frac{\pi}{8}.$
Shifting the apex to the origin by the obvious translation
$$(41D^2-320B^2-256AB-576A^2)z^2-10496Byz-10496Axz\\-94464y^2+41984xy-52480x^2=0\tag2$$
we see that we have a homogeneous form.
Remember $\frac{\sqrt{2+\sqrt2}}{2}=\cos(\frac{\pi}{8})=\frac{n_x x+ n_y y+n_z z}{\sqrt{x^2+y^2+z^2}}$ where $\hat{n}=(n_x,n_y,n_z)$ is a unit vector along the axis. We get $$\frac{2+\sqrt2}{4}(x^2+y^2+z^2)=(n_x x+ n_y y+n_z z)^2\tag3$$
Comparing coefficients in $(2)$ and $(3)$, where $k$ is a scaling factor, we get the following system, solving it in maxima CAS, precomputed a grobner basis in M2, all the while remembering that $$n_x^2+n_y^2+n_z^2=1$$
s:sqrt(2);
solve([n1*B+4*n3, A^2-2*A*B-B^2, 83968*k*A+s*B+209920*k*B+2*B, n2*A+4*n3, n1*A-n2*B+8*n3,2*s*A-9*s*B-1721344*k*B+4*A-18*B, 7774978036465664*k^2-19040*A*B-6272*B^2+287*D^2+430336*s+252941172736*k+1674112, 9033613312*s*k+2720*A*B+896*B^2-41*D^2-18067226624*k-146944, n2*n3-5248*k*B, 16*n1*n3+s*B+209920*k*B+2*B,4*n2^2-s-377856*k-2, n1*n2+20992*k, s*n2+83968*n1*k+209920*n2*k+2*n2, 4*n1^2-s-209920*k-2,s*n1+377856*n1*k+83968*n2*k+2*n1, s^2-2, 41*n2*D^2+3584*n3*A+3712*n3*B+20992*n1+52480*n2,41*n1*D^2+10880*n3*A+3584*n3*B+94464*n1+20992*n2,7*s*D^2+3253760*k*D^2+20992*n3^2+14*D^2+5248*s+3084648448*k+10496, 1312*B^3-85*A*D^2+198*B*D^2-94464*A+209920*B,1312*A*B^2+28*A*D^2-85*B*D^2+20992*A-94464*B, 146944*k*B^2-6724*k*D^2-340*n3^2+41*s+82, n2*B^2+4*n3*A-8*n3*B,7*s*B^2+67240*k*D^2+2952*n3^2+14*B^2-410*s-820, 2720*n3*A*B+896*n3*B^2-41*n3*D^2+5248*n2*B-94464*n3,20992*n2*k*B-s*n3-377856*n3*k-2*n3, 7904411648*n3^2*k+2205472*k*D^2+203360*n3^2-203*A*B+196*B^2-13448*s-26896,656*s*n3^2-1312*n3^2+5*A*B-2*B^2, 2720*A*B*D^2+896*B^2*D^2-41*D^4+3022848*A*B+671744*B^2-146944*D^2-110166016,6724*k*B*D^2+112*n3^2*A+116*n3^2*B-41*s*B-82*B,4299999936512*n3^4+47812050944*n3^2*B^2-988051456*n3^2*D^2-14193536*B^2*D^2+93275*D^4+30371007954944*k*D^2-740756291584*n3^2+1156743168*A*B-13918535680*B^2+334297600*D^2-185189072896*s-119750459392,1000195*k*D^4+257152*n3^2*B^2+56063*n3^2*D^2+3946396288*k*D^2+78609792*n3^2-2205472*s+2687499960320*k-4410944],[n1,n2,n3,A,B,D,k]);
And one of the solutions to this gives the positive octant apex:
n1:sqrt(13*sqrt(2)+16)/sqrt(82);
n2:((sqrt(2)-1)*sqrt(13*sqrt(2)+16))/sqrt(82);n3:sqrt(1-n1^2-n2^2);%r8:n3;
A:(sqrt(13*sqrt(2)+16)*(3*2^(3/2)-20)*sqrt(82)*%r8)/41;
B:((sqrt(2)-1)*sqrt(13*sqrt(2)+16)*(3*2^(3/2)-20)*sqrt(82)*%r8)/41;
D:(2^(5/2)*sqrt((2*(sqrt(2)-1)^2*(13*sqrt(2)+16)*(85*sqrt(2)+113)*(3*2^(3/2)-20)^2*%r8^2)/41-41*2^(9/2)-2296))/sqrt(41);
k:-(3*sqrt(2)+10)/1721344;
the above solution substituted into $(1)$ gives the apex
$$({{\sqrt{
13\,\sqrt{2}+16}\,\sqrt{35-5\,2\sqrt2}\,\left(1856-2^5\sqrt2\right)\,\sqrt{82}}\over{1681\,\sqrt{2}\,\sqrt{2344-
23\,2^5\sqrt2}}},{{\sqrt{13\,\sqrt{2}+16}\,\sqrt{35-5\,2\sqrt2}\,\left(59\,2^5\sqrt2-1920\right)\,\sqrt{82}
}\over{1681\,\sqrt{2}\,\sqrt{2344-23\,2^5\sqrt2}}},{{2^4\sqrt{82}}\over{\sqrt{2344-23\,2^5\sqrt2}}})$$
$$\approx(5.116868190800648,2.119476201505111,4.013576710399762).$$
Best Answer
Using the Rodrigues formula for rotation $(\theta)$ around an axis $\vec k$
$$ p^{\theta} = p\cos\theta+\vec k\times p\sin\theta+(\vec k\cdot p)\vec k(1-\cos\theta) $$
the rotated cube around $(A,\vec k)$ can be obtained calculating
$$ p^{\theta}_k = p_k\cos\theta+\vec k\times p_k\sin\theta+(\vec k\cdot p_k)\vec k(1-\cos\theta) $$
because $A=(0,0,0)$.
Now regarding the rotated edge containing $C$ we have
$$ \frac 12(p^{\theta}_t+p^{\theta}_b)\cdot(0,0,1)=0 $$
where $p^{\theta}_t,p^{\theta}_b$ are respectively the rotated edge extrema. Regarding the point $B$ we need to determine $0\le \lambda\le 1$ such that
$$ \lambda p^{\theta}_t+(1-\lambda)p^{\theta}_b= B $$
where $p^{\theta}_t,p^{\theta}_b$ are respectively the corresponding rotated edge extrema, and finally, the normalization condition $\|\vec k \|=1$. This gives us four conditions and five unknowns $(\vec k, \theta,\lambda)$. This can be solved as a minimization procedure as follows
$$ \min_{\vec k,\theta,\lambda}\|\lambda p^{\theta}_{t_1}+(1-\lambda)p^{\theta}_{b_1}- B\| \ \ \text{s. t}\ \ \ \cases{0\le \lambda\le 1\\ \|\vec k\| = 1\\ \frac 12(p^{\theta}_{t_2}+p^{\theta}_{b_2})\cdot(0,0,1)=0} $$
giving
$$ \vec k = (0.130848,-0.447689,0.884564), \theta = -0.751834, \lambda = 0.708333, E = (31.7616, 3.94883, -11.2963) $$
In blue the original cube, in red, rotated and in black the point $C$
Follows a MATHEMATICA script to perform this minimization