As you can see from the title my problem is rather complex, so I'll try to break it apart and explain it step by step.
My original problem is to find the formula of the volume of a triangular prism (if the hypotenuse is $a$ then the corresponding altitude is $\frac{a}{2}$ and its length is also $a$ (if it was $\frac{a}{\sqrt{2}}$ it'd make it a half cube)) which is cut by a sphere (its diameter is $a$ and its radius is $R$) tangent to its 3 longitudinal edges and the 2 triangle faces of the triangular prism.
I searched for the answer for a while, but I figured it out it's better off finding a similar, but more symmetrical problem, because it's way too unique to find the answer for that, so I came up with the idea of a cube cut by a sphere tangent to all of its edges (the cube's edges is not equal to $a$, instead its face diagonal is $a$), which is similar enough to my problem, but I didn't find the answer for that either.
Finally, the title is the extension of the second, more symmetrical problem, where the cuboid is a lengthened cube which makes it a square prism, which length is $a$ (but the square's edges not equal to $a$) and the half of its volume is equals to what I originally looking for so it's practically identical with my original problem.
For me the whole problem is too complex. I can easily look up the formulas for the volume of a sphere or a triangular prism, but I can't deal with the section of them, although I've had an idea. If we take a square prism which is circumscribed about the half of the sphere from my original problem and its volume is $V_\Box=2R*2R*R=4 R^3$ then the hemisphere's volume is $V_\bigcirc=\frac{\pi}{6}V_\Box$ and the triangular prism's volume is $V_\bigtriangleup=\frac{1}{2}V_\Box$, so my theory is the volume I'm looking for is $\frac{\pi}{12}V_\Box$? But honestly, I don't think this layout would be that symmetrical.
I don't know how clear I was, so just in case I made a sketch about it. In the draw the dashed lines are the square prism circumscribed about the hemisphere and the shaded areas are the body which volume I want to find.
Best Answer
How absurd is that 4 days later the answer just struck me out of nowhere. Basically, my approach to the whole thing was wrong. I thought that since what's cut off of the triangular prism is amorphous thus what's left from it (the body which volume I'm interested in) is also amorphous, but this is not true.
If we have a sphere what's cut by a triangular prism which edges tangent to the sphere's surface (just like in my sketches), so basically doing the same as before, but vice versa throw new light upon the whole problem. This way what was cut off is 2 “spherical cap” (and of course, the half of the whole sphere, which also a spherical cap by the way), which volume is easy to calculate!
Now let's talk about the solution, shall we? First, what should we know about the spherical cap? The most important part is that we know its volume! If there's a spherical cap of a sphere with radius $R$ (the diameter is $a$), which height is $h$ and the radius of its base is $r$, then its volume is $V_{cap}=\frac{\pi h}{6}(3r^2+h^2)=\frac{\pi h^2}{3}(3R-h)$. We know that $R=\frac{a}{2}$, $r=\frac{\sqrt{2}}{2}R=\frac{\sqrt{2}}{4}a$ and $h=R-\sqrt{R^2-r^2}$ $=\frac{2-\sqrt{2}}{2}R$ $=\frac{2-\sqrt{2}}{4}a$, so we can express the volume as a function of $a$, as I originally wanted! The result is $V_{cap}=\frac{8-5\sqrt{2}}{96}\pi a^3=\frac{8-5\sqrt{2}}{12}\pi R^3$.
We're out of the woods now, because the only thing left to do is to subtract twice of that from the volume of the hemisphere, which means $V=\frac{1}{2}V_\bigcirc-2V_{cap}=\frac{1}{12}\pi a^3-\frac{8-5\sqrt{2}}{48}\pi a^3=\frac{5\sqrt{2}-4}{48}\pi a^3$ $=\frac{5\sqrt{2}-4}{6}\pi R^3$ and which means it's $\frac{5\sqrt{2}-4}{24}\pi V_\Box \approx \frac{3}{24}\pi V_\Box \neq \frac{1}{12}\pi V_\Box$. So, that's it! This was what I wanted to know: the formula of the volume of that body as a function of the sphere's diameter.