Hint
From the perimeter, you can extract $l$. From the surface, you can extract $b$. So, now, $V$ is only a function of $h$ and you want to maximize $V$. Then, ...
I am sure that you can take from here.
How absurd is that 4 days later the answer just struck me out of nowhere. Basically, my approach to the whole thing was wrong. I thought that since what's cut off of the triangular prism is amorphous thus what's left from it (the body which volume I'm interested in) is also amorphous, but this is not true.
If we have a sphere what's cut by a triangular prism which edges tangent to the sphere's surface (just like in my sketches), so basically doing the same as before, but vice versa throw new light upon the whole problem. This way what was cut off is 2 “spherical cap” (and of course, the half of the whole sphere, which also a spherical cap by the way), which volume is easy to calculate!
Now let's talk about the solution, shall we? First, what should we know about the spherical cap? The most important part is that we know its volume! If there's a spherical cap of a sphere with radius $R$ (the diameter is $a$), which height is $h$ and the radius of its base is $r$, then its volume is $V_{cap}=\frac{\pi h}{6}(3r^2+h^2)=\frac{\pi h^2}{3}(3R-h)$. We know that $R=\frac{a}{2}$, $r=\frac{\sqrt{2}}{2}R=\frac{\sqrt{2}}{4}a$ and $h=R-\sqrt{R^2-r^2}$ $=\frac{2-\sqrt{2}}{2}R$ $=\frac{2-\sqrt{2}}{4}a$, so we can express the volume as a function of $a$, as I originally wanted! The result is $V_{cap}=\frac{8-5\sqrt{2}}{96}\pi a^3=\frac{8-5\sqrt{2}}{12}\pi R^3$.
We're out of the woods now, because the only thing left to do is to subtract twice of that from the volume of the hemisphere, which means $V=\frac{1}{2}V_\bigcirc-2V_{cap}=\frac{1}{12}\pi a^3-\frac{8-5\sqrt{2}}{48}\pi a^3=\frac{5\sqrt{2}-4}{48}\pi a^3$ $=\frac{5\sqrt{2}-4}{6}\pi R^3$ and which means it's $\frac{5\sqrt{2}-4}{24}\pi V_\Box \approx \frac{3}{24}\pi V_\Box \neq \frac{1}{12}\pi V_\Box$. So, that's it! This was what I wanted to know: the formula of the volume of that body as a function of the sphere's diameter.
Best Answer
For a parallelopiped with sides $A $, $ B $ and $ C $, the surface area will be given by $2 (AB + AC+BC) $, which yields $ a=AB+AC+BC $. Volume $ V $ of a parallelopiped is given by $ ABC $. Solving for $ C $ from the previous expression and substituting the rusult into the expression for volume, we get $ V (A, B)=\frac {AB (a-AB)}{A+B} $. Now all you need is to use differentiation.