The data given does not determine the volume of the pyramid.
Note that since $ADM$ and $BCM$ are perpendicular to $ABCD$, it follows that $M$ lies above the intersection $P$ of $AD$ and $BC$. Let $h$ be the height of $M$ above $P$, let $x$ be the perpendicular distance from $P$ to $AB$, and let $d$ be the perpendicular distance between $CD$ and $AB$. Then
$$S_{\triangle BCM}=\frac12 h|BC|=9$$
and
$$S_{\triangle ABM}=\frac12\sqrt{x^2+h^2}|AB|=20\;,$$
and with $|AB|=5$ this yields
$$x^2+h^2=64\;.$$
We also have
$$\frac{x-d}x=\frac35\;,$$
so $x=\frac52d$, and thus
$$\left(\frac52d\right)^2+h^2=64\;.$$
Writing $d=|BC|\cos\alpha$, with $\alpha$ the angle between $BC$ and the perpendicular on $AB$ and $CD$, we arrive at
$$\left(\frac52|BC|\cos\alpha\right)^2+\left(\frac{18}{|BC|}\right)^2=64\;.$$
Solving for $\cos\alpha$ yields
$$\cos\alpha=\frac{\sqrt{64-(18/|BC|)^2}}{\frac52|BC|}\;.$$
Setting this to zero yields $|BC|\gt\frac94$, and the value is below $1$ in that entire range with a maximum of $32/45$ at $|BC|=\frac94\sqrt2$.
With $V=\frac13hd(|AB|+|CD|)/2=\frac43hd=\frac43h|BC|\cos\alpha=24\cos\alpha$, the volume of the pyramid can take any value between $0$ and $256/15\approx17$.
The angle between two planes $\alpha$ and $\beta$, having a line of intersection $r$, is defined as the angle formed by two lines $a\subset \alpha$ and $b\subset\beta$, both perpendicular to $r$ at the same point $M$.
In your case, to find the angle between $OAB$ and $OBC$ you can for instance draw the altitudes $AM$ and $CM$ of those two faces: then the angle is $\angle AMC$. To compute it you can find the lengths of $AM$, $CM$ and $AC$ (by some standard trigonometry) and then use the cosine law to get the angle.
The angle between $OAB$ and $ABCD$ is even easier: if $OK$ is another altitude of face $OAB$, then the angle is just $\angle OKH$, where $H$ is the center of square base.
Best Answer
Let $A = (2,0,0)$, $B = (1,0,0)$, $C = (0,1,0)$, $D = (0,2,0)$, and $S = (0,0,2)$.
Then, $ABCD$ is a trapezoid (with $BC \parallel AD$ and $AB \not\parallel CD$) which lies in the $xy$-plane.
Also, $SAB$ lies in the $xz$-plane and $SCD$ lies in the $yz$-plane, which are each orthogonal to the $xy$-plane.