For arbitrary subspaces $U,V$ of an arbitrary vector space $W$, we have the dimension formula
$$\dim U + \dim V = \dim (U+V) + \dim (U\cap V).$$
(Take a basis $B_1$ of $U\cap V$, and extend it by systems $B_2$ to a basis of $U$, and $B_3$ to a basis of $V$. Then $B_1 \cup B_2 \cup B_3$ is a basis of $U+V$.)
That yields the
$$\dim F + \dim F^\perp = \dim (F + F^\perp) + \dim (F\cap F^\perp)$$
part. It remains to see that $\dim F + \dim F^\perp = \dim E$. Let's denote the non-degenerate bilinear form by $\beta$. $\beta$ induces a linear map $\Phi \colon E \to E^\ast$ ($E^\ast$ is the dual space of $E$, the space of all linear maps $E\to K$) via
$$\Phi(x)(y) = \beta(x,y).$$
The non-degeneracy of $\beta$ is equivalent to the injectivity of $\Phi$. Since the spaces are finite-dimensional, we have $\dim E^\ast = \dim E$, and thus $\Phi$ is an isomorphism. For a subspace $F \subset E$, the image of $F^\perp$ under $\Phi$ is the annihilator of $F$,
$$\Phi(F^\perp) = F^0 = \{ \lambda \in E^\ast : F\subset \ker\lambda\}.$$
If you already know that $\dim F + \dim F^0 = \dim E$ for all subspaces of finite-dimensional spaces, that's it. Otherwise, to see that, choose a basis $B_0 = \{v_1,\dotsc, v_f\}$ of $F$, extend it to a basis $B = B_0 \cup B_1 = \{v_1,\dotsc,v_f,v_{f+1},\dotsc,v_e\}$ of $E$, and consider the dual basis $B^\ast = \{\lambda_1,\dotsc,\lambda_e\}$ of $E^\ast$, defined by
$$\lambda_i(v_j) = \delta_{ij} = \begin{cases}1 &, i = j\\ 0 &, i \neq j. \end{cases}$$
It is then easy to see that $F^0 = \operatorname{span} \{\lambda_{f+1},\dotsc,\lambda_e\}$, whence $\dim F^0 = e - f = \dim E - \dim F$.
Here's an alternate proof. Given a non-degenerate $G$-invariant bilinear form $F$, define $F_s(x,y) = \frac{1}{2}( F(x,y) + F(y,x))$ and $F_a(x,y) = \frac{1}{2}(F(x,y)-F(y,x))$. So, $F_s$ is the symmetric part of $F$ and $F_a$ is the antisymmetric part.
Because $F$ is $G$-invariant, it follows easily that both $F_s$ and $F_a$ are $G$-invariant so that we obtain $G$-equivariant maps $\phi_a,\phi_s:V\rightarrow V^\ast$ defined by$\phi_s(x) = F_s(x,\cdot)$ and $\phi_a(x) = F_a(x,\cdot)$. Because the representations are irreducible, each of $\phi_a,\phi_s$ is either an isomorphism or the zero map.
If they're both the zero map, then this implies that both $F_s$ and $F_a$ are zero, which then implies $F = F_s + F_a$ is the zero form, contradicting non-degeneracy. If exactly one is the zero map, then $F = F_s$ or $F= F_a$, so we get the result you want.
Thus, there is one case left to consider: both $\phi_a$ and $\phi_s$ are isomorphisms. If this happens, consider the composition $\phi_a^{-1}\circ \phi_s:V\rightarrow V$, which is a $G$-equivariant isomorphism of $V$. By Schur's Lemma, this composition is a non-zero multiple $\lambda$ of the identity map.
Said another way, $\phi_s = \lambda \phi_a$ for some complex number $\lambda$. That is, for any $v\in V$, we have $F_s(v,\cdot) = \lambda F_a(v,\cdot)$. Or, to say it even simpler, $F_s = \lambda F_a$. This implies that $F_s$ is anti-symmetric, which then implies $F_s = 0$, giving a contradiction.
Best Answer
In the language of representation theory, a (finite dim) group representation is self-dual if and only if it has an invariant bilinear form. This is because an invariant non-degenerate bilinear form on a representation $V$ is the same as an injective map of representations $V\to V^*.$
In the language of the paper, a hypergeometric group is specified by matrices $h_0,h_1,h_\infty.$ Since we're working with matrices, the dual transformations are transposes. Applying the inverse transpose map $g\mapsto (g^{-1})^T$ gives another hypergeometric group by $(h_0^{-1})^T,(h_1^{-1})^T,(h_\infty^{-1})^T.$ In the case you're interested in, these two hypergeometric groups have the same parameter sets. So by Theorem 3.5, they are simultaneously conjugate: there exists a matrix $\alpha$ such that $\alpha h_i\alpha^{-1}=(h_i^{-1})^T$ for $i=0,1,\infty.$ The nondegenerate bilinear form given by $(u,v)=u^T\alpha v$ is invariant under each $h_i$: $$(h_iu,h_iv)=u^Th_i^T\alpha h_iv=u^Th_i^T(h_i^{-1})^T\alpha v=u^T\alpha v=(u,v).$$