The matrix $\begin{bmatrix}1 & 0 \\ 0 & -1\end{bmatrix}$ is orthogonal and indefinite.
$\begin{bmatrix}1 & 0 \\ 0 & 2\end{bmatrix}$ is positive definite and not orthonormal.
and the Identity matrix $I$ is of course both orthogonal and positive definite. Let $S$ be the intersection of orthogonal matrices and positive definite matrices. We have seen that $S$ is non-empty.
By a positive definite matrix, I mean either a symmetric or asymmetric matrix $M \in \mathbb{R}^{p^2}$ whose quadratic form satisfies $\forall x \in \mathbb{R}^p \setminus \{0\}: x^TMx > 0$. Let $P$ denote the set of all positive definite matrices. Then $P$ is a convex cone.
$S$ is not a convex cone, unlike $P$. Also unlike $P$, $S$ is closed under multiplication The product of any two matrices in $S$ is guaranteed to be at least PSD.[1]
I am interested in complete characterizations of this space $S$, which globally behaves more like the space of orthogonal matrices. My real motivation is that I want to know whether there are efficient procedures for testing whether a matrix $M $ is in $S$ that are faster than testing for positive definiteness which requires the calculation of eigenvalues? E.g. such procedures might take only $O(p^2)$ computations. I tried to google for resources but nothing relevant came up.
[1] Orthogonal matrices are of course closed under multiplication. The product of two PD matrices is PD PSD matrices is PSD iff their product itself is normal, which is true in $S$.
Reference: On a product of positive semidefinite matrices, A.R. Meenakshi, C. Rajian, Linear Algebra and its Applications, Volume 295, Issues 1–3, 1 July 1999, Pages 3–6.
In case someone is wondering, the real real reason, why I am interested in this question is because I want to "efficiently" find a "reasonably good" positive definite approximation of a matrix whose $QR$ decomposition is known to me. The details of this part are best left for future.
Best Answer
You may find the Cayley transform to be useful here:
As is well-known and easy to prove, every orthogonal $n$-by-$n$ matrix $R$ that does not have $-1$ as an eigenvalue can be written uniquely in the form $$ R = (I-A)(I+A)^{-1} $$ for some anti-symmetric matrix $A$ for which $I+A$ is invertible, and, conversely, if $A$ is any anti-symmetric matrix such that $(I+A)$ is invertible, the matrix $R$ in the above formula is orthogonal and $I+R$ is invertible.
In fact, $A = (I-R)(I+R)^{-1}$, so $A$ is easy to find. The two matrices $R$ and $A$ are said to be Cayley transforms of each other, and this provides a 'rational parametrization' of the orthogonal group minus the hypersurface consisting of the orthogonal matrices that have $-1$ as an eigenvalue. (Note that $R$ and $A$ commute.)
Now $R$ satisfies the stronger condition that $x^TRx>0$ for all nonzero $x\in\mathbb{R}^n$ if and only if (setting $y=(I+A)^{-1}x$ or, equivalently $x = (I+A)y$), we have $$ 0 < x^TRx = y^T(I+A)(I-A)y = y^T(I-A^2)y = |y|^2-|Ay|^2 $$ for all $y\in \mathbb{R}^n$. Equivalently, the matrix norm of $A$, i.e., $\|A\|$, should be strictly less than $1$.
Thus, via the Cayley transform, your set $S$ is parametrized by the open convex set in the vector space of anti-symmetric $n$-by-$n$ matrices consisting of those anti-symmetric matrices $A$ whose matrix norm is less than $1$.