For typing convenience, let me substitute Latin in place of your Greek letters
$$\eqalign{
X &= \Xi \cr
x &= \vartheta = \operatorname{vec}(X) \cr
W &= \Omega = I + XX^T = W^T \cr\cr
}$$
Then for your first function, the differential and gradient can be calculated as
$$\eqalign{
f &= \log\det W \cr
\cr
df &= d\log\det W = d\operatorname{tr}\log W \cr
&= W^{-T}:dW \cr
&= W^{-T}:(dX\,X^T+X\,dX^T) \cr
&= \big(W^{-T}X+W^{-1}X\big):dX \cr
&= 2\,W^{-1}X:dX \cr
&= 2\,\operatorname{vec}(W^{-1}X)\cdot\,dx \cr
\cr
\frac{\partial f}{\partial x} &= 2\,\operatorname{vec}(W^{-1}X) \cr\cr
}$$
In your second function, I don't quite understand the definition of $\,H(\vartheta)\,$ so all I can offer is a partial solution
$$\eqalign{
M &= W^{-1} = M^T \cr
f &= H:MH \cr
\cr
df &= dH:MH + H:M\,dH + H:dM\,H \cr
&= (M+M^T)H:dH + HH^T:dM \cr
&= 2\,MH:dH - HH^T:M\,dW\,M \cr
&= 2\,MH:dH + MHH^TM:dW \cr
&= 2\,MH:dH + MHH^TM:(dX\,X^T+X\,dX^T) \cr
&= 2\,MH:dH + 2\,MHH^TMX:dX \cr
\cr
}$$
You can finish off the solution by expanding $dH$ in terms of $dX$, then vectorizing.
In the above, a colon denotes the double-dot (aka Frobenius) product, which is merely a product notation for the trace, i.e. $$A:B=\operatorname{tr}(A^TB)$$
Update
I'm guessing that $H$ is a partitioned matrix: $H=[\,X, I\,]$
Expanding that term in the differential
$$\eqalign{
MH:dH &= [\,MX, M\,]:[\,dX, 0\,] \cr
&= MX:dX + M:0 \cr
}$$
So, continuing with the full differential
$$\eqalign{
df &= 2\,\Big(MX + MHH^TMX\Big):dX \cr
&= 2\,\operatorname{vec}(MX + MHH^TMX)\cdot dx \cr
\cr
\frac{\partial f}{\partial x} &= 2\,\operatorname{vec}(MX + MHH^TMX) \cr
}$$
Best Answer
For typing convenience, define the matrices $$\eqalign{ Y &= XW \\ J &= 1_{n\times K} \qquad&({\rm all\,ones\,matrix}) \\ S &= {\rm sign}(Y) \\ A &= S\odot Y \qquad&({\rm absolute\,value\,of\,}Y) \\ B &= A-J \\ Y &= S\odot A \qquad&({\rm sign\,property}) \\ }$$ where $\odot$ denotes the elementwise/Hadamard product and the sign function is applied element-wise. Use these new variables to rewrite the function, then calculate its gradient. $$\eqalign{ \phi &= \|B\|_F^2 \\&= B:B \\ d\phi &= 2B:dB \\ &= 2(A-J):dA \\ &= 2(A-J):S\odot dY \\ &= 2S\odot(A-J):dY \\ &= 2(Y-S):dY \\ &= 2(Y-S):X\,dW \\ &= 2X^T(Y-S):dW \\ \frac{\partial\phi}{\partial W} &= 2X^T(Y-S) \\ }$$ where a colon denotes the trace/Frobenius product, i.e. $$\eqalign{ A:B = {\rm Tr}(A^TB) = {\rm Tr}(AB^T) = B:A }$$ The cyclic property of the trace allows such products to be rearranged in various ways $$\eqalign{ A:BC &= B^TA:C \\ &= AC^T:B \\ }$$ Finally, when $(A,B,C)$ are all the same size, their Hadamard and Frobenius products commute with each other $$\eqalign{ A:B\odot C &= A\odot B:C \\\\ }$$ NB: When an element of $\,Y$ equals zero, the gradient is undefined. This behavior is similar to the derivative of $\,|x|\,$ in the scalar case.