It's not the derivative with respect to a matrix really. It's the derivative of $f$ with respect to each element of a matrix and the result is a matrix.
Although the calculations are different, it is the same idea as a Jacobian matrix. Each entry is a derivative with respect to a different variable.
Same goes with $\frac{\partial f}{\partial \mu}$, it is a vector made of derivatives with respect to each element in $\mu$.
You could think of them as $$\bigg[\frac{\partial f}{\partial \Sigma}\bigg]_{i,j} = \frac{\partial f}{\partial \sigma^2_{i,j}} \qquad \text{and}\qquad \bigg[\frac{\partial f}{\partial \mu}\bigg]_i = \frac{\partial f}{\partial \mu_i}$$
where $\sigma^2_{i,j}$ is the $(i,j)$th covariance in $\Sigma$ and $\mu_i$ is the $i$th element of the mean vector $\mu$.
For typing convenience, let me substitute Latin in place of your Greek letters
$$\eqalign{
X &= \Xi \cr
x &= \vartheta = \operatorname{vec}(X) \cr
W &= \Omega = I + XX^T = W^T \cr\cr
}$$
Then for your first function, the differential and gradient can be calculated as
$$\eqalign{
f &= \log\det W \cr
\cr
df &= d\log\det W = d\operatorname{tr}\log W \cr
&= W^{-T}:dW \cr
&= W^{-T}:(dX\,X^T+X\,dX^T) \cr
&= \big(W^{-T}X+W^{-1}X\big):dX \cr
&= 2\,W^{-1}X:dX \cr
&= 2\,\operatorname{vec}(W^{-1}X)\cdot\,dx \cr
\cr
\frac{\partial f}{\partial x} &= 2\,\operatorname{vec}(W^{-1}X) \cr\cr
}$$
In your second function, I don't quite understand the definition of $\,H(\vartheta)\,$ so all I can offer is a partial solution
$$\eqalign{
M &= W^{-1} = M^T \cr
f &= H:MH \cr
\cr
df &= dH:MH + H:M\,dH + H:dM\,H \cr
&= (M+M^T)H:dH + HH^T:dM \cr
&= 2\,MH:dH - HH^T:M\,dW\,M \cr
&= 2\,MH:dH + MHH^TM:dW \cr
&= 2\,MH:dH + MHH^TM:(dX\,X^T+X\,dX^T) \cr
&= 2\,MH:dH + 2\,MHH^TMX:dX \cr
\cr
}$$
You can finish off the solution by expanding $dH$ in terms of $dX$, then vectorizing.
In the above, a colon denotes the double-dot (aka Frobenius) product, which is merely a product notation for the trace, i.e. $$A:B=\operatorname{tr}(A^TB)$$
Update
I'm guessing that $H$ is a partitioned matrix: $H=[\,X, I\,]$
Expanding that term in the differential
$$\eqalign{
MH:dH &= [\,MX, M\,]:[\,dX, 0\,] \cr
&= MX:dX + M:0 \cr
}$$
So, continuing with the full differential
$$\eqalign{
df &= 2\,\Big(MX + MHH^TMX\Big):dX \cr
&= 2\,\operatorname{vec}(MX + MHH^TMX)\cdot dx \cr
\cr
\frac{\partial f}{\partial x} &= 2\,\operatorname{vec}(MX + MHH^TMX) \cr
}$$
Best Answer
Jacobi's formula for the derivative of the determinant is usually stated as follows:
Theorem. For a path $t\mapsto A(t)$ of matrices, one has \begin{equation} \partial_t\det A(t)=\mathrm{Tr}\left(\mathrm{adj}(A(t))\,\dot{A}(t)\right). \end{equation}
This Wikipedia page explains where this formula comes from. To use this result for your question, consider your parameter-dependent matrix as a matrix-valued function $A:\mathbb{R}^3\to M_{n\times n}(\mathbb{C})$. The derivative $\partial_{v}A$ is the directional derivative of this function, so for any point $p\in\mathbb{R}^3$ we want to compute \begin{equation} \partial_v\det A(p)=\partial_t\Bigr|_{t=0}\det A(p+t\,v). \end{equation} Let $B(t)=A(p+t\,v)$ and we can apply Jacobi's formula to get \begin{equation} \partial_v\det A(p)=\partial_t\Bigr|_{t=0}\det B(t)=\mathrm{Tr}\left(\mathrm{adj}(B(0))\,\partial_t\Bigr|_{t=0}B(t)\right)=\mathrm{Tr}\left(\mathrm{adj}(A)\,\partial_vA\right)(p). \end{equation}
A more geometric way to think about this: "$\det$" is a function from the set of $n$ by $n$ matrices to the complex (or real) numbers. Jacobi's formula is telling us what the differential of this function is, namely \begin{equation} \begin{split} \mathrm{d}(\mathrm{det})_A:T_AM_{n\times n}(\mathbb{C})&\to\mathbb{C}\\ \dot{A}&\mapsto\mathrm{Tr}\left(\mathrm{adj}(A)\,\dot{A}\right). \end{split} \end{equation} What you want to compute is the differential of $\det$ composed with some function $A:\mathbb{R}^3\to M_{n\times n}(\mathbb{C})$, and the differential of the composition is the composition of the differentials \begin{equation} \mathrm{d}\big((x,y,z)\mapsto\det A(x,y,z)\big)_p(v)=\mathrm{d}(\mathrm{det})_{A(p)}\left(\partial_vA\right)=\mathrm{Tr}\left(\mathrm{adj}(A)\,\partial_vA\right)(p). \end{equation}
For example, assume that $A(x,y,z)$ is the simple matrix-valued function $$A(x,y,z)=\begin{pmatrix} x & 0 & 0 \\ 0 & y & 0\\ 0& 0 & z\end{pmatrix}.$$ Then the determinant is $\det A(x,y,z)= x y z$. Let's fix the vector $v=(a,b,c)$, then the derivative of the determinant is $\partial_v\left(\det A(x,y,z)\right)=ayz+bxz+cxy$. We can check that this agrees with the formula above: since $$\partial_vA(x,y,z)= \begin{pmatrix} a & 0 & \\ 0 & b & 0\\ 0& 0 & c\end{pmatrix}$$ we get \begin{equation} \begin{split}\partial_v&\left(\det A(x,y,z)\right)=\det (A)\,\mathrm{Tr}\left(A^{-1}\partial_vA\right)=\\ &=xyz\,\mathrm{Tr}\left( \begin{pmatrix} x^{-1} & 0 & 0 \\ 0 & y^{-1} & 0\\ 0& 0 & z^{-1}\end{pmatrix}\begin{pmatrix} a & 0 & \\ 0 & b & 0\\ 0& 0 & c\end{pmatrix} \right)=ayz+bxz+cxy. \end{split} \end{equation}