It's not the derivative with respect to a matrix really. It's the derivative of $f$ with respect to each element of a matrix and the result is a matrix.
Although the calculations are different, it is the same idea as a Jacobian matrix. Each entry is a derivative with respect to a different variable.
Same goes with $\frac{\partial f}{\partial \mu}$, it is a vector made of derivatives with respect to each element in $\mu$.
You could think of them as $$\bigg[\frac{\partial f}{\partial \Sigma}\bigg]_{i,j} = \frac{\partial f}{\partial \sigma^2_{i,j}} \qquad \text{and}\qquad \bigg[\frac{\partial f}{\partial \mu}\bigg]_i = \frac{\partial f}{\partial \mu_i}$$
where $\sigma^2_{i,j}$ is the $(i,j)$th covariance in $\Sigma$ and $\mu_i$ is the $i$th element of the mean vector $\mu$.
For typing convenience, let me substitute Latin in place of your Greek letters
$$\eqalign{
X &= \Xi \cr
x &= \vartheta = \operatorname{vec}(X) \cr
W &= \Omega = I + XX^T = W^T \cr\cr
}$$
Then for your first function, the differential and gradient can be calculated as
$$\eqalign{
f &= \log\det W \cr
\cr
df &= d\log\det W = d\operatorname{tr}\log W \cr
&= W^{-T}:dW \cr
&= W^{-T}:(dX\,X^T+X\,dX^T) \cr
&= \big(W^{-T}X+W^{-1}X\big):dX \cr
&= 2\,W^{-1}X:dX \cr
&= 2\,\operatorname{vec}(W^{-1}X)\cdot\,dx \cr
\cr
\frac{\partial f}{\partial x} &= 2\,\operatorname{vec}(W^{-1}X) \cr\cr
}$$
In your second function, I don't quite understand the definition of $\,H(\vartheta)\,$ so all I can offer is a partial solution
$$\eqalign{
M &= W^{-1} = M^T \cr
f &= H:MH \cr
\cr
df &= dH:MH + H:M\,dH + H:dM\,H \cr
&= (M+M^T)H:dH + HH^T:dM \cr
&= 2\,MH:dH - HH^T:M\,dW\,M \cr
&= 2\,MH:dH + MHH^TM:dW \cr
&= 2\,MH:dH + MHH^TM:(dX\,X^T+X\,dX^T) \cr
&= 2\,MH:dH + 2\,MHH^TMX:dX \cr
\cr
}$$
You can finish off the solution by expanding $dH$ in terms of $dX$, then vectorizing.
In the above, a colon denotes the double-dot (aka Frobenius) product, which is merely a product notation for the trace, i.e. $$A:B=\operatorname{tr}(A^TB)$$
Update
I'm guessing that $H$ is a partitioned matrix: $H=[\,X, I\,]$
Expanding that term in the differential
$$\eqalign{
MH:dH &= [\,MX, M\,]:[\,dX, 0\,] \cr
&= MX:dX + M:0 \cr
}$$
So, continuing with the full differential
$$\eqalign{
df &= 2\,\Big(MX + MHH^TMX\Big):dX \cr
&= 2\,\operatorname{vec}(MX + MHH^TMX)\cdot dx \cr
\cr
\frac{\partial f}{\partial x} &= 2\,\operatorname{vec}(MX + MHH^TMX) \cr
}$$
Best Answer
Define the variables $$\eqalign{ F &= {\large\tt 1} - I \cr P &= I+kF \cr S &= \Sigma = P\circ vv^T \cr }$$ Write the function of interest in terms of these variables and find its differential. $$\eqalign{ \phi &= \log(\det(S)) \cr d\phi &= S^{-1}:dS \cr }$$ Expand $dS$ in terms of $dv$ and find the gradient wrt $v$. $$\eqalign{ d\phi &= S^{-1}:P\circ(dv\,v^T+v\,dv^T) \cr &= 2P\circ S^{-1}:dv\,v^T \cr &= 2(P\circ S^{-1})v:dv \cr \frac{\partial \phi}{\partial v} &= 2(P\circ S^{-1})v \cr }$$ Expand $dS$ in terms of $dk$ and find the gradient wrt $k$. $$\eqalign{ d\phi &= S^{-1}:dP\circ vv^T \cr &= S^{-1}:(dk\,F)\circ vv^T \cr &= S^{-1}:({\large\tt 1}-I)\circ vv^T\,dk \cr &= S^{-1}:(vv^T-I\circ vv^T)\,dk \cr \frac{\partial\phi}{\partial k} &= S^{-1}:(vv^T-I\circ vv^T) \cr\cr }$$ In the steps above, a colon denotes the trace/Frobenius product, i.e. $$A:B = {\rm Tr}(A^TB)$$ which commutes with the elementwise/Hadamard product $$A:B\circ C = A\circ B:C$$ and, of course, the Hadamard and Frobenius products are themselves commutative $$\eqalign{ A:B &= B:A \cr A\circ B &= B\circ A \cr }$$