Assume $X, Y$ are positive definite and $\lambda \in [0,1]$.
We know $Z(\lambda) = (1-\lambda) X + \lambda Y$ is positive definite and so does
$Z^{-1}(\lambda)$. Let us use $(\ldots)'$ to represent derivative with respect to $\lambda$, we have:
$$
Z Z^{-1} = I_n
\implies Z'Z^{-1} + Z ( Z^{-1} )' = 0_n
\implies (Z^{-1})' = - Z^{-1} Z' Z^{-1}
$$
Differentiate one more time and notice $Z'' = 0_n$, we get:
$$(Z^{-1})'' = - (Z^{-1})' Z' Z^{-1} - Z^{-1}Z' (Z^{-1})' = 2 Z^{-1}Z' Z^{-1} Z' Z^{-1}\tag{*1}$$
Pick any random non-zero vector $u$ and consider following pair of vector/matrix valued functions:
$$v(\lambda) = Z'(\lambda) Z^{-1}(\lambda) u\quad\quad\text{ and }\quad\quad
\varphi(\lambda) = u^T Z^{-1}(\lambda) u$$
$(*1)$ tell us
$$\varphi''(\lambda) = u^T (Z^{-1})''(\lambda) u = 2 v^T(\lambda) Z^{-1}(\lambda) v(\lambda) \ge 0\tag{*2}$$
because $Z^{-1}(\lambda)$ is positive definite. From this we can conclude $\varphi(\lambda)$ is a convex function for $\lambda$ over $[0,1]$. As a result, for any $\lambda \in (0,1)$, we have:
$$\begin{align}&(1-\lambda)\varphi(0) + \lambda\varphi(1) - \varphi(\lambda) \ge 0\\
\iff& u^T \left[ (1-\lambda) X^{-1} + \lambda Y^{-1} - ((1-\lambda) X + \lambda Y)^{-1}\right] u \ge 0\tag{*3}
\end{align}$$
Since $u$ is arbitrary, this implies the matrix within the square bracket in $(*3)$
is positive semi-definite and hence:
$$(1-\lambda) X^{-1} + \lambda Y^{-1} \succeq ((1-\lambda) X + \lambda Y)^{-1}$$
Please note that when $Z' = Y - X$ is invertible, $v(\lambda)$ is non-zero for non-zero $u$. The inequalities in $(*2)$ and $(*3)$ become strict and the matrix within
the square bracket in $(*3)$ is positive definite instead of positive semi-definite.
Best Answer
If $A$ is symmetric and positive-definite, then the answer is yes.
It follows, e.g., straight from the matrix inversion Lemma (e.g., Matrix Analysis by Horn or this mathoverflow answer) or standard blockwise inverse results (as the one in Appendix).
Lemma. Let $A\in\mathbb{S}_{++}^N$ be an $N\times N$ positive-definite symmetric matrix. Let $S\subset \left\{1,2,\ldots,N\right\}$. Then,
$$\left[A^{-1}\right]_{S}\succeq \left(A_S\right)^{-1},$$
where $B_{S}$ or $\left[B\right]_S$ is the $\left|S\right|\times \left|S\right|$ submatrix formed with the elements of $B$ indexed by $S\times S$.
Proof.
From the inversion Lemma (or refer to the Appendix below), you have
$$\left(A_S\right)^{-1}=\left[A^{-1}\right]_S-\underbrace{\left[A^{-1}\right]_{SS'}\left(\left[A^{-1}\right]_{S'}\right)^{-1}\left[A^{-1}\right]_{S'S}}_{=:\mathcal{E}_S},$$
where $S'$ is the complement of $S$ and $\mathcal{E}_S$ may be interpreted as the error matrix that stems from commuting the projection $\left[\cdot\right]_{S}$ and the inversion $\left(\cdot\right)^{-1}$ operations.
Therefore, $\left[A^{-1}\right]_S \succeq \left(A_S\right)^{-1}$ if and only if $\mathcal{E}_S\succeq 0$.
It is straighforward to check that $\mathcal{E}_S\succeq 0$. Indeed, if $A\succ 0$, then $\left(\left[A^{-1}\right]_{S'}\right)^{-1}\succ 0$, as the positive-definiteness property is preserved under projection $\left[\cdot\right]_S$ and inversion $\left(\cdot\right)^{-1}$. Further, since $A$ is symmetric, then $\left[A^{-1}\right]_{SS'}$ is the transpose of $\left[A^{-1}\right]_{S'S}$.
----------------- Appendix: Blockwise inversion -------------------------
Since $A$ is invertible and symmetric, we can write it as $A=\left[\begin{array}{cc} \widetilde{A} & \widetilde{B}\\ \widetilde{B}^{\top} & \widetilde{D} \end{array}\right]^{-1}.$
From this inverse block matrix result in equation (2), -- that relates the sub-block $A_S$ with the Schur Complement of $A^{-1}$ -- we have
$$A_S=\left(\left[A^{-1}\right]_S-\left[A^{-1}\right]_{SS'}\left(\left[A^{-1}\right]_{S'}\right)^{-1}\left[A^{-1}\right]_{S'S}\right)^{-1},$$
since $\left[A^{-1}\right]_S=\widetilde{A}$; $\left[A^{-1}\right]_{S'}=\widetilde{D}$; $\left[A^{-1}\right]_{SS'}=\widetilde{B}$ and $\left[A^{-1}\right]_{SS'}=\widetilde{B}^{\top}$. Therefore,
$$\left(A_S\right)^{-1}=\left[A^{-1}\right]_S-\left[A^{-1}\right]_{SS'}\left(\left[A^{-1}\right]_{S'}\right)^{-1}\left[A^{-1}\right]_{S'S}.$$