In the first place, it is unclear to me what a sheaf of $\mathcal{A}$-objects on $X$ should be. The obvious candidate, if you are used to the functor definition of sheaf, is that a sheaf of $\mathcal{A}$-objects on $X$ is a functor $\textrm{Open} (X)^\textrm{op} \to \mathcal{A}$ that satisfies the usual sheaf condition. I think this is the wrong definition and will fail to give an abelian category in general – at least, it is not obvious to me where cokernels are going to come from. On the other hand, there is another candidate, more in line with the original definition of sheaf in terms of stalks, and this one yields an abelian category. In more detail:
Let $\mathcal{A}$ be an abelian category and suppose $\mathcal{A}$ is an exact full subcategory of an abelian category $\mathcal{B}$ that satisfies Grothendieck's axioms AB3, AB3*, and AB5. (For example, if $\mathcal{A}$ is a Grothendieck abelian category we may take $\mathcal{B} = \mathcal{A}$; or if $\mathcal{A}$ is an essentially small abelian category we may take $\mathcal{B} = \textbf{Ind} (\mathcal{A})$.)
Definition. A sheaf of $\mathcal{A}$-objects on $X$ is a functor $F : \textrm{Open} (X)^\textrm{op} \to \mathcal{B}$ with the following properties:
- $F : \textrm{Open} (X)^\textrm{op} \to \mathcal{B}$ satisfies the sheaf condition.
- For all $x \in X$, the stalk $F_x$ (computed in $\mathcal{B}$) is (isomorphic to) an object in $\mathcal{A}$.
Remark. It does not follow that $F (U)$ is (isomorphic to) an object in $\mathcal{A}$! For example, if $X$ is an infinite discrete set and $\mathcal{A}$ is the category of finite abelian groups (and $\mathcal{B}$ is the category of all abelian groups), then $F (X)$ can be infinite even if every $F_x$ is finite.
Proposition. The category of sheaves of $\mathcal{A}$-objects on $X$ is an abelian category.
Proof. The category of sheaves of $\mathcal{B}$-objects on $X$ is a full subcategory of the category of all functors $\textrm{Open} (X)^\textrm{op} \to \mathcal{B}$ and has an exact reflector. The category of all functors $\textrm{Open} (X)^\textrm{op} \to \mathcal{B}$ is an abelian category, so it follows that the category of sheaves of $\mathcal{B}$-objects on $X$ is also an abelian category. Taking stalks is also exact, so the category of sheaves of $\mathcal{A}$-objects is an exact full subcategory of the category of sheaves of $\mathcal{B}$-objects, hence is also an abelian category. ■
What is not clear to me is whether the category of sheaves of $\mathcal{A}$-objects on $X$ defined in this way is independent of the choice of embedding $\mathcal{A} \hookrightarrow \mathcal{B}$. If it is then I could be convinced this is the correct definition.
Yes this is indeed such a correspondance. The main reason is that, if you have a $t$-structure with heart $\mathcal{B}$ and if $H$ denote the corresponding cohomology, then a triangle in $\mathcal{D}$ gives rise to a long exact sequence in $\mathcal{B}$.
Now, if $A\to B\to C\to A[1]$ is a distinguished triangle in $\mathcal{D}$, such that $A,B,C\in\mathcal{B}$, then the long exact sequence gives in particular
$$H^{-1}(C)\to H^0(A)\to H^0(B)\to H^0(C)\to H^1(A)$$
but since $A,B,C$ belong to the heart, this five terms exact sequence is actually
$$0\to A\to B\to C\to 0$$
so you have indeed a short exact sequence in $\mathcal{B}$.
Conversely, let $0\to A\to B\to C\to 0$ a short exact sequence in $\mathcal{B}$. Let $C'$ be the cone of $A\to B$. Because the composition $A\to B\to C$ is zero, the morphism $B\to C$ factors through $C'$ (not necessarily uniquely). So we have a commutative diagram :
$$\require{AMScd}
\begin{CD}
A@>>>B@>>>C'@>>>A[1]\\
@|@|@VVV\\
A@>>>B@>>>C
\end{CD}
$$
The top row is a distinguished triangle, we want to show that $C'\to C$ is an isomorphism. Now apply the cohomology functor, by naturality we have a commutative diagram :
$$\require{AMScd}
\begin{CD}
0@>>>H^{-1}(C')@>>>A@>>>B@>>>H^0(C')@>>>0\\
@.@.@|@|@VVV\\
@.0@>>>A@>>>B@>>>C@>>>0
\end{CD}
$$
The top row is exact by the long exact sequence of cohomology, with the outer $0$ because $A$ and $B$ are in the heart. The bottom row is also exact by assumption. By diaram chase, $H^{-1}(C')=0$ and $H^0(C')=C$. The long exact sequence also give $H^i(C')=0$ for all $i\neq -1, 0$. It follows that $C'$ is concentrated in degree $0$ and thus that $C'= H^0(C')= C$.
Best Answer
I should have thought about this question just a little bit more.
The answer is no. Let $\mathcal{A}$ be the category of finite dimensional vector spaces over a field $k$. Then the product $\mathcal{A}\times \mathcal{A}$ embedded in $\mathcal{D}$ as the subcategory of complexes concentrated in degrees 0 and 1 is not a heart of a $t$-structure since there may be nonzero elements of $\text{Hom}(X,Y[1])$ for $X,Y\in\mathcal{A}\times\mathcal{A}\subset\mathcal{D}$.