Let $\pi_1$ be the fundamental group of a closed aspherical manifold of dimension $n$. In particular, $\pi_1$ is finitely presented, torsion-free and its cohomology is finitely generated and satisfies Poincaré duality.
Now, we take the center $\mathcal{Z}\pi_1$ (which I assume is finitely generated) and consider the inner automorphism group $Inn(\pi_1)=\pi_1/\mathcal{Z}\pi_1$. I would like prove that $Inn(\pi_1)$ always have an element of infinite order unless $\pi_1\cong\mathbb{Z}^n$.
By a theorem of Schur, if $Inn(\pi_1)$ is finite then $[\pi_1,\pi_1]$ is also finite and therefore $[\pi_1,\pi_1]$ is trivial, since $\pi_1$ is torsion-free. In this case $\pi_1$ is abelian and hence $\pi_1\cong \mathbb{Z}^n$ and the manifold is a torus.
So we only need to exclude the case where $Inn(\pi_1)$ is infinite periodic group (all elements are torsion elements).
Note that if $\pi_1$ is centerless then $Inn(\pi_1)\cong\pi_1$ which is torsion-free, so I was looking onto cases where the center is large (for example $\mathbb{Z}^{n-1}$).
I was mainly trying to use some cohomological arguments. For example, $H_2(Inn(\pi_1),\mathbb{Z})$ needs to be finitely generated. In this paper, they prove that $H_2(B(a,b),\mathbb{Z})$ of the free Burnside group with odd $b\geq 665$ has countable rank (in conseqeunce $Inn(\pi_1)\ncong B(a,b)$). A similar argument from this paper, can be used to show that $Inn(\pi_1)$ cannot be an infinite 2-group of bounded exponent.
Finally, $Inn(\pi_1)$ is finitely presented, but it seems that the Burnside problem for finitely presented groups is still open (per this mathoverflow question).
Maybe there is a trivial reason for $Inn(\pi_1)$ to have always elements of infintie order that has slipped out of my mind, but I don't see it rigth now.
Best Answer
If $G=Inn(\pi_1)$ is a 2-group, then it must be finite (and then you are done by Schur’s argument above).
As you describe, $\pi_1$ is a central extension of $G$ by $\mathbb{Z}^k$ for some $k\leq n$, since you are assuming $Z(\pi_1)$ is finitely generated. We may take the quotient $\pi_1/2\mathbb{Z}^k$ to get a central extension of $G$ by $(\mathbb{Z}/2\mathbb{Z})^k$.
For each element $a\in \pi_1$, $Z_a=\langle a, \mathbb{Z}^k\rangle$ is an abelian subgroup of $\pi_1$ isomorphic to $\mathbb{Z}^k$. Hence $Z_a/2\mathbb{Z}^k$ is a non-trivial (abelian) central extension of $<a>$ by $(\mathbb{Z}/2\mathbb{Z})^k$ with some factor isomorphic to $\mathbb{Z}/2|a|\mathbb{Z}$. Modding out by the appropriate subgroup of $Z(\pi_1)\cong \mathbb{Z}^k$, we get a central extension of $G$ by $\mathbb{Z}/2\mathbb{Z}$ inducing a non-trivial central extension of $<a>$. Hence the map $H^2(G;\mathbb{F}_2)\to H^2(<a>;\mathbb{F}_2)$ is non-trivial.
Now specialize to $a$ an involution. Then we must have $H^1(G;\mathbb{F}_2)\to H^1(<a>;\mathbb{F}_2)\cong \mathbb{F}_2$ non-trivial, since the cohomology ring of $H^*(<a>;\mathbb{F}_2) \cong \mathbb{F}_2[x]$. Hence $<a> \to H_1(G)\cong G/[G,G]$ is non-trivial. If every involution of $G$ injects into $H_1(G)$, then every non-trivial element of $G$ injects into $H_1(G)$ since $G$ is assumed to be a 2-group and hence has a power that is an involution. Hence $G$ injects into $H_1(G)$, so it is finite and abelian since it is finitely generated.