First of all, let $X$ be a reasonably nice space, say, metrizable and locally compact. Define
$$
H^i(Ends(X))=\lim_K H^i(X-K),
$$
where the direct limit is taken over compact subsets $K$ in $X$. (Similarly, one defines $H_i(Ends(X))$ by taking the inverse limit.)
In fact, these groups are the Chech cohomology groups of the space of ends of $X$ but I will not need this.
The space $X$ has more than one end if and only if
$$
\tilde{H}^0(Ends(X))\ne 0,
$$
where I am using the reduced cohomology. On the other hand, the cohomology with local support of $X$ satisfies
$$
H^1_c(X)\cong \lim_K H^1(X, X-K).
$$
Assume now that $X$ is acyclic as in your case. Then by the long exact sequence of a pair,
$$
\lim_K H^1(X, X-K) \cong \lim_K \tilde{H}^0(X-K)\cong \tilde{H}^0(Ends(X)).
$$
By the Alexander duality, assuming that $X$ is an $n$-dimensional homology manifold,
$$
H^1_c(X)\cong H_{n-1}(X).
$$
Hence, since $X$ is acyclic, $$\tilde{H}^0(Ends(X))\cong H^1_c(X)\cong H_{n-1}(X)=0,$$ i.e. $X$ has exactly one end.
Let $M$ be a closed manifold of dimension $n\ge1$. Choose a point $p\in U\subseteq M$, where $(\overline{U},U)\cong(D^n,B^n)$. Consider the open cover $M=M\setminus\{p\}\cup U$, where $M\setminus\{p\}\cap U=U\setminus\{p\}\simeq S^{n-1}$. The reduced M-V-sequence of this open cover immediately implies that the map $\tilde{H}_k(M\setminus\{p\})\rightarrow\tilde{H}_k(M)$ induced by the inclusion is an isomorphism if $k\neq n-1,n$. As you've noted, $M\setminus\{p\}$ is non-compact (here, we use that $M$ is not zero-dimensional), so $\tilde{H}_n(M\setminus\{p\})=0$. The critical part of this sequence reads
$$0=\tilde{H}_n(M\setminus\{p\})\rightarrow\tilde{H}_n(M)\rightarrow\tilde{H}_{n-1}(S^{n-1})\rightarrow\tilde{H}_{n-1}(M\setminus\{p\})\rightarrow\tilde{H}_{n-1}(M)\rightarrow\tilde{H}_{n-2}(S^{n-1})=0.$$
I claim that if $M$ is orientable, the map $\tilde{H}_n(M)\rightarrow\tilde{H}_{n-1}(S^{n-1})$ is an isomorphism. From that, it follows that $\tilde{H}_{n-1}(M\setminus\{p\})\rightarrow\tilde{H}_{n-1}(M)$ is an isomorphism also.
To see this, consider the quotient $M/(M\setminus U)\cong S^n$ (here, we use that $(\overline{U},U)\cong(D^n,B^n)$ in full strength). The quotient map $M\rightarrow M/(M\setminus U)$ takes our open cover of $M$ to the standard cover of a sphere by the complements of two respective points (the points being $p$ and $\partial U/\partial U$). It induces an isomorphism on $\tilde{H}_n$ (the induced map is the composite of the isomorphism $\tilde{H}_n(M)\rightarrow H_n(M,M\setminus U)$, from the general theory of top homology, and the isomorphism $H_n(M,M\setminus U)\rightarrow H_n(M/(M\setminus U),(M\setminus U)/(M\setminus U))\cong\tilde{H}_n(M/(M\setminus U))$, from excision) and the boundary map of the M-V-sequence of the sphere is known to be an isomorphism, so by naturality the boundary map $\tilde{H}_n(M)\rightarrow\tilde{H}_{n-1}(S^{n-1})$ of our original M-V-sequence is an isomorphism as well.
(If $M$ is triangulable, here's an argument with more geometric flavor: the simplices of a triangulation with appropriate signs give a chain that represents a generator of $\tilde{H}_n(M)$. Assume WLOG that $p$ is an interior point of one (and necessarily only one) simplex of the triangulation and that this simplex is contained in $U$ (subdivide if necessary). The boundary map of the M-V-sequence then maps the fundamental class to the homology class represented by the boundary of that simplex, which is a generator of $\tilde{H}_{n-1}(S^{n-1})$.)
Furthermore, if $n\ge2$, the inclusion induces a surjection $\pi_1(M\setminus\{p\})\rightarrow\pi_1(M)$ (basepoints implicit) by the Seifert-van-Kampen theorem (applied to the same open cover $M=M\setminus\{p\}\cup U$; we need $n\ge2$ so that the intersection $M\setminus\{p\}\cap U=U\setminus\{p\}\simeq S^{n-1}$ is connected).
In total, if $M$ is a homology sphere, which is not a sphere, it has dmiension $n=\dim(M)\ge2$ by classification, is compact since it has non-vanishing top homology and is orientable, since $H_1(M)=0$ implies that the orientation character is trivial. Thus, $\tilde{H}_k(M\setminus\{p\})=\tilde{H}_k(M)=0$ for $k=0,\dotsc,n-1$ by the preceeding discussion and $\tilde{H}_k(M\setminus\{p\})=0$ for $k\ge n$ by general theory, so $M\setminus\{p\}$ is acyclic. Furthermore, $\pi_1(M)\neq1$ (otherwise $M$ would be a sphere), so $\pi_1(M\setminus\{p\})\neq1$, since it surjects onto $\pi_1(M)$. It follows that $M\setminus\{p\}$ is not contractible.
Best Answer
A manifold with boundary is defined to be orientable if $int(M) = M \setminus \partial M$ is orientable. Since simply connected manifolds are orientable, it suffices to show that $int(M)$ is simply connected.
In fact $int(M)$ is contractible. It is known that $\partial M$ has an open collar neighborhood $N$ in $M$ (see Hatcher, Proposition 3.42). Let $h : \partial M \times [0,1) \to N$ be a homeomorphism such that $h(x,0) = x$ for all $x$. Define $M' = M \setminus h(\partial M \times [0,1/2)) \subset int(M)$. This is a homeomorphic copy of $M$. But obviously $M'$ is a strong deformation retract of $int(M)$, hence $int(M) \simeq M' \simeq \ast$.
By the way, open collar neighborhoods exist for any manifold with boundary. Hatcher proves it only in the compact case.