To your first bullet point, you certainly could make it hyperbolic near the point. The (nonunique) geodesic connecting (-1,0) to (1,0) will go around the cusp and look semicircularish (but I don't think it will actually be a semicircle, just approximately one).
Perhaps something easier to visualize is that $\mathbb{R}^2 - \{pt\}$ is diffeomorphic to $S^1\times\mathbb{R}$. If you give this space the product metric of the usual metrics, then you can easily see and work out the details.
To you second bullet point, you should modify the statement slightly (and somewhat pedantically). Any proper nonempty open subset $U$ of a complete connected manifold $M$ is incomplete. To see this, let $p\in U$ and $q\in M-U$. By Hopf-Rinow, since $M$ is complete there is a geodesic $\gamma$ starting at $p$ and ending at $q$. Since $U$ is an open subset, it is totally geodesic: what $U$ thinks are geodesics are precisely what $M$ does. Thus $U$ thinks $\gamma$ is a geodesic which doesn't stay in $U$ for all time, hence $U$ is incomplete.
To the third bullet point, (you say "surface" then use $\mathbb{R}^n$), the theorem is true for $\mathbb{R}^n$ from Cheeger and Gromoll's Soul Theorem together with Perelman's proof of the Soul Conjecture. The soul theorem states that if $M$ is complete and has nonnegative sectional curvature, then $M$ has a compact totally convex totally geodesic submanifold $K$ (called the soul) so that $M$ is diffeomorphic to the normal bundle of $K$. The Soul conjecture asks: If $M$ has nonnegative curvature everywhere and a point with all sectional curvatures positive, must $K$ be a point?
Perelman proved the answer is yes: under these hypothesis, $K$ is a point. But a normal bundle of a point in a manifold is diffeomorphic to $\mathbb{R}^n$.
Finally, I have been told, but I have no idea about references/proofs/etc that every noncompact surface has some metric (necessarily incomplete if the surface isn't $\mathbb{R}^2$ by the above) of positive curvature. I'll try to dig up a reference tomorrow, if I remember to.
If $M$ is a closed, complete hyperbolic manifold, then the hyperbolic structure lifts to its universal cover, $\widetilde{M}$. This means that $\Gamma = \pi_1(M)$ acts on $\widetilde{M}$ by isometries, and $M = \widetilde{M}/\Gamma$. Since $\widetilde{M}$ is isometric to $\mathbb{H}^n$, we have $M = \mathbb{H}^n/\Gamma$. If $\Gamma$ had torsion, then the action would not be fixed-point free and the quotient would have a cone point and not be a manifold, so we can conclude that $\Gamma$ has no torsion.
(Here's a short discussion of bigger picture because it's interesting! Study of hyperbolic surfaces is especially relevant for dimensions $2$ and $3$, where "most" manifolds are hyperbolic. For orientable closed surfaces, the Euler characteristic and Gauss-Bonnet imply that only $S^2$ and $T^2$ are not hyperbolic. Geometrization (Thurston-Perelman are the two biggest names here) states that if a closed $3$-manifold $M$ has: $\pi_1M$ infinite (i.e., $M$ is not covered by $S^3$), $\pi_1(M)$ not a free product (i.e., no incompressible spheres - $M$ is prime), and $\pi_1(M)$ contains no $\mathbb{Z}^2$ (i.e., no incompressible tori), then $M$ is hyperbolic. In three dimensions, once a space is hyperbolic, its geometric invariants become topological invariants by Mostow rigidity: If two hyperbolic manifolds are homotopy-equivalent, then that homotopy-equivalence was induced by an isometry.)
Best Answer
Take the hyperbolic plane $\mathbb H^n$ and remove one point. If $n> 2$ it is simply connected and thus is the universal cover of itself.