First definition (algebraic): A monoid is a pair $(M,b)$, where $M$ is a set (called the underlying set of the monoid) and $b\colon M\times M\to M$ is a mapping (called the binary operation of the monoid; for $m_1,m_2\in M$ denote $b(m_1,m_2)=m_1\bullet m_2$), which satisfy the following two properties:
1). for any $m_1,m_2,m_3\in M$ the following equality holds (associativity):
$$(m_1\bullet m_2)\bullet m_3=m_1\bullet(m_2\bullet m_3).$$
2). there exists such $e\in M$ (called the identity of the monoid), that for any $m\in M$ the following equalities hold: $e\bullet m=m\bullet e=m$.
It is a standard definition of a monoid. There are a lot of examples of monoids; for example, any group is a monoid. However, there are monoids, which are not groups.
You are right, $(\mathbb{Z},+)$, $(\mathbb{Z},\cdot)$ are monoids, but $(\mathbb{Z},-)$ is not (because, for example, $1-(1-1)\ne(1-1)-1$).
If you are familiar with category theory, then you can get a lot of natural examples in different categories. The reason is following:
Let $A$ be a category, $a\in A$ be its object. Then any morphism $f\colon a\to a$ is called an endomorphism of $a$. Thus we can consider the set of all endomorphisms of the object $a$, denote it by $end(a)$. Note, that we can composite any two morphisms in $end(a)$, therefore we get a binary operation on $end(a)$!
$$
\bullet\colon end(a)\times end(a)\to end(a);\qquad f\bullet g=f\circ g.
$$
It is a monoid by the definition of category.
Now you can take any category, for example $\mathbf{Set}$, take any its object -- for example, $\mathbb{N}$ (which is considered without its algebraic structure), -- and you get the monoid $\mathbb{N}^{\mathbb{N}}$ of all functions $f\colon\mathbb{N}\to\mathbb{N}$, which, of course, is not a group (check it).
Now it is easy to believe in the following definition of a monoid:
Second definition (category-theoretic): Monoid is a category with one object.
Indeed, denote by $x$ its single object, then $end(x)$ is a corresponding monoid. Conversely, if you have a monoid $(M,b)$, you can define a category $\mathbf{M}$ with single object, such that $Arr(\mathbf{M})=M$, and composition defined by the following rule:
$$
\forall f,g\in M:\;f\circ g=f\bullet g.
$$
But it was, of course, intuitive reasoning. In order to make an exact statement, I have to give a few more definitions:
1). Let $(M,\bullet_M)$ and $(N,\bullet_N)$ be monoids. Monoid homomorphism is a mapping $f\colon M\to N$, such that for all $m_1,m_2\in M$ the equalities $f(m_1\bullet_M m_2)=f(m_1)\bullet_N f(m_2)$ and $f(e_M)=e_N$ hold. It's easy to check that all small monoids and their homomorphisms form a category, called $\mathbf{Mon}$.
2). Let's denote by $\mathbf{S}$ the full subcategory of $\mathbf{Cat}$, which objects are categories with fixed single object.
Now I can formulate the exact statement (equivalence of two definitions):
Statement: The category $\mathbf{Mon}$ of monoids is isomorphic to the category $\mathbf{S}$.
You're right: there probably is no term. Almost nobody (or nobody at all?) deals with operations that "aren't closed."
I would recommend looking at it this way: from your original set $X$ and "operation" "$\circ$", let $Y=X\cup \{x\circ y\mid x,y\in X\}$. Then you could call $(Y,\circ)$ a "magma with a partial operation" or something like that.
For your specific example, you would actually have a full-fledged magma with $(\mathbb Z, -)$.
I was going to suggest "magmoid" since I've read that "groupoids" sometimes refer to partially defined operations, but I found people using magmoid for something else already. Well, being explicit with "magma with a partial operation" is probably better anyhow.
Best Answer
These are called magmas, not groupoids.
The ``midpoint'' operation $s\ast t=\frac{s+t}{2}$ on $\mathbb{R}$ makes it a magma which is not a semigroup.