The pair / tuple notation used both for gcds and ideals serves to highlight their similarity. Just as in the domain $\,\Bbb Z,\,$ in any PID we have the ideal equality $\,(a,b) = (c)\iff \gcd(a,b) \cong c,\,$ where the congruence means "associate", i.e. they divide each other (differ by only a unit factor). Thus in a PID we can equivalently view $\,(a,b)\,$ as denoting either a gcd or an ideal, and the freedom to move back-and-forth between these viewpoints often proves useful.
Gcds and ideals share many properties, e.g. associative, commutative, distributive laws, and
$$ b\equiv b'\!\!\!\pmod{\!a}\,\Rightarrow\, (a,b) = (a,b')$$
Using the shared properties and notation we can give unified proofs of theorems that hold true for both gcds and ideals, e.g. in the proofs below we can read the tuples either as gcds or ideals
$$(a,b)\,(a^2,b^2)\, =\, (a,b)^3\ \ \ {\rm so}\ \ \ (a,b)=1\,\Rightarrow\, (a^2,b^2) = 1$$
$\quad \color{#c00}{ab = cd}\ \Rightarrow\ (a,c)\,(a,d)\, =\ (aa,\color{#c00}{cd},ac,ad)\, =\, \color{#c00}a\,(a,\color{#c00}b,c,d)\,\ [= (a)\ \ {\rm if}\ \ (a,c,d) = 1] $
Such abstraction aids understanding generalizations and analogies in more general ring-theoretic contexts - which will become clearer when one studies divisor theory, e.g. see the following
Friedemann Lucius. Rings with a theory of greatest common divisors.
manuscripta math. 95, 117-36 (1998).
Olaf Neumann. Was sollen und was sind Divisoren?
(What are divisors and what are they good for?) Math. Semesterber, 48, 2, 139-192 (2001).
Best Answer
Usually the notation $\mathbb{Z}_p$ or $\mathbb{Z}/p\mathbb{Z}$ mean the integers modulo $p$, that is $\{0,\ldots,p-1\}$ where you add and multiply as usually and then take the reminder modulo $p$.
The notation of $\mathbb{Z}_p^\times$ is for those numbers which have a multiplicative inverse modulo $p$, namely all $n$ such that exists $m$ such that $n\cdot m$ is $1$ modulo $p$. These are the numbers coprime to $p$, the greatest common divisor of them and $p$ is $1$.
A subgroup of $\mathbb{Z}_p^\times$ is a subset of these numbers which is closed under multiplication (but not necessarily addition), and every number in this subset also has its multiplicative inverse in there.
For example,
$\mathbb{Z}_5 = \{0,1,2,3,4\}$
$\mathbb{Z}_5^\times = \{1,2,3,4\}$
$\{1,4\}$ is a subgroup of $\mathbb{Z}_5^\times$. Can you see why?