Disclaimer: It is very possible that the notation I will be using is not formally correct. I am not a mathematician, and I'm just trying to write down this idea in a reasonable way. I wasn't able to find literature addressing this topic
As an example, take the family of univariate normal distributions with the sum operation. Let see if it satisfies the four defining properties of a group:
Closure: Given $X \sim \mathcal{N}(\mu_1,\,\sigma_1^{2})$ and $Y \sim \mathcal{N}(\mu_2,\,\sigma_2^{2})$, $Z = X+Y$ is also normally distributed, with $Z \sim \mathcal{N}(\mu_1+\mu_2,\,\sigma_1^{2}+\sigma_2^{2})$
Associativity: It is clear that $(X + Y) + Z = X + (Y + Z)$ in distribution, with $X, Y, Z$ each independently and normally distributed with given means and variances.
Identity element: The distribution $\mathcal{N}(0,\,0)$ is the identity element, because any $I \sim \mathcal{N}(0,\,0)$ satisfies the property that $Y = X + I = I + X$ is distributed as $X$, with $X \sim \mathcal{N}(\mu_1,\,\sigma_1^{2})$.
Inverse element: In this case for any $X \sim \mathcal{N}(\mu_1,\,\sigma_1^{2})$, there exists $-X$ such that $X-X = -X+X = I$, with $I \sim \mathcal{N}(0,\,0)$.
The four properties seem to be satisfied, therefore couldn't we say that the family of univariate normal distributions with the sum operation constitutes a group? And the same with other several families of distributions and different operations such as sum, product, ratio, etc.
Edit: As pointed out below, I should have added that $X$ and $Y$ have to be independent in the first point, otherwise the expression for the variance of the addition is not correct.
Best Answer
You neglected independence.
As it stands, this is not true. It is true if $X,Y$ are independent. Thus your group operation corresponds to addition of independent random variables. There is no inverse element because there is no probability distribution for which, if $Y$ has that distribution and $X$ is as above and $X,Y$ are independent, then $X+Y \sim \mathcal N(0,0).$ In particular, $X$ and $-X$ are not independent. So this is a semigroup but not a group.
Variances are always nonnegative, and the variance of the sum of two independent random variables is at least as big as either of their variances, so you won't get back to $0.$