Suppose $G$ is a group, $Z(G)=\{g \mid\ \forall a \in G,\ gag^{-1}a^{-1}=e\}$ is the center of $G$, and $K(G)=\{gag^{-1}a^{-1}\mid a,g\in G\}$ is the subset of all commutators.
I am wondering whether there is any way to find $|K(G)|$ based on $|G|$ and $|Z(G)|$.
I have tried to use group actions, but, since $K(G)$ ranges over two variables ($g$ and $a$), I could not succeed. It seems it needs a generalization of group actions so that two elements of a group act on an element of a set simultaneously.
If we consider $K(a)= \{gag^{-1}a^{-1}\mid g\in G\}$ then $|K(a)|=|\bar{a}|=\frac{|G|}{|Z_G(a)|}$, where $Z_G(a)=\{g \mid gag^{-1}a^{-1}=e\}$ is the centralizer of $a$ and $\bar{a}=\{gag^{-1}\mid g\in G\}$ is the conjugacy class of $a$.
How can we find a formula among $|K(G)|$,$|G|$, and $|Z(G)|$ like $|K(a)|=\frac{|G|}{|Z_G(a)|}$?
Edited
I changed the notations from $C(G)$ to $Z(G)$, $C_G(a)$ to $Z_G(a)$, and from $Z$ to $K(G)$ so that they become more standard.
Best Answer
There is no formula, because you can have two groups in which two of the quantities agree but the third is different.
I will use the usual notation: let $G$ be a group. We use $Z(G)$ to denote the center of $G$, $Z(G) = \{z\in G\mid zx=xz\text{ for all }x\in G\}$, and $K(G)$ to denote the set of commutators of $G$, $$K(G) = \{ [x,y]\mid x,y\in G\},\qquad\text{where }[x,y]=xyx^{-1}y^{-1}.$$ Then the commutator subgroup of $G$ is $[G,G]=\langle K(G)\rangle$.
I had the most trouble finding groups of the same size, with centers of the same size but commutator set of different sizes, though that is the most relevant for your purposes. I will leave it last.
Groups $G_1$ and $G_2$ with $|Z(G_1)|=|Z(G_2)|$, $|K(G_1)|=|K(G_2)|$, but $|G_1|\neq|G_2|$
Take $G_1=S_3\times C_3$, where $C_3$ is the cyclic group of order $3$, and let $G_2$ be the Heisenberg group of order $3^3$. Then $H$ has $K(H)=[H,H]=Z(H)$ of order $3$. So both $G_1$ and $G_2$ have center of order $3$, set of commutators (which equals the commutator subgroup) of order $3$, but $|G_1|=18$ and $|G_2|=27$.
Groups $G_1$ and $G_2$ with $|G_1|=|G_2|$, $|K(G_1)|=|K(G_2)|$, but $|Z(G_1)|\neq|Z(G_2)|$
Take $G_1=S_3\times S_3\times C_{81}$, and $G_2=H\times H\times C_4$, where $H$ is the Heisenberg group of order $27$. Both groups have order $2^2\times 3^{6}$, and commutator subgroup equal to the set of commutators, of order $9$. But the center of $G_1$ has order $81$ and the center of $G_2$ has order $36$.
Groups $G_1$ and $G_2$ with $|G_1|=|G_2|$, $|Z(G_1)|=|Z(G_2)|$, but $|K(G_1)|\neq |K(G_2)|$
Let $G_1 = D_{16}\times M$, where $D_{16}$ is the dihedral group of order $16$, and $M$ is a group of order $3^4$ that has center of order $3$ and is of class $3$ (that is, of maximal class). Such a group has $[M,M]$ of order $3^2$.
It is a theorem of Guralnick that the smallest groups in which $K(G)\neq [G,G]$ have order $92$ (and there two such nonisomorphic groups), so it follows from this (or from direct computation) that $[M,M]=K(M)$, so $K(M)$ has order $3^2$.
We have that that $K(G_1) = K(D_{16})\times K(M)$, which has order $2^2\times 3^2 = 36$; and $Z(G_1) = Z(D_{16})\times Z(M)$, which has order $2\times 3 = 6$.
Now consider $G_2 = S_3\times D_8\times H$, where $H$ is the Heisenberg group of order $3^3$. We have that $K(G_2) = K(S_3)\times K(D_8)\times K(H)$, which has order $3\times 2\times 3 = 18$. On the other hand, the center is $Z(G_2) = Z(S_3)\times Z(D_8)\times Z(H)$, which has order $2\times 3$.
Thus, we have $|G_1|=|G_2| = 2^4\times 3^4$, $|Z(G_1)|=|Z(G_2)|=6$, but $|K(G_1)| = 36\neq 18=|K(G_2)|$.
Thus, there can be no formula that yields the value of one of $|G|$, $|Z(G)|$, and $|K(G)|$ in terms of the other two.
Now, you can bound $|K(G)|$ in terms of $|G|$ and $|Z(G)|$: since $[xz,y]=[x,yz]=[x,y]$ for any $x,y\in G$ and $z\in Z(G)$, there is a surjective map $$\frac{G}{Z(G)}\times \frac{G}{Z(G)}\to K(G),\qquad (xZ(G),yZ(G))\longmapsto [x,y].$$ Now, this immediately yields $|K(G)|\leq \frac{|G|^2}{|Z(G)|^2}$, but this is too rough and we can easily do better: any pair in which any of the elements is $eZ(G)$ will yield the trivial commutator, as will any pair of the form $[x,x]$. Thus, we have $$|K(G)| \leq \left(\frac{|G|}{|Z(G)|}-1\right)\left(\frac{|G|}{|Z(G)|}-1\right) - \frac{|G|}{|Z(G)|}+2,$$ where we add $2$ to account for the trivial commutator. This is usually overkill: for example with $G=S_3$, we have $|Z(G)|=1$, so the bound would tells us that $|K(G)|$, which is $3$, is less than or equal to $$(5)(5) - (5) + 2 = 22,$$ which is true but far too large. You can do a bit better since we also know that $[x^n,x^m]=1$ for any integers $n$ and $m$, but that would require knowing the order of elements modulo the center, so that is more knowledge than just the orders of $G$ and $Z(G)$. More generally, we know that all elements in $[xZ(G),C_G(x)]$ are trivial, so each coset of $Z(G)$ contributes at most $|G|/|C_G(x)|$ commutators, where $x$ is any representative of the coset.