Use a generating function for the dice. Normal dice have faces with 1 to 6 dots, so they are represented by the polynomial:
$\begin{align}
D(z)
&= z + z^2 + z^3 + z^4 + z^5 + z^6 \\
&= z \frac{1 - z^6}{1 - z}
\end{align}$
Throwing two dice gives the distribution:
$\begin{align}
D^2(z)
&= z^2 + 2 z^3 + 3 z^4 + 4 z^5 + 5 z^6 + 6 z^7
+ 5 z^8 + 4 z^9 + 3 z^{10} + 2 z^{11} + z^{12}
\end{align}$
We want polynomials $D_1(z)$ and $D_2(z)$ that give:
$\begin{align}
D_1(z) D_2(z) = D^2(z)
\end{align}$
It is clear that $D_i(1)$ is the number of faces of the die, so we want:
$\begin{align}
D_1(1) = D_2(1) = 6
\end{align}$
If we want no blank faces, it means $D_i(0) = 0$ (no constant term).
Let's factor:
$\begin{align}
D(z)
&= z (z + 1) (z^2 - z + 1) (z^2 + z + 1)
\end{align}$
If we evaluate the factors at $z = 1$:
$\begin{align}
D(1)
&= 1 \cdot 2 \cdot 1 \cdot 3
\end{align}$
Thus, in order to have $D_i(1) = 6$, each has to get a factor $z + 1$ and a factor $z^2 + z + 1$. As we want $D_i(0) = 0$, each one has to get a factor $z$ too. This leaves two factors $z^2 - z + 1$ (one from each normal die) to distribute, giving the dice:
$\begin{align}
D_1(z)
&= z (z + 1) (z^2 - z + 1)^2 (z^2 + z + 1) \\
&= z + 2 z^2 + 2 z^3 + z^4 \\
D_2(z)
&= z (z + 1) (z^2 + z + 1) \\
&= z + z^3 + z^4 + z^5 + z^6 + z^8
\end{align}$
These is dice with faces $\{1, 2, 2, 3, 3, 4\}$ and $\{1, 3, 4, 5, 6, 8\}$
The only other solution is normal dice (and exchanging them, obviously).
They are called Sicherman dice.
What you did at the very beginning is correct, and leads to the correct answer.
$[x^{14}](x+x^2+x^3+x^4+x^5+x^6)^5=540$
See https://www.wolframalpha.com/input/?i=Coefficient((x%2Bx%5E2%2Bx%5E3%2Bx%5E4%2Bx%5E5%2Bx%5E6)%5E5,x%5E14)
Then $${540\over6^5}=.06944444444$$ is the probability.
I don't understand what you did after the first paragraph of your question. I think you may have wanted to do something like
$$(1+x+x^2+x^3+x^4+x^5)^5=\left({1-x^6\over1-x}\right)^5=\left(1-x^6\right)^5(1-x)^{-5}$$ so that $$G(x)=x^5H(x)J(x)$$ where $H,J$ can be computed by the binomial formula. As it is, you've replaced $G$ by some other function.
What you say at the end about the dice perhaps not being distinguishable is also confusing. The dice are distinguishable, as you have assumed. I don't know what the problem would mean if the dice weren't distinguishable. How could they show different numbers?
Best Answer
For fixed $n \ge 1$ the number of pairs $(i,j) \in \mathbf{N}^2$ such that $\max(i,j) = n$ is $2n-1$. So the polynomial (generating function) whose coefficient of $x^n$ counts the number of ways to roll two (six-sided) dice and get a max of $n$ is $x + 3x^2 + 5x^3 + 7x^4 + 9x^5 + 11x^6$.