The test that you're thinking of is Rabin's test for irreducibility, which can be stated as follows.
Let $f(x)$ be a polynomial of degree $n$ over $\mathbb{F}_p$. Then $f$ is irreducible over $\mathbb{F}_p$ if and only if
$f(x)$ divides $x^{p^n}-x$, and
$\mathrm{gcd}\bigl(f(x),x^{p^{n/q}}-x\bigr)=1$ for each prime divisor $q$ of $n$.
For $f(x) = x^{10}+x^3+1 \in \mathbb{F}_2[x]$, we must check that $f(x)$ divides $x^{2^{10}}-x = x^{1024}-x$, and that it is relatively prime with $x^{32}-x$ and $x^4-x$.
For $f(x) = x^5 + x^4+x^3+x^2+x-1 \in\mathbb{F}_3[x]$, we must check that $f(x)$ divides $x^{243}-x$ and that it is relatively prime with $x^3-x$.
All of this is easy to do on a computer. In Mathematica, the PolynomialExtendedGCD
command can check polynomial GCD's modulo any prime.
In[1] := PolynomialExtendedGCD[x^10 + x^3 + 1, x^1024 - x, x, Modulus -> 2][[1]]
Out[1] := 1 + x^3 + x^10
In[2] := PolynomialExtendedGCD[x^10 + x^3 + 1, x^32 - x, x, Modulus -> 2][[1]]
Out[2] := 1
In[3] := PolynomialExtendedGCD[x^10 + x^3 + 1, x^4 - x, x, Modulus -> 2][[1]]
Out[3] := 1
This establishes that $x^{10}+x^3+1$ is irreducible over $\mathbb{F}_2$. For the other polynomial, we can try:
In[1] := PolynomialExtendedGCD[x^5+x^4+x^3+x^2+x-1, x^243 - x, x, Modulus -> 3][[1]]
Out[1] := 1
As you can see, the GCD is $1$, so it doesn't divide. Indeed,
$$
x^5+x^4+x^3+x^2+x-1 \;=\; \bigl(x^2-x-1\bigr)\bigl(x^3-x^2+x+1\bigr)
$$
over $\mathbb{F}_3$, so the second polynomial isn't irreducible.
Edit: As Jyrki Lahtonen and Alex M. point out, one can always just use Mathematica directly to test whether a polynomial is irreducible, so there's not a practical reason to use Rabin's test in Mathematica for these two polynomials. Also, if you have access to a computer, you can always in principle check whether a polynomial is irreducible simply by enumerating all possible factors and then attempting to divide by each factor. If you don't have access to a computer, the methods suggested in their answers are certainly better than doing Rabin's method by hand.
However, Rabin's test is important from an algorithmic point of view as one possible way to check for irreducibility. Indeed, for polynomials of high degree, Rabin's test clearly outperforms trial division by all possible factors, though there are other competitive algorithms. The two given problems are simple enough that almost any algorithm you use on a computer will give the desired results very quickly, but it is still instructive to see how Rabin's test can be applied to these examples.
Best Answer
$P$ is reducible over $\mathbb F_{64}$, because reducible over the subfield $\mathbb F_4\subset \mathbb F_{64}$ ($2\mid6$).