The test that you're thinking of is Rabin's test for irreducibility, which can be stated as follows.
Let $f(x)$ be a polynomial of degree $n$ over $\mathbb{F}_p$. Then $f$ is irreducible over $\mathbb{F}_p$ if and only if
$f(x)$ divides $x^{p^n}-x$, and
$\mathrm{gcd}\bigl(f(x),x^{p^{n/q}}-x\bigr)=1$ for each prime divisor $q$ of $n$.
For $f(x) = x^{10}+x^3+1 \in \mathbb{F}_2[x]$, we must check that $f(x)$ divides $x^{2^{10}}-x = x^{1024}-x$, and that it is relatively prime with $x^{32}-x$ and $x^4-x$.
For $f(x) = x^5 + x^4+x^3+x^2+x-1 \in\mathbb{F}_3[x]$, we must check that $f(x)$ divides $x^{243}-x$ and that it is relatively prime with $x^3-x$.
All of this is easy to do on a computer. In Mathematica, the PolynomialExtendedGCD command can check polynomial GCD's modulo any prime.
In[1] := PolynomialExtendedGCD[x^10 + x^3 + 1, x^1024 - x, x, Modulus -> 2][[1]]
Out[1] := 1 + x^3 + x^10
In[2] := PolynomialExtendedGCD[x^10 + x^3 + 1, x^32 - x, x, Modulus -> 2][[1]]
Out[2] := 1
In[3] := PolynomialExtendedGCD[x^10 + x^3 + 1, x^4 - x, x, Modulus -> 2][[1]]
Out[3] := 1
This establishes that $x^{10}+x^3+1$ is irreducible over $\mathbb{F}_2$. For the other polynomial, we can try:
In[1] := PolynomialExtendedGCD[x^5+x^4+x^3+x^2+x-1, x^243 - x, x, Modulus -> 3][[1]]
Out[1] := 1
As you can see, the GCD is $1$, so it doesn't divide. Indeed,
$$
x^5+x^4+x^3+x^2+x-1 \;=\; \bigl(x^2-x-1\bigr)\bigl(x^3-x^2+x+1\bigr)
$$
over $\mathbb{F}_3$, so the second polynomial isn't irreducible.
Edit: As Jyrki Lahtonen and Alex M. point out, one can always just use Mathematica directly to test whether a polynomial is irreducible, so there's not a practical reason to use Rabin's test in Mathematica for these two polynomials. Also, if you have access to a computer, you can always in principle check whether a polynomial is irreducible simply by enumerating all possible factors and then attempting to divide by each factor. If you don't have access to a computer, the methods suggested in their answers are certainly better than doing Rabin's method by hand.
However, Rabin's test is important from an algorithmic point of view as one possible way to check for irreducibility. Indeed, for polynomials of high degree, Rabin's test clearly outperforms trial division by all possible factors, though there are other competitive algorithms. The two given problems are simple enough that almost any algorithm you use on a computer will give the desired results very quickly, but it is still instructive to see how Rabin's test can be applied to these examples.
Best Answer
Being a quartic, this polynomial is reducible if and only if it has a linear or quadratic factor with integer coefficients.
A linear factor implies an integer root. The only possible roots are $1,-1,5,-5$. Checking, none of them works. Note that for $x=\pm5$ you don't need to do all the calculations since it is easy to see that then $$x^4+x^3-4x^2-5x-5\equiv-5\pmod{25}$$ and so LHS${}\ne0$.
So try $$x^4+x^3-4x^2-5x-5=(x^2+ax+b)(x^2+cx+d)\ .$$ Expanding and equating coefficients, $$a+c=1\ ,\quad ac+b+d=-4\ ,\quad ad+bc=-5\ ,\quad bd=-5\ .$$ The last equation gives four possibilities for $b$ and $d$, (in fact only two, as we may assume by symmetry that $b=\pm1$ and $d=\pm5$) and then it's easy to find the other coefficients:
and we then check that $$x^4+x^3-4x^2-5x-5=(x^2+x+1)(x^2-5)\ .$$
Note that if our second attempt above had failed, this would be enough to conclude that the polynomial is irreducible.