A polynomial $f(X)$ is said to be reciprocal if whenever $\alpha$ is a root, then $1/\alpha$ is also a root. We suppose that $f$ has coefficients in a real subfield $k$ of the complex numbers. If $f$ is irreducible over $k$, and has a nonreal root of absolute value $1$, show that $f$ is reciprocal of even degree.
Proof:
Let $f(X)\in k[X]$ with $\text{deg}f(X)=n\geq 1$. Since $\beta \in \mathbb{C}-\mathbb{R}$ and $f(\beta)=0$ then $f\left(\overline{\beta}\right)=0$ and $\beta\neq \overline{\beta}$. It means that $(X-\beta)\left(X-\overline{\beta}\right)\mid f(X)$, but $(X-\beta)\left(X-\overline{\beta}\right)=X^2-\left(\beta+\overline{\beta}\right)X+1$ because $|\beta|=1$.
The following cases are possible:
1. If $\text{deg} f(X)=2$ then $f(X)=c\left(X^2-\left(\beta+\overline{\beta}\right)X+1\right)$ for some $c\in k$. Then we see that $f$ has even degree and is reciprocal because its roots are $\beta$ and $\overline{\beta}$.
2. If $\text{deg} f(X)>2$ then $\beta+\overline{\beta}\notin k$ because $f(X)$ is irreducible over $k$. Consider an extension $k\left(\beta+\overline{\beta}\right)$ then we see that $k\leq k\left(\beta+\overline{\beta}\right)\leq k(\beta)$. Using multiplicativity of towers and taking into account that $[k(\beta):k]=n$ then $$n=\left[k(\beta):k\left(\beta+\overline{\beta}\right)\right]\left[k\left(\beta+\overline{\beta}\right):k\right].$$
We can show that $k(\beta)=k\left(\beta+\overline{\beta}\right)(\beta)$. Consider the polynomial $X^2-\left(\beta+\overline{\beta}\right)X+1\in k\left(\beta+\overline{\beta}\right)[X]$ and we see that $\beta$ its root. Hence $\left[k(\beta):k\left(\beta+\overline{\beta}\right)\right]\leq 2$.
If this value was equal to $1$ then $k(\beta)=k\left(\beta+\overline{\beta}\right)$ i.e. $\beta \in k\left(\beta+\overline{\beta}\right)$ but $\beta$ is nonreal complex and $k\left(\beta+\overline{\beta}\right)$ is real subfield. Hence $\left[k(\beta):k\left(\beta+\overline{\beta}\right)\right]=2$.
Therefore, we have shown that $n$ is divisible by $2$, i.e. $n=2m$.
Let's show that $f(X)$ is reciprocal polynomial.
Since $\beta$ has absolute value $1$ then $k(\beta)=k\left(\beta^{-1}\right)$ and $[k(\beta):k]=\left[k\left(\beta^{-1}\right):k\right]=\text{deg} f(X)=2m$.
Define the new polynomial $\hat{f}(X)=X^{2n}f(1/X)\in k[X].$ Then I can show that $\hat{f}$ is irreducible over $k$.
Also note that $\hat{f}(\beta)=\beta^{2n}f(1/\beta)=\beta^{2n}f\left(\overline{\beta}\right)=\beta^{2n}\overline{f(\beta)}=0.$
Thus $f,\hat{f}\in k[X]$ are both irreducible polynomials over $k$ with common root $\beta$. Since minimal polynomial is unique then $f(X)=\hat{f}(X)$, i.e. $$f(X)=X^{2n}f(1/X).$$
And from here its easy to deduce that if $\alpha$ root of $f$ then $1/\alpha$ is also root of $f$.
Remark: When I read this problem this seemed to me very interesting and I solved it but spent couple hours for its solution.
Let me ask you two questions:
1) Is the solution correct?
2) When I was proving that $f(X)$ is reciprocal where did use that $f(X)$ has even degree? I was not able to figure it out by myself.
Best Answer
Your solution looks just fine to me.
[Edit: In hindsight, I missed a critical error! For my part, I should've thought a bit more carefully about what $\hat f$ actually looked like, and not taken you at your word that you were able to prove it to be irreducible. However, even if $\hat f$ had been irreducible, there would still be a minor issue, which I missed.
Rather, we would only be able to conclude from this that $\hat f$ is some non-zero constant multiple of $f.$ That would've still been enough to prove that $f$ is reciprocal, but since $\hat f$ isn't irreducible, we need to work a bit harder. More below.]
You didn't use the fact that $f$ has even degree to show that $f$ is reciprocal; that was just another thing you had to prove.
In general, reciprocal polynomials need not have even degree--after all, $\pm 1$ can be roots, too, so it will have odd degree if only one of those two is a root of odd multiplicity. In this case, though, $f$ was irreducible, so neither of these was a root.
Added: As it turns out, $\hat f$ is not equal to $f.$ To see this, let's consider what we know about $f.$
Since $f\in k[X]$ is irreducible, of degree $n,$ and has $\beta$ as a root, then there is a non-zero $a\in k$ such that $f(X)=a\cdot g(X),$ where $g$ is the minimal polynomial of $\beta$ over $k.$ Without loss of generality, we can assume that $f=g,$ so that $a=1$. (Do you see why we can assume this?) Note in particular that $f$ is monic.
Now, we know that $$f(X)=\sum_{j=0}^na_jX^j,\tag{$\star$}$$ where each $a_j\in k,$ and $a_n=1,$ in particular. Moreover, since $f$ is irreducible, we must also have $a_0\ne 0.$ (Do you see why?)
Thus, $$f\left(X^{-1}\right)=\sum_{j=0}^na_jX^{-j}$$ $$f\left(X^{-1}\right)=X^{-n}\sum_{j=0}^na_jX^{n-j}$$ $$f\left(X^{-1}\right)=X^{-n}\sum_{j=0}^na_{n-j}X^j,\tag{1}$$ so since $a_0\ne 0,$ then $\hat f$ would have degree $2n,$ and so can't be equal to $f$! Fortunately, this isn't a problem.
By $(1)$, we can see that $$\hat f(X)=X^n\sum_{j=0}a_{n-j}X^j,\tag{2}$$ and you're quite right that $\hat f(\beta)=0.$ Since $\hat f$ is a polynomial with $k$-coefficients having $\beta$ as a root, then the minimal polynomial of $\beta$ over $k$ divides $\hat f$ in $k[X].$ But $f$ is that minimal polynomial! Thus, there is a polynomial $h$ with $k$-coefficients such that $$\hat f(X)=f(X)h(X),$$ so by $(2)$ we have $$X^n\sum_{j=0}a_{n-j}X^j=f(X)h(X).\tag{3}$$
Now, given any root $\alpha$ of $f,$ we have that $\alpha\ne 0$ since $f$ is irreducible over $k,$ and so $\alpha^n\ne 0$ since $k$ is a real subfield. Since $$\alpha^n\sum_{j=0}a_{n-j}\alpha^j=f(\alpha)h(\alpha)=0\cdot h(\alpha)=0,$$ then $$\sum_{j=0}a_{n-j}\alpha^j=0,$$ so $$f\left(\alpha^{-1}\right)=\alpha^{-n}\sum_{j=0}a_{n-j}\alpha^j=0,$$ meaning that $\alpha^{-1}$ is a root of $f.$ Since $\alpha$ was an arbitrary root of $f,$ then $f$ is reciprocal, as was to be shown.
Bonus Content: It turns out that we can go a little further, still! Note that $0$ is a root of multiplicity at least $n$ of the left-hand side of $(3),$ so since $0$ isn't a root of $f,$ then $0$ is a root of $h$ of the same multiplicity. However, the left-hand side of $(3)$ has degree $2n,$ so since $f$ has degree $n,$ then $h$ has degree $n,$ as well, whence $h=bX^n$ for some non-zero $b\in k.$ Thus, it follows by $(3)$ that $$\sum_{j=0}a_{n-j}X^j=b\cdot f(X),$$ so by $(\star),$ we see that $$\sum_{j=0}a_{n-j}X^j=\sum_{j=0}^nba_jX^j,$$ so that $a_{n-j}=ba_j$ for each $j.$ In particular, $$a_n=a_{n-0}=ba_0=ba_{n-n}=b^2a_n,$$ so $b^2=1,$ and so either $b=1$ or $b=-1.$
I claim that $b=1.$ We prove this by way of contradiction, supposing instead that $b=-1.$ By this assumption, $a_{n-j}=-a_j$ for each $j.$ In particular, recalling that $n=2m,$ we have $$a_m=a_{n-m}=-a_m,$$ whence $a_m=0.$ Thus, we have by $(\star)$ that $$\begin{eqnarray}f(X) &=& \sum_{j=0}^na_jX^j\\ &=& \sum_{j=0}^{m-1}a_jX^j+\sum_{j=m+1}^na_jX^j\\ &=& \sum_{j=0}^{m-1}a_jX^j+\sum_{j=0}^{m-1}a_{n-j}X^{n-j}\\ &=& \sum_{j=0}^{m-1}a_jX^j-\sum_{j=0}^{m-1}a_jX^{n-j}\\ &=& \sum_{j=0}^{m-1}a_j\left(X^j-X^{n-j}\right)\\ &=& \sum_{j=0}^{m-1}a_j\left(X^j-X^{2m}X^{-j}\right).\end{eqnarray}$$ Consequently, we have that $$f(-1)=\sum_{j=0}^{m-1}a_j\left((-1)^j-(-1)^{2m}(-1)^{-j}\right)=\sum_{j=0}^{m-1}a_j\left((-1)^j-(-1)^{-j}\right)=0.$$ Since $-1\in k$ and $f$ is monic and irreducible over $k,$ then $f(X)=x-1,$ which is impossible, since $f$ has even degree.
Thus, $b=1,$ meaning that $a_{n-j}=a_j$ for each $j,$ so that $f$ is a palindromic polynomial, and $$f(X)=\sum_{j=0}a_{n-j}X^j.$$ Equivalently, by $(1),$ we have $F(X)=X^nf\left(X^{-1}\right).$
The only other adjustment I can think to make to your proof, for simplicity's sake, is that instead of $\hat f,$ you consider the function $$F(X):=X^nf\left(X^{-1}\right)=\sum_{j=0}^na_{n-j}X^j.$$ We can show that this is an element of $k[X]$ of degree $n,$ and that $$F(\beta)=\beta^n\overline{f(\beta)}=0,$$ whence there is a non-zero scalar $c\in k$ such that $F(X)=cf(X).$ Then for any root $\alpha$ of $f,$ we have $$\sum_{j=0}^na_{n-j}\alpha^j=F(\alpha)=cf(\alpha)=0,$$ whence $f\left(\alpha^{-1}\right)=0$ as above.
This also lets us prove that $F(X)=f(X)$ even more directly.