Hermitian operator
Your definition is ok and equivalent to
$\langle a| A b \rangle = \langle A a| b \rangle$
Eigenspace of a Hermitian operator
It's possible to prove that a Hermitian operator has eigenvectors that form an orthogonal vector base, associated with real eigenvalues
$A | n \rangle = \lambda_n | n \rangle$.
Than you can normalize eigenvectors to get a orthonormal basis.
Generic wave function
Given a generic wave function $| \psi \rangle$ you can write it using the basis $\{ |n\rangle\}_n$, as a superposition of states
$| \psi \rangle = a_n |n\rangle $
so that you can write the action of the operator $A$ as
$ A | \psi \rangle = a_n A |n\rangle = a_n \lambda_n |n\rangle $.
Normalization condition and interpretation as a probability
With the normalization condition for the wave function $\langle \psi|\psi \rangle = 1$ (holding also for eigenvectors), you readily get
$1 =\langle \psi|\psi \rangle = \langle a_n n | a_m m \rangle = \sum_n a_n^2$.
If a system is ina state descirbe by wavefunction $|\psi \rangle$, the probability of measuring state $| m \rangle$ is
$|\langle \psi | m \rangle|^2 = |\langle a_n n | m \rangle|^2 = a_m^2$.
Eigenvalues and result of measurement process
On the other hand, eigenvalues of an operator are connected to the values you get from measurement process for the physical quantity associated with that oeprator. See Born's rule, https://en.m.wikipedia.org/wiki/Born_rule
is it true in general that 𝑓(𝜎(𝑇))=𝜎(𝑓(𝑇))?
The answer is negative already for continuous functions $f: \mathbb{R} \to \mathbb{R}$.
Elementary conterexample. Consider the Hamiltonian $H$ of the harmonic oscillator and then focus on $H^{-1}$. The spectrum of the latter includes the further point $0$ which does not belong to $1/\sigma(H)$. As a matter of fact, the spectrum of $H^{-1}$ admits also $0$ as (unique) element of the continuous part of spectrum $\sigma_c(H^{-1})$. $\qquad\blacksquare$
As a general fact, if 𝑇 is selfadjoint and $f: \mathbb{R} \to \mathbb{C}$ is Borel-measurable, then 𝑓(𝑇) is closed and normal.
In particular 𝑓(𝑇) is selfadjoint as well, if 𝑓 as above is real-valued. Obviously continuous functions are Borel-measurable so everything applies to that case.
For continuous functions, the general relation is
$$\sigma(f(T))=\overline{f(\sigma(T))}\:.$$
𝜎(𝑓(𝑇)) may be different from 𝑓(𝜎(𝑇)). The former is always closed for the very definition of spectrum, the latter may not, even if 𝑓 is continuous. The conterexample above is an elementary case.
However,
PROPOSITION.
$$\sigma(f(T))=f(\sigma(T))$$
if $T=T^*$ is bounded and $f: \mathbb{R} \to \mathbb{R}$ is continuous.
Proof. 𝑇 bounded is equivalent to 𝜎(𝑇) is bounded since $||T||= \sup |\sigma(T)|$. In that case 𝜎(𝑇) is compact (closed and bounded set in $\mathbb{R}$), and thus 𝑓(𝜎(𝑇)) is compact as well because $f$ is continuous, hence 𝑓(𝜎(𝑇)) is closed since it is a compact subset of $\mathbb{R}$. In that case,
$$\sigma(f(T))=\overline{f(\sigma(T))}=f(\sigma(T))\:.$$
QED
It is therefore clear that, when $f: \mathbb{R} \to \mathbb{R}$ is continuous, problems may pop up only for unbounded (seldafjoint) operators.
Restricting to the point part of the spectrum, as a general fact we have that $f(\sigma_p(T)) \subset \sigma_p(f(T))$, the proof essentially is the one you wrote. It is valid for every (Borel-measurable) function $f: \mathbb{R} \to \mathbb{R}$. The converse inclusion is false in general. To this end consider the position operator $X$ whose point spectrum is empty. Next consider the map $f: \mathbb{R}\ni x \mapsto 1 \in \mathbb{R}$. Evidently $f(X)=I$. Therefore $\sigma_p(f(X)) = \sigma(I) = \{1\}$, but $f(\sigma_p(X))= \emptyset$ since $\sigma_p(X)= \emptyset$. Therefore $f(\sigma_p(X)) \subsetneq \sigma_p(f(X))$.
Best Answer
I think the easiest way to solve these problems when you first encounter them is to convert them into a matrix problem, where it is easier to proceed.
Since $|a\rangle$ and $|b\rangle$ form an orthonormal basis, one way of proceeding is to represent them as the column vectors: $$ |a\rangle := \begin{bmatrix} 1 \\ 0 \end{bmatrix} \ \ \text{and} \ \ |b\rangle := \begin{bmatrix} 0 \\ 1 \end{bmatrix} $$ You can pick any vectors you like (as long as $|a\rangle$ and $|b\rangle$ are orthonormal), so we may as well pick the above simple case.
You can look up the rules for outer products on wikipedia, what you get is: $$ |a\rangle\langle b| = \begin{bmatrix} 1 \\ 0 \end{bmatrix}\begin{bmatrix} 0 & 1 \end{bmatrix} = \begin{bmatrix} 0 & 1 \\ 0 & 0 \end{bmatrix} \\ |b\rangle\langle a| = \begin{bmatrix} 0 \\ 1 \end{bmatrix}\begin{bmatrix} 1 & 0 \end{bmatrix} = \begin{bmatrix} 0 & 0 \\ 1 & 0 \end{bmatrix} \\ $$ which means that your Hamiltonian is the matrix: $$ H = \sigma \big( |a\rangle\langle b|+|b\rangle\langle a| \big) = \begin{bmatrix} 0 & \sigma \\ \sigma & 0 \end{bmatrix} $$ This is probably going to be useful for you, because it is less abstract and I assume you've diagonalized a $2\times 2$ matrix before. You should check this yourself, but the eigenvalues turn out to be $\lambda_{\pm} = \pm \sigma$ and the (normalized) eigenvectors are $| \lambda_{\pm} \rangle = \frac{1}{\sqrt{2}} \begin{bmatrix} 1 \\ \pm 1 \end{bmatrix}$.
What you do at the end of the day is notice that the eigenvectors can be writen in terms of $|a\rangle := \begin{bmatrix} 1 \\ 0 \end{bmatrix}$ and $|b\rangle := \begin{bmatrix} 0 \\ 1 \end{bmatrix}$, where we get: $$ | \lambda_{\pm} \rangle = \frac{1}{\sqrt{2}}|a\rangle \pm \frac{1}{\sqrt{2}}|b\rangle $$ You should double check that these satisfy $H |\lambda_{\pm}\rangle = \lambda_{\pm} |\lambda_{\pm}\rangle$ in terms of the abstract vectors (no longer written as column vectors).
The magic of the above is that you would have found exactly the same answer, even if you used a different representation of $|a\rangle$ and $|b\rangle$ (for example, if you want to give yourself a headache, try the same calculation with the different representation $|a\rangle := \frac{1}{\sqrt{17^2 + \pi^2}} \begin{bmatrix} 17 \\ \pi \end{bmatrix}$ and $|b\rangle := \frac{1}{\sqrt{17^2 + \pi^2}} \begin{bmatrix} \pi \\ - 17 \end{bmatrix}$. You'll get the same abstract answer $| \lambda_{\pm} \rangle = \frac{1}{\sqrt{2}}|a\rangle \pm \frac{1}{\sqrt{2}}|b\rangle$ with $\lambda_{\pm} = \pm \sigma$ at the end of the day!)