Let me address your first question. First, I want to argue that there is no precise meaning of "involving numbers only". For example, given a finite field $F$ of size $4$ constructed in the usual manner (quotient of a polynomial ring over $\mathbb{Z}/2\mathbb{Z}$), I can choose a set of numbers, say
$$S=\{37,\tfrac{5}{19},\pi,e\}$$
and, choosing a bijection of $S$ with $F$, use transport of structure to give $S$ the structure of a field. The field structure does not depend in any way on what the underlying set is "made of".
However, along the lines of what I think you are ultimately after, you can obtain finite fields of any possible order using larger rings of integers. For example, $\mathbb{Z}[i]/(3)$ is a finite field of size $9$, and $\mathbb{Z}[i]$ consists of very reasonable numbers,
$$\mathbb{Z}[i]=\{a+bi\mid a,b\in\mathbb{Z}\}.$$
Now let me addressr your second question. Let's use $\mathbb{F}_p$ to mean $\mathbb{Z}/p\mathbb{Z}$, a finite field of order $p$ - it is a very common notation that is slightly less cumbersome, but doesn't mean anything different, they are exact synonyms.
A finite field of order $p^n$ is often constructed by taking the polynomial ring $\mathbb{F}_p[x]$, choosing an irreducible polynomial $f\in \mathbb{F}_p[x]$ of degree $n$, and then making the field
$$F=\mathbb{F}_p[x]/(f).$$
Now, the division algorithm for polynomials tells you that each equivalence class in this quotient can be uniquely identified by a representative of degree $<n$. In other words,
$$\begin{align*}
F&=\{a_0+a_1x+\cdots +a_{n-1}x^{n-1}+(f)\mid a_0,a_1,\ldots,a_{n-1}\in\mathbb{F}_p\}\\\\
&=\left\{\,\overline{a_0+a_1x+\cdots +a_{n-1}x^{n-1}}\,\;\middle\vert\;a_0,a_1,\ldots,a_{n-1}\in\mathbb{F}_p\right\}\\\\\\
&=\{a_0+a_1\overline{x}+\cdots +a_{n-1}\overline{x}^{n-1}\mid a_0,a_1,\ldots,a_{n-1}\in\mathbb{F}_p\}
\end{align*}$$
Letting the symbol $\alpha$ be a stand-in for $\overline{x}$, you can think of $F$ as being $\mathbb{F}_p$ with a new element "$\alpha$" added in, where $\alpha$ is a root of $f$, and you can write $F=\mathbb{F}_p[\alpha]$.
Now, the prime subfield of $F$ is just the "constant" polynomials, i.e. the ones with no $\alpha$'s in them:
$$\text{the prime subfield of }F=\{a_0+0\alpha+\cdots+0\alpha^{n-1}\mid a_0\in\mathbb{F}_p\}$$
and for each divisor $d\mid n$, the unique subfield of $F$ of order $p^d$ is the collection of polynomials in $\alpha$ whose terms are those of exponents that are multiples of $n/d$:
$$\text{the subfield of }F\text{ of order }p^d=\{a_0+a_1\alpha^{n/d}+\cdots+a_{d-1}\alpha^{(d-1)n/d}\mid a_0,a_1,\ldots,a_{d-1}\in\mathbb{F}_p\}$$
(clearly, the above set has cardinality $p^d$, because it takes $d$ elements of $\mathbb{F}_p$ to specify a given element of the above set, namely, each of the coefficients of the powers of $\alpha$. To see that it is a field, remember that $(a+b)^p=a^p+b^p$ in a field of characteristic $p$.)
Best Answer
Once you prove that the intersection of all subfields of a field is itself a field, you see that every field has a single smallest subfield. The function that Lubin suggests gives you a way to understand the structure of that smallest field. Lubin is defining the function $\varphi: \mathbb{Z} \rightarrow F$ (where $F$ is your original field) by $$\varphi(n) = \underbrace{1 + 1 + \dots + 1}_{\text{$n$ times}}.$$
Equivalently, you can also define this function as the group homomorphism $\varphi: (\mathbb{Z}, +) \rightarrow (F, +)$ with the property $\varphi(1) = 1_F$. (Since $1$ generates $\mathbb{Z}$ as a group, knowing where $\varphi$ sends $1$ is enough to define the whole function.)
At this point, you have to prove that $\varphi$ is also a ring homomorphism (which I'll leave to you).
Consider the kernel of $\varphi$. Since $\mathbb{Z}$ is a principal ideal domain, we have that $\ker \varphi = p\mathbb{Z}$ for some nonnegative integer $p$. This $p$ will, in fact, be the characteristic of the field $F$, meaning $p = 0$ or is a prime.
Now, suppose $p = 0$. Then consider the subset $$S = \left\{\varphi(a) \varphi(b)^{-1}: a, b \in \mathbb{Z}, b \not = 0\right\}$$ of $F$. You can show that $S$ is isomorphic to $\mathbb{Q}$. Since $\mathbb{Q}$ has no proper subfields, $S$ must be the smallest subfield of $F$.
On the other hand, suppose $p$ is a prime. Then $\varphi(\mathbb{Z}) \cong \mathbb{Z}/p\mathbb{Z}$. Since $\mathbb{Z}/p\mathbb{Z}$ has no proper subfields, $\varphi(\mathbb{Z})$ must be the smallest subfield of $F$.
Now, how do we know that a field $F$ with $p^n$ elements (where $p$ is prime) has characteristic $p$ (and hence a smallest subfield isomorphic to $\mathbb{Z}/p\mathbb{Z}$)? Since $F$ is a finite field, it must have some non-zero prime characteristic, $q$. Thus, $F$ has a subfield isomorphic to $\mathbb{Z}/q\mathbb{Z}$, and hence $F$ is a vector space over $\mathbb{Z}/q\mathbb{Z}$. Let $m = \dim F$ as a $\mathbb{Z}/q\mathbb{Z}$-vector space. Because $F$ is finite, $m$ must be finite as well. That means $p^n = q^m$. Hence $p = q$ and $m = n$.