To the first point: you have $k$-linear maps $\theta\colon A \to k^n$ and $\theta'\colon {\frak a}/{\frak a^2} \to k^n$ and the second an isomorphism, so to compute the dimension of $\theta(\mathfrak{b})$ you could instead compute the dimension of $\theta'^{-1}(\theta(\mathfrak{b}))$. The latter is indeed $(\mathfrak{b} + \mathfrak{a}^2)/\mathfrak{a}^2$ since that ideal has the right image.
For the last bit: in general, if $R$ is a ring with maximal ideal $M$ then $M/M^2 \simeq (MR_M)/(MR_M)^2$. Here's how I think about this: $(MR_M)/(MR_M)^2 \simeq (M/M^2)_M$ but $M/M^2$ is a vector space over $R/M$, so the elements of $R - M$ already act invertibly and there is no need to localize. In our situation $R = A/\mathfrak{b}$, $M = \mathfrak{a}/\mathfrak{b}$, and $MR_M = \mathfrak{m}$.
You also want to use the fact that, in the notation of this Wikipedia article, $I^eJ^e = (IJ)^e$. In our case, we use this to show that $M^2 = (\mathfrak{a}/\mathfrak{b})^2 = (\mathfrak{a}^2 + \mathfrak{b})/\mathfrak{b}$. Then $M/M^2 \simeq \mathfrak{a}/(\mathfrak{a}^2 + \mathfrak{b})$ by one of the isomorphism theorems.
$\newcommand{\ideal}[1]{\mathfrak{#1}}$
The isomorphism is very simple: Let $A=k[x_1,\ldots,x_n]$ be the coordinate ring of $\mathbb{A}^n$ and $\ideal{b} \subseteq A$ be the ideal $I(Y)$ of the variety $Y$. Furthermore let $p \in \mathbb{A}^n$ be a closed point which lies in $Y$ too. Let $\ideal{m} \subseteq A$ be the ideal $I(p)$ of this point in $\mathbb{A}^n$. Then $A/\ideal{b}$ is the coordinate ring of $Y$ and $A_\ideal{m} = \mathcal{O}_{\mathbb{A}^n,p}$ and $(A/\ideal{b})_\ideal{m} = \mathcal{O}_{Y,p}$ are the local coordinate rings.
Now the maximal ideal of $(A/\ideal{b})_\ideal{m} = C$ is
$$(\ideal{m} + \ideal{b}) C = \ideal{m} C.$$
Its square in $C$ is $(\ideal{m}^2 + \ideal{b}) C$. Note that for every $\ideal{a} \subseteq A$, we have $\ideal{a} C = (\ideal{a} + \ideal{b}) C$.
Now
$$ \ideal{m}C/\ideal{m^2} C =\ideal{m}C/(\ideal{m}^2+\ideal{b})C = (\ideal{m}/(\ideal{m}^2+\ideal{b})) C$$
For the last step one uses that we have $(I/J) R/K = (I (R/K)) / (J (R/K))$ for $K \subseteq I,J$ and $(I/J) R_K = I R_K/ J R_K$ for a prime ideal $K \supseteq I,J$
and the chain $A \to A/\ideal{b} \to (A/\ideal{b})_\ideal{m} = C$.
Note: What you call $\ideal{a}_p$ I call $\ideal{m}$ or $\ideal{m} (A/\ideal{b})$ when viewed as a point in $Y$. What you call $\ideal{m}$ is $\ideal{m} C$ for me.
Note added for clarification:
The $A$-module $\ideal{m}/(\ideal{m}^2 + \ideal{b})$ is an $A/\ideal{b}$-module because of $\ideal{b} \ideal{m} \subseteq \ideal{m}^2 + \ideal{b}$ and by an analogous argument also an $A/\ideal{m} = k$-module, all compatible with $A \to A/\ideal{b} \to A/\ideal{m}$. As $\ideal{m}/(\ideal{m}^2 + \ideal{b})$ is a $k$-module via $A \to A/\ideal{m}$ it does not change under localization with $\ideal{m}$. So we have
$$
\begin{multline}\ideal{m}/(\ideal{m}^2+\ideal{b}) = \ideal{m}_\ideal{m}/(\ideal{m}^2 + \ideal{b})_\ideal{m} = \\
= (\ideal{m}_\ideal{m}/\ideal{b}_\ideal{m})/(
(\ideal{m}^2 + \ideal{b})_\ideal{m}/\ideal{b}_\ideal{m}) =
\ideal{m} C/ (\ideal{m}^2 + \ideal{b}) C = \ideal{m} C/\ideal{m}^2 C.
\end{multline}$$
Especially the second and third quality follow from
$$I/J = (I/K)/(J/K) = (I R/K)/(J R/K)$$
for $K \subseteq J \subseteq I \subseteq R$ ideals of $R$. It is an equality of $R/K$-modules.
In total, this seems to be what is stated in Hartshorne, p. 32. (remember my change in notation).
Best Answer
By definition, the rank of a matrix is equal to the dimension of the span of its columns.
The columns of the $n \times n$ matrix $J$ are precisely $\theta(f_1), \dots , \theta(f_t)$, where each $\theta(f_i)$ is viewed as a vector in $k^n$. Therefore, the rank of $J$ is equal to the dimension of $\text{Span}(\theta(f_1), \dots , \theta(f_t)) \subset k^n$.
I claim that $\theta(\mathfrak b) = \text{Span}(\theta(f_1), \dots , \theta(f_t))$.
To summarise, the rank of $J$ is equal to the dimension of $\text{Span}(\theta(f_1), \dots , \theta(f_t))$ as a vector subspace of $k^n$, and $\text{Span}(\theta(f_1), \dots , \theta(f_t)) = \theta(\mathfrak b)$. Hence the rank of $J$ is equal to the dimension of $\theta(\mathfrak b)$ as a vector subspace of $k^n$.