You seem to mean two slightly different things by the $p=\ell$ case. Your first theorem is about a field of characteristic $p$, and the second is about $\mathbb C_p$, which is a field of characteristic zero. Let's talk first about the significance of this.
For a field of actual characteristic $p$, the proof of "Artin vanishing" for mod $p$ coefficients is even easier than you gave. One doesn't need any perfectoid spaces, as you can just use the Artin-Schreier sequence $0 \to \mathbb F_p \to \mathcal O_X \to \mathcal O_X \to 0$ to handle the case of affine schemes.
By a Lefschetz principle argument, your proof over $\mathbb C_p$ can be adapted to work over an arbitrary field of characteristic zero. Of course, by the same argument, and the comparison theorem between etale and analytic cohomology, the Artin affine theorem can also be proved by analytic arguments over fields of characteristic zero (say, using a Morse function).
Characteristic zero really is pretty much always easier than characteristic $p$.
In actual characteristic $p$, the $\ell=p$ case is not necessarily always easier. I think a good description is that it is more degenerate. The cohomology groups are usually much smaller, in fewer degrees. It's not surprising that vanishing results are easier to prove in this setting! But other results we want, like Poincare duality or Euler characteristic formulas, are simply not true for mod $p$ coefficients.
There is certainly something unsatisfying about these somewhat ad hoc arguments, but they're not THAT bad. I think getting over some of your discomfort with them might be easier than finding a better argument.
It's perhaps worthwhile to point out that Grothendieck famously wanted to solve problems by setting up theories that make the problems easy rather than the concrete calculations favored by mathematicians like Serre and Deligne, which were later necessary to solve problems like the Weil conjectures. If you find the arguments in the part of etale cohomology worked out by Grothendieck to have too many concrete calculations and not enough big theories then I am tempted to say you have gone too far, but I perhaps shouldn't because many fields of mathematics have trended in that direction since Grothendieck's day, and that perspective does work for at least some mathematicians.
Anyways, the use of $\mathbb P^1$ in this argument is certainly not a pure application of general theory, but it is also not ad hoc. It is part of a systematic strategy in the foundations of etale cohomology, which is to first prove the desired statement for curves using the powerful theorems available for curves (precise descriptions of $H^1$ and $H^2$, together with dimension estimates), and then find an inductive approach to the general case based on varieties mapping to a curve with fibers mapping to a curve, etc., suitable to the particular problem.
For different problems, it makes sense to try slightly different curves, and put them together in slightly different ways.
For the affine vanishing theorem, we of course want to look at affine curves, so we might look first at the simplest such, $\mathbb A^1$, and then observe we can prove the theorem for arbitrary products of $\mathbb A^1$, or $\mathbb A^n$. Can we reduce from the general case to $\mathbb A^n$? Yes, by Noether normalization.
For the comparison of etale and analytic cohomology, it's convenient to work with affine curves, and fibrations of affine curves by affine curves, so that all our cohomology will match the cohomology of the fundamental group, as then we can take advantage of the comparison of the etale and analytic fundamental groups. In this case, it suffices to check that smooth varieties are covered by open subsets isomorphic to an iterated fibration of curves, and this we can do.
For the proper base change theorem, we of course want to start with proper curves, and so we try the simplest such, $\mathbb P^1$. There's no clear reason to take nontrivial fibrations of $\mathbb P^1$ over $\mathbb P^1$, so we just look at powers, $(\mathbb P^1)^n$. Can we reduce from a general projective variety to $(\mathbb P^1)^n$? Not directly, as we only know that projective varieties map to $\mathbb P^n$, but using the map $(\mathbb P^1)^n \to \mathbb P^n$ we can close the gap.
So in some sense it's all the same idea, just adapated in different ways to different contexts.
The Weil conjectures are one place this strategy failed, as Grothendieck tried to prove them by showing every smooth projective variety is covered by a product of curves, but this is false. (Deligne was later, in Weil II, able to solve the problem by an inductive argument, using the theory of weights of constructible sheaves.)
Best Answer
Note : [BBD]=Astérisque 100 by Beilinson-Bernstein-Deligne
The answer to the first question is basically Deligne's "generic base change", by which I mean theorem 1.9 of SGA 4 1/2 [Th. finitude].
I suppose that you are in part (b) of the second proof of corollary 5.3.2 in [BBD]. So you're assuming that X is affine, and a "projection" is just an embedding of X into some $\mathbb{A}^n$ followed by a projection on one of the coordinates. So $f^{-1}(v)$ is just the intersection of $X$ and of a hyperplane varying in some pencil.
Let's start with the fact that $i^*\mathcal{F}[-1]$ is perverse for generic $v$. Note that the inclusion of the complement of $f^{-1}(v)$ (in $U$ or $X$) is an open affine embedding, so $i^*\mathcal{F}$ is concentrated in perverse degrees $0$ and $-1$, and saying that $i^*\mathcal{F}[-1]$ is perverse is the same as saying that ${}^p H^0 i^*\mathcal{F}=0$, and it is also the same as saying that the adjunction $k_!k^*\mathcal{F}\rightarrow\mathcal{F}$ is surjective, where $k$ is the embedding of the complement of $f^{-1}(v)$. This is proved in the proof of lemma 3.3 of Beilinson's "On the derived category of perverse sheaves". Basically, you base change the whole situation over the space of all hyperplanes in $\mathbb{A}^n$, you prove that the obvious analogue of the statement you want is true there (it follows easily from smooth base change), and then you use Deligne's generic base change to go back.
Actually, you can apply this proof to a finite family of perverse sheaves on $U$ or $X$, not just one (you'll get a dense open subset of good $v$ for each sheaf, and you just take their intersection). So, after restricting $v$ a bit more, you get the perversity of $i^*(j_{!*}\mathcal{F})[-1]$. In fact, what this really show is that you can make the functor $i^*[-1]$ exact on any finite diagram of perverse sheaves on $X$ or $U$ by taking $v$ generic enough.
Now for the isomorphism. I won't assume that $j$ is affine. First, by generic base change (again !), if $v$ is generic enough then the base change map $i^*j_*\mathcal{F}\rightarrow j_*i^*\mathcal{F}$ is an isomorphism. Using the remark above, we can also assume that $i^*{}^p H^kj_*\mathcal{F}[-1]$ is perverse for every $k$ (and that $i^*\mathcal{F}[-1]$ is perverse), so the two spectral sequences that give the ${}^p H^k i^*j_*\mathcal{F}={}^p H^k j_*i^*\mathcal{F}$ are concentrated on one line or one column, and we easily get $i^*{}^p H^kj_*\mathcal{F}[-1]={}^p H^k j_* (i^*\mathcal{F}[-1])$. In the same way, using base change and the remark above, we can make sure that $i^*{}^p H^kj_!\mathcal{F}[-1]={}^p H^k j_! (i^*\mathcal{F}[-1])$.
Remember that $j_{!*}\mathcal{F}$ is the image of the natural map ${}^p H^0j_*\mathcal{F}\rightarrow {}^p H^0 j_!\mathcal{F}$. Applying $i^*[-1]$ to that map and using what I wrote above, we get for $v$ generic enough the map ${}^p H^0j_* (i^*\mathcal{F}[-1])\rightarrow {}^p H^0 j_!(i^*\mathcal{F}[-1])$, whose image is $j_{!*}(i^*\mathcal{F}[-1])$. Of course $i^*[-1]$ is not exact in general, so why would it preserve images ? But by the remark above, I can make this functor exact on any finite diagram by restricting $v$ a bit more, so that's it.
Second question : You need to know that $j_*$ commutes to restricting to the fibers of $\overline{X}\rightarrow Y$, and you also need to know that restricting to the fibers will preserve perversity (up to a shift) for all the perverse shaves you're using. The first is true by lemma 2.1.10 in SGA 7 XIII if you assume that $\mathcal{F}$ is tamely ramified along $\overline{X}-X$; it's not true in general. If you assume that $\mathcal{F}$ is tamely ramified, then I think that you're okay. (Not sure why you would need to cut $Y$ with divisors.)