Dear Akhil,
This is a big topic, although one that has been discussed at various times here, e.g.
In what setting does one usually define mixed sheaves and weights for them?
The idea is that for constant coeffients smooth projective
varieties should be cohomologically the simplest. And by approximating more general
varieties by these, via simplicial techniques etc., we get a weight filtration on
cohomology which measures the deviation from the simplest case.
How to make this precise? Well
- In positive characteristic, we can say that smooth projective
varieties are one on which Frobenius acts
with expected bounds eigenvalues. So the weight filtration is defined via eigenspaces of these.
- Over $\mathbb{C}$, smooth projective varietes carry classical
Hodge decompositions. The weight filtration needs to be (nontrivially)
inserted into this picture via mixed Hodge theory.
The compatibility of the weights comes either by construction* or via the (somewhat
conjectural) story of mixed motives.
For perverse coefficients, the story is already much more complex. The "simplest"
cases should be intersection cohomology complexes with coefficients in direct images
of families of smooth projective varieties.
The analogue of (1) is BBD, and of (2) is Saito's theory that Uhlirch mentions.
*(Added) Perhaps I can say what I mean "compatible by construction".
I'll take two examples, which give a sense of what's going behind the scenes.
A) take $X$ to be the complement of two points $p,q$ in smooth projective curve $\bar X$.
Then have an exact sequence
$$ 0\to W_1= H_1(\bar X, \mathbb{Q}) \to W_2=H^1(X, \mathbb{Q})\to \mathbb{Q}(-1)\to 0$$
The last map can be thought of as sort of residue at $p$.
The symbol
$\mathbb{Q}(-1)$ means the one dim vector space shifted into weight $2$, so this sequence also displays the weight filtration,
There is an entirely analogous sequence in the $\ell$-adic world which gives the weights
there. So these are compatible (pretty much by design).
B) For the second example, let us use $\bar X$ as above but with coefficients in the intersection cohomology $L=j_\ast R^i f_\ast\mathbb{Q}$, where $f:Y\to X$ is smooth projective.
Then $L$ carries variation of Hodge structure of weight $i$. By Zucker [Ann. Math 1979]
$H^1(\bar X, L)$ has a pure Hodge structure of weight $1+i$.
In the $\ell$-adic world, the analgous statement is Deligne's purity theorem [Weil II].
Note that Zucker's theorem was one of the key analytic inputs in Saito's work, analogous to
the role of Weil II in BBD.
Some References: Matt is correct that Saito's work isn't easy to get into.
Aside from some expositions by Saito, I might suggest looking at the last few
chapters of Peters and Streenbrink's book on mixed Hodge theory, which gives a pretty good
introduction. I'm also linking my own, not quite successful,
attempt to go through some of this:
http://www.math.purdue.edu/~dvb/preprints/tifr.pdf
Note : [BBD]=Astérisque 100 by Beilinson-Bernstein-Deligne
The answer to the first question is basically Deligne's "generic base change", by which I mean theorem 1.9 of SGA 4 1/2 [Th. finitude].
I suppose that you are in part (b) of the second proof of corollary 5.3.2 in [BBD]. So you're assuming that X is affine, and a "projection" is just an embedding of X into some $\mathbb{A}^n$ followed by a projection on one of the coordinates. So $f^{-1}(v)$ is just the intersection of $X$ and of a hyperplane varying in some pencil.
Let's start with the fact that $i^*\mathcal{F}[-1]$ is perverse for generic $v$.
Note that the inclusion of the complement of $f^{-1}(v)$ (in $U$ or $X$) is an open affine embedding, so $i^*\mathcal{F}$ is concentrated in perverse degrees $0$ and $-1$, and saying that $i^*\mathcal{F}[-1]$ is perverse is the same as saying that ${}^p H^0 i^*\mathcal{F}=0$, and it is also the same as saying that the adjunction $k_!k^*\mathcal{F}\rightarrow\mathcal{F}$ is surjective, where $k$ is the embedding of the complement of $f^{-1}(v)$. This is proved in the proof of lemma 3.3 of Beilinson's "On the derived category of perverse sheaves". Basically, you base change the whole situation over the space of all hyperplanes in $\mathbb{A}^n$, you prove that the obvious analogue of the statement you want is true there (it follows easily from smooth base change), and then you use Deligne's generic base change to go back.
Actually, you can apply this proof to a finite family of perverse sheaves on $U$ or $X$, not just one (you'll get a dense open subset of good $v$ for each sheaf, and you just take their intersection). So, after restricting $v$ a bit more, you get the perversity of $i^*(j_{!*}\mathcal{F})[-1]$. In fact, what this really show is that you can make the functor $i^*[-1]$ exact on any finite diagram of perverse sheaves on $X$ or $U$ by taking $v$ generic enough.
Now for the isomorphism. I won't assume that $j$ is affine. First, by generic base change (again !), if $v$ is generic enough then the base change map $i^*j_*\mathcal{F}\rightarrow j_*i^*\mathcal{F}$ is an isomorphism. Using the remark above, we can also assume that $i^*{}^p H^kj_*\mathcal{F}[-1]$ is perverse for every $k$ (and that $i^*\mathcal{F}[-1]$ is perverse), so the two spectral sequences that give the ${}^p H^k i^*j_*\mathcal{F}={}^p H^k j_*i^*\mathcal{F}$ are concentrated on one line or one column, and we easily get $i^*{}^p H^kj_*\mathcal{F}[-1]={}^p H^k j_* (i^*\mathcal{F}[-1])$. In the same way, using base change and the remark above, we can make sure that $i^*{}^p H^kj_!\mathcal{F}[-1]={}^p H^k j_! (i^*\mathcal{F}[-1])$.
Remember that $j_{!*}\mathcal{F}$ is the image of the natural map ${}^p H^0j_*\mathcal{F}\rightarrow
{}^p H^0 j_!\mathcal{F}$. Applying $i^*[-1]$ to that map and using what I wrote above, we get for $v$ generic enough the map ${}^p H^0j_* (i^*\mathcal{F}[-1])\rightarrow {}^p H^0 j_!(i^*\mathcal{F}[-1])$, whose image is $j_{!*}(i^*\mathcal{F}[-1])$. Of course $i^*[-1]$ is not exact in general, so why would it preserve images ? But by the remark above, I can make this functor exact on any finite diagram by restricting $v$ a bit more, so that's it.
Second question : You need to know that $j_*$ commutes to restricting to the fibers of $\overline{X}\rightarrow Y$, and you also need to know that restricting to the fibers will preserve perversity (up to a shift) for all the perverse shaves you're using. The first is true by lemma 2.1.10 in SGA 7 XIII if you assume that $\mathcal{F}$ is tamely ramified along $\overline{X}-X$; it's not true in general. If you assume that $\mathcal{F}$ is tamely ramified, then I think that you're okay. (Not sure why you would need to cut $Y$ with divisors.)
Best Answer
A great book which contains basic information about perverse sheaves (although not so much about the topics you are especially looking for) is
D-Modules, Perverse Sheaves, and Representation Theory
by Hotta, Takeuchi and Tanisaki. Its even available for free online:
http://www.math.harvard.edu/~gaitsgde/grad_2009/Hotta.pdf