The Short question: Where can I find a book for the theory of probability and statistics that teaches from scratch in a rigorous (very important condition) way? The book must not be elementary, but it has to start from scratch. (For example, I think the Lang/Hungerford algebra texts begin by defining what a group is: in that sense they start from scratch.)
The long question: I only took an engineering course in probability and statistics. In my opinion, it is very lousy/non-rigorous. You may assume I have no knowledge of probability and statistics. I have to take an independent study statistics course this year. I am allowed to choose a book for the course. It has to be a statistics course. My instructor assumes I know probability because I took the course mentioned above. (I admit I have a poor understanding of probability and this irritates me a lot.) I'd like to have a book that:
1) Is mathematically oriented and rigorous
2) Has a significant statistics part
3) Teaches the amount of probability needed to do statistics.
Best Answer
As discussed on Meta, I think it's a rather unusual person who could follow a mathematically rigorous treatment of Probability & Statistics without having previously studied them in a shallower, more heuristic fashion. If you're that person ...
For rigour the pair
would be hard to beat, but would still leave you needing a book on Probability—so three books, not one.
is one I'd like to recommend, for clarity of exposition & for their take on foundational issues, but it tends to assume a fair amount of familiarity with the basics, has nothing on Probability; & though it's thorough, I couldn't put my hand on my heart & say "rigorous".
On the whole I'd probably go with
; its opening chapters cover enough Probability & the later ones on inference are still pretty rigorous. Rather scanty on Bayesian analysis though.