I think one reason it is so hard to answer this is that R is so powerful and flexible that a real introduction to R programming goes well beyond what is normally needed in an introduction to statistics. The books that teach statistics using MiniTab, JMP or SPSS are doing relatively straightforward things with the software that barely scratch the surface of what R is capable of when it comes to data manipulation, simulations, custom-built functions, etc.
Having said that, I think that Wilcox's Modern Statistics for the Social and Behavioral Sciences: A Practical Introduction (2012) is a brilliant new book. It assumes no statistical knowledge and takes you from scratch right through to a big range of modern robust techniques; and assumes not much more R knowledge than the ability to open it up and load a dataset. It covers many of the classical techniques too including ANOVA (mentioned in the OP).
I would see this book as the equivalent of the books that introduce stats and a stats package like SPSS at the same time. However, it won't teach you to program in R - only how to do modern statistical analysis with it, with an emphasis on robust techniques that address the known problems with classical analysis that are sidelined by most other approaches to teaching statistics.
The three problems with classical methods that this book particularly addresses right from the beginning are sampling from heavy-tailed distributions; skewness; and heteroscedasticity.
Wilcox uses R because "In terms of taking advantage of modern statistical techniques, R clearly dominates. When analyzing data, it is undoubtedly the most important software development during the last quarter of a century. And it is free. Although classic methods have fundamental flaws, it is not suggested that they be completely abandoned... Consequently, illustrations are provided on how to apply standard methods with R. Of particular importance here is that, in addition, illustrations are provided regarding how to apply modern methods using over 900 R functions written for this book."
This book is so excellent that after we bought a copy for work I purchased my own copy at home.
The chapter headings are:
- numerical and graphical summaries of data;
- probability and related concepts;
- sampling distributions and confidence intervals;
- hypothesis testing;
- regression and correlation;
- bootstrap methods;
- comparing two independent groups;
- comparing two dependent groups;
- one-way ANOVA;
- two-way and three-way designs;
- comparing more than two dependent groups;
- multiple comparisons;
- some multivariate methods;
- robust regression and measures of association;
- basic methods for analyzing categorical data;
Further edit - having checked out the David Moore example of what you are looking for, I really think Wilcox's book meets the need.
I personally thought that Modern Applied Statistics with S-Plus ticks all of the boxes you have outlined. Every example has R code, they give good references to other sources, and Venables and Ripley have a wonderfully terse and explanatory writing style which I really appreciated. I tend to re-read the book every so often, and each time I get more from it. Of course, your mileage may vary.
Best Answer
As you said, it's not necessarily the case that a mathematician may want a rigorous book. Maybe the goal is to get some intuition of the concepts quickly, and then fill in the details. I recommend two books from CMU professors, both published by Springer: "All of Statistics" by Larry Wasserman is quick and informal. "Theory of Statistics" by Mark Schervish is rigorous and relatively complete. It has decision theory, finite sample, some asymptotics and sequential analysis.
Added 7/28/10: There is one additional reference that is orthogonal to the other two: very rigorous, focused on learning theory, and short. It's by Smale (Steven Smale!) and Cucker, "On the Mathematical Foundations of Learning". Not easy read, but the best crash course on the theory.