If $4$ vertices of a regular octagon are selected at random, then the probability that the quadrilateral formed by them is a rectangle is:
(A) $1\over8$
(B) $2\over21$
(C) $1\over32$
(D) $1\over35$
I went about it this way:
We have two types of rectangles, i.e. the blue and green.
If we count the number of rectangles manually, we get 6 rectangles in total.
Counting the number of ways to choose $4$ of $8$ vertices, we get $^8C_4=70$.
So the answer should be $\frac {6}{70}=\frac {3}{35}$. But this is not in the options at all. The answer is (D).
Where is my mistake?
Okay, so I asked my teacher and he solved it like this:
To calculate $n(S)$, where $S$ is sample space, first we select any one vertice, say $A$ in $^8C_1$ ways. Then, for the remaining $3$ vertices of the quadrilateral, there are $^7C_3$ ways. We divide this by $4$ since $A$ can be any of the four vertices of the chosen quadrilateral.
$$n(S) = ^8C_1 \times ^7C_3 \times \frac 1 4 = 70$$Now we count the number of quadrilaterals using $A$ as a vertice. To avoid repetition, we only go counting counterclockwise (for example, we will count $ABEF$ but not $AHED$.
Now there are only $2$ quadrilaterals using $A$. So $n(E)=2$, where $E$ is the event of a rectangle.
Then $$P(E) = \frac {n(E)}{n(S)} = \frac 2 {70} = \frac 1 {35}$$
Which of these approaches is correct? And what is the error in the incorrect one?
Best Answer
Yours is right.
A big problem with your teacher's argument is that they forgot about the quadrilateral ADEH. Even so, it is not quite right, since I am not even sure what they are counting.
So, their n(S) is precisely 70, since they are actually just choosing 4 vertices out of 8, and then giving them a well-defined order. If they want to count it on a per-vertex basis, they would multiply that by $4$ (to allow for rotations), and say that each vertex gives rise to $3$ quadrilaterals, giving a final answer of $\frac{3*8}{70*4}$ which matches your answer.
I think the question askers got the 1/35 by using the following argument: you can pick any vertex as the first vertex, and then you can pick one of 6 of 7, and then 1/6 and then 1/5 vertices, giving you 1/35. This is wrong, and the fundamental difference is that not every permutation of the selection of the vertices gives rise to a rectangle, so selecting them one by one is indeed different than selecting 4 from a set and then checking. It is quite subtle.