Solved – What will be the correct answer, if we modify the “Best statistics question ever”

paradoxprobability

There is a popular question, called "Best statistics question ever".

If you choose an answer to this question at random, what is the chance you will be correct?

A) 25%
B) 50%
C) 60%
D) 25%

This task is not very difficult, the correct answer is 0%. But if we modify it like this:

If you choose an answer to this question at random, what is the chance you will be correct?

A) 50%
B) 25%
C) 60%
D) 50%

What will be the correct answer? Do we have two correct answers: 25% and 50%, or there is no correct answer, as with this two correct answers the chance of choosing the correct answer is in fact 75% (but we do not have 75% written on the desk)?

By the way. Does the answer 0% remains correct answer, the third correct answer in this case?

Best Answer

The apparent paradoxes (of logic or probability) can be resolved by framing the questions clearly and carefully.

The following analysis is motivated by the idea of defending an answer: when a test-taker can exhibit a possible state of affairs (consistent with all available information) in which their answer indeed is correct, then it should be marked as correct. Equivalently, an answer is incorrect when no such defense exists; it is considered correct otherwise. This models the usual interactions between (benevolent, rational) graders and (rational) test-takers :-). The apparent paradox is resolved by exhibiting multiple such defenses for the second question, only one of which could apply in any instance.


I will take the meaning of "random" in these questions in a conventional sense: to model a random choice of answer, I will write each answer on a slip of paper ("ticket") and put it in a box: that will be four tickets total. Drawing a ticket out of the box (after carefully and blindly shuffling the box's contents) is a physical model for a "random" choice. It motivates and justifies a corresponding probability model.

Now, what does it mean to "be correct"? In my ignorance, I will explore all possibilities. In any case, I take it as definite that zero, one, or even more of the tickets may be "correct." (How might I know? I simply consult the grading sheet!) I will mark the "correct" answers as such by writing the value $1$ on each correct ticket and writing $0$ on the others. That's routine and should not be controversial.

An obvious but important thing to notice is that the rule for writing $0$ or $1$ must be based solely on the answer written on each ticket: mathematically, it is a mapping (or reassignment) sending the set of listed answers ($\{.25, .50, .60\}$ in both questions) into the set $\{0,1\}$. This rule is needed for self-consistency.

Let's turn to the probabilistic element of the question: by definition, the chance of being correct, under a random drawing of tickets, is the expectation of the values with which they have been marked. The expectation is computed by summing the values on the tickets and dividing that by their total number. It will therefore be either $0$, $.25$, $.50$, $.75$, or $1$.

A marking will make sense provided that only the tickets whose answers equal the expectation are marked with $1$s. This also is a self-consistency requirement. I claim that this is the crux of the matter: to find and interpret the markings that make sense. If there are none, then the question itself can be branded as being meaningless. It there is a unique marking, then there will be no controversy. Only if two or more markings make sense will there be any potential difficulty.

Which markings make sense?

We don't even need to make an exhaustive search. In the first question, the expectations listed on the tickets are 25%, 50% and 60%. The latter is impossible with four tickets. The first would require exactly one ticket to be marked; the second, two tickets. That gives at most $3+3=6$ possible markings to explore. The only marking that makes sense puts $0$s on each ticket. For this marking, the expectation is $(0+0+0+0)/4 = 0$. That justifies the stated answer to the first question. (Arguably, the sole correct response to the first question is not to select any answer!)

In the second question, the same answers appear and once again there are six markings to explore. This time, three markings are self-consistent. I tabulate them:

Solution 1                Solution 2                Solution 3
Ticket Answer Mark        Ticket Answer Mark        Ticket Answer Mark
     A    50%    1             A    50%    0             A    50%    0
     B    25%    0             B    25%    1             B    25%    0
     C    60%    0             C    60%    0             C    60%    0
     D    50%    1             D    50%    0             D    50%    0

Therefore, there are three distinct possible definitions of "correct" in the second problem, leading to either A or D being correct (in solution 1) or only B being correct (in solution 2), or none of the answers being correct (in solution 3).

One way to interpret this state of affairs is that for each of the answers A, B, and D, there exists at least one way of marking the tickets that makes those answers correct. This does not imply that all three are simultaneously correct: they couldn't be, because $.25 \ne .50$. If you were the grader of the test, then if you marked any of A, B, or D correct, then you would not get an argument from the test-taker; but if you marked any of them incorrect, the test-taker would have a legitimate basis to dispute your scoring: they would invoke either solution 1 or solution 2. Indeed, if a test-taker refused to answer the question, solution 3 would give them a legitimate basis to argue that their non-response ought to get full credit, too!

In summary, this analysis addresses the second part of the question by concluding that any of the following responses to question 2 should be marked correct because each of them are defensible: A, B, D, A and D, and nothing. No other response can be defended and therefore would not be correct.

Related Question