A student is asked to answer 50 true/false questions and he would get 35 right and 15 incorrect if he had to put his best guesses for each question down.
Now, for each question he has a certain confidence of getting the problem correct and if we start imposing penalties for getting a question wrong, he can easily adapt by only answering questions with a confidence level tantamount to the relation [point penalty if wrong]/[point reward if right], and leave all the other one's blank.
Is there a way for the test designer to design a reward/penalty scheme that maximizes skill penetration and reduces the role of luck and, if so, what factors does the designer have to look out for?
Assume that the test questions have increasing difficulty in a fashion that we cover the full spectrum of confidence on the student.
EDIT: Another idea would be to let the student gamble with points. Also setting a limit on each question and a global gambling limit.
Is there any reasonable scheme to reduce the role of luck in general on true and false questions?
Best Answer
To maximize skill penetration, award 1 point for each correct answer, and -1000 for each incorrect answer. Note: this is not a joke, this is how to only get answers that the student is entirely confident of.
In your example, the student's knowledge is net zero, getting the same score as random guessing. Such a student is unlikely to adapt his or her test-taking strategy to subtle rewards and penalties.
Additional commentary regarding gambling: Suppose you let each student give a weight to each question, of any real number in $[0,1]$, as well as the T/F answer. If right, the student gains that many points, if wrong, loses. Naively one might think that this would encourage students to weigh questions more if they are more confident, and less otherwise. In fact this is not true. If I am 51% confident that the answer is true, I will maximize my score by weighing it 1 rather than anything else.
Suppose instead you allow students to weight as above, but this time if they are wrong they lose DOUBLE their wager. If I am right with probability $p$ my expected score is $pw-2(1-p)w=w(3p-2)$. All this change does is move the threshhold from $p=\frac{1}{2}$ to $p=\frac{2}{3}$. If I believe that $p>\frac{2}{3}$, I should bet the maximum, otherwise I should bet 0.
Result: under all similar gambling schemes, there is never any reason to bet anything other than the maximum (or zero/leave the question blank).