mcnemar-testproportion;repeated measures
Is there a test I can use to compare two proportions from the same sample that are not mutually exclusive?
For example, I would like see if the proportion of people that answered "yes" to a particular question is significantly different from the proportion of people that answered "yes" to a different question.
What you're asking about is a test of dependent (or paired) proportions. Please see this article on McNemar's test or this calculator site (its language isn't the clearest, but it will help you calculate the result you're looking for).
All participants answered two questions. One question was answered correctly by 85% and the other question was answered correctly by 65%. I am interested in whether the proportion of correct answers is significantly larger for the first than the second question.
That would be a paired test.
Why is wrong to use a two-proportions z test in this case?
Because the independent-sample proportions test relies on ... independence. Specifically, the (normal approximation of the) distribution of the test statistic under the null hypothesis is computed on the basis that the observations are independent.
Does it also depend on the question one would like to answer with the statistical test?
No, at least not for any of the questions that occur to me.
What are the consequences of using the procedure nonetheless (e.g. will the significance values be systematically too high or low)?
If you do it with samples that are paired (and so positively correlated within the pairs), as in your example, then the variance of the difference in proportions will be different from what the independence assumption would suggest.
As a result, your true significance level will be larger than you chose it to be so you'll reject more often (much more often) than you should.
Below is the results of a simulation, first when the two columns are independent, and second when the variables are correlated (to get correlated binary variables I generated correlated standard normals with $\rho=0.6$ and dichotomized them by recording $1$ if they were less than 0.1**; the independent variables were created the same way but from independent normals).
** I chose a $p$ that was not exactly 1/2, in case there was any thought that $p$=1/2 might be a special case
These are 10000 simulations at n=100 for a two-tailed two sample proportions test (here done via a chi-square using R's default settings; the chi-square should be the square of the z-test done with the same settings). The true distribution of the test statistic is discrete and the chi-square (and the corresponding z-test) is approximate. The small spike in the left-side plot is due to that discreteness (and leads to mild conservatism in the test with independent proportions); ideally it should look uniform. In the right hand plot, correlated binaries (as described above) were used. There, about 98% of the tables generated had p-value <0.05. This is when the null hypothesis is true.
A small amount of effect might be tolerable, but this is quite dramatic.
Best Answer
The within-subjects test of equality of proportions is McNemar's test. I discuss it fairly thoroughly here: What is the difference between McNemar's test and the chi-squared test, and how do you know when to use each? The answer also provides an example demo in
R, should you use that, but it should be easy to do in any software.