I am using a chi-sq test between two proportions (http://statistic-on-air.blogspot.com/2009/07/comparison-of-two-proportions.html). I am using this over the z-test for proportions because I do not think my data is normal.
My question is how do I get the difference (shift) estimate & confidence intervals for this test.
For example,
Data1: 193/252=.77
Data2: 154/227=.68
P-value=.032 (I used the formula in the link above. I will also add my exact code below)
Shift Estimate: I assume its the difference between proportions, ie .77-.68=.09. Is this correct?
Confidence Interval: ??????????????????
Code for generating P-value (matlab):
% Pooled estimate of proportion
p0 = (n1+n2) / (N1+N2);
% Expected counts under H0 (null hypothesis)
n10 = N1 * p0;
n20 = N2 * p0;
% Chi-square test, by hand
observed = [n1 N1-n1 n2 N2-n2];
expected = [n10 N1-n10 n20 N2-n20];
chi2stat = sum((observed-expected).^2 ./ expected);
p = 1 - chi2cdf(chi2stat,1);
H=0; if(p<.05), H=1; end
Best Answer
Just for statistically intimidated people (wait, it should be 'intimidated by statistics') that can identify themselves with this joke:
A patient asks his surgeon what the odds are of him surviving an impending operation. The doctor replies that the odds are usually 50-50. "But there is no need to worry," the doctor explains.
"The first fifty have already died.
It is meant to be a quick reference for those confused with the terminology and overlapping tests - I am not addressing confidence intervals.
COMPARING PROPORTIONS BETWEEN SAMPLES:
I will use the following toy tabulated data:
This is what it looks like (with marginals):
So we have
368
patients:178
onDrug A
, and190
onDrug B
and we try to see if there are differences in the proportion of heartburn symptoms between drug A and B, i.e. $p1 = 64/178$ vs $p2 = 92/190$.1. FISHER EXACT TEST: There is a discussion on Wikipedia about "Controversies". Based on the hypergeometric distribution, it is probably most adequate when the expected values in any of the cells of a contingency table are below 5 - 10. The story of the RA Fisher and the tea lady is great, and can be reproduced in [R] by simply grabbing the code here. [R] seems to tolerate without a pause the large numbers in our data (no problem with factorials):
2. CHI-SQUARE TEST OF HOMOGENEITY: Otherwise known as the goodness of fit Pearson's chi squared test. For larger samples (> 5 expected frequency count in each cell) the $\chi^2$ provides an approximation of the significance value. The test is based on calculating the expected frequency counts obtained by cross-multiplying the marginals (assuming normal distribution of the marginals, it makes sense that we end up with a $\chi^2$ distributed test statistic, since if $X\sim N(\mu,\sigma^)$, then $X^2\sim \chi^2(1))$:
The degrees of freedom will be calculated as the {number of
populations
(Heartburn
sufferers andNormals
, i.e.2
)minus 1
} * {number of levels in thecategorical variable
(Drug A
andDrug B
, i.e.2
)minus 1
}. Therefore, in a2x2
table we are dealing with1 d.f
. And crucially, a $\chi^2$ of $1\,df$ is exactly asquared
$N \sim (0,1)$ (proof here), which explains the sentence "a chi-square test for equality of two proportions is exactly the same thing as a z-test. The chi-squared distribution with one degree of freedom is just that of a normal deviate, squared. You're basically just repeating the chi-squared test on a subset of the contingency table" in this post.The Test Statistic is calculated as:
$\chi^2=\frac{(64-75)^2}{75} + \frac{(92-81)^2}{81} +\frac{(114-103)^2}{103} + \frac{(98-109)^2}{109} = 5.39$
This is calculated in
R
with the functionprop.test()
orchisq.test()
, which should yield the same result, as indicated here:or..
3. G-TEST: The Pearson's chi-test statistic is the second order Taylor expansion around 1 of the G test; hence they tend to converge. In
R
:4. Z-TEST OF PROPORTIONS: The normal distribution is a good approximation for a binomial when $np>5$ and $n(1-p)>5$. When the occurrences of successes are small in comparison with the total amount of observations, it is the actual number of expected observations that will determine if a normal approximation of a poisson process can be considered ($\lambda \geq 5$).
Although the post hyperlinked is old, I haven't found in CV an
R
function for it. This may be due to the fact explained above re: $\chi^2_{(df=1)}\sim \, N_{(0,1)}^2$.The Test Statistic is:
$ \displaystyle Z = \frac{\frac{x_1}{n_1}-\frac{x_2}{n_2}}{\sqrt{p\,(1-p)(1/n_1+1/n_2)}}$ with $\displaystyle p = \frac{x_1\,+\,x_2}{n_1\,+\,n_2}$, where $x_1$ and $x_2$ are the number of "successes" (in our case, sadly, heartburn), over the number of subjects in that each one of the levels of the categorical variable (
Drug A
andDrug B
), i.e. $n_1$ and $n_2$.In the linked page there is an ad hoc formula. I have been toying with a spin-off with a lot of loose ends. It defaults to a two-tailed alpha value of
0.05
, but can be changed, as much as it can be turned into a one tailedt = 1
:In our case:
Giving the same z value as the function in the R-Bloggers:
z.prop(64, 178 , 92, 190) [1] -5.44273
OK... Hope it helps somebody out there, and I'm sure mistakes will be pointed out...