probability distributionsstatistics
I got a question about the use of Chi Square test. Let's assume I am conducting a survey. And I have a question: "Have you ever heard of the Internet"?
The possible answers are: "Yes", "No", "Not sure" and I have 3 different age groups for my respondents: "<18", "18
When I want to do a Chi Square test to see if the age factor has an effect, can I just conduct this test on the "Yes" part? Or Do I need to include all the variables?
This sounds like a good case for using Fisher's Exact Test. It sounds like you could assign your data into a 2x2 contingecy table. If that is the case, I would read up on the Tea Drinker Exp w/r Fisher's Test.
Question A:
Question B Control Treatment Totals
Control 2 4 6
Treatment 5 6 11
Totals 7 10 17
p-value = R1!*R2!*C1!*C2! / { (grand total)! times (factorials of internals) }
p-value = (6!*11!*7!*10!) / (17!*2!*4!*5!*6!)
One more thing, MH test is a more specific example of the the Fisher's Test when used with multiple confounding levels, for example: 2x2xk matrices. Also I would not rule out the chi-square test but it assumes that you data is normal and this can be done but the warning is that you must have a large data set for that assumption of normality. The Fisher's Exact Test does not depend on the assumption of normality, ie good for small data sets.
This seems to be a two-sample test with Groups 1 (of size $n_1$) and 2 (of size $n_2$). Your data are scores from 1 to 10 on the question.
Welch t test. If $n_1$ and $n_2$ are large enough (perhaps both above 20), you might be able to get a reliable answer using a Welch 2-sample t-test.
Wilcoxon test. You are almost sure to have lots of ties (repeated scores) even if both sample sizes are relatively small. Thus you will get error messages about ties when trying to do a Wilcoxon rank-sum test, along with an approximated P-value or a statement that a P-value is not available (depending of the software you use).
Permutation test. Perhaps it is best to do a permutation test. Under the null hypothesis that the two groups tend to give the same responses to the question, the argument is that the scores could be permuted between Groups A and B without effect. So if we choose some measure of difference such as the difference $D = \bar X_1 - \bar X_2$ between the two sample means, we can use either combinatorics or simulation to get the null permutation distribution of $D$, and judge whether your observed value of $D$ is consistent with the null distribution.
Example. I will illustrate each kind of test using fake data with 25 subjects in each group (although none of the tests require sample sizes to be equal).
Here are listings and summaries of some fake data to use for testing.
x1; summary(x1)
## 9 6 4 10 5 5 8 8 8 8 8 9 8 6 4 7 8 9 8 6 8 8 5 8 9
## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 4.00 6.00 8.00 7.28 8.00 10.00
x2; summary(x2)
## 10 9 10 7 7 8 8 10 8 5 7 7 7 5 8 10 10 10 9 7 9 10 10 10 9
## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 5.0 7.0 9.0 8.4 10.0 10.0
A quick look shows means to be greater in Group 2 than in Group 1. Is this difference statistically significant?
t test: A Welch 2-sample t test in R statistical software finds a significant difference. (P-value $\approx$ 2%.) The only doubt is whether data are sufficiently nearly normally distributed for the t test to give accurate results. (Data for both groups spectacularly fail a Shapiro-Wilk test with P-values < .01. But sample sizes may be large enough for the t test to be useful anyhow.)
t.test(x1, x2)
## Welch Two Sample t-test
## data: x1 and x2
## t = -2.434, df = 47.853, p-value = 0.01872
## alternative hypothesis: true difference in means is not equal to 0
## sample estimates:
## mean of x mean of y
## 7.28 8.40
Wilcoxon test: The Wilcoxon test, for a difference in medians gives a (tentative) P-value of about 2%, but warns that it may not be accurate. However, there are only seven uniquely different values among the 50 subjects. So the number of ties is 'massive' and the Wilcoxon test is based on a comparison of ranks, which can be problematic when there are many ties. I would not want to trust the result of the Wilcoxon test.
wilcox.test(x1, x2)
## Wilcoxon rank sum test with continuity correction
## data: x1 and x2
## W = 200.5, p-value = 0.02702
## alternative hypothesis: true location shift is not equal to 0
## Warning message:
## In wilcox.test.default(x1, x2) : cannot compute exact p-value with ties
Permutation test. It would be tedious to derive the exact permutation distribution of $D$ for this example. The usual cure is to simulate a large number of permutations and to approximate the P-value from simulation results. Here is a brief program in R to find the approximate P-value (2.1%) of the permutation test. (You may get a slightly different P-value at each run of the program, but not enough different to matter in the interpretation. For this program, subsequent runs all gave values rounding to 2%)
m = 10^4; d.perm = numeric(m)
all = c(x1, x2); d.obs = mean(x1) - mean(x2)
n1 = n2 = 25
for (i in 1:m) {
perm = sample(all, n1+n2)
d.perm[i] = mean(perm[1:n1]) - mean(perm[(n1+1):(n1+n2)])
}
mean(abs(d.perm) >= abs(d.obs))
## 0.0215
Here is a histogram of the approximate permutation distribution. The solid red line at the left is the observed value of $D$ for the data above. The dotted red line at the right is just as extreme (far from 0) as the observed value of $D.$ The P-value of this 2-sided permutation test is the percentage of values in the permutation distribution outside these red lines, in this case, 2.1%.
Conclusion: The two groups differ significantly. The t test is probably OK, because, for samples this large, the Central Limit Theorem tends to make the sample means very nearly normal even if the data are not normal. For groups as small as ten, I would certainly insist on seeing permutation test results before drawing a conclusion.
You can read more about permutation tests in this paper by Eudey. The two-sample test above is discussed, with additional examples, in Section 4.
Almost certainly, your data will look different than my fake data. Please let me know if you have trouble relating my answer to your specific data.
Note: The fake data above were generated from populations with respective means about 3/5 and 5/6 using the R code below. (So it is appropriate that the tests found a significant difference.) By using the same seed I used, you should get exactly the same data.
set.seed(1234)
x1 = ceiling(10*rbeta(25, 3, 2))
x2 = ceiling(10*rbeta(25, 5, 1))
Addendum (Your Data from Comment). Your result in the Comment seems OK. Significant at 9.3% < 10% level; sometimes optimistically called "suggestive" of significance.
If you honestly expected (before seeing data) Gp2 scores to be higher, then maybe this should be a left-sided test of $H_0: \mu_1 \ge \mu_2$ vs. $H_a: \mu_1 < \mu_2.$ if so, P-value would be 3.8% < 5% for significance at the 5% level.
x1 <- c(0,7,10,0,9,5,10,6,8,7,8,2,2,8,10,7,10)
x2 <- c(7,4,10,10,9,10,10,9,10,7,5,10,10,10,10,5,10,2)
all = c(x1, x2); gp = rep(0:1; times = c(17,18))
stripchart(all~gp, meth="stack", pch=19, col=c("blue", "green3"))
Welch t-test gives P-value 0.09024. Repeat of permutation test with
m = 10^6 iterations to reduce possibility of simulation error.
x1 <- c(0,7,10,0,9,5,10,6,8,7,8,2,2,8,10,7,10)
x2 <- c(7,4,10,10,9,10,10,9,10,7,5,10,10,10,10,5,10,2)
m = 10^6; d.perm = numeric(m)
all = c(x1, x2); d.obs = mean(x1) - mean(x2)
n1 = length(x1); n2 = length(x2)
for (i in 1:m) { perm = sample(all, n1+n2)
d.perm[i] = mean(perm[1:n1]) - mean(perm[(n1+1):(n1+n2)]) }
mean(abs(d.perm) >= abs(d.obs))
## 0.093149
## 0.093183 # 2nd run with m=10^6
mean(d.perm < d.obs)
## 0.038349 # P-value of LEFT SIDED test
length(unique(d.perm))
## 75 # uniquely different sim. values of D (enough)
hist(d.perm, prob=T, col="skyblue2", main="Simulated Permutation Distribution")
abline(v=d.obs, col="red", lwd=2)
abline(v=-d.obs, col="red", lwd=2, lty="dashed")
Note: If this is for a reviewed paper, you might get criticism (as noted by @Nameless) that the permutation test involves taking sample means of ordinal data. Possible nonparametric, ordinal-oriented alternatives:
(a) Use
median instead of mean in the permutation test when finding
d.obs and (within the loop) when finding d.perm, but not at
the end when finding the P-value. (In R, the mean of a logical vector
is the proportion of its TRUEs.) Trouble is I got only only about 20
uniquely different values of d.perm that way; not quite enough
for my taste. One-sided P-value 0.047.
(b) Do a Welch t test
on rank-transformed data. (Ranks are appropriate for ordinal data, their means are
likely not far from normal with sample sizes above 15.) From
t.test(rank(all) ~ gp, alte="less"), I get (Welch, one-sided) P-value 0.03457.
Best Answer
Inorder to use Chi Square test you need to have two variables. In your example these variables are age and use/no use of internet. Hence you have to use 'yes','no,'not sure'.