The question is in the title: In both cases I use R² and Chi-Square to test if a fit/model is good enough. Up to now I only know that R² is used for models (?) and Chi-Square for Fits/functions (?). Is this true? And how exactly do they differ?
Thank you!
Solved – Difference between R² and Chi-Square
chi-squared-testr-squared
Related Solutions
Another reason that continuity corrections for contingency tables have gone out of fashion is that they only make a noticeable difference when the cell counts are small, and modern computing power has made it feasible to calculate 'exact' p-values for such tables.
Exact tests for 2×2 tables for R are provided by the exact2x2 package written by Michael Fay. There's an accompanying vignette about the exact McNemar's test and matching confidence intervals.
The very limited information you have is certainly a severe constraint! However, things aren't entirely hopeless.
Under the same assumptions that lead to the asymptotic $\chi^2$ distribution for the test statistic of the goodness-of-fit test of the same name, the test statistic under the alternative hypothesis has, asymptotically, a noncentral $\chi^2$ distribution. If we assume the two stimuli are a) significant, and b) have the same effect, the associated test statistics will have the same asymptotic noncentral $\chi^2$ distribution. We can use this to construct a test - basically, by estimating the noncentrality parameter $\lambda$ and seeing whether the test statistics are far in the tails of the noncentral $\chi^2(18, \hat{\lambda}) $ distribution. (That's not to say this test will have much power, though.)
We can estimate the noncentrality parameter given the two test statistics by taking their average and subtracting the degrees of freedom (a methods of moments estimator), giving an estimate of 44, or by maximum likelihood:
x <- c(45, 79)
n <- 18
ll <- function(ncp, n, x) sum(dchisq(x, n, ncp, log=TRUE))
foo <- optimize(ll, c(30,60), n=n, x=x, maximum=TRUE)
> foo$maximum
[1] 43.67619
Good agreement between our two estimates, not actually surprising given two data points and the 18 degrees of freedom. Now to calculate a p-value:
> pchisq(x, n, foo$maximum)
[1] 0.1190264 0.8798421
So our p-value is 0.12, not sufficient to reject the null hypothesis that the two stimuli are the same.
Does this test actually have (roughly) a 5% reject rate when the noncentrality parameters are the same? Does it have any power? We'll attempt to answer these questions by constructing a power curve as follows. First, we fix the average $\lambda$ at the estimated value of 43.68. The alternative distributions for the two test statistics will be noncentral $\chi^2$ with 18 degrees of freedom and noncentrality parameters $(\lambda-\delta, \lambda+\delta)$ for $\delta = 1, 2, \dots, 15$. We'll simulate 10000 draws from these two distributions for each $\delta$ and see how often our test rejects at, say, the 90% and 95% level of confidence.
nreject05 <- nreject10 <- rep(0,16)
delta <- 0:15
lambda <- foo$maximum
for (d in delta)
{
for (i in 1:10000)
{
x <- rchisq(2, n, ncp=c(lambda+d,lambda-d))
lhat <- optimize(ll, c(5,95), n=n, x=x, maximum=TRUE)$maximum
pval <- pchisq(min(x), n, lhat)
nreject05[d+1] <- nreject05[d+1] + (pval < 0.05)
nreject10[d+1] <- nreject10[d+1] + (pval < 0.10)
}
}
preject05 <- nreject05 / 10000
preject10 <- nreject10 / 10000
plot(preject05~delta, type='l', lty=1, lwd=2,
ylim = c(0, 0.4),
xlab = "1/2 difference between NCPs",
ylab = "Simulated rejection rates",
main = "")
lines(preject10~delta, type='l', lty=2, lwd=2)
legend("topleft",legend=c(expression(paste(alpha, " = 0.05")),
expression(paste(alpha, " = 0.10"))),
lty=c(1,2), lwd=2)
which gives the following:
Looking at the true null hypothesis points (x-axis value = 0), we see that the test is conservative, in that it doesn't appear to reject as often as the level would indicate, but not overwhelmingly so. As we expected, it doesn't have much power, but it's better than nothing. I wonder if there are better tests out there, given the very limited amount of information you have available.
Best Answer
Found this after a quick google: "R^2 is used to quantify the amount of variability in the data that is explained by your model. It's useful for comparing the fits of different models.
The Chi-square goodness of fit test is used to test if your data follows a particular distribution. It's more useful for testing model assumptions rather than comparing models."
sounds like Chi-square is more useful if you have a function you are trying to test (or a model you are trying to fit to your data) as opposed to the R^2 which tells you how much variability there is in your data, and therefore how much the best model fits.