Solved – How to test that the difference between two sample means is greater that a given value (in Matlab)

MATLABt-test

I (unfortunately) only have an elementary knowledge of statistics, and so please excuse me for my following basic question.

I have two independent samples (with sample size of 60 and 765 respectively) and I need to test if the mean of first sample is greater than the mean of second sample by a non-zero given value (the two samples have unequal variance). Since, Matlab is the tool that is used in my workplace, I need to do this in Matlab (I had looked at the "ttest2" function in Matlab, but from the help document,I could not figure out how to use ttest2 to test whether the difference between two sample means is greater than a given non-zero value).

Many thanks in advance.

-Tinku

Best Answer

I am not a Matlab expert, but looking at the documentation for ttest2 it looks like there is not an option to change the null value (what the difference is under the null hypothesis).

But it can return a confidence interval on the difference between the 2 means and you can use a confidence interval to do a hypothesis test (the yes/no part, not the exact p-value). Create the confidence interval and look to see if the difference that you are interested in is within the interval. If it is in the interval then that means the result is not statistically significant, if it is outside of the interval then it is statistically significant.

Another option is that if you subtract the null value difference from each value in one set (the one you expect to be higher) then that adjusts the mean appropriately and does not change the variances or other values. So you can pass the new vectors (one with the subtraction done) to the ttest2 function and it will give you the appropriate p-value.

Related Solutions

Hypothesis Testing – How to Test for Differences Between Non-Normally Distributed Group Means

The idea that the t-test is only for small samples is a historical hold over. Yes it was originally developed for small samples, but there is nothing in the theory that distinguishes small from large. In the days before computers were common for doing statistics the t-tables often only went up to around 30 degrees of freedom and the normal was used beyond that as a close approximation of the t distribution. This was for convenience to keep the t-table's size reasonable. Now with computers we can do t-tests for any sample size (though for very large samples the difference between the results of a z-test and a t-test are very small). The main idea is to use a t-test when using the sample to estimate the standard deviations and the z-test if the population standard deviations are known (very rare).

The Central Limit Theorem lets us use the normal theory inference (t-tests in this case) even if the population is not normally distributed as long as the sample sizes are large enough. This does mean that your test is approximate (but with your sample sizes, the appromition should be very good).

The Wilcoxon test is not a test of means (unless you know that the populations are perfectly symmetric and other unlikely assumptions hold). If the means are the main point of interest then the t-test is probably the better one to quote.

Given that your standard deviations are so different, and the shapes are non-normal and possibly different from each other, the difference in the means may not be the most interesting thing going on here. Think about the science and what you want to do with your results. Are decisions being made at the population level or the individual level? Think of this example: you are comparing 2 drugs for a given disease, on drug A half the sample died immediatly the other half recovered in about a week; on drug B all survived and recovered, but the time to recovery was longer than a week. In this case would you really care about which mean recovery time was shorter? Or replace the half dying in A with just taking a really long time to recover (longer than anyone in the B group). When deciding which drug I would want to take I would want the full information, not just which was quicker on average.

Solved – Can a two-sample t-test be used with data that doesn’t follow a normal distribution

One simple way to convince yourself that the CLT applies or does not apply is with some simulations.

Here is some R code:

testfun <- function(n1=19, n2=15) {
    x <- rexp(n1, 1/3)
    y <- rt(n1, 5) + 3
    t.test(x,y)$p.value
}

out <- replicate(10000, testfun(n1=19, n2=15))
hist(out)
abline(v=0.05, col='red')
mean( out <= 0.05 )

This code defines a function (testfun) that generates data from 2 different distributions (t with 5 df and exponential ) that have the same mean (3 in this case) and runs the built in t.test function and returns the p-value.

The replicate then runs this 10,000 times and we look at the results. The histogram should be close to uniform, but in this case we see an excess of values close to 0. The mean function calculates the type I error rate (since the null is true in the simulations), for my run this was a little of 7% when it should be 5%. Is that far enough to cause you concern? or are you happy with that as a "close enough" approximation?

Of course you should probably run this generating data from distributions that are more reasonable for your study, it may be that for something less skewed than the exponential that the differences would be small enough to not worry about.

Best Answer

Related Solutions

Hypothesis Testing – How to Test for Differences Between Non-Normally Distributed Group Means

Solved – Can a two-sample t-test be used with data that doesn’t follow a normal distribution

Related Question