I was not able to reproduce the results you got from WebPower using the pilot data you supplied. I was able to reproduce your R code however.
You are correct that you can't use the $\eta^2$ for Cohen's f, but $f^2 = \frac{\eta^2}{1-\eta^2}$
"However, how should I compute the effect size from the pilot study" - use the $\eta^2$ from the pilot study.
"Why are there interaction effect sizes, i.e, the effect size for group x vs group y?" Those are the effect sizes for the pair-wise comparisons (if you were using a t-test or a TukeyHSD)
require(dplyr)
require(reshape2)
pilot <- data.frame(option1 = c(6.3, 2.8, 7.8, 7.9, 4.9),
option2 = c(9.9, 4.1, 3.9, 6.3, 6.9),
option3 = c(5.1, 2.9, 3.6, 5.7, 4.5),
option4 = c(1.0, 2.8, 4.8, 3.9, 1.6))
pilot2 <- pilot %>%
reshape2::melt(value.name = "y") %>%
dplyr::rename("option" = "variable")
lm1 <- lm(y ~ option, data = pilot2)
aov1 <- aov(lm1)
means <- apply(pilot, 2, mean)
vs <- apply(pilot, 2, var)
# cohen's f for overall anova
# eta^2 = SSR / SST
eta.sq <- anova(lm1)$`Sum Sq`[2] / sum(anova(lm1)$`Sum Sq`)
f <- sqrt(eta.sq / (1-eta.sq))
# cohen's d for pairwise
d <- abs(means[c(1,1,1,2,2,3)] - means[c(2,3,4,3,4,4)]) / sqrt(((5-1)*vs[c(1,1,1,2,2,3)] + (5-1)*vs[c(2,3,4,3,4,4)])/ (5+5))
names(d) <- c("1-2", "1-3", "1-4", "2-3", "2-4", "3-4")
require(pwr)
# with 5 samples, we have the power to detect effect size f = 0.835
# i.e. with only 5 samples, we need a large effect to detect
pwr::pwr.anova.test(k = 4, n = 5, sig.level = 0.05, power = 0.80)
#>
#> Balanced one-way analysis of variance power calculation
#>
#> k = 4
#> n = 5
#> f = 0.8352722
#> sig.level = 0.05
#> power = 0.8
#>
#> NOTE: n is number in each group
# since we have a really large effect in the pilot for f = 1.2,
# we only need 3 per group to detect with 80% power
pwr::pwr.anova.test(k = 4, f = 1.2414, sig.level = 0.05, power = 0.80)
#>
#> Balanced one-way analysis of variance power calculation
#>
#> k = 4
#> n = 2.950833
#> f = 1.2414
#> sig.level = 0.05
#> power = 0.8
#>
#> NOTE: n is number in each group
Best Answer
In
glht, "tukey" doesn't refer to Tukey's HSD. It just means "do all pairwise comparisons". By default,ghltuses a "single-step" correction method, but other correction methods could be used.As far as I can tell, the
TukeyHSDfunction uses the Tukey-Kramer procedure. The code for the function can be found on GitHub. See also the example on RPubs.At least for the simple case of a one-way design with equal variances in groups (but potentially unequal sample sizes), it appears that the results of
TukeyHSDwill match those ofemmeanswith a Tukey adjustment, and those ofglhtwith a "single-step" adjustment.Likewise, it appears that the results of
emmeanswith no adjustment and those ofglhtwith no adjustment, will match those ofpairwise.t.testwith no adjustment.