library(tidyverse)
set.seed(301)
# Five groups, ALL from the same population - no real differences at all
false_positive <- function(k) {
groups <- map(1:k, \(i) rnorm(20, 100, 15))
pairs <- combn(k, 2)
ps <- apply(pairs, 2, \(idx) t.test(groups[[idx[1]]], groups[[idx[2]]])$p.value)
any(ps < 0.05) # did ANY comparison "find" something?
}
rates <- map_dbl(2:8, \(k) mean(replicate(400, false_positive(k))))
tibble(groups = 2:8, rate = rates) %>%
ggplot(aes(groups, rate)) +
geom_col(fill = "#E74C3C") +
geom_hline(yintercept = 0.05, linetype = "dashed") +
scale_y_continuous(labels = scales::percent) +
labs(title = "Chance of at least one false positive when NOTHING is going on",
x = "Number of groups compared", y = "False positive rate") +
theme_minimal(base_size = 12)ANOVA Hypothesis Testing
The Role of p-values in Making Statistical Decisions
Hey, Is There A, Like, a Huge and Ugly Type-Error Chart??
There Sure Is! Error Types as a Matrix
Assumptions of ANOVA
Tip
- Independence of Observations: Each observation should be independent of the others.
- Normality: The residuals (errors) should be normally distributed.
- Homogeneity of Variance: The variances across groups should be approximately equal.
Why Not Just Run Many t-tests? (code)
Why Not Just Run Many t-tests (plot)?
Tip
Tip: When you have more than two groups, you should use ANOVA instead of multiple t-tests to avoid inflating the Type I error rate. Type I errors increase with the number of comparisons made. ANOVA controls for this by testing all groups simultaneously.
Letโs Use an Example
The Dog Show: Description of Problem
Important
- Imagine that you are hosting a dog show and you want to determine if there are significant differences in the average weights of different dog breeds
- For this, you will need specific characteristics of the dogs (e.g., breed, weight)
- You will form groups based on the characteristics of the dogs, and then use ANOVA to compare the average weights across these groups.
What are the best ways to arrange groups?!
The Dog Show: Letโs Talk About Grouping The Data
For Successful group, we have two main requirements:
- Homogeneity of Variance: The variances across groups should be approximately equal.
- Each group is to be homogeneous or similar by some characteristic.
- Distinct means: The mean of each group is to be different from the others
- If groups have the same mean, then they are likely not separate in any meaningful way.
The Dog Show: Bad Group, No Biscuit!
What if the groups are not well-defined?
The Dog Show: Bad Groups
- Not a fit because all groups are too similar (homogenous). Same mean and same variance.
- Young vs old, and short-haired vs long-haired
- This grouping fails to explain the variation in the overall distribution (yellow-orange)
The Dog Show: Better Groups
- A better fit because the groups are distinct and homogeneous. The means are different, and the variances are similar.
- Pet vs Working breed and less athletic vs more athletic
- The significant overlap of distributions causes difficulty in distinguishing between the groups.
The Dog Show: Best Grouping
- A Very good fit because the groups are distinct and homogeneous. The means are different, and the variances are similar.
- Weight by breed
- We can discriminate between groups since the distributions are distinct and homogeneous.
Good Groupings
Ideal groupings are distinct and homogeneous
Dogs Groups Selected From a Population
So What Test?
Important
Want to distinguish between the means of three or more groups.
- ANOVA is the appropriate test to use in this case.
- T-tests, works fine for two means, but not more than two means
- The risks of type error increase with each mean that we are comparing.
- ANOVA is the appropriate test to use in this case.
Letโs Hit the Code!
Pos_hoc: WHICH Groups Differ? (plot)
Want to ask: โWhich pairs of iris species have statistically significant differences in their mean sepal lengths?โ
Important
Tukeyโs HSD adjusts for all the pairwise comparisons, so these p-values are already corrected. Any interval that misses zero is a real difference. โ
Two-Way ANOVA & Interaction ๐
Df Sum Sq Mean Sq F value Pr(>F)
supp 1 205.4 205.4 15.572 0.000231 ***
dose 2 2426.4 1213.2 92.000 < 2e-16 ***
supp:dose 2 108.3 54.2 4.107 0.021860 *
Residuals 54 712.1 13.2
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
ANOVA Diagnostics ๐ฉบ
Residuals vs Fitted โ should be a shapeless cloud (equal variance)
Q-Q Residuals โ points should hug the line (normality)
Scale-Location โ flat red line means constant spread
Residuals vs Leverage โ points past Cookโs distance are influential
And Now Onto the Ethics of Testing Responsibly
