Normal Curves: Sexy Science, Serious Statistics

Statistical power is one of the most important ideas in statistics—and one of the most misunderstood. In this episode, we explain what statistical power really means, why researchers need to think about it before they collect their data, and what goes wrong when they calculate “post hoc power” after the study is over. Along the way, we hunt for UFOs, test Kristin’s psychic abilities, explore the ethics of underpowered studies, and discover why post hoc power is really just a p-value wearing a fake mustache. Then statistician and powerlifter Andrew Althouse joins us for the inside story of a years-long scientific feud over post hoc power, where we discuss what happens when being wrong about statistics turns into doubling down, angry letters to the editor, and debates about how science corrects itself.

Statistical topics
Effect size
False negatives
P-values
Post hoc power
Power calculations
Research ethics
Sample size
Scientific controversy
Sensitivity analysis
Statistical power
Statistical significance
Study design
Type II error

Methodologic Morals
“Big questions need studies powerful enough to answer them.”
“Post hoc power is circular reasoning masquerading as new information.”
“Know when to cut your losses. A mistake gets more expensive every time you double down.”
References
Althouse AD, Chow ZR. Comment on “Post-hoc power: If you must, at least try to understand.” Ann Surg. 2019;270:e78-e79.
Althouse AD. Post hoc power: Not empowering, just misleading. J Surg Res. 2021;259:A3-A6.
Altman DG. Statistics and ethics in medical research: III. How large a sample? BMJ. 1980;281:1336-1338.
Bababekov YJ, Hung YC, Hsu YT, et al. Is the power threshold of 0.8 applicable to surgical science?—empowering the underpowered study. J Surg Res. 2019;241:235-239.
Bababekov YJ, Stapleton SM, Mueller JL, et al. A proposal to mitigate the consequences of type 2 error in surgical science. Ann Surg. 2018;267:621-622.
Chang DC, Stapleton SM. Response: The proliferation and misinterpretation of “as safe as” statements in surgical science: A call for professional discourse to search for a solution. J Surg Res. 2021;259:A12-A15.
Freiman JA, Chalmers TC, Smith H Jr, et al. The importance of beta, the type II error and sample size in the design and interpretation of the randomized control trial: Survey of 71 negative trials. N Engl J Med. 1978;299:690-694. 
Hoenig JM, Heisey DM. The abuse of power: The pervasive fallacy of power calculations for data analysis. Am Stat. 2001;55:19-24.
Nuzzo RL. Statistical Power. PM&R. 2016;8(9):907-912. doi:10.1016/j.pmrj.2016.08.004
Nuzzo RL. Post hoc Power. PM&R. 2021;13(4):422-424. doi:10.1002/pmrj.12476

Kristin and Regina’s online courses: 
Demystifying Data: A Modern Approach to Statistical Understanding  
Clinical Trials: Design, Strategy, and Analysis 
Medical Statistics Certificate Program  
Writing in the Sciences 
Epidemiology and Clinical Research Graduate Certificate Program 
Programs that we teach in:
Epidemiology and Clinical Research Graduate Certificate Program 

Find us on:
Kristin -  LinkedIn & Twitter/X
Regina - LinkedIn & ReginaNuzzo.com

(00:00) - Intro

(00:43) - What is Statistical Power?

(03:22) - The UFO Telescope Analogy

(06:19) - The Psychic Ability Experiment

(10:36) - Planning Power Before You Study

(12:46) - History and the Landmark 1978 Paper

(17:42) - Enter Post-Hoc Power: The Problem

(19:26) - Three Problems with Post-Hoc Power

(24:33) - The Surgery Journal Disaster

(29:59) - Guest Interview: Andrew Althouse

(32:30) - The 2018 Paper and the Controversy Begins

(39:54) - Doubling Down and the Cost of Being Right

(53:03) - Rating the Claim

Can weed protect your brain as you get older? We unpack a study linking lifetime cannabis use to larger brain volumes and better cognitive test scores in older adults—and investigate how those findings became headlines urging readers to “pass the pot to grandma.” Along the way, we explore collapsing categorical variables, confounding, clinically versus statistically significant results, multiple testing, and how a press release subtly changed a story. We also meet permanently stoned mice, learn about Carl Sagan’s surprising connection to cannabis research, and discover how a single missing word can change the entire meaning of a study.
Statistical topics
Categorical variables
Confounding
Effect size
False discovery rate
Healthy volunteer bias
Measurement error
Multiple testing
Observational studies
Recoding variables
Science communication
Statistical vs practical significance

Methodologic Morals
“Always follow the chain from data to result to conclusions and pay attention to the words along the way.”
“Huge samples can give tiny differences delusions of grandeur.”

References
Guha A, Fu Z, Calhoun V, Hutchison KE. Lifetime Cannabis Use Is Associated With Brain Volume and Cognitive Function in Middle-Aged and Older Adults. J Stud Alcohol Drugs. 2026;87(4):739-751. doi:10.15288/jsad.25-00346
Press Release: https://news.cuanschutz.edu/news-stories/study-finds-cannabis-usage-in-middle-aged-and-older-adults-associated-with-larger-brain-volume-better-cognitive-function
Bilkei-Gorzo A, Albayram O, Draffehn A, et al. A chronic low dose of Δ9-tetrahydrocannabinol (THC) restores cognitive function in old mice. Nat Med. 2017;23(6):782-787. doi:10.1038/nm.4311
NY Post: https://nypost.com/2026/02/07/health/cannabis-may-benefit-aging-brains-study-finds/
David Grinspoon: https://www.nytimes.com/2020/07/02/science/lester-grinspoon-dead.html?unlocked_article_code=1.7lA.CyS1.neeqGGNgx8g1&smid=url-share

Kristin and Regina’s online courses: 
Demystifying Data: A Modern Approach to Statistical Understanding  
Clinical Trials: Design, Strategy, and Analysis 
Medical Statistics Certificate Program  
Writing in the Sciences 
Epidemiology and Clinical Research Graduate Certificate Program 
Programs that we teach in:
Epidemiology and Clinical Research Graduate Certificate Program 

Find us on:
Kristin -  LinkedIn & Twitter/X
Regina - LinkedIn & ReginaNuzzo.com

(00:00) - Opening: The Brain on Drugs

(00:37) - The Cannabis Headline

(03:38) - The Claim: Brain Benefits for Older Adults

(06:53) - The Mouse Study: Pothead Mice vs. Control Mice

(11:25) - From Mice to Humans: UK Biobank

(15:01) - Measurement Problems and Category Collapse

(21:39) - Results: Statistically Significant, Clinically Meaningless

(27:13) - Individual Brain Regions and Cherry-Picking

(32:32) - Cognitive Tests: What Actually Matters

(40:09) - How It Gets Lost in Translation

(45:32) - Rating the Claim

Can the meningitis vaccine do double duty against gonorrhea? We examine a fascinating randomized trial inspired by years of encouraging observational studies—and what happened when those earlier results were finally put to the test. Along the way, we discuss ecological studies, case-control studies, why statisticians distrust phrases like “trend toward significance,” how intention-to-treat differs from per-protocol analysis, and why no amount of statistical adjustment can fully substitute for randomization. Along the way we celebrate outrageously named clinical trials, ask whether “flirting with statistical significance” belongs in a romance novel or a medical journal, and discover that “practice safe statistics” may be the best advice of the episode.

Statistical topics
Case-control studies
Confounding
Ecological studies
Generalizability
Intention-to-treat
Meta-analysis
Null results
Observational studies
P-values
Per-protocol analysis
Randomized clinical trials
Survival analysis

Methodologic Morals
“Don't mistake a p-value of 0.06 for a p-value of 0.04 with bad luck.”
 “Practice safe statistics; randomize whenever possible.”
References
Seib KL, Donovan B, Jin F, et al. Meningococcal B Vaccine to Prevent Neisseria gonorrhoeae Infection. N Engl J Med. 2026; 395: 349-61. 
Petousis-Harris H, Paynter J, Morgan J, et al. Effectiveness of a group B outer membrane vesicle meningococcal vaccine against gonorrhoea in New Zealand: a retrospective case-control study. The Lancet. 2017; 390: 1603–10.
Wang, B., Mohammed, H., Andraweera, P., McMillan, M., Marshall, H., 2024. Vaccine effectiveness and impact of meningococcal vaccines against gonococcal infections: A systematic review and meta-analysis. Journal of Infection. 2024; 89: 106225. 
Molina JM, Bercot B, Assoumou, L, et al. Doxycycline prophylaxis and meningococcal group B vaccine to prevent bacterial sexually transmitted infections in France (ANRS 174 DOXYVAC): a multicentre, open-label, randomised trial with a 2 × 2 factorial design. The Lancet Infectious Diseases. 2024; 24: 1093–1104.
Thng C, Eskandari S, Jin F, et al. Efficacy of the meningococcal vaccine against Neisseria gonorrhoeae: a randomised clinical trial (MenGO). npj Vaccines. 2026.

Still Not Significant | Probable Error: List of funny disguises scientists use to describe p>.05

Kristin and Regina’s online courses: 
Demystifying Data: A Modern Approach to Statistical Understanding  
Clinical Trials: Design, Strategy, and Analysis 
Medical Statistics Certificate Program  
Writing in the Sciences 
Epidemiology and Clinical Research Graduate Certificate Program 
Programs that we teach in:
Epidemiology and Clinical Research Graduate Certificate Program 

Find us on:
Kristin -  LinkedIn & Twitter/X
Regina - LinkedIn & ReginaNuzzo.com

(00:15) - - Introduction

(00:51) - - The Claim: Can a Meningitis Vaccine Prevent Gonorrhea?

(04:10) - - The Biology Behind the Idea

(10:37) - - How Observational Studies Made the Link

(14:44) - - Earlier Clinical Trials: MENGO and DOXYVAC

(17:30) - - Flirting with Significance: A Statistical Rant

(22:40) - - GoGoVax: Study Design and Analysis

(33:31) - - The Shocking Null Result

(35:38) - - Why These Results Don't Generalize

(37:42) - - Rating the Claim

Can you improve your endurance by sitting in a hot tub instead of doing another hard workout? We dig into a study that put elite runners through five weeks of what we affectionately call “endurance hot tubbing” to see whether heat alone could trigger the same cardiovascular adaptations as altitude training. Along the way, we explore crossover study designs, multiple testing, why significant physiological changes don’t necessarily translate into better athletic performance, and how an exploratory regression analysis ended up answering the wrong scientific question. We also discover that serious hot tub research involves surprisingly little relaxation, debate whether bubbles should count as an experimental condition, and learn why distance runners pay good money to sleep in tents with some of the oxygen sucked out.

Statistical topics
best subsets regression
causal inference
crossover studies
exploratory analyses
mediation / mechanisms
model specification
multiple testing
peer review
sample size

(00:15) - Introduction

(00:44) - Can Hot Tubs Replace Exercise?

(01:27) - Crazy Runners and Crazy Training

(05:44) - The Amazon Delivery System Metaphor

(12:12) - From Heat Training to Hot Tubs

(13:33) - The Study: 10 Runners in Hot Water

(22:31) - Results and the VO2 Max Question

(29:26) - Statistical Sleuthing: Best Subsets Regression

(36:50) - The Peer Review Problem

(41:39) - Rating the Claim

Methodologic Morals
“A model can be very good at answering the wrong question.”
 “The label ‘peer reviewed’ does not guarantee ‘carefully reviewed’.”
References
Jenkins EJ, Killick JA, Zerilli O, et al. Long-term passive heat acclimation enhances maximal oxygen consumption via haematological and cardiac adaptation in endurance runners. J Physiol. Published online November 20, 2025. doi:10.1113/JP289874
Stembridge M, Jenkins E. Marathon training: Why hot baths might help you run faster. The Conversation. Published March 16, 2026. doi:10.64628/AB.6vknm9esh

Kristin and Regina’s online courses: 
Demystifying Data: A Modern Approach to Statistical Understanding  
Clinical Trials: Design, Strategy, and Analysis 
Medical Statistics Certificate Program  
Writing in the Sciences 
Epidemiology and Clinical Research Graduate Certificate Program 
Programs that we teach in:
Epidemiology and Clinical Research Graduate Certificate Program 

Find us on:
Kristin -  LinkedIn & Twitter/X
Regina - LinkedIn & ReginaNuzzo.com

(00:15) - Introduction

(00:44) - Can Hot Tubs Replace Exercise?

(01:27) - Crazy Runners and Crazy Training

(05:44) - The Amazon Delivery System Metaphor

(12:12) - From Heat Training to Hot Tubs

(13:33) - The Study: 10 Runners in Hot Water

(22:31) - Results and the VO2 Max Question

(29:26) - Statistical Sleuthing: Best Subsets Regression

(36:50) - The Peer Review Problem

(41:39) - Rating the Claim

Could four one-minute bursts of exercise really improve blood sugar? We try “exercise snacks” ourselves before taking a close look at the clinical trial that inspired headlines. We explain why the study’s main result wasn’t statistically significant, how 34 secondary outcomes complicated the story, and what pre-registration can reveal about a study after it’s published. Along the way, we compare notes on our own exercise-snacking adventures, debate continuous glucose monitors, and ask how much evidence a single study should generate before it becomes health news.

Statistical topics
Crossover design
Multiple testing
Pre-registration
Primary vs secondary outcomes
Randomized controlled trial
Research transparency
Methodologic Morals
“It's good relationship advice to be transparent. It's also good research advice.”
“If the primary outcome is not significant, say it up top.”
References
Babir FJ, Marcotte-Chénard A, Sandilands RE, et al. Exercise snacks performed in real-world settings reduce postprandial hyperglycaemia and glycaemic variability in individuals living with type 2 diabetes: a randomised crossover study. Diabetologia. 2026;69(8):2200-2211. doi:10.1007/s00125-026-06741-2
https://www.washingtonpost.com/wellness/2026/05/28/4-minutes-exercise-day-could-help-control-blood-sugar/
clinicaltrials.gov pre-registration with changes: https://clinicaltrials.gov/study/NCT06382246?term=NCT06382246&rank=1&tab=history&a=1&b=2#version-content-panel

Kristin and Regina’s online courses: 
Demystifying Data: A Modern Approach to Statistical Understanding  
Clinical Trials: Design, Strategy, and Analysis 
Medical Statistics Certificate Program  
Writing in the Sciences 
Epidemiology and Clinical Research Graduate Certificate Program 
Programs that we teach in:
Epidemiology and Clinical Research Graduate Certificate Program 

Find us on:
Kristin -  LinkedIn & Twitter/X
Regina - LinkedIn & ReginaNuzzo.com

(00:00) - Intro

(04:06) - The claim: four minutes a day

(07:05) - Our own N of 1 experiments

(13:24) - The study

(21:54) - Primary outcome: complete miss

(25:16) - Secondary outcomes to the rescue?

(35:02) - Statistical sleuthing and transparency

(44:23) - Rating the claim