Post Hoc Power: What It Is & Why It Matters (Explained)

Understanding Post Hoc Power: A Comprehensive Guide

This article aims to clarify the concept of post hoc power, its limitations, and why understanding its role in statistical analysis is crucial. We’ll break down the key elements, addressing common misconceptions and providing practical context.

Defining Post Hoc Power

Post hoc power, also known as observed power, is the statistical power calculated after a study has been conducted and results are known. It estimates the probability of finding a statistically significant effect given the sample size, the effect size observed in the study, and the chosen significance level (alpha).

Key Differences from A Priori Power

It’s important to distinguish post hoc power from a priori power (prospective power).

• A priori power: Calculated before a study, used to determine the necessary sample size to detect a specific effect size with a desired level of power (typically 80%). Its main goal is to design a study with a sufficient chance of finding a statistically significant effect if it truly exists.

• Post hoc power: Calculated after a study, using the observed effect size to estimate the power the study had to detect that effect.

Formula for Post Hoc Power Calculation

While the specific formula depends on the statistical test used (t-test, ANOVA, etc.), the general form remains consistent:

Post Hoc Power = Function (Observed Effect Size, Sample Size, Significance Level)

Where:

• Observed Effect Size: The magnitude of the effect found in the study (e.g., Cohen’s d for t-tests).
• Sample Size: The number of participants or observations in the study.
• Significance Level (alpha): The probability of rejecting the null hypothesis when it is actually true (typically set at 0.05).

Why Post Hoc Power Matters (and Its Limitations)

The primary reason for discussing post hoc power is to understand its misuse and inherent limitations. While it may seem intuitive to assess the "power" of a non-significant result, doing so can be misleading.

Common Misinterpretations

• Interpreting a high post hoc power for a non-significant result as evidence supporting the null hypothesis: A high post hoc power simply means that if the effect size observed in the study is the true effect size, the study had a good chance of detecting it. However, a non-significant result can occur for various reasons, including a true effect size close to zero, even with high power.

• Interpreting a low post hoc power for a non-significant result as conclusive evidence that there is no effect: This is perhaps the most common error. A low post hoc power only suggests that the study might have been underpowered to detect the observed effect size. It does not rule out the possibility of a real effect that is smaller than the study was designed to detect.

The Dependency on the p-value

Post hoc power is fundamentally tied to the p-value. It’s essentially a transformation of the p-value, providing the same information in a different form. If the p-value is already known, calculating post hoc power adds no new insight.

• Relationship Explained: A smaller p-value will result in a higher post hoc power. Conversely, a larger p-value will result in a lower post hoc power. Because of this direct relationship, post hoc power cannot provide information independent of the p-value.

The Problem of "Observed Effect Size"

The effect size used in the post hoc power calculation is observed within the specific sample. This observed effect size is subject to sampling error. If the true effect size is different from the observed effect size, the post hoc power calculation will be inaccurate. Relying solely on the observed effect size from a single study to estimate power is statistically unsound.

Alternatives to Post Hoc Power

Instead of focusing on post hoc power, consider the following approaches:

• Confidence Intervals: Report confidence intervals for the effect size. This provides a range of plausible values for the true effect size, offering a more informative perspective than a single point estimate like the observed effect size used in post hoc power calculations.

• Equivalence Testing: If the goal is to demonstrate the absence of a practically meaningful effect, use equivalence testing. This involves defining a region of effect sizes that are considered practically equivalent to zero and testing whether the observed effect size falls within that region.

• Bayesian Analysis: Employ Bayesian methods to update prior beliefs about the effect size in light of the observed data. This approach provides a more nuanced assessment of the evidence for and against the null hypothesis.

Illustrative Examples

To further clarify the concept, consider these examples:

Example 1: Clinical Trial for a New Drug

A clinical trial is conducted to test the efficacy of a new drug. The p-value for the primary outcome is 0.06, and the calculated post hoc power is 55%.

• Incorrect Interpretation: "The study had low power (55%), so we can conclude the drug is ineffective."

• Correct Interpretation: "The p-value is above the conventional significance level of 0.05, suggesting the observed effect could be due to chance. The post hoc power indicates that the study had a moderate chance of detecting the observed effect size, assuming the observed effect size accurately represents the true population effect. Further research with a larger sample size might be warranted."

Example 2: Psychology Experiment on Memory

A researcher conducts an experiment on memory recall. The p-value for the difference between two groups is 0.01, and the post hoc power is 95%.

• Incorrect Interpretation: "The high post hoc power (95%) confirms the effect is real and meaningful."

• Correct Interpretation: "The p-value is statistically significant, indicating a strong evidence against the null hypothesis of no difference. The post hoc power of 95% suggests that, given the observed effect size, the study was well-powered to detect it. However, it is still essential to consider the magnitude and practical significance of the effect, not solely rely on the high power value."

So, there you have it – a (hopefully!) clear look at post hoc power. It’s a tricky concept, but understanding its limitations is key to good research. Happy analyzing!