Calculate p-value from z-score or t-test to determine statistical significance.
Enter values to see the result.
Guide: P-Value and Statistical Significance
P-value is the probability of obtaining results at least as extreme as the observed ones, assuming the null hypothesis is true. In other words, it measures how much the data contradicts the hypothesis that there is no effect or difference.
Interpreting p-values: A low p-value (typically below 0.05) means the result is unlikely if the null hypothesis were true. This leads to rejecting the null hypothesis and accepting the alternative. A high p-value (above 0.05) means we do not have enough evidence to reject the null hypothesis.
Z-test vs t-test: Z-test is used when the population standard deviation is known or when we have a large sample (n greater than 30). T-test is used when the population standard deviation is unknown and we are working with a smaller sample. T-test is more conservative with small samples.
Statistical vs. practical significance: Statistical significance does not always mean practical significance. With large samples, even small effects can be statistically significant. For example, a 0.01 difference in average price may be statistically significant with a million observations, but it would not have practical importance.
Limitations: P-value does not measure the size of the effect or the probability that the null hypothesis is true. You also cannot compare p-values across different experiments without context. Always report effect size and confidence intervals along with p-values.
p < 0.05 does not mean a 5% chance of being wrong
The p-value answers a narrow question: if the null hypothesis were true, how often would data this extreme appear? It says nothing about how often your significant findings are actually false. That second number depends on how plausible your hypotheses were to begin with, and it is usually far worse than 5%.
How it works
- Converts a z-score or t-statistic into a p-value for one-tailed or two-tailed tests.
- Compares the result against a chosen significance threshold.
- Reports the statistic, not a verdict — the interpretation is where the errors live.
p = P(data at least this extreme | null hypothesis true) it is NOT P(null hypothesis true | data) false discovery rate = false positives ÷ all significant results false positives = (1 − prior) × alpha true positives = prior × power
Worked example
A thousand hypotheses tested at α = 0.05 with 80% power, where one in ten is genuinely true.
- 100 true hypotheses × 80% power = 80 true positives
- 900 false hypotheses × 5% alpha = 45 false positives
- significant results: 80 + 45 = 125
- false discovery rate: 45 ÷ 125 = 36%
More than a third of the significant findings are wrong, despite every one of them clearing p < 0.05. If only one hypothesis in a hundred were true, 86% of the significant results would be false.
Reading the result
- The prior matters enormously and is rarely stated. Testing plausible, theory-driven hypotheses gives a false discovery rate near 6%; trawling through implausible ones at a 1% prior pushes it to 86% with the identical threshold.
- Low power compounds it. At a 10% prior, dropping power from 80% to 20% takes the false discovery rate from 36% to 69% — an underpowered study is not merely likely to miss real effects, it makes its own positives untrustworthy.
- Testing many outcomes multiplies the problem. Twenty independent tests at α = 0.05 give roughly a 64% chance of at least one false positive, which is why correction methods exist and why undisclosed multiple testing is so damaging.
- Statistical significance is not effect size. With a large enough sample, a trivial difference reaches p < 0.001 while remaining irrelevant in practice — always read the confidence interval alongside the p-value.
Common questions
- So what does p = 0.03 actually mean?
- That if the null hypothesis were true, data this extreme or more so would occur 3% of the time. It is a statement about the data given a hypothesis, not about the hypothesis given the data — and those are not interchangeable.
- Should I use a stricter threshold?
- It helps but does not solve it. Moving to α = 0.005 cuts false positives fivefold, at the cost of missing more real effects. The more effective levers are testing plausible hypotheses, running adequately powered studies, and pre-registering what you intend to test.