Three Samples per Group: What We Learned Switching from the t-Test to limma

Omics experiments are often run with three samples per group (3 vs 3). This post sets out what statistics can and cannot do under that condition, and what we deliberately did not change when we switched our default test.

Everything below comes from published literature and a simulation on synthetic data whose script and output are public. No client data is used. (한국어판: 군마다 시료가 셋뿐일 때)

Three weighings

Suppose you step on a scale three times and get 60.1, 60.3 and 59.9 kg. That looks stable. But three readings cannot tell you whether this scale really wobbles that little, or whether those three just happened to agree. The next three might read 58, 61 and 63.

A t-test divides "the difference between group means" by "the wobble within groups". With three samples per group, you are dividing by a wobble you barely know. A proteomics statistics paper puts it this way [1]:

"proteins exhibiting a large fold change are often declared non-significant because of a large sample variance, while at the same time small observed fold changes might be declared statistically significant, because of a small sample variance."

Big changes get missed, and small changes get flagged because their variance happened to come out small. That is the first thing to keep in mind when reading a 3-vs-3 results table.

1. Non-parametric tests cannot reach 0.05 at all

"The samples are few and we don't know the distribution, so let's use a non-parametric test (Mann–Whitney)" is a common suggestion. At 3 vs 3 that road is closed.

There are only C(6,3) = 20 ways to split six samples into two groups of three. Even the cleanest possible separation (one group ranked 1-2-3, the other 4-5-6) gives a two-sided p of 2/20 = 0.10 at best.

Samples per group	Possible splits	Smallest attainable p (two-sided)
3	20	0.100
4	70	0.029
5	252	0.008

Whatever the data, p < 0.05 is impossible. At 3 vs 3, t-type tests are the only option.

2. Power: even large effects are missed half the time

The probability that a 3-vs-3 two-sided t-test (α = 0.05) detects a real difference. Effect size d is the mean difference in units of the within-group standard deviation.

Effect size d	Probability of detection
1 (difference = one SD)	16%
2	46%
3	78%

Even when the means differ by two standard deviations, the test misses more often than not. So "not significant" must never be read as "no difference". To reach 80% power you would need about 17 samples per group at d = 1, and 9 at d = 1.5.

3. Hence the advice: don't select on p alone

The MAQC Consortium examined this problem at scale with microarray data and issued recommendations [2] (emphasis added):

"Reports of DEG list instability in the literature are often a direct consequence of comparing DEG lists derived from a simple t-statistic when the sample size is small and variability in variance estimation is large. Therefore, the practice of using P alone for gene selection should be discouraged."

"Using FC and P together balances reproducibility, specificity, and sensitivity. Control of specificity and sensitivity can be accomplished with a P criterion, while reproducibility is enhanced with an FC criterion."

"FC-ranking should be used in combination with a non-stringent P threshold to select a DEG list that is reproducible, specific, and sensitive, and a joint rule is recommended as a baseline practice."

The widely used "p < 0.05 and at least 2-fold" rule is exactly this kind of joint rule.

But there is a counter-argument

Bolting on a fold-change (FC) cut-off has a real weakness [3]:

"The disadvantage of an ad-hoc FC criterion is that it does not take into account error variance and therefore offers no statistical confidence about future results."

In that paper (a NanoString study, 6 mutant vs 12 control mice), a regular t-test at p < 0.01 combined with a 1.5-fold cut-off selected 49 genes, of which 20 were "in reality most likely not DE" [3]. An FC cut-off reduces false positives, but it does not tell you by how much.

4. A synthetic experiment where we know the answer

In real data you never know which differences are genuine. So we built synthetic data with a known truth:

• 400 metabolites, of which only 40 truly change (2.8-fold, log2 change ±1.5)
• 3 samples per group; the within-group SD differs between metabolites (log-normal, median 0.5 on the log2 scale)
• Averaged over 500 repetitions; Student's t-test vs limma with default eBayes

Rule	Test	Selected	False	False share	True found (of 40)
p < 0.05 only	t-test	46.1	18.0	39%	28.1
	limma	51.2	19.0	37%	32.2
p < 0.05 and ≥ 2-fold	t-test	33.3	5.9	18%	27.5
	limma	42.1	10.8	26%	31.3
Multiple-testing corrected (BH 5%)	t-test	6.9	0.4	6%	6.5
	limma	16.1	0.9	6%	15.1

What the table says

1. Selecting on p alone, nearly 40% of the list is chance. The 18 false hits are exactly what you expect from 360 unchanged metabolites × 0.05 = 18.
2. Adding the 2-fold rule cuts false hits from 18 to 6. The effect described in the MAQC recommendations shows up in synthetic data too.
3. limma finds more true changes (27.5 → 31.3) but also more false ones (5.9 → 10.8). Because it borrows variance information across metabolites, some genuinely noisy metabolites appear less noisy, and those then cross the 2-fold line by chance; that appears to be the mechanism. Combined with the 2-fold rule, limma's false share is higher (18% → 26%).
4. limma more than doubles the number that survive multiple-testing correction (6.9 → 16.1), with the same 6% false share inside that list. At 3 vs 3 a t-test commonly leaves next to nothing after correction; with limma you get a usable corrected list.

5. Why limma survives correction

A t-test estimates each metabolite's variance from that metabolite's six samples alone. limma uses the fact that variances across metabolites tend to fall in a similar range, and lends a little information from all metabolites to each individual estimate (variance moderation, the moderated t). Metabolites whose variance happened to be measured as tiny are less likely to become falsely significant, and the test gains degrees of freedom.

A proteomics paper reports, for a 3-vs-3 cell-line comparison (8,625 proteins, 1% FDR) [4]:

"t test identified a single significant DEP after correction for multiple testing (1% FDR), whereas 21 DEPs were identified using ROTS. DEqMS and Limma generated 201 and 120 DEPs respectively"

The same paper points out a second difference [4]:

"all these methods calculate pooled variances for each protein using all samples in all sample groups, whereas t test use only the samples in the two groups compared."

We measured the same pattern on a known-ratio proteomics benchmark in limma vs DEqMS at small n.

But metabolomics has fewer variables

The DEqMS example had 8,625 proteins. Variance moderation is not automatically a win on smaller panels. In Vaes et al.'s NanoString study of 558 genes, limma's moderated test against a fold-change threshold (TREAT) gave more false discoveries than its plain t-based counterpart at fixed p cut-offs, and overall performance (AUC) showed no difference [3]. The authors attributed this to counts that were not very noisy and to the small panel, and wrote [3]:

"some of the gene-wise variances may be shrunk (or blasted) by too large a factor when applying TREAT on our NanoString data, as the hierarchical Bayesian model is expecting a few outlier variances."

Metabolomics datasets often quantify only a few hundred compounds. So "limma is always better" cannot be claimed. You have to compare on your own data, which is what we did before switching.

6. What we decided

Changed: the group-comparison test, from the t-test to limma (moderated t, using the samples of the two groups being compared). Two reasons:

1. Results that survive multiple-testing correction. "Nothing left after correction" is common at 3 vs 3; with limma we can provide a corrected list alongside.
2. It is where the common tooling is heading. The development version of MetaboAnalystR switched its t-test default (Ttests.Anal) to limma in July 2026 (commit e028f54); the latest tagged release (v4.2.0) still runs a classical t-test.

Not changed: the "p < 0.05 and ≥ 2-fold" rule. As MAQC recommends, the FC cut-off clearly reduces false positives.

Not claimed: "it's more accurate now". As the synthetic experiment shows, combined with the 2-fold rule limma's false share is not lower than the t-test's. What we gained is "finding more" and "a corrected list", not "fewer false positives".

Applies to new projects. Recomputing past results every time a default changes would leave two sets of numbers for the same experiment.

7. Sentences worth including when reporting 3-vs-3 results

• "This is an exploratory analysis with three samples per group. The significant list consists of candidates for follow-up validation."
• "Even with no true differences, about N features would be expected to pass this rule by chance." (Obtainable by shuffling group labels and recomputing.)
• "Multiple-testing corrected (BH) results are shown alongside."
• "Do not interpret non-significant features as 'unchanged' (power is low)."

Closing

In a 3-vs-3 experiment, the biggest statistical decision is not which test to use, but how many samples to collect. Changing the test changes the length of the list; it does not change the fact that three samples cannot measure variability. The MAQC recommendation quoted above continues with the same point: "Sensitivity can also be improved by … increased sample sizes." [2]

So even as we changed the test, we decided to deliver this sentence with every result: "This list is a set of candidates. Some may be chance, and we tell you roughly how many."

Reproduce it

three_vs_three.R prints every number in sections 1, 2 and 4 in about 20 seconds (R 4.3.3, limma 3.58.1). Simulation settings: 400 metabolites (40 true changes, log2 ±1.5, alternating up and down), 3 samples per group, per-metabolite within-group SD from a log-normal distribution with median 0.5 and sdlog 0.6 on the log2 scale, 500 repetitions, seed 1.

References

1. Kammers K, Cole RN, Tiengwe C, Ruczinski I. Detecting Significant Changes in Protein Abundance. EuPA Open Proteomics 2015;7:11–19. doi:10.1016/j.euprot.2015.02.002
2. Shi L, Jones WD, Jensen RV, et al. The balance of reproducibility, sensitivity, and specificity of lists of differentially expressed genes in microarray studies. BMC Bioinformatics 2008;9(Suppl 9):S10. doi:10.1186/1471-2105-9-S9-S10
3. Vaes E, Khan M, Mombaerts P. Statistical analysis of differential gene expression relative to a fold change threshold on NanoString data of mouse odorant receptor genes. BMC Bioinformatics 2014;15:39. doi:10.1186/1471-2105-15-39
4. Zhu Y, Orre LM, Zhou Tran Y, Mermelekas G, Johansson HJ, Malyutina A, Anders S, Lehtiö J. DEqMS: A Method for Accurate Variance Estimation in Differential Protein Expression Analysis. Molecular & Cellular Proteomics 2020;19:1047–1057. doi:10.1074/mcp.TIR119.001646

All quotations were checked word for word against the full text of the cited papers; bold inside quotations is added for emphasis.