It Almost Worked: Four Ways a Multi-Omics Pipeline Nearly Reported a False Success

I'm building a pipeline that filters biomarker candidates out of multi-omics data. Several statistical and machine-learning methods run separately, and only genes that more than one of them point to become candidates. Those candidates then get tested against confounders — whether the signal is really just a difference in tissue composition between patient groups, for example. The last gate is replication: does the same conclusion hold in a separate group of patients?

I ran it on renal cell carcinoma data, splitting patients by whether they carried a particular driver mutation. Candidates came out. They survived the composition test. But the verdict came back PARTIAL, because there was only one cohort and nothing had been replicated.

That's where this post starts. One more cohort and the verdict becomes SUPPORTED. And a second cohort was already sitting right there.

This is a personal research project. Everything below runs on open-source software, and every dataset is public. I've listed the tools and the papers behind them at the end so any of this can be checked rather than taken on faith.

---

The stack

Nothing here is exotic, and that's deliberate. The interesting part of the system is not which methods it runs but what it refuses to conclude from them.

Six evidence sources contribute to the consensus. Differential expression uses limma's moderated t statistics [1]. Latent structure comes from MOFA+ [2] and DIABLO, the multi-block discriminant method in mixOmics [3,4]. Feature interaction is a random forest with Gini importance [5]. Distributed representation is a small PyTorch MLP [6] scored by out-of-fold permutation importance [7]. Network evidence comes from STRING hub connectivity [8]. A pathway impact analysis via ROntoTools [9] is also configured, though in the run described here it contributed nothing that survived multiple-testing correction. Co-expression modules from WGCNA [10] are in the configuration too, and contributed zero of the top twenty — which is its own story, and one I'll come back to at the end.

One further source lives in the same repository but is not part of this run. MOGONET, a graph-convolutional method for multi-omics patient classification and biomarker identification [12], is vendored here the same way the other third-party implementations are, and I evaluated it as an additional evidence source. The bar for keeping it was written down before the results came back: it had to contribute genes the others missed, hold a stable cross-validated AUC across seeds, and justify its cost. When I evaluated it, it contributed exactly one unique gene, its AUC swung between roughly 0.63 and 0.80 depending on seed, and it took about forty minutes per job. It missed the bar, so it moved to opt-in rather than being deleted — the code and its checksum test stay, and one flag turns it back on. That decision took no argument at all, because the bar existed before the numbers did. The co-expression method I mentioned above is the harder case precisely because its bar didn't.

This is a negative result on one dataset, not a verdict on the method.

Multiple testing is controlled with Benjamini–Hochberg [11].

Data access is entirely through public APIs and packages. Proteogenomic data comes through the cptac Python package [13], which wraps the CPTAC clear cell renal cell carcinoma proteogenomic dataset [14]. The independent cohort comes from UCSC Xena [15] for the expression and copy-number matrices, and the cBioPortal API [16,17] for mutation labels against the TCGA kidney renal clear cell carcinoma study [18].

Execution is a Python orchestrator that fans jobs out to worker processes on a local cluster. Each method runs three seeds. Methods that score a single omics matrix are dispatched once per block, while the multi-block methods take the blocks together, so the job count is not a simple product of methods × seeds × cohorts. The two-cohort run described here came to sixty jobs. R-based methods are invoked as subprocesses with a wall-clock limit — currently three hours, raised from one hour after real runs hit the ceiling.

Two rules about the runner matter for what follows.

No fallbacks. If a method fails, the job fails and is recorded as failed. It is never silently replaced by a similar method. The moment you allow substitution, one source of evidence gets counted twice under two names, and the consensus rule that the whole pipeline rests on becomes meaningless.

Third-party implementations are vendored unmodified and checksum-verified. When the pipeline claims it ran a published method, a test compares the vendored source against a recorded sha256. If someone patched it locally, the claim of "we ran the original" fails loudly rather than quietly becoming false.

---

Writing the stopping rule before the download

The data package I was using carried a few other source values for the same cancer type. One of them was distributed as a confirmatory study, and it listed 110 cases — about the same size as my discovery cohort. It looked like exactly the independent cohort I needed.

Before downloading it, I wrote down one line.

If the patient-ID overlap exceeds 50%, stop immediately.

I wrote it first because a rule decided after seeing the result is not a rule. Suppose I had pulled the data, run the whole analysis, and only then found a 60% overlap. Someone — me — would have said "but the quantification pipelines are different, so it's still partly independent." I know that sentence would have arrived, because it always does. This is the same reasoning behind preregistration in the experimental sciences [19], and the same failure mode catalogued as researcher degrees of freedom [20]: the flexibility isn't in the analysis, it's in the decision about what counts as success, made after you can see which answer you'd prefer.

Then I compared the patient IDs.

discovery tumors      110
confirmatory tumors   110
intersection          110      ← ratio 1.0

Every single one was the same person.

Looking further, the 185 samples on the confirmatory side weren't more patients. They were the same 110 patients plus 75 matched normal tissues. Against another source the sample index was identical too.

The cause was mundane. That source field did not separate cohorts. It separated the centers that did the analysis. The same patient specimens, quantified by different labs through different pipelines. I had been reading it as different patient populations.

---

Why this was the worst possible failure

What happens if you count the same patients as two cohorts? Everything replicates perfectly.

The numbers wobble a little because the quantification differs — that difference is a batch effect [21], not independent evidence — but the biological signal is the same signal from the same people. The replication rate would have looked beautiful. The verdict would have flipped cleanly to SUPPORTED, and I would have written that down as a result.

This has a name. Hurlbert called it pseudoreplication in 1984: treating measurements that are not independent as though they were, and thereby inflating your evidence without adding any [22]. His examples were ecological field experiments. The mechanism is identical here. Two quantifications of one patient are two measurements, not two patients.

And it gets caught later. Anyone who knows the data source only has to compare case IDs. There are tools built specifically to detect this kind of sample identity overlap [23]. What collapses at that point isn't one conclusion. It's every conclusion the pipeline has ever produced.

The replication guard cannot catch this. The guard asks whether genes reappear, not whether patients are the same people. It was never built to ask the second question.

What stopped me wasn't clever code. It was a number written down before the download.

---

Then I checked the guard, and the guard was broken too

Once something surprises you, you start doubting the thing next to it. So I opened the replication guard.

It worked like this: for each method, collect which cohorts it passed in, take the union, and if that union has two or more cohorts, conclude this isn't a single-cohort result.

The hole should be visible.

Say method A points to gene X in cohort 1, and method B points to gene Y in cohort 2. The union of passing cohorts is {1, 2}. Size two, so the guard opens. But the number of genes that replicated across cohorts is zero. Nothing was reproduced, and the replication gate passed anyway.

I built a synthetic case to make it happen on purpose. Every consensus gene appeared in exactly one cohort, and the verdict came back SUPPORTED.

The fix is not sophisticated. The unit of the claim is a gene, so ask the question per gene.

# Gene-level replication guard. The single-cohort guard above can be bypassed:
# cohorts_supported is the UNION of per-method passing cohorts, so method A hitting
# only GENE_X in C1 and method B hitting only GENE_Y in C2 yields {C1, C2} and opens
# the guard — while the number of replicated genes is zero. That is exactly the
# pseudo-replication we meant to block. The unit of the claim is the gene, so we ask
# per gene. Different methods picking the same gene in different cohorts still counts
# as replication (orthogonal evidence is stronger, not weaker).
replicated_feats = [f for f in consensus_feats if (f.get("n_cohorts") or 0) >= 2]

(Comment translated from the original source.)

What matters is that this was fixed before the second cohort went in. Had the order been reversed, I would have used that pass as evidence, and I would have had no reason to look at the guard at all. Nobody audits a test they just passed.

---

The real second cohort

I ended up pulling a different cohort of the same cancer from a public database. It uses an entirely different patient ID scheme, and I confirmed the intersection was zero before running anything. This time the check came before the assumption.

The numbers, stated precisely: 525 patients had both omics blocks, 402 had mutation calls, and 398 satisfied both and were actually usable as labeled samples. People write "cohort n = 398," but that isn't the size of the cohort. It's the number of samples the analysis could use. Skip that distinction and you will eventually misread your own figure.

One structural limitation is worth stating plainly, because it doesn't go away. The two cohorts share exactly one data layer. One has proteomics and transcriptomics; the other has transcriptomics and copy number. So what replicates is a gene-level conclusion, not the multi-omics structure itself. That's a property of what's publicly available, not something a better analysis fixes.

I ran it. SUPPORTED. Nine features replicated.

---

The third place I had to stop

I would have liked to end there, but review caught one more thing.

Of the nine replicated features, exactly one had been a candidate before. The other eight were new — genes that only surfaced once both cohorts were analyzed together. Most of the original candidates, including the gene that defines the patient grouping in the first place, did not reappear in the second cohort.

"Nine replicated" is true. And that true statement hides the number that actually matters: of the candidates the discovery cohort had proposed, exactly one came back.

I treated this as an instrumentation problem and changed the instrument: report the denominator alongside the count, compute recovery against the prior candidate list separately, and write the warning into the report itself rather than trusting the reader.

As for the eight new ones: they passed the composition test. But they came out of a single analysis that looked at both cohorts together. They were not proposed in one and confirmed in the other. When discovery and confirmation happen in the same run, that isn't replication, it's agreement. Until a third cohort says otherwise, they're a hypothesis, not a result.

---

A fourth one, smaller, about counting evidence

There's a related trap in the consensus rule itself, and it's worth describing because it's the reason the composition test exists at all.

The top of the consensus list was full of endothelial markers — genes expressed in blood vessels. Eight of them. Eight separate genes, agreed on by multiple independent methods, is exactly what a consensus rule is supposed to reward.

They were not eight findings. They were one signal counted eight times. Patients in the two groups differ in how much vascular tissue their tumor samples contain, and every one of those eight genes is reading that same difference. Correct for per-patient composition and they collapse together.

This is a known hazard in bulk tissue profiling, and deconvolution methods exist precisely because cell-type proportions dominate bulk expression [24,25]. What's specific to a consensus pipeline is the interaction: agreement across methods gives you no protection at all when the methods share an input matrix and a confounder sits in that matrix. Algorithms with different objectives will happily agree on the same artifact. Not unanimously, either — in this run no gene was picked by every source, and the top twenty were backed by three to five of them. Partial agreement is what a consensus rule actually sees, and partial agreement is exactly what one shared confounder produces.

So the composition test runs after consensus, not before, and it excludes any marker that overlaps the candidate being tested — otherwise you'd be correcting a gene against an axis that contains the gene itself, and the attenuation would be arithmetic rather than evidence.

Worth being precise about what that test does and does not establish. It answers "is this explained by other vascular signal." It does not establish independence from composition in general, because the markers left in the axis still co-express with the ones removed.

---

On reproducibility, and one loose end

A few details that only matter if you're trying to reproduce something like this.

Every method runs three seeds, and seeds are recorded per job. A run reports how many jobs succeeded and how many failed, and failures are listed individually with method, seed, and reason.

That last part earned its keep. One run failed three of sixty jobs, all the same method, all wall-clock timeouts at the one-hour limit. I raised the limit to three hours and re-ran the whole thing. Sixty of sixty passed, and four of that method's twelve runs took longer than the old limit — the longest a little over 85 minutes. So the limit had genuinely been too low, and it wasn't masking some other defect.

The conclusions were identical: same verdict, same replicated set, same ordering in the top twenty, same per-source contribution. Which is the answer I wanted, but note that I only know it because I re-ran rather than argued.

Writing this post turned up a third thing I hadn't noticed. The network-evidence step calls its public API without pinning a version, so the endpoint resolves to whatever release is currently live. Everything else here is version-locked — packages, vendored source, checksums — and then one step quietly depends on a database that updates on someone else's schedule. Nothing in this run leans on it heavily, but "we ran the same pipeline" stops being true the moment that database ships a new release, and the code would give no sign. It's now on the list to pin.

Now the loose end. That method — the co-expression one — contributed zero genes to the top twenty both before and after the fix, while being by a wide margin the most expensive thing in the run. Before the re-run I couldn't tell whether its zero contribution was caused by the timeouts. Now I can: it isn't. It simply doesn't put anything into the consensus on this dataset.

The tempting move is to drop it. I haven't, and the reason is the theme of this whole post: that decision would be made after seeing the result, on a single dataset, which is exactly the kind of reasoning the rest of the pipeline exists to prevent. Co-expression methods find modules, and a method that reports modules may simply not surface in a ranking built around single genes. So the criterion goes in the file first — if it contributes nothing on the next dataset either, it becomes opt-in — and the decision waits for the data.

---

What these have in common

All of these failures wore the face of success.

Counting the same patients twice looks like perfect replication. A union-based guard opening looks like a pass. "Nine replicated" looks like a good number. Eight vascular genes agreeing looks like overwhelming consensus. In every case the pipeline stays alive and shows a green light.

Real failures are loud. Jobs die, logs pile up, nothing comes out the other end. They're easy to notice. Fake successes are quiet. Results appear, the numbers are pretty, and you move on to the next step. The published record suggests this is not a personal failing — irreproducibility in preclinical work is well documented [26,27], and the failures are rarely dramatic ones.

So when I write a check now, the first question isn't "does this catch failure." It's "does this catch a failure that looks like success." The first question tends to answer itself. The second one is the work.

One more thing. Two of the decisions above were settled without argument because the criterion had been written down first. Without that 50% overlap rule I would almost certainly have downloaded the data. Once you've downloaded it there's sunk cost, and sunk cost makes criteria flexible. Writing the rule in advance is a declaration that you don't trust your future self, and in my case that turned out to be correct.

---

References

Methods

1. Ritchie ME, Phipson B, Wu D, Hu Y, Law CW, Shi W, Smyth GK. limma powers differential expression analyses for RNA-sequencing and microarray studies. Nucleic Acids Research 2015;43(7):e47.

2. Argelaguet R, Arnol D, Bredikhin D, Deloro Y, Velten B, Marioni JC, Stegle O. MOFA+: a statistical framework for comprehensive integration of multi-modal single-cell data. Genome Biology 2020;21:111.

3. Rohart F, Gautier B, Singh A, Lê Cao K-A. mixOmics: An R package for 'omics feature selection and multiple data integration. PLoS Computational Biology 2017;13(11):e1005752.

4. Singh A, Shannon CP, Gautier B, Rohart F, Vacher M, Tebbutt SJ, Lê Cao K-A. DIABLO: an integrative approach for identifying key molecular drivers from multi-omics assays. Bioinformatics 2019;35(17):3055–3062.

5. Breiman L. Random Forests. Machine Learning 2001;45(1):5–32.

6. Paszke A, Gross S, Massa F, et al. PyTorch: An Imperative Style, High-Performance Deep Learning Library. Advances in Neural Information Processing Systems 32 (NeurIPS) 2019, pp. 8024–8035.

7. Altmann A, Toloşi L, Sander O, Lengauer T. Permutation importance: a corrected feature importance measure. Bioinformatics 2010;26(10):1340–1347.

8. Szklarczyk D, Nastou K, Koutrouli M, et al. The STRING database in 2025: protein networks with directionality of regulation. Nucleic Acids Research 2025;53(D1):D730–D737.

9. Tarca AL, Draghici S, Khatri P, et al. A novel signaling pathway impact analysis. Bioinformatics 2009;25(1):75–82.

10. Langfelder P, Horvath S. WGCNA: an R package for weighted correlation network analysis. BMC Bioinformatics 2008;9:559.

11. Benjamini Y, Hochberg Y. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society, Series B 1995;57(1):289–300.

12. Wang T, Shao W, Huang Z, Tang H, Zhang J, Ding Z, Huang K. MOGONET integrates multi-omics data using graph convolutional networks allowing patient classification and biomarker identification. Nature Communications 2021;12:3445.

Data sources

13. Lindgren CM, Adams DW, Kimball B, et al. Simplified and Unified Access to Cancer Proteogenomic Data. Journal of Proteome Research 2021;20(4):1902–1910.
14. Clark DJ, Dhanasekaran SM, Petralia F, et al. Integrated Proteogenomic Characterization of Clear Cell Renal Cell Carcinoma. Cell 2019;179(4):964–983.
15. Goldman MJ, Craft B, Hastie M, et al. Visualizing and interpreting cancer genomics data via the Xena platform. Nature Biotechnology 2020;38:675–678.
16. Cerami E, Gao J, Dogrusoz U, et al. The cBio Cancer Genomics Portal: An Open Platform for Exploring Multidimensional Cancer Genomics Data. Cancer Discovery 2012;2(5):401–404.
17. Gao J, Aksoy BA, Dogrusoz U, et al. Integrative analysis of complex cancer genomics and clinical profiles using the cBioPortal. Science Signaling 2013;6(269):pl1.
18. The Cancer Genome Atlas Research Network. Comprehensive molecular characterization of clear cell renal cell carcinoma. Nature 2013;499(7456):43–49.

On the failure modes described here

19. Nosek BA, Ebersole CR, DeHaven AC, Mellor DT. The preregistration revolution. PNAS 2018;115(11):2600–2606.
20. Simmons JP, Nelson LD, Simonsohn U. False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant. Psychological Science 2011;22(11):1359–1366.
21. Leek JT, Scharpf RB, Bravo HC, et al. Tackling the widespread and critical impact of batch effects in high-throughput data. Nature Reviews Genetics 2010;11(10):733–739.
22. Hurlbert SH. Pseudoreplication and the Design of Ecological Field Experiments. Ecological Monographs 1984;54(2):187–211.
23. Lee S, Lee S, Ouellette S, Park W-Y, Lee EA, Park PJ. NGSCheckMate: software for validating sample identity in next-generation sequencing studies within and across data types. Nucleic Acids Research 2017;45(11):e103.
24. Newman AM, Liu CL, Green MR, et al. Robust enumeration of cell subsets from tissue expression profiles. Nature Methods 2015;12:453–457.
25. Avila Cobos F, Alquicira-Hernandez J, Powell JE, Mestdagh P, De Preter K. Benchmarking of cell type deconvolution pipelines for transcriptomics data. Nature Communications 2020;11:5650.
26. Begley CG, Ellis LM. Drug development: Raise standards for preclinical cancer research. Nature 2012;483(7391):531–533.
27. Ioannidis JPA. Why Most Published Research Findings Are False. PLoS Medicine 2005;2(8):e124.

---

Related

• A MOGONET-style multi-omics biomarker pipeline — an earlier write-up in the same line of work, on synthetic data
• Multi-omics data integration: a practical guide
• Biomarker discovery: a practical guide for researchers
• limma vs DEqMS for small-n proteomics
• PPI network hub analysis — the STRING side

---

This is a record of work in progress. The figures reflect the current state of the runs and may change as criteria or data change. All software named above is open source and was used here for personal research; all datasets are publicly available through the sources cited. The specific candidate genes and their biological interpretation are a separate discussion.