The Number That Survived a Full Day of Review: What Two Agents Couldn't See

I run a dry-lab pipeline with two AI agents attached to it, and they have deliberately different access.

One sits next to the cluster. It launches jobs, watches them finish, and can open every artifact a run produces — the JSON reports, the per-job logs, the intermediate matrices.

The other sits next to the repository. It can read every line of code and every commit message, search the literature, and check claims against sources. It cannot reach the cluster at all. The output directory is in .gitignore, so run artifacts never arrive on its side.

I am the channel between them. Or I was, until yesterday.

That arrangement caught six errors in a working day, in both directions. It also let one wrong number sit untouched through the entire day while both agents reviewed the document it was in. The difference between those two outcomes is the whole point of this post.

---

What the asymmetry caught

The cluster-side agent, against mine. A public-facing page I had drafted described the pipeline as combining eight evidence sources. The report says otherwise: only six contributed anything to the top of the consensus list. One of the two remaining had recorded, in its own output, that it contributed nothing at all. And no gene in the top twenty had been picked by every source — the top twenty were backed by three to five each.

That correction matters more than a word count. "Eight independent methods agree" is the sentence the whole system rests on. Anyone who received a report could have refuted it in a minute by counting the per-gene source lists. It would have been refuted from outside, publicly, rather than internally.

The cluster-side agent again, against my fix. I changed eight to six and listed the six. I got two of them wrong: I included the co-expression method, which had contributed zero of the top twenty, and left out the network method, which had contributed twelve. The two errors cancelled. The count was still six, so every check I ran on the number passed.

That one bothers me the most. I verified the arithmetic and stopped there, because the arithmetic was the thing I had just been corrected on.

My side, against the cluster's. The cluster agent reported that one row of a results table had the wrong condition label. I read the eight-row dump it had sent to support that claim and found a second row with the same defect, in the other cohort. It had caught the instance it was looking at and not the class.

My side again, on rounding. Two p-values had been transcribed as .078 and .228. The stored values were 0.07737 and 0.22698, which round to .077 and .227. Both had been reviewed and passed as matching. Small, but this is a document whose entire argument is that you should be able to check its numbers.

Me, wrong, refuted by data. I noticed that a gene eliminated by the composition test also appeared to be a member of the marker set used to build the correction axis. If true, its attenuation would be arithmetic rather than evidence, and the elimination would be circular. I flagged it and removed it from the argument. The cluster agent opened the run artifact: the tool had already excluded every marker overlapping the tested candidates — ten of thirty-three, including that gene and its alias — and built the axis from the remaining twenty-three. My concern was reasonable and wrong. It went back into the argument.

The cluster agent, wrong, self-retracted. It had flagged a gene as suspiciously recurrent across cancer types, which would have suggested the pipeline emits it regardless of biology. Across all nine combinations of the sweep it appeared in one. It withdrew its own concern in the next report without being asked.

Six errors, both directions, one day. The arrangement works.

---

And then the one that didn't get caught

A summary document written by one agent contained a p-value: 0.00087, described as the smallest pathway-level p-value in a run, nominally significant but not surviving multiple-testing correction.

I read that document. I used the number. It went into a page intended for outside readers. Both agents reviewed that page — one for factual accuracy against the run record, one for internal consistency — and it survived both passes and a full day.

The actual value in the run artifact is 0.00028, with a corrected value of 0.083.

The number 0.00087 does not appear in any output file. Not as a different statistic, not in another run. It has no source. It was written into a summary at some point, and from then on it was simply what everyone knew.

It was caught the moment someone opened the JSON.

---

Why one class of error survived and the other didn't

Line the seven up and they separate cleanly.

The six that were caught were all cases where the two agents were looking at different material. One had the artifact; the other had the code, the commit history, or the literature. Each could see something the other structurally could not.

The one that survived was the case where both were reading the same paragraph. Neither had independent access to the underlying value, so neither could disagree with it. Two reviewers of the same summary are not two reviewers. They are one reviewer, consulted twice.

This is not a new result, it is just a new place to find it. Knight and Leveson tested the assumption behind N-version programming — that independently written implementations of the same specification fail independently — by having twenty-seven versions written from one specification and running a million tests against them. Versions failed together far more often than independence predicts, because the specification, and the parts of the problem that are hard, were common to all of them [1]. Redundancy buys you nothing against a fault that all copies share.

And there is an uncomfortable symmetry with the technical content of the pipeline itself. The first post in this series described eight vascular genes that looked like eight independent findings agreed on by multiple methods, and turned out to be one tissue-composition signal counted eight times, because every method was reading the same input matrix. The methods were different; the input was shared, so the agreement was worth about as much as one measurement.

Same structure, one layer up. Two agents were different; the prose summary was shared, so the review was worth about as much as one review.

There is a well-studied human version of this too. Reviewers given an automated aid tend to check less, not more, and errors of omission rise when the aid is usually right [2,3]. Neither agent here was being lazy. But when a number arrives already integrated into a coherent narrative, the cost of doubting it is much higher than the cost of accepting it, and nothing in the workflow was paying that cost.

---

The fix is boring, which is the point

The channel had to stop carrying prose.

The pipeline now writes its own state as machine-readable records, committed to the repository where the code-side agent can read them directly: what is running right now, an index of what has finished, and one record per run. No agent and no human retypes a number on the way.

The required fields were not chosen abstractly. They are a list of the places where this actually went wrong: per-source contribution counts, the exact list of markers excluded from a correction axis, the raw and corrected pathway p-values, the label split per cohort, and every failed job with its method, seed and reason. Each of those corresponds to a specific error above.

Two rules do most of the work.

A missing required field fails the run. It does not become null. Silent absence is how a number goes from "not measured" to "apparently fine" without anyone deciding.

A partial record must declare that it is partial, and say which fields are missing. This one justified itself within hours. The first batch of run records was emitted for a sweep that had started before the state hook existed, so the consensus lists were not available. I ran a check across all of them for whether a particular set of genes reappeared. Every run came back with nothing found — a clean, plausible negative across nine experiments.

The arrays were empty. There was no result to find. A declared partial_reason field is the only reason that took five seconds to notice instead of becoming the next paragraph of a public page. An empty result and a negative result are different things, and the difference is invisible unless something is required to declare it.

None of this is novel engineering. "Avoid manual data manipulation steps" is rule two of ten in a well-known set of guidelines for reproducible computational research, and rule nine of the same ten — "connect textual statements to underlying results" — describes the fix in this post almost exactly [4]. The most detailed public forensic reconstruction of a bioinformatics failure traces a chain of consequential errors that begins with numbers moving between steps by hand [5]. Knowing the rule and having a system that enforces it are different states, and I was in the first one.

---

What still didn't work

Writing a rule down is not the same as the rule taking effect, and I have a fresh example.

Two runs in a sweep had degenerate labels — one had a minority group of twelve out of two hundred and seventy-six, and one had a minority group of zero. The second should not have been possible to run at all, and the pipeline ran it anyway and produced output, which a person then had to label invalid after the fact. So we wrote a threshold: below ten, hard failure; below twenty or below five percent, exploratory only and not usable as evidence; above both, usable.

The point of writing it in the middle of a sweep was to have it on record before the remaining results were visible. That is only half true. By the time it was committed, four of the six runs in that arm had already started, and the sweep did not stop. Of nine combinations, seven were compared against the threshold after their results existed. Two were checked before.

So the honest statement is that the criterion was pre-registered for two runs and post-hoc for seven, and that is what the record says. It happens to agree with the judgments already made, which is mildly reassuring and not evidence of anything. A criterion that arrives after the data is a description, not a test [6,7].

The gate is now in code, and it fails the run before a report exists. The next sweep will be the first one where the rule is actually load-bearing.

---

What I would keep

If you are running more than one agent against the same work, the temptation is to make them symmetric — same context, same tools, same view — because it is simpler and it feels like it should be more capable.

The value here came from the opposite. One could see the cluster and not the literature. The other could see the repository and not the runs. Every error that got caught was caught across that gap, by whichever one could see the thing the other could not. The single error that survived was the one where the gap had been closed by a summary.

Two things follow, and they pull in opposite directions, which is why both need saying.

Preserve the asymmetry. It is not an inconvenience to be engineered away. It is the only reason the review is worth anything.

Never let a number cross the gap as prose. The asymmetry gives you two viewpoints; a summary collapses them back into one. Numbers should travel in a form neither side can retype, and the format should be required to admit what it does not contain.

The failure I would not have predicted is that the second one is harder. Building the state files took an afternoon. Noticing that a fluent, confident, internally consistent paragraph was the weakest link in the system took a day of two agents agreeing with each other.

---

References

1. Knight JC, Leveson NG. An experimental evaluation of the assumption of independence in multiversion programming. IEEE Transactions on Software Engineering 1986;SE-12(1):96–109.
2. Parasuraman R, Riley V. Humans and Automation: Use, Misuse, Disuse, Abuse. Human Factors 1997;39(2):230–253.
3. Skitka LJ, Mosier KL, Burdick M. Does automation bias decision-making? International Journal of Human-Computer Studies 1999;51(5):991–1006.
4. Sandve GK, Nekrutenko A, Taylor J, Hovig E. Ten Simple Rules for Reproducible Computational Research. PLoS Computational Biology 2013;9(10):e1003285.
5. Baggerly KA, Coombes KR. Deriving chemosensitivity from cell lines: Forensic bioinformatics and reproducible research in high-throughput biology. Annals of Applied Statistics 2009;3(4):1309–1334.
6. Nosek BA, Ebersole CR, DeHaven AC, Mellor DT. The preregistration revolution. PNAS 2018;115(11):2600–2606.
7. Simmons JP, Nelson LD, Simonsohn U. False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant. Psychological Science 2011;22(11):1359–1366.

---

Related

• It Almost Worked: Four Ways a Multi-Omics Pipeline Nearly Reported a False Success — the first post in this series
• A MOGONET-style multi-omics biomarker pipeline
• Multi-omics data integration: a practical guide
• Biomarker discovery: a practical guide for researchers

---

Second in a series on building a dry-lab pipeline that filters candidates rather than producing them. The first post described four ways the pipeline nearly reported a false success. Figures here reflect the current state of the runs and may change as criteria or data change.