Why You Must NOT Merge Species FASTA Databases in Cross-Species Proteomics (Shared Peptide Problem)
In cross-species LC-MS/MS proteomics (Porcine ECM vs Mouse Matrigel, human vs mouse cells, host-pathogen, etc.) merging species FASTA into one search database breaks protein quantification. This is the shared-peptide ambiguity problem. Here's why it happens, how to set up species-separated searches in MaxQuant and DIA-NN, and how to handle peptides that genuinely match both species.
The Single Most Common Cross-Species Proteomics Mistake
You have two sets of LC-MS/MS samples: Porcine esophageal ECM (n=5) and Mouse Matrigel (n=3). You want to compare protein composition. The instinct — open MaxQuant, load Sus scrofa + Mus musculus FASTA files into one search, hit run.
Don't. This breaks the quantification at the peptide-to-protein assignment step, and the failure mode is silent — your numbers come out, they just don't mean what you think.
This guide walks through why a merged-FASTA search fails in cross-species comparisons, the correct species-separated search workflow in MaxQuant and DIA-NN, and what to do with the small fraction of peptides that legitimately match both species. It's the principle that underpins the entire cross-species ECM proteomics workflow described in the Park et al. 2026 reproduction post, but worth its own focused write-up.
Why Merging Breaks Cross-Species Quantification
Tryptic peptides are often species-shared
Many tryptic peptides — particularly from highly conserved proteins like histones, ribosomal subunits, actin, GAPDH — are identical across mammalian species. Pig actin and mouse actin share most tryptic peptides at the sequence level.
In a single-species search, this is fine. The peptide maps unambiguously to "this species' actin."
In a merged Pig+Mouse FASTA search, the same MS/MS spectrum matches:
ACTB_PIGACTB_MOUSE
The search engine has to decide which protein "owns" the peptide. MaxQuant calls these razor peptides and assigns each one, once for the whole experiment, to the protein group with the most identified peptides. If every peptide of the pig and mouse protein is shared, the two proteins collapse into a single group instead. DIA-NN's protein inference faces the same ambiguity. The assignment is arbitrary from a biological standpoint.
What goes wrong downstream
When you compute the protein-level quantity:
- In a Porcine-only sample, all peptides match
_PIGproteins → quantification correct - In a Mouse-only sample, all peptides match
_MOUSEproteins → quantification correct - In the merged search, peptides matching both get assigned to one or the other based on tie-breakers → some peptides measured in porcine samples get credited to the mouse protein (or vice versa), or the two orthologs end up in one mixed protein group
The result: your protein quantities are wrong by a variable, sample-dependent amount. The volcano plot looks fine. Nothing visibly breaks. But the fold changes you publish reflect peptide assignment artifacts as much as biology.
Why it's silent
This is the dangerous part. No software warns you. Both MaxQuant and DIA-NN will happily run a merged search and produce a proteinGroups.txt or report.pg_matrix.tsv. The numbers look reasonable. The downstream statistics run. You publish — and the result isn't reproducible.
The Correct Setup — Species-Separated Searches
General principle
Run one search per species. Each search uses only that species' FASTA. Then combine the per-species protein quantities for cross-species comparison at the ortholog level, not at the peptide level.
Porcine samples (5 files) → MaxQuant search w/ Sus scrofa FASTA → porcine pg_matrix
Mouse samples (3 files) → MaxQuant search w/ Mus musculus FASTA → mouse pg_matrix
↓
Join via BioMart 1:1 ortholog table
↓
Cross-species comparison
MaxQuant — concrete setup: two runs
[Corrected October 2026. The first version split the species into two parameter groups with a different FASTA each. MaxQuant cannot do that: the FASTA list and match-between-runs are global settings (in mqpar.xml they sit outside <parameterGroups>; I checked this in my own 2.8.1 parameter file). One MaxQuant run searches every raw file against the same database and writes one proteinGroups.txt.]
So run MaxQuant twice:
- Run A (porcine): load the 5 porcine raw files only. FASTA:
sus_scrofa_UP000008227.fasta. Standard modifications (Oxidation M, Acetyl N-term; Carbamidomethyl C), Trypsin/P, 2 missed cleavages, LFQ/iBAQ as needed, MBR on if you want it. - Run B (mouse): load the 3 mouse raw files only, same settings, FASTA
mus_musculus_UP000000589.fasta. - Keep the two
combined/txtfolders separate, and join the twoproteinGroups.txtfiles at the ortholog level afterwards.
Match-between-runs then stays within each species automatically, because each run only contains one species.
DIA-NN — concrete setup
DIA-NN doesn't have MaxQuant's "parameter groups" but achieves the same result via separate runs:
# Porcine run: predict the library from the porcine FASTA and search in one call
diann --f porcine_1.d --f porcine_2.d ... \
--fasta sus_scrofa_UP000008227.fasta --fasta-search --predictor --gen-spec-lib \
--out-lib porcine/lib.parquet --out porcine/report.parquet \
--threads 16
# Mouse run (separate invocation)
diann --f mouse_1.d --f mouse_2.d ... \
--fasta mus_musculus_UP000000589.fasta --fasta-search --predictor --gen-spec-lib \
--out-lib mouse/lib.parquet --out mouse/report.parquet \
--threads 16
Create the porcine/ and mouse/ folders first; DIA-NN does not create output folders. Without --fasta-search, DIA-NN does not digest the FASTA at all.
Critically: do not enable cross-run MBR between species — the equivalent in DIA-NN is the global library/index. Keep them separate.
What About Peptides That Genuinely Match Both Species?
It is not a small fraction. I digested the two UniProt reference proteomes in silico (pig UP000008227, 46,004 entries; mouse UP000000589, 54,855 entries; Trypsin/P, 7–30 residues, no missed cleavages, I and L merged because they have the same mass):
| Pig | Mouse | |
|---|---|---|
| Unique tryptic peptides | 532,849 | 551,062 |
| Also present in the other species | 173,071 (32.5%) | 173,071 (31.4%) |
| Proteins with at least one shared peptide | 84.1% | 73.0% |
| Proteins with every peptide shared | 1.8% | 2.9% |
About a third of all tryptic peptides are identical between pig and mouse, and most proteins have at least one. Allowing one missed cleavage changes these numbers by about a percentage point. Almost every protein still has species-unique peptides to quantify from. The options for the shared ones:
Option A — Drop shared peptides (most conservative)
After species-separated quantification, when joining via orthologs, only use species-unique peptides for protein quantification. Most modern tools support filtering by peptide uniqueness within their species reference.
- Pro: cleanest biological interpretation
- Con: lower protein coverage, especially for highly conserved proteins
Option B — Use shared peptides only for ID, not for quantification
Allow shared peptides to contribute to protein identification (boost confidence) but quantify proteins from species-unique peptides only. MaxQuant's Unique + Razor vs Unique columns in proteinGroups.txt controls this.
- Pro: better ID coverage, clean quantification
- Con: more bookkeeping
Option C — Accept shared-peptide quantification but report it
If you must include shared peptides (e.g., for histones where unique peptides are rare), report it explicitly in Methods and note it as a limitation in Discussion. Reviewers in the proteomics field will accept this if it's transparent.
For most cross-species ECM, immune, or signaling proteomics, Option A or B is preferred — most proteins of interest (collagens, laminins, cytokines) have plenty of species-unique peptides.
Common Variants of the Same Mistake
The shared-peptide problem appears in multiple cross-species contexts:
When one sample really contains both species
Everything above assumes each sample comes from one species (porcine ECM samples, mouse Matrigel samples). Host-pathogen infections, patient-derived xenografts and co-cultures are different: a single sample contains both proteomes. [Corrected October 2026: the first version recommended separate searches here too.]
For mixed samples the usual practice is the opposite: search one combined database, so that every spectrum competes against both proteomes, and quantify each protein from its species-specific peptides only. Searching a mixed sample against one species at a time lets conserved peptides be credited to whichever species you happened to search. Report how you handled the shared peptides.
Spike-in standards
Spike-in benchmarks such as the UPS1/UPS2 standards (48 human proteins, usually spiked into a yeast or E. coli background) are designed to be searched with the spike-in sequences in the same database as the background. Include them in the search, then quantify from peptides unique to each.
Quick Sanity Checks for Existing Datasets
If you inherited a cross-species dataset and aren't sure how it was searched:
- Open
proteinGroups.txt— look at theProtein IDscolumn. If you see entries likesp|P12345|ACTB_PIG;sp|P67890|ACTB_MOUSE, the search merged FASTAs. The quantification is suspect. - Check the
Fasta headerscolumn — should be one species' UniProt entries per protein group in a properly separated search. - Compare top-100 quantified proteins between samples — in a merged search you'll see suspiciously consistent "shared" proteins across species samples; in a clean separated search, conserved proteins still appear but their per-species intensities are independent.
- In DIA-NN reports — the
Protein.Groupcolumn similarly reveals whether assignments crossed species boundaries.
If the dataset was merged-FASTA searched, the only fix is re-running the searches species-separated. There's no post-hoc statistical correction.
What This Means for Reproducibility
When you publish cross-species proteomics, your Methods should explicitly state:
Database searches were performed separately for each species, using Sus scrofa UniProt UP000008227 for porcine samples and Mus musculus UniProt UP000000589 for mouse samples (downloaded YYYY-MM-DD). Each species was processed in a separate run, so match-between-runs operated within species only. Cross-species comparisons were performed at the ortholog level using Ensembl BioMart Release [N] 1:1 ortholog tables.
This sentence is small but it's how reviewers and future replicators know your quantification is valid. Its absence is the easiest red flag for a careful reviewer.
FAQ
Q: My samples are all the same species but include trace contamination from another species. Same rules? Mostly yes. If you suspect contamination (e.g., mouse cells in human-only experiment), add the suspected contaminant species (and cRAP) to the search database and inspect what gets identified from it. MaxQuant's built-in contaminant list already covers common lab contaminants. Use it for QC; base the main quantification on your target species.
Q: Can I post-process to fix a merged search? No statistically valid way. You can filter to species-unique peptides retrospectively, but the FDR was computed against the merged database, so significance estimates are off. Re-running the search separately is the only clean fix.
Q: What about closely related strains — e.g., two different mouse strains? For nearly-identical references, separate searches add complexity for minimal benefit. One unified FASTA per species is fine. The shared-peptide problem only matters when species (or significantly diverged reference proteomes) are mixed.
Q: Does this also apply to RNA-seq cross-species analysis? Conceptually yes — read mapping ambiguity is the analogue. But standard practice in cross-species RNA-seq already aligns to species-specific references separately and joins at the ortholog level. The MS proteomics community has been slower to standardize because tools allow merged searches without warning.
Q: How do I find Ensembl BioMart 1:1 orthologs to join species after separated searches? That's a separate workflow — see BioMart Pig ↔ Mouse 1:1 Ortholog Mapping for Cross-Species Proteomics (R + Python Tutorial).
Closing — The One Sentence
When each sample comes from one species: search each species separately, join at the ortholog level afterwards, and do not merge the species FASTA files into one search. When one sample contains both species, do the opposite: one combined search, quantified from species-specific peptides. The shared-peptide problem makes merged-FASTA quantification silently wrong, and there's no post-hoc fix.
Related posts:
- Reproducing Park et al. 2026 — Cross-Species ECM Proteomics, Three Iterations
- From DIA-NN Output to Paper Draft: AI-Assisted Proteomics Workflow
- LC-MS/MS Proteomics: Complete Workflow Guide 2026
- BioMart Pig ↔ Mouse 1:1 Ortholog Mapping Tutorial
References:
- Cox, J. & Mann, M. (2008). MaxQuant enables high peptide identification rates. Nature Biotechnology, 26, 1367-1372.
- Demichev, V. et al. (2020). DIA-NN: neural networks and interference correction. Nature Methods, 17, 41-44.
- Park, S. M. et al. (2026). Decellularized esophageal ECM. (Yonsei, Cho Lab)
Related
BioMart Pig ↔ Mouse 1:1 Ortholog Mapping for Cross-Species Proteomics (R + Python Tutorial)
May 23 · 12 min read
ProteomicsIs 1% FDR Really 1%? I Hid 14,567 Arabidopsis Proteins in MaxQuant, FragPipe and DIA-NN Searches
Nov 14 · 13 min read
ProteomicsProteomics CV: What Replicate CVs Look Like in MaxQuant, FragPipe and DIA-NN (Measured on a Known-Ratio Benchmark)
Oct 16 · 8 min read
ProteomicsDIA-NN 2.3 on a Known-Answer Benchmark: How It Compares With MaxQuant and FragPipe (and Why the Comparison Has a Catch)
Oct 13 · 11 min read