Proteomics

Why You Must NOT Merge Species FASTA Databases in Cross-Species Proteomics (Shared Peptide Problem)

In cross-species LC-MS/MS proteomics (Porcine ECM vs Mouse Matrigel, human vs mouse cells, host-pathogen, etc.) merging species FASTA into one search database breaks protein quantification. This is the shared-peptide ambiguity problem. Here's why it happens, how to set up species-separated searches in MaxQuant and DIA-NN, and how to handle peptides that genuinely match both species.

·10 min read
#cross-species proteomics#MaxQuant#DIA-NN#FASTA database#shared peptide#razor peptide#Sus scrofa#Mus musculus#host pathogen proteomics#Matrigel#search engine

Species-separated FASTA search

The Single Most Common Cross-Species Proteomics Mistake

You have two sets of LC-MS/MS samples: Porcine esophageal ECM (n=5) and Mouse Matrigel (n=3). You want to compare protein composition. The instinct — open MaxQuant, load Sus scrofa + Mus musculus FASTA files into one search, hit run.

Don't. This breaks the quantification at the peptide-to-protein assignment step, and the failure mode is silent — your numbers come out, they just don't mean what you think.

This guide walks through why a merged-FASTA search fails in cross-species comparisons, the correct species-separated search workflow in MaxQuant and DIA-NN, and what to do with the small fraction of peptides that legitimately match both species. It's the principle that underpins the entire cross-species ECM proteomics workflow described in the Park et al. 2026 reproduction post, but worth its own focused write-up.

Why Merging Breaks Cross-Species Quantification

Tryptic peptides are often species-shared

Many tryptic peptides — particularly from highly conserved proteins like histones, ribosomal subunits, actin, GAPDH — are identical across mammalian species. Pig actin and mouse actin share most tryptic peptides at the sequence level.

In a single-species search, this is fine. The peptide maps unambiguously to "this species' actin."

In a merged Pig+Mouse FASTA search, the same MS/MS spectrum matches:

  • ACTB_PIG
  • ACTB_MOUSE

The search engine has to decide which protein "owns" the peptide. MaxQuant calls these razor peptides and assigns each one, once for the whole experiment, to the protein group with the most identified peptides. If every peptide of the pig and mouse protein is shared, the two proteins collapse into a single group instead. DIA-NN's protein inference faces the same ambiguity. The assignment is arbitrary from a biological standpoint.

What goes wrong downstream

When you compute the protein-level quantity:

  • In a Porcine-only sample, all peptides match _PIG proteins → quantification correct
  • In a Mouse-only sample, all peptides match _MOUSE proteins → quantification correct
  • In the merged search, peptides matching both get assigned to one or the other based on tie-breakers → some peptides measured in porcine samples get credited to the mouse protein (or vice versa), or the two orthologs end up in one mixed protein group

The result: your protein quantities are wrong by a variable, sample-dependent amount. The volcano plot looks fine. Nothing visibly breaks. But the fold changes you publish reflect peptide assignment artifacts as much as biology.

Why it's silent

This is the dangerous part. No software warns you. Both MaxQuant and DIA-NN will happily run a merged search and produce a proteinGroups.txt or report.pg_matrix.tsv. The numbers look reasonable. The downstream statistics run. You publish — and the result isn't reproducible.

The Correct Setup — Species-Separated Searches

General principle

Run one search per species. Each search uses only that species' FASTA. Then combine the per-species protein quantities for cross-species comparison at the ortholog level, not at the peptide level.

Porcine samples (5 files) → MaxQuant search w/ Sus scrofa FASTA → porcine pg_matrix
Mouse samples (3 files)   → MaxQuant search w/ Mus musculus FASTA → mouse pg_matrix
                                                ↓
                                   Join via BioMart 1:1 ortholog table
                                                ↓
                                   Cross-species comparison

MaxQuant — concrete setup: two runs

[Corrected October 2026. The first version split the species into two parameter groups with a different FASTA each. MaxQuant cannot do that: the FASTA list and match-between-runs are global settings (in mqpar.xml they sit outside <parameterGroups>; I checked this in my own 2.8.1 parameter file). One MaxQuant run searches every raw file against the same database and writes one proteinGroups.txt.]

So run MaxQuant twice:

  1. Run A (porcine): load the 5 porcine raw files only. FASTA: sus_scrofa_UP000008227.fasta. Standard modifications (Oxidation M, Acetyl N-term; Carbamidomethyl C), Trypsin/P, 2 missed cleavages, LFQ/iBAQ as needed, MBR on if you want it.
  2. Run B (mouse): load the 3 mouse raw files only, same settings, FASTA mus_musculus_UP000000589.fasta.
  3. Keep the two combined/txt folders separate, and join the two proteinGroups.txt files at the ortholog level afterwards.

Match-between-runs then stays within each species automatically, because each run only contains one species.

DIA-NN — concrete setup

DIA-NN doesn't have MaxQuant's "parameter groups" but achieves the same result via separate runs:

# Porcine run: predict the library from the porcine FASTA and search in one call
diann --f porcine_1.d --f porcine_2.d ... \
      --fasta sus_scrofa_UP000008227.fasta --fasta-search --predictor --gen-spec-lib \
      --out-lib porcine/lib.parquet --out porcine/report.parquet \
      --threads 16

# Mouse run (separate invocation)
diann --f mouse_1.d --f mouse_2.d ... \
      --fasta mus_musculus_UP000000589.fasta --fasta-search --predictor --gen-spec-lib \
      --out-lib mouse/lib.parquet --out mouse/report.parquet \
      --threads 16

Create the porcine/ and mouse/ folders first; DIA-NN does not create output folders. Without --fasta-search, DIA-NN does not digest the FASTA at all.

Critically: do not enable cross-run MBR between species — the equivalent in DIA-NN is the global library/index. Keep them separate.

What About Peptides That Genuinely Match Both Species?

It is not a small fraction. I digested the two UniProt reference proteomes in silico (pig UP000008227, 46,004 entries; mouse UP000000589, 54,855 entries; Trypsin/P, 7–30 residues, no missed cleavages, I and L merged because they have the same mass):

PigMouse
Unique tryptic peptides532,849551,062
Also present in the other species173,071 (32.5%)173,071 (31.4%)
Proteins with at least one shared peptide84.1%73.0%
Proteins with every peptide shared1.8%2.9%

About a third of all tryptic peptides are identical between pig and mouse, and most proteins have at least one. Allowing one missed cleavage changes these numbers by about a percentage point. Almost every protein still has species-unique peptides to quantify from. The options for the shared ones:

Option A — Drop shared peptides (most conservative)

After species-separated quantification, when joining via orthologs, only use species-unique peptides for protein quantification. Most modern tools support filtering by peptide uniqueness within their species reference.

  • Pro: cleanest biological interpretation
  • Con: lower protein coverage, especially for highly conserved proteins

Option B — Use shared peptides only for ID, not for quantification

Allow shared peptides to contribute to protein identification (boost confidence) but quantify proteins from species-unique peptides only. MaxQuant's Unique + Razor vs Unique columns in proteinGroups.txt controls this.

  • Pro: better ID coverage, clean quantification
  • Con: more bookkeeping

Option C — Accept shared-peptide quantification but report it

If you must include shared peptides (e.g., for histones where unique peptides are rare), report it explicitly in Methods and note it as a limitation in Discussion. Reviewers in the proteomics field will accept this if it's transparent.

For most cross-species ECM, immune, or signaling proteomics, Option A or B is preferred — most proteins of interest (collagens, laminins, cytokines) have plenty of species-unique peptides.

Common Variants of the Same Mistake

The shared-peptide problem appears in multiple cross-species contexts:

When one sample really contains both species

Everything above assumes each sample comes from one species (porcine ECM samples, mouse Matrigel samples). Host-pathogen infections, patient-derived xenografts and co-cultures are different: a single sample contains both proteomes. [Corrected October 2026: the first version recommended separate searches here too.]

For mixed samples the usual practice is the opposite: search one combined database, so that every spectrum competes against both proteomes, and quantify each protein from its species-specific peptides only. Searching a mixed sample against one species at a time lets conserved peptides be credited to whichever species you happened to search. Report how you handled the shared peptides.

Spike-in standards

Spike-in benchmarks such as the UPS1/UPS2 standards (48 human proteins, usually spiked into a yeast or E. coli background) are designed to be searched with the spike-in sequences in the same database as the background. Include them in the search, then quantify from peptides unique to each.

Quick Sanity Checks for Existing Datasets

If you inherited a cross-species dataset and aren't sure how it was searched:

  1. Open proteinGroups.txt — look at the Protein IDs column. If you see entries like sp|P12345|ACTB_PIG;sp|P67890|ACTB_MOUSE, the search merged FASTAs. The quantification is suspect.
  2. Check the Fasta headers column — should be one species' UniProt entries per protein group in a properly separated search.
  3. Compare top-100 quantified proteins between samples — in a merged search you'll see suspiciously consistent "shared" proteins across species samples; in a clean separated search, conserved proteins still appear but their per-species intensities are independent.
  4. In DIA-NN reports — the Protein.Group column similarly reveals whether assignments crossed species boundaries.

If the dataset was merged-FASTA searched, the only fix is re-running the searches species-separated. There's no post-hoc statistical correction.

What This Means for Reproducibility

When you publish cross-species proteomics, your Methods should explicitly state:

Database searches were performed separately for each species, using Sus scrofa UniProt UP000008227 for porcine samples and Mus musculus UniProt UP000000589 for mouse samples (downloaded YYYY-MM-DD). Each species was processed in a separate run, so match-between-runs operated within species only. Cross-species comparisons were performed at the ortholog level using Ensembl BioMart Release [N] 1:1 ortholog tables.

This sentence is small but it's how reviewers and future replicators know your quantification is valid. Its absence is the easiest red flag for a careful reviewer.

FAQ

Q: My samples are all the same species but include trace contamination from another species. Same rules? Mostly yes. If you suspect contamination (e.g., mouse cells in human-only experiment), add the suspected contaminant species (and cRAP) to the search database and inspect what gets identified from it. MaxQuant's built-in contaminant list already covers common lab contaminants. Use it for QC; base the main quantification on your target species.

Q: Can I post-process to fix a merged search? No statistically valid way. You can filter to species-unique peptides retrospectively, but the FDR was computed against the merged database, so significance estimates are off. Re-running the search separately is the only clean fix.

Q: What about closely related strains — e.g., two different mouse strains? For nearly-identical references, separate searches add complexity for minimal benefit. One unified FASTA per species is fine. The shared-peptide problem only matters when species (or significantly diverged reference proteomes) are mixed.

Q: Does this also apply to RNA-seq cross-species analysis? Conceptually yes — read mapping ambiguity is the analogue. But standard practice in cross-species RNA-seq already aligns to species-specific references separately and joins at the ortholog level. The MS proteomics community has been slower to standardize because tools allow merged searches without warning.

Q: How do I find Ensembl BioMart 1:1 orthologs to join species after separated searches? That's a separate workflow — see BioMart Pig ↔ Mouse 1:1 Ortholog Mapping for Cross-Species Proteomics (R + Python Tutorial).

Closing — The One Sentence

When each sample comes from one species: search each species separately, join at the ortholog level afterwards, and do not merge the species FASTA files into one search. When one sample contains both species, do the opposite: one combined search, quantified from species-specific peptides. The shared-peptide problem makes merged-FASTA quantification silently wrong, and there's no post-hoc fix.


Related posts:

References:

  • Cox, J. & Mann, M. (2008). MaxQuant enables high peptide identification rates. Nature Biotechnology, 26, 1367-1372.
  • Demichev, V. et al. (2020). DIA-NN: neural networks and interference correction. Nature Methods, 17, 41-44.
  • Park, S. M. et al. (2026). Decellularized esophageal ECM. (Yonsei, Cho Lab)

Related