bcftools vs GATK HaplotypeCaller, Scored Against GIAB HG002: A Tie on SNPs, a Clear Gap on Indels, and Hard Filters That Cost Recall
We called germline variants on GIAB HG002 chromosome 20 with bcftools and GATK HaplotypeCaller and scored both against the GIAB v4.2.1 truth set with hap.py. On SNPs they tied (F1 0.992). On indels GATK was far ahead (0.992 vs 0.937), and most of bcftools' false indels had the right allele but the wrong genotype. GATK's recommended hard filters removed 580 true SNPs, mostly in low-mappability regions.
Everything here was scored against a public truth set. The run script, the trimmed hap.py tables, the scoring script and the figure are public, and CI re-runs the scoring and checks that it reproduces these numbers exactly. The first run was in a 4-core cloud container. Before writing this, we downloaded the inputs again (md5 sums matched) and re-ran every step on our own server with the same tool versions: all four hap.py tables came out byte for byte identical. No client data was used. (한국어판: bcftools와 GATK, GIAB 정답으로 채점해 보니)
If you are choosing a germline variant caller, or wondering whether the one in your pipeline is good enough, the most useful comparison is against a sample whose variants are already known. Genome in a Bottle publishes exactly that for HG002, so we took one public alignment of it and ran the two callers people most often reach for without a GPU or a cluster: bcftools and GATK HaplotypeCaller.
On single-nucleotide variants they came out the same. On indels they did not: bcftools missed eight times as many true indels and got far more genotypes wrong. GATK's recommended hard filters, which many pipelines apply by default, removed more true SNPs than false ones.

What we ran
The input is the HG002 (NA24385) Illumina NovaSeq PCR-free whole genome at 35× coverage, aligned with BWA-MEM to GRCh38 (no-alt) with duplicates marked. This is the BAM Google uses in its DeepVariant case study, cut to chromosome 20. We also made a 15× version by downsampling the same BAM.
Every caller read the same alignment, so this compares callers, not aligners.
| Call set | How it was run |
|---|---|
| bcftools, all calls | bcftools 1.25: mpileup -q 20 -Q 20 -d 250 -a AD,DP piped into call -mv |
| bcftools, QUAL ≥ 20 | the same calls without those flagged LowQual |
| GATK HaplotypeCaller | GATK 4.6.2.0, default settings, chr20 split into four pieces run in parallel and joined |
| GATK + hard filters | GATK's recommended hard filters. SNPs: QD < 2, FS > 60, MQ < 40, MQRankSum < −12.5, ReadPosRankSum < −8, SOR > 3. Indels: QD < 2, FS > 200, ReadPosRankSum < −20, SOR > 10. The guide's QUAL < 30 is left out: HaplotypeCaller does not emit calls below QUAL 30 by default |
We did not run VQSR, which needs far more variants than one chromosome of one sample. DeepVariant is not in this round; we plan to add it.
Scoring used hap.py 0.3.15 with the vcfeval engine against the GIAB v4.2.1 benchmark for HG002. Only the benchmark regions count, which on chr20 cover 57.0 Mb with 71,333 true SNPs and 11,256 true indels. hap.py also reports each GIAB v3.3 "difficult region" stratum separately: tandem repeats and homopolymers, segmental duplications, and low-mappability regions.
Results at 35×
| Call set | SNP recall | SNP precision | SNP F1 | Indel recall | Indel precision | Indel F1 |
|---|---|---|---|---|---|---|
| bcftools, all calls | 0.994 | 0.990 | 0.9920 | 0.931 | 0.942 | 0.9365 |
| bcftools, QUAL ≥ 20 | 0.993 | 0.992 | 0.9928 | 0.925 | 0.945 | 0.9349 |
| GATK HaplotypeCaller | 0.994 | 0.990 | 0.9921 | 0.991 | 0.993 | 0.9923 |
| GATK + hard filters | 0.986 | 0.995 | 0.9905 | 0.991 | 0.994 | 0.9927 |
SNPs are a tie. Both callers missed about 430 of the 71,333 true SNPs (bcftools 437, GATK 424) and made about 710 false calls (714 and 705). Outside every difficult stratum, SNP F1 was 0.999 for both.
Indels are not. Of 11,256 true indels, bcftools missed 781 and GATK 97. bcftools made 675 false indel calls, and 592 of them had the right allele with the wrong genotype, typically heterozygous called as homozygous or the reverse. GATK made 80 false indel calls in total. A comparison that checked only whether the allele was found would rate bcftools far higher than this one does.
Hard filters help precision and cost recall. On SNPs, GATK's filters removed 356 false calls but also 580 true ones, so F1 went down, from 0.9921 to 0.9905. Most of the lost true SNPs (413 of 580) sit in low-mappability regions, where SNP recall fell from 0.867 to 0.715. MQ < 40 flagged 382 of the 580 and SOR > 3 flagged 211 (a call can fail both): reads in low-mappability regions tend to have low mapping quality. On indels the filters helped a little (F1 0.9923 to 0.9927).
Where the callers struggle
SNP and indel F1 by region at 35×:
| Call set | Easy regions | All difficult | Tandem repeats, homopolymers | Segmental duplications | Low mappability |
|---|---|---|---|---|---|
| SNP, bcftools | 0.9991 | 0.953 | 0.968 | 0.860 | 0.883 |
| SNP, GATK | 0.9987 | 0.956 | 0.989 | 0.846 | 0.867 |
| SNP, GATK + filters | 0.9983 | 0.946 | 0.988 | 0.877 | 0.805 |
| Indel, bcftools | 0.983 | 0.917 | 0.916 | 0.919 | 0.830 |
| Indel, GATK | 0.998 | 0.990 | 0.993 | 0.889 | 0.862 |
| Indel, GATK + filters | 0.999 | 0.990 | 0.993 | 0.903 | 0.878 |
On SNPs, every call set was worst in segmental duplications and low-mappability regions, where F1 dropped to 0.81-0.88. In segmental duplications the problem is mostly false positives: SNP precision there was 0.77-0.83 at recall 0.92-0.95, most likely because reads from near-identical copies elsewhere in the genome are placed there. GATK did clearly better in tandem repeats and homopolymers (SNP F1 0.989 against 0.968, indel F1 0.993 against 0.916), which is where its local reassembly of reads is meant to help.
At 15×
| Call set | SNP F1 | Indel F1 |
|---|---|---|
| bcftools, all calls | 0.9842 | 0.9172 |
| bcftools, QUAL ≥ 20 | 0.9813 | 0.9077 |
| GATK HaplotypeCaller | 0.9843 | 0.9573 |
| GATK + hard filters | 0.9767 | 0.9575 |
At lower coverage both callers lose SNPs at the same rate, the indel gap stays, and filtering costs more. The hard filters and the QUAL ≥ 20 cut were designed around typical coverage; at 15× they remove a larger share of true calls (GATK SNP recall 0.982 unfiltered, 0.965 filtered).
Run time
On chr20 with four threads, bcftools took 3.7 minutes at 35× (107 MB of memory), and GATK 17.4 minutes of wall time and 58 minutes of CPU time (3.1 GB per process). bcftools mpileup is single-threaded; --threads only speeds up compression. These times are from the cloud run. Our re-run shared the server with two proteomics searches, so its times are not comparable; it took longer, with the same results.
What we would do with this
For SNPs in well-behaved regions, either caller is fine, and bcftools is several times cheaper. For anything where indels matter, GATK HaplotypeCaller is the safer default. Much of the difference is in genotypes, so check genotype concordance, not just allele presence, when you validate a pipeline. Treat hard filters as something to test rather than a default: on this sample they made SNP calls worse, and the loss concentrated in exactly the regions where calls are already least reliable. And whatever the caller, results from segmental duplications and low-mappability regions deserve a second look before anyone acts on them.
Limits
One sample, one chromosome, one library type (NovaSeq PCR-free). PCR-amplified libraries produce many more indel errors, mostly in homopolymers (Fang et al. 2014), so the indel numbers here are a best case. GATK ran as four pieces joined afterwards; we checked ±300 bp around the three boundaries, and the 8 truth variants there were each called once. No VQSR, no joint genotyping, and DeepVariant still to come.
Reproduce it
Download script, run script (bcftools, GATK, DeepVariant and hap.py steps), trimmed hap.py tables, scores and figure: sbmlab-benchmarks/variant-calling-giab. download.sh fetches about 2.2 GB. The tools come from bioconda (gatk4 4.6.2.0, bcftools 1.25, samtools 1.25, hap.py 0.3.15, rtg-tools 3.13).
References
- Wagner J, et al. Benchmarking challenging small variants with linked and long reads. Cell Genomics 2022;2(5):100128. (GIAB v4.2.1)
- Krusche P, et al. Best practices for benchmarking germline small-variant calls in human genomes. Nat Biotechnol 2019;37:555-560. (hap.py)
- Danecek P, et al. Twelve years of SAMtools and BCFtools. GigaScience 2021;10(2):giab008.
- Poplin R, et al. Scaling accurate genetic variant discovery to tens of thousands of samples. bioRxiv 2017:201178. (GATK HaplotypeCaller)
- Fang H, et al. Reducing INDEL calling errors in whole genome and exome sequencing data. Genome Med 2014;6:89.
Related
Related
STRING Confidence Score and Hub Proteins, Measured: What 0.4 vs 0.7 Changes and Whether Hubs Are Biology or Attention (STRING 12.5)
Oct 19 · 15 min read
ProteomicsIs 1% FDR Really 1%? I Hid 14,567 Arabidopsis Proteins in MaxQuant, FragPipe and DIA-NN Searches
Nov 14 · 13 min read
ProteomicsDIA-NN 2.7.0 vs 2.3.2: What Changed on a Known-Ratio Benchmark, and How to Upgrade
Nov 1 · 12 min read
ProteomicsDIA-NN Quantification Modes Scored on Known Ratios: QuantUMS, Legacy MaxLFQ and Fragment-Level Re-Quantification, With and Without MBR
Oct 17 · 17 min read