Bioinformatics

bcftools vs GATK HaplotypeCaller, Scored Against GIAB HG002: A Tie on SNPs, a Clear Gap on Indels, and Hard Filters That Cost Recall

We called germline variants on GIAB HG002 chromosome 20 with bcftools and GATK HaplotypeCaller and scored both against the GIAB v4.2.1 truth set with hap.py. On SNPs they tied (F1 0.992). On indels GATK was far ahead (0.992 vs 0.937), and most of bcftools' false indels had the right allele but the wrong genotype. GATK's recommended hard filters removed 580 true SNPs, mostly in low-mappability regions.

·8 min read
#variant calling#bcftools#GATK#HaplotypeCaller#GIAB#HG002#hap.py#germline variants#benchmark#whole genome sequencing

Everything here was scored against a public truth set. The run script, the trimmed hap.py tables, the scoring script and the figure are public, and CI re-runs the scoring and checks that it reproduces these numbers exactly. The first run was in a 4-core cloud container. Before writing this, we downloaded the inputs again (md5 sums matched) and re-ran every step on our own server with the same tool versions: all four hap.py tables came out byte for byte identical. No client data was used. (한국어판: bcftools와 GATK, GIAB 정답으로 채점해 보니)

If you are choosing a germline variant caller, or wondering whether the one in your pipeline is good enough, the most useful comparison is against a sample whose variants are already known. Genome in a Bottle publishes exactly that for HG002, so we took one public alignment of it and ran the two callers people most often reach for without a GPU or a cluster: bcftools and GATK HaplotypeCaller.

On single-nucleotide variants they came out the same. On indels they did not: bcftools missed eight times as many true indels and got far more genotypes wrong. GATK's recommended hard filters, which many pipelines apply by default, removed more true SNPs than false ones.

Errors per call set and region for SNPs and indels at 35x, and F1 at 15x vs 35x

What we ran

The input is the HG002 (NA24385) Illumina NovaSeq PCR-free whole genome at 35× coverage, aligned with BWA-MEM to GRCh38 (no-alt) with duplicates marked. This is the BAM Google uses in its DeepVariant case study, cut to chromosome 20. We also made a 15× version by downsampling the same BAM.

Every caller read the same alignment, so this compares callers, not aligners.

Call setHow it was run
bcftools, all callsbcftools 1.25: mpileup -q 20 -Q 20 -d 250 -a AD,DP piped into call -mv
bcftools, QUAL ≥ 20the same calls without those flagged LowQual
GATK HaplotypeCallerGATK 4.6.2.0, default settings, chr20 split into four pieces run in parallel and joined
GATK + hard filtersGATK's recommended hard filters. SNPs: QD < 2, FS > 60, MQ < 40, MQRankSum < −12.5, ReadPosRankSum < −8, SOR > 3. Indels: QD < 2, FS > 200, ReadPosRankSum < −20, SOR > 10. The guide's QUAL < 30 is left out: HaplotypeCaller does not emit calls below QUAL 30 by default

We did not run VQSR, which needs far more variants than one chromosome of one sample. DeepVariant is not in this round; we plan to add it.

Scoring used hap.py 0.3.15 with the vcfeval engine against the GIAB v4.2.1 benchmark for HG002. Only the benchmark regions count, which on chr20 cover 57.0 Mb with 71,333 true SNPs and 11,256 true indels. hap.py also reports each GIAB v3.3 "difficult region" stratum separately: tandem repeats and homopolymers, segmental duplications, and low-mappability regions.

Results at 35×

Call setSNP recallSNP precisionSNP F1Indel recallIndel precisionIndel F1
bcftools, all calls0.9940.9900.99200.9310.9420.9365
bcftools, QUAL ≥ 200.9930.9920.99280.9250.9450.9349
GATK HaplotypeCaller0.9940.9900.99210.9910.9930.9923
GATK + hard filters0.9860.9950.99050.9910.9940.9927

SNPs are a tie. Both callers missed about 430 of the 71,333 true SNPs (bcftools 437, GATK 424) and made about 710 false calls (714 and 705). Outside every difficult stratum, SNP F1 was 0.999 for both.

Indels are not. Of 11,256 true indels, bcftools missed 781 and GATK 97. bcftools made 675 false indel calls, and 592 of them had the right allele with the wrong genotype, typically heterozygous called as homozygous or the reverse. GATK made 80 false indel calls in total. A comparison that checked only whether the allele was found would rate bcftools far higher than this one does.

Hard filters help precision and cost recall. On SNPs, GATK's filters removed 356 false calls but also 580 true ones, so F1 went down, from 0.9921 to 0.9905. Most of the lost true SNPs (413 of 580) sit in low-mappability regions, where SNP recall fell from 0.867 to 0.715. MQ < 40 flagged 382 of the 580 and SOR > 3 flagged 211 (a call can fail both): reads in low-mappability regions tend to have low mapping quality. On indels the filters helped a little (F1 0.9923 to 0.9927).

Where the callers struggle

SNP and indel F1 by region at 35×:

Call setEasy regionsAll difficultTandem repeats, homopolymersSegmental duplicationsLow mappability
SNP, bcftools0.99910.9530.9680.8600.883
SNP, GATK0.99870.9560.9890.8460.867
SNP, GATK + filters0.99830.9460.9880.8770.805
Indel, bcftools0.9830.9170.9160.9190.830
Indel, GATK0.9980.9900.9930.8890.862
Indel, GATK + filters0.9990.9900.9930.9030.878

On SNPs, every call set was worst in segmental duplications and low-mappability regions, where F1 dropped to 0.81-0.88. In segmental duplications the problem is mostly false positives: SNP precision there was 0.77-0.83 at recall 0.92-0.95, most likely because reads from near-identical copies elsewhere in the genome are placed there. GATK did clearly better in tandem repeats and homopolymers (SNP F1 0.989 against 0.968, indel F1 0.993 against 0.916), which is where its local reassembly of reads is meant to help.

At 15×

Call setSNP F1Indel F1
bcftools, all calls0.98420.9172
bcftools, QUAL ≥ 200.98130.9077
GATK HaplotypeCaller0.98430.9573
GATK + hard filters0.97670.9575

At lower coverage both callers lose SNPs at the same rate, the indel gap stays, and filtering costs more. The hard filters and the QUAL ≥ 20 cut were designed around typical coverage; at 15× they remove a larger share of true calls (GATK SNP recall 0.982 unfiltered, 0.965 filtered).

Run time

On chr20 with four threads, bcftools took 3.7 minutes at 35× (107 MB of memory), and GATK 17.4 minutes of wall time and 58 minutes of CPU time (3.1 GB per process). bcftools mpileup is single-threaded; --threads only speeds up compression. These times are from the cloud run. Our re-run shared the server with two proteomics searches, so its times are not comparable; it took longer, with the same results.

What we would do with this

For SNPs in well-behaved regions, either caller is fine, and bcftools is several times cheaper. For anything where indels matter, GATK HaplotypeCaller is the safer default. Much of the difference is in genotypes, so check genotype concordance, not just allele presence, when you validate a pipeline. Treat hard filters as something to test rather than a default: on this sample they made SNP calls worse, and the loss concentrated in exactly the regions where calls are already least reliable. And whatever the caller, results from segmental duplications and low-mappability regions deserve a second look before anyone acts on them.

Limits

One sample, one chromosome, one library type (NovaSeq PCR-free). PCR-amplified libraries produce many more indel errors, mostly in homopolymers (Fang et al. 2014), so the indel numbers here are a best case. GATK ran as four pieces joined afterwards; we checked ±300 bp around the three boundaries, and the 8 truth variants there were each called once. No VQSR, no joint genotyping, and DeepVariant still to come.

Reproduce it

Download script, run script (bcftools, GATK, DeepVariant and hap.py steps), trimmed hap.py tables, scores and figure: sbmlab-benchmarks/variant-calling-giab. download.sh fetches about 2.2 GB. The tools come from bioconda (gatk4 4.6.2.0, bcftools 1.25, samtools 1.25, hap.py 0.3.15, rtg-tools 3.13).

References

  1. Wagner J, et al. Benchmarking challenging small variants with linked and long reads. Cell Genomics 2022;2(5):100128. (GIAB v4.2.1)
  2. Krusche P, et al. Best practices for benchmarking germline small-variant calls in human genomes. Nat Biotechnol 2019;37:555-560. (hap.py)
  3. Danecek P, et al. Twelve years of SAMtools and BCFtools. GigaScience 2021;10(2):giab008.
  4. Poplin R, et al. Scaling accurate genetic variant discovery to tens of thousands of samples. bioRxiv 2017:201178. (GATK HaplotypeCaller)
  5. Fang H, et al. Reducing INDEL calling errors in whole genome and exome sequencing data. Genome Med 2014;6:89.

Related