Coverage: investigate and better handle tabix indexing concatenated bgzipped files.

[grosscol]

Current State

During coverage aggregation, bgzipped results are concatenated together as the process runs.

bravo_data_prep/workflows/coverage/depth_statistics.nf

Line 85 in 3349238

template 'mlr_agg.sh'

bravo_data_prep/workflows/coverage/templates/mlr_agg.sh

Lines 28 to 31 in 3349238

	# Aggregate depths from depth file chunks
	mlr -N --tsv 'nest' --ivar ";" -f 3 \${PIPES[@]} |\
	sort --numeric-sort --key=2 |\
	bgzip >> ${result_file}

Tabix sometimes fails to produce a valid index for concatenated summary data occasionally. An index gets written, has the contig name, but can't be used to get data by region. tabix file.tsv.gz chr22:10000100-10000200 | wc -l gets 0.

Work around currently involves re-writing entire bgzipped file.

Action items

• Generate small reproducible example of tabix not producing an index of data.
• Sort out solution that is more efficient that re-writing the entire gzipped file.

Reference

Suspected to be related to:

• https://sourceforge.net/p/samtools/mailman/samtools-help/thread/2EDFB36701A9ED4790EA0B76B3E2FC198AAB389B%40UKCH-PRD-EXMB01.illumina.com/#msg34108936
• An empty BGZF block elsewhere in the file stream causes premature EOF. samtools/htslib#45