success of all jobs

[fergsc]

Hi,
I am trying to lastz align some primate genomes and came across your tool, and am excited to use it.

I am trying to work out if all jobs compeleted sucessfully or not. If they dont all complete will XXX.final.chain.gz be created?
If jobs dont finish sucessfully how can we find out the reason for failure? I can not see this information in the logs.

The reason I ask is due to the large number of jobs that give this output.

executor >  local (8238)
executor >  local (8238)
[c2/6dbd2f] process > execute_jobs (8238)  [ 13%] 13151 of 97287, failed: 7, ...
[c2/6dbd2f] process > execute_jobs (8238)  [ 13%] 13151 of 97287, failed: 7, ...

When I look at temp_lastz_run/.nextflow.log I have this output

~> TaskHandler[id: 8238; name: execute_jobs (8238); status: RUNNING; exit: -; error: -; workDir: /make_lastz_chains/temp_lastz_run/work/c2/6dbd2f55240d4986404c6f5a503940]
~> TaskHandler[id: 8238; name: execute_jobs (8238); status: RUNNING; exit: -; error: -; workDir: /make_lastz_chains/temp_lastz_run/work/c2/6dbd2f55240d4986404c6f5a503940]
~> TaskHandler[id: 8238; name: execute_jobs (8238); status: RUNNING; exit: -; error: -; workDir: /make_lastz_chains/temp_lastz_run/work/c2/6dbd2f55240d4986404c6f5a503940]
~> TaskHandler[id: 8238; name: execute_jobs (8238); status: RUNNING; exit: -; error: -; workDir: /make_lastz_chains/temp_lastz_run/work/c2/6dbd2f55240d4986404c6f5a503940]
~> TaskHandler[id: 8238; name: execute_jobs (8238); status: RUNNING; exit: -; error: -; workDir: /make_lastz_chains/temp_lastz_run/work/c2/6dbd2f55240d4986404c6f5a503940]
~> TaskHandler[id: 8238; name: execute_jobs (8238); status: RUNNING; exit: -; error: -; workDir: /make_lastz_chains/temp_lastz_run/work/c2/6dbd2f55240d4986404c6f5a503940]
~> TaskHandler[id: 8238; name: execute_jobs (8238); status: RUNNING; exit: -; error: -; workDir: /make_lastz_chains/temp_lastz_run/work/c2/6dbd2f55240d4986404c6f5a503940]
~> TaskHandler[id: 8238; name: execute_jobs (8238); status: RUNNING; exit: -; error: -; workDir: /make_lastz_chains/temp_lastz_run/work/c2/6dbd2f55240d4986404c6f5a503940]
~> TaskHandler[id: 8238; name: execute_jobs (8238); status: RUNNING; exit: -; error: -; workDir: /make_lastz_chains/temp_lastz_run/work/c2/6dbd2f55240d4986404c6f5a503940]
~> TaskHandler[id: 8238; name: execute_jobs (8238); status: RUNNING; exit: -; error: -; workDir: /make_lastz_chains/temp_lastz_run/work/c2/6dbd2f55240d4986404c6f5a503940]
~> TaskHandler[id: 8238; name: execute_jobs (8238); status: RUNNING; exit: -; error: -; workDir: /make_lastz_chains/temp_lastz_run/work/c2/6dbd2f55240d4986404c6f5a503940]
~> TaskHandler[id: 8238; name: execute_jobs (8238); status: RUNNING; exit: -; error: -; workDir: /make_lastz_chains/temp_lastz_run/work/c2/6dbd2f55240d4986404c6f5a503940]
~> TaskHandler[id: 8238; name: execute_jobs (8238); status: RUNNING; exit: -; error: -; workDir: /make_lastz_chains/temp_lastz_run/work/c2/6dbd2f55240d4986404c6f5a503940]
~> TaskHandler[id: 8238; name: execute_jobs (8238); status: RUNNING; exit: -; error: -; workDir: /make_lastz_chains/temp_lastz_run/work/c2/6dbd2f55240d4986404c6f5a503940]
~> TaskHandler[id: 8238; name: execute_jobs (8238); status: RUNNING; exit: -; error: -; workDir: /make_lastz_chains/temp_lastz_run/work/c2/6dbd2f55240d4986404c6f5a503940]
~> TaskHandler[id: 8238; name: execute_jobs (8238); status: RUNNING; exit: -; error: -; workDir: /make_lastz_chains/temp_lastz_run/work/c2/6dbd2f55240d4986404c6f5a503940]
~> TaskHandler[id: 8238; name: execute_jobs (8238); status: RUNNING; exit: -; error: -; workDir: /make_lastz_chains/temp_lastz_run/work/c2/6dbd2f55240d4986404c6f5a503940]
~> TaskHandler[id: 8238; name: execute_jobs (8238); status: RUNNING; exit: -; error: -; workDir: /make_lastz_chains/temp_lastz_run/work/c2/6dbd2f55240d4986404c6f5a503940]
~> TaskHandler[id: 8238; name: execute_jobs (8238); status: RUNNING; exit: -; error: -; workDir: /make_lastz_chains/temp_lastz_run/work/c2/6dbd2f55240d4986404c6f5a503940]
~> TaskHandler[id: 8238; name: execute_jobs (8238); status: RUNNING; exit: -; error: -; workDir: /make_lastz_chains/temp_lastz_run/work/c2/6dbd2f55240d4986404c6f5a503940]
~> TaskHandler[id: 8238; name: execute_jobs (8238); status: RUNNING; exit: -; error: -; workDir: /make_lastz_chains/temp_lastz_run/work/c2/6dbd2f55240d4986404c6f5a503940]
~> TaskHandler[id: 8238; name: execute_jobs (8238); status: RUNNING; exit: -; error: -; workDir: /make_lastz_chains/temp_lastz_run/work/c2/6dbd2f55240d4986404c6f5a503940]
~> TaskHandler[id: 8238; name: execute_jobs (8238); status: RUNNING; exit: -; error: -; workDir: /make_lastz_chains/temp_lastz_run/work/c2/6dbd2f55240d4986404c6f5a503940]
~> TaskHandler[id: 8238; name: execute_jobs (8238); status: RUNNING; exit: -; error: -; workDir: /make_lastz_chains/temp_lastz_run/work/c2/6dbd2f55240d4986404c6f5a503940]
~> TaskHandler[id: 8238; name: execute_jobs (8238); status: RUNNING; exit: -; error: -; workDir: /make_lastz_chains/temp_lastz_run/work/c2/6dbd2f55240d4986404c6f5a503940]
~> TaskHandler[id: 8238; name: execute_jobs (8238); status: RUNNING; exit: -; error: -; workDir: /make_lastz_chains/temp_lastz_run/work/c2/6dbd2f55240d4986404c6f5a503940]
~> TaskHandler[id: 8238; name: execute_jobs (8238); status: RUNNING; exit: -; error: -; workDir: /make_lastz_chains/temp_lastz_run/work/c2/6dbd2f55240d4986404c6f5a503940]
~> TaskHandler[id: 8238; name: execute_jobs (8238); status: RUNNING; exit: -; error: -; workDir: /make_lastz_chains/temp_lastz_run/work/c2/6dbd2f55240d4986404c6f5a503940]

Thanks!

[MichaelHiller]

Hi Scott,

aligning primates should definitely work.
Pls check whether ref and queries are both independently repeatModeled and repeatMasked.
We also have alignments with hg38 as a ref for a lot of species, including publicly-available primates (happy to share).

Chaining should only start once all local alis are done, as the chains will otherwise be incomplete.
From the logging, it looks like many jobs succeeded.
Could you pls dig into the dir to find more info on which jobs crashed why? Could be a runtime limit on your scheduler (some lastz jobs take a long time). Sometimes we also get out of mem.

Ways to fix that is

• breaking the genome into even smaller chunks
• adding WindowMasking to the softmask. This could be relevant for very complete (long-read) genomes where centromers are well assembled. They often get only partially masked by RepeatMasker. WM does a better job here.

[fergsc]

Hi Michael,

I am currently testing make lastz chains on some T2T primates, downloaded from UCSC, I believe/hope the repeat masking on them is suitable. I will be running lastz on my own genomes, when masking on them is finished.

I am currently chunking at 10 Mbp with 10 Kbp of overlap. I don't expect these genomes to hit the lastz index limit, but I just want to be able to recognise if they do. If it's nextflow time/RAM issue that's great, I can increase those.

Looking at the logs, I think it is nextflow time/RAM.

temp_lastz_run/work/ad/b8f0216e629fe82a46a5ea639e4753/.exitcode
0

these next files are all empty

temp_lastz_run/work/ad/b8f0216e629fe82a46a5ea639e4753/.command.err
temp_lastz_run/work/ad/b8f0216e629fe82a46a5ea639e4753/.command.log
temp_lastz_run/work/ad/b8f0216e629fe82a46a5ea639e4753/.command.out

example of /make_lastz_chains/temp_lastz_run/.nextflow.log

Aug-10 18:48:04.625 [Task submitter] INFO  nextflow.Session - [ad/b8f021] Submitted process > execute_jobs (196)
~> TaskHandler[id: 196; name: execute_jobs (196); status: RUNNING; exit: -; error: -; workDir: /make_lastz_chains/temp_lastz_run/work/ad/b8f0216e629fe82a46a5ea639e4753]
~> TaskHandler[id: 196; name: execute_jobs (196); status: RUNNING; exit: -; error: -; workDir: /make_lastz_chains/temp_lastz_run/work/ad/b8f0216e629fe82a46a5ea639e4753]
~> TaskHandler[id: 196; name: execute_jobs (196); status: RUNNING; exit: -; error: -; workDir: /make_lastz_chains/temp_lastz_run/work/ad/b8f0216e629fe82a46a5ea639e4753]
~> TaskHandler[id: 196; name: execute_jobs (196); status: RUNNING; exit: -; error: -; workDir: /make_lastz_chains/temp_lastz_run/work/ad/b8f0216e629fe82a46a5ea639e4753]
Aug-10 19:11:23.790 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 196; name: execute_jobs (196); status: COMPLETED; exit: 0; error: -; workDir: /make_lastz_chains/temp_lastz_run/work/ad/b8f0216e629fe82a46a5ea639e4753]

I think that nextflow is retrying the jobs with increased time and RAM, and finishing successfully.

Thanks for the tip of WindowMasker and the quick response.

Scott.

[MichaelHiller]

Looks all fine to me.
Just a note. I downloaded the repeatMask that UCSC created for 2 non-primate mammals (I think a hippopotamus and another one) and that resulted in very very large chains, indicative of many unmasked repeats.
I then did repeatModeler and masking on our system and got a chain file much smaller, more comparable to what you would expect based on related species. We couldn't figure out where the difference is coming from as UCSC seem to use the same RepeatModeler version.

Lets try WM first. If that doesn't do it, send me one of these primates (if they are on NCBI already, I can fetch it) and I can send you my repeatMask. Then we can check if that works better.

[fergsc]

I think your right in your assessment of the ucsc masking, my jobs timed out after 80 hours on 56 cpus. And many jobs were running for 3+ hours.

Some of my genomes have finished repeatmodeler and masker. I will run a few test alignements over the weekend, give them a longer runtime. And set up WM as well.

For our "real" TE annotations we are using https://github.com/unavailable-2374/Pan_TE Do you have any opions on this tool? I guess I should run a test on these maskings.

When you use WM you run both WM and repeatmodeler+masker seperatly and merge the beds?

[MichaelHiller]

We never tested Pan_TE.
Pls merge the RM and WM mask, as just WM doesn't do it (typically catches just similar parts of TEs, leaving other parts unmasked.

[ohdongha]

Hello Scott and Michael,

I saw WM is mentioned. If WM is windowmasker, you could mask the genome "more aggressively" than the default by adjusting these parameters:

$ windowmasker -help
...
 -t_high_pct <Real, 0..100>
   maximum useful unit score as percentage
 -t_thres_pct <Real, 0..100>
   window score threshold used to trigger masking as percentage
 -t_extend_pct <Real, 0..100>
   window score above which it is allowed to extend masking as percentage
...

As far as I understand, windowmasker first creates a "stat" file that contains scores based on the frequency of Nmer in the genome. Then, it scans the genome and masks if the window it is looking at has scores >=99.5 percentile (i.e., -t_thresh_pct 99.5) by default (and extends masking if it does not go below 99.0 percentile) (ref). The genome FASTAs available on NCBI FTP are already soft-masked in this way.

If you want to mask more than the default, these default parameters (=99.5p) could be lowered to, e.g., 98.5p, 97.5p, and so on. Below is the percentage of a few human "T2T" (or "T2T-ish") genomes masked with decreasing -t_thresh_pct values:

#AssmAcc         genome_size  default  WMt98.5p  WMt97.5p  WMt96.5p
GCA_018852605.3  2947895164   39.82    46.23     51.84     53.06
GCA_018852615.3  3051529554   39.13    46.29     49.01     52.33
GCF_009914755.1  3117275501   40.27    47.46     51.95     53.82

As we decrease the -t_thresh_pct values, fewer additional percentages of the genomes are masked. For example, "WMt97.5p" -> "WMt96.5p" masked only a few more percent. But masking those additional a few percent of the genome often substantially reduces the number of make_lastz_chains alignments without losing the alignment coverage of coding regions (which are more likely conserved) by much.

For aligning mammalian-size genomes, "WMt97.5p" alone worked well for distances similar to or beyond human vs. marmoset (e.g., human vs. more distant primates or non-primate mammals). "WMt96.5p" may be needed for particularly large or repetitive genomes.

For very close pairs, e.g., within the same species, human vs. chimpanzee or other apes, etc., I wondered if the sensitivity of make_lastz_chains could be attenuated a bit. Or masking WM combined with Repeat Modeler/Masker may do the trick - I haven't tried this option.

...

You can create the windowmasker stat file and mask the genome with different thresholds, as in this example:

## Pass 1 to create windowmasker "stat" file:
windowmasker -in ${genome.fa} -mk_counts -smem 1024 -out ${windowmasker_pct.stat}

## Pass 2 to (soft-)mask the genome with "WMt97.5p" setting:
windowmasker -in ${genome.fa} -t_thres_pct 97.5 -t_high_pct 97.8 -t_extend_pct 97.0 -ustat ${windowmasker_pct.stat} -outfmt fasta -out ${genome_masked_WMt97.5p}.fa 

I think the default values for -t_thres_pct, -t_high_pct, and -t_high_extend are 99.5, 99.8, and 99.0, respectively. As we move -t_thres_pct, the other two parameters can be adjusted accordingly.

...

Older windowmasker versions may not have these parameters - instead, they have -t_high, -t_thres, and -t_extend parameters that accept integer values. In this case, you will have to look into the "stat" file created at the Pass 1 to find out score values that correspond to the percentile values you want to use.

Cheers,
Dong-Ha

[simone-says]

@ohdongha thanks for sharing this! Do you know where the newer version of windowmasker can be found? I can only seem to locate the binary for v1.0.0 from NCBI's toolkit.

[MichaelHiller]

probably https://github.com/goeckslab/WindowMasker ?

[ohdongha]

@ohdongha thanks for sharing this! Do you know where the newer version of windowmasker can be found? I can only seem to locate the binary for v1.0.0 from NCBI's toolkit.

@simone-says I am sorry that I have missed this post.

The NCBI C++ toolkit includes the windowmasker source code with the -t_thres_pct and related options included. You can compile it following these instructions:
https://www.ncbi.nlm.nih.gov/IEB/ToolBox/CPP_DOC/lxr/source/src/app/winmasker/README.build
https://www.ncbi.nlm.nih.gov/IEB/ToolBox/CPP_DOC/lxr/source/src/app/winmasker/README

Or NCBI's annotation tool, EGAPx, also includes windowmasker in its image.

...
[Edit:] I think NCBI's BLAST+ package also has it.
https://anaconda.org/channels/bioconda/packages/blast/overview

Please check windowmasker -help and see if -t_thres_pct, -t_high_pct, -t_low_pct, and -t_extend_pct are available. Even when the version is 1.0.0, a newer build will have these options.

[ohdongha]

[Edit:] I think NCBI's BLAST+ package also has it.

@simone-says I checked the BLAST+ package, and it does have the updated windowmasker binary. For Linux, you can download the tarball and unpack:
https://ftp.ncbi.nlm.nih.gov/blast/executables/LATEST/ncbi-blast-2.17.0+-x64-linux.tar.gz

The /bin folder includes the updated windowmasker binary:

$ ncbi-blast-2.17.0+/bin/windowmasker -version
windowmasker: 1.0.0
 Package: blast 2.17.0, build Jul  1 2025 08:59:18

...
[Edit:] If you want a standalone binary, try this link (rename to remove the date suffix and make it executable):
https://ftp.ncbi.nlm.nih.gov/pub/agarwala/windowmasker/winmasker.2026.april.02