TaskVine at the LPC

[cmoore24-24]

Hello! I'm opening a topic that is about getting something to run smoothly at the FNAL LPC.

I was able to successfully run a small-scale application at the LPC with taskvine, which read input files from Notre Dame via XRootD. Here's some information about that run:

I've attachd a tar file with the vine logs from the run. I've also includes the print out of the warning I mentioned during the meeting (below). I've also included a text file that has the output of the factory associated with the run. Perhaps notable is the text error that appears which I've never seen on Notre Dame, but it didn't stop anything from working. I've also included a screenshot of the manager from the catalog, which does show a fair few failed tasks.

2026[/02/26](http://localhost:8126/02/26) 20:19:25.46 vine_manager[1846253]notice: Releasing worker source because of repeated 131.225.204.179:51460 transfer failures: 5[/5](http://localhost:8126/5)
2026[/02/26](http://localhost:8126/02/26) 20:19:33.74 vine_manager[1846253]notice: Releasing worker source because of repeated 131.225.207.249:39316 transfer failures: 5[/5](http://localhost:8126/5)
2026[/02/26](http://localhost:8126/02/26) 20:19:42.36 vine_manager[1846253]notice: Releasing worker source because of repeated 131.225.204.179:47488 transfer failures: 5[/5](http://localhost:8126/5)
2026[/02/26](http://localhost:8126/02/26) 20:19:45.93 vine_manager[1846253]notice: Releasing worker source because of repeated 131.225.204.154:44056 transfer failures: 5[/5](http://localhost:8126/5)
2026[/02/26](http://localhost:8126/02/26) 20:19:45.93 vine_manager[1846253]notice: Releasing worker destination because of repeated 131.225.207.249:54922 transfer failures: 10[/10](http://localhost:8126/10)

successful_lpc_logs.tar.gz

[JinZhou5042]

Hello @cmoore24-24, we're glad to hear that your recent run at the LPC with TaskVine was successful, and we appreciate you sharing the detailed description of your experience along with these very helpful materials.

Here are some observations after taking a look at the debug log file -

• Although the workflow completed successfully, the worker concurrency is quite poor. It appears that the manager repeatedly connects to new workers, while each worker only lasts for a very short period of time. As a result, the workflow falls into a loop where workers are continuously connected and then removed.

334 destination and 876 source workers were removed during the run:

(dagvine-env) [jzhou24@condorfe vine-logs]$ grep "Releasing worker destination because of repeated" debug | wc -l
334
(dagvine-env) [jzhou24@condorfe vine-logs]$ grep "Releasing worker source because of repeated" debug | wc -l
876

• There were in total 6127 peer transfers initialized, only 315 of them succeeded, the rest 5812 failed.

(dagvine-env) [jzhou24@condorfe vine-logs]$ grep puturl debug | wc -l
6127

(dagvine-env) [jzhou24@condorfe vine-logs]$ bash show_good_puturl.sh
2026/02/26 20:34:57.13 vine_manager[1846253]vine: tx to cjmoore-2119001-0-cmswn2238.fnal.gov (131.225.204.218:54268): puturl workerip://131.225.204.218:1025/file-meta-904172a2877dba43c2b74e4e5d251f9b file-meta-904172a2877dba43c2b74e4e5d251f9b 0 17062 0755 0013CA22-8006-4248-9904-8D7135D212E2
2026/02/26 20:25:13.78 vine_manager[1846253]vine: tx to cjmoore-58309770-0-cmswn2238.fnal.gov (131.225.204.218:35564): puturl workerip://131.225.204.218:1026/file-meta-b8d864c571a0e8fc97a477ac4a7ef284 file-meta-b8d864c571a0e8fc97a477ac4a7ef284 0 227 0755 00342E34-AB56-463F-B405-094D67496FA2
2026/02/26 20:36:35.03 vine_manager[1846253]vine: tx to cjmoore-58309768-0-cmswn2238.fnal.gov (131.225.204.218:43684): puturl workerip://131.225.204.218:1025/file-meta-b8d864c571a0e8fc97a477ac4a7ef284 file-meta-b8d864c571a0e8fc97a477ac4a7ef284 0 227 0755 00830460-0CF0-4FBD-B201-F0D77901DFF8
......

Total: 315

(dagvine-env) [jzhou24@condorfe vine-logs]$ bash show_bad_puturl.sh
2026/02/26 20:44:27.17 vine_manager[1846253]vine: tx to cjmoore-58309767-0-cmswn2234.fnal.gov (131.225.204.179:51080): puturl workerip://131.225.204.35:1024/file-meta-b8d864c571a0e8fc97a477ac4a7ef284 file-meta-b8d864c571a0e8fc97a477ac4a7ef284 0 227 0755 0003BC04-10C3-4503-8155-796968EA1D47
2026/02/26 20:33:58.41 vine_manager[1846253]vine: tx to cjmoore-58309772-0-cmswn2260.fnal.gov (131.225.204.35:39320): puturl workerip://131.225.207.197:1024/file-meta-b8d864c571a0e8fc97a477ac4a7ef284 file-meta-b8d864c571a0e8fc97a477ac4a7ef284 0 227 0755 0003BC18-1275-426F-90F3-275A91BED15B
2026/02/26 20:33:34.55 vine_manager[1846253]vine: tx to cjmoore-58309766-0-cmswn2215.fnal.gov (131.225.207.249:35722): puturl workerip://131.225.204.218:1025/file-meta-904172a2877dba43c2b74e4e5d251f9b file-meta-904172a2877dba43c2b74e4e5d251f9b 0 17062 0755 00045680-CECB-4AB4-99F8-804276486325
......

Total: 5812

• Key insight: all successful puturl (peer-to-peer) transfers occurred on the same node (seems only on node 131.225.204.218), meaning the source and destination workers shared the same IP address, whereas the failed transfers were cross-node.

Here is the file which records the succeeded puturl transfers: https://drive.google.com/file/d/14et5oNW6YY3RkIM1fPtN1G1JjQkPuojA/view?usp=drive_link
And this file records the failed transfers: https://drive.google.com/file/d/1QT_MDKiqEPjRijDwybjbySjbV8yzhgPu/view?usp=drive_link

That said, my hypothesis is that cross-node transfers are blocked on the LPC cluster, or that the node 131.225.204.218 has less restrictions on same-node transfer. In other words, you can’t simply bind an arbitrary port and open a connection to a remote node, and only same-node transfers are allowed. I’m not sure what the right fix is yet, but it may come down to a cluster-specific network restriction.

[dthain]

We have learned that internal communications in the Fermi LPC also need to be limited to ports 10000-11000. @JinZhou5042 can you arrange to add a ‘—transfer-port’ option to the worker (and then the factory) to make this work?

[JinZhou5042]

This sounds doable, will work on it and propose a PR!

[JinZhou5042]

Supported in #4375