MPAS model hangs

[AndersJensen-NOAA]

MPAS hangs randomly (and sometimes not at all) when MPAS global print statements are set to true. The global print statements are config_print_global_minmax_vel, config_print_detailed_minmax_vel, config_print_global_minmax_sca. Additional model hangs have also been reported when setting these global print statements to false.

These print statements use the subroutine mpas_dmpar_maxattributes_real in /src/framework/mpas_dmpar.F, and the subroutine relies on MPI_Allreduce. When using (older) intel compilers, there are reports that MPI_Allreduce and MPI_Allgather can hang.

@SamuelTrahanNOAA recently debugged a hanging version of MPAS and found that one MPI task was stuck in a different part of the code than the other tasks.

Solutions:

1. Set global config print statements to false. This may help, but not completely fix the issue.
2. Use newer intel compilers if possible or gfortran. More testing will follow.

[SamuelTrahanNOAA]

The global MPAS HFIP 2025 runs had a similar problem. My workaround was to automatically detect the hanging job, kill it, and resubmit it. The logic to implement this reliably in mpas_app is somewhat complicated. However, it was a 100% reliable workaround.

[SamuelTrahanNOAA]

I've opened a ticket to the Ursa helpdesk (RDHPCS#2026011454000206) asking for a GNU+OpenMPI build of their parallel-netcdf 1.12.3. Once we have this, we can test without the Intel MPI.

EDIT: The parallel-netcdf version is 1.12.3, not 1.12.4

[SamuelTrahanNOAA]

OpenMPI may not be an option due to this limitation which the developers say won't be fixed:

• TCP BTL tries binding to address already in use open-mpi/ompi#7246

We're investigating using OpenMPI with UCX to bypass that limitation. Otherwise, we may choose a different MPI.

I'm working with the RDHPCS admins to find a way to test this without Intel MPI.

[clark-evans]

I exchanged emails with Craig Schwartz and Falko Judt at NCAR/MMM about this issue, asking if they had encountered anything similar in their global 3-4 km MPAS forecasts. Neither have encountered anything like it.

Falko shared their module set, noting that derecho is cray-based (like gaea, but not ursa):

set modules = ( ncarenv/23.09 \
                craype/2.7.23 \
                intel-oneapi/2023.2.1 \
                ncarcompilers/1.0.0 \
                cray-mpich/8.1.27 \
                hdf5-mpi/1.12.2 \
                netcdf-mpi/4.9.2 \
                parallelio/2.6.2 \
                parallel-netcdf/1.12.3 \
                grib-util/1.2.4 \
                wgrib2/3.1.1 \
                cmake/3.26.3 \
                nco/5.1.6 \
                mkl/2023.2.0 \
                esmf/8.6.0 \
                ncl/6.6.2 )

My recommendation is that we continue different compiler and MPI implementations and also potentially test NCAR (i.e., unmodified) MPAS code. We considered the latter during our 2025 HFIP debugging but did not pursue it once Sam's workaround was in place.

[clark-evans]

@guoqing-noaa reports seeing this issue with the 1200 UTC 16 February 2026 real-time regional 3.5 km atmospheric river forecast on ursa. From Guoqing:

the last few lines of log.atmosphere.out:

 Begin timestep 2026-02-19_06:29:35
 --- time to run the LW radiation scheme L_RADLW = F
 --- time to run the SW radiation scheme L_RADSW = F
 --- time to apply limit to accumulated radiation diags. L_ACRADT   = F
 --- time to calculate additional physics_diagnostics               = F
 --- enter subroutine mynnsfclay_driver:
 --- call subroutine ruc_land:
 --- call subroutine ruc_ice:
 --- call subroutine sfcdiags_ruclsm:
  split dynamics-transport integration 3
 Computing hourly max reflectivity
 Computing accumulated max hail diameter at the surface

  global min w: -8.74820 k=16, 37.8643 lat, -104.980 lon

It looks like the MPAS-Model just hung there since 07:39 unti 10:39 when the job reached the 5-hour walltime limit.

[clark-evans]

@guoqing-noaa was recommended by the ursa helpdesk to try export I_MPI_COLL_INTRANODE=pt2pt, and he reports that setting this environment variable allowed him to overcome a hanging forecast.

The Intel MPI developer reference:

https://cdrdv2-public.intel.com/671270/intelmpi-2018-developer-reference-linux.pdf

Indicates that this environment variable defines the intranode communication type for collective operations. As noted in the first post of this thread, the model has been hanging in collective operations like MPI_Allreduce. The reference specifically calls out switching from the default shared memory collective (shm) to point-to-point collectives (pt2pt) when there are a lot of communicators. That jives with our experience where these hangs occur for particularly resource-intensive forecasts. More testing will ensue.

[guoqing-noaa]

To update, I resumed a backup realtime AR3.5km run on Ursa. So far, it looks like export I_MPI_COLL_INTRANODE=pt2pt helped. In the past 3 days (6 cycles, 00/12z each day), the forecasts completed successfully without hangs.