Skip to content

Fix cell-family detection on empty MPI ranks - #691

Merged
lmoresi merged 1 commit into
underworldcode:developmentfrom
gthyagi:bugfix/collective-empty-rank-cell-family
Sep 6, 2026
Merged

Fix cell-family detection on empty MPI ranks#691
lmoresi merged 1 commit into
underworldcode:developmentfrom
gthyagi:bugfix/collective-empty-rank-cell-family

Conversation

@gthyagi

@gthyagi gthyagi commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Summary

Determine a distributed mesh's cell family collectively when some MPI ranks own no cells.

DMPlexIsSimplex() is rank-local. On an empty rank it reports False, while populated ranks correctly report the actual family. UW3 then constructed different coordinate finite elements on different ranks. That desynchronised PETSc communication and caused a later labeled HDF5 mesh load to hang in PetscSFSetUp_Basic.

Change

  • Read the local simplex result on each rank.
  • Contribute None from ranks with no cells.
  • Collect the populated-rank results and use one cell family on every rank.
  • Use that collective result when selecting triangle, tetrahedron, quadrilateral, or hexahedron element metadata.
  • Document the distributed construction invariant.

The serial/populated-rank behavior is unchanged.

Regression

tests/parallel/test_0781_empty_rank_mesh_sequence.py constructs one simplex or tensor cell in 2-D and 3-D on eight ranks, forcing seven empty partitions. It verifies:

  • identical mesh.isSimplex values on all ranks;
  • identical element-family metadata on all ranks;
  • successful construction and integration on a second labeled HDF5-backed mesh, reproducing the sequence that previously hung.

Validation

mpirun -np 8 python -m pytest --with-mpi \
  tests/parallel/test_0781_empty_rank_mesh_sequence.py -v

4 passed per rank in about 23 s

This is split from #689 so the generic distributed-mesh correction can be reviewed and merged independently of Eulerian SUPG transport.

Underworld development team with AI support from Claude Code

DMPlexIsSimplex returns false on an empty partition. Using that local value for coordinate FE construction mixed simplex and tensor bases across the same communicator, consuming different PETSc message tags. A subsequent mesh HDF5 labelsLoad then blocked in PetscSFSetUp_Basic/MPI_Waitall.

Gather cell-family decisions from populated ranks before FE setup and use the same classification for element metadata. Preserve the constructor hint for an entirely empty mesh; do not change SUPG algorithms, solver tolerances or MPI providers.

Add single-cell triangle/tet/quad/hex regressions that require empty ranks and subsequently load another mesh, checking P2 volume and boundary integrals. The triangle regression failed before the fix. The original SUPG migration/partition sequence plus all four regressions passed on eight Mac ranks: 15 tests in 26.05 s, 32.30 s including launcher, 2.74 GiB peak process-tree RSS. The pre-fix sequence hung and the mesh-only reproducer also hung. Gadi rerun remains pending.
@gthyagi

gthyagi commented Sep 6, 2026

Copy link
Copy Markdown
Contributor Author

This independent MPI mesh fix is now a prerequisite for completing the Eulerian SUPG integration stack:

  1. merge Fix cell-family detection on empty MPI ranks #691 and Make mesh.cell_size() local and partition independent #692 into development;
  2. sync feature/eulerian-supg-transport with updated development;
  3. refresh and validate Integrate CitcomS predictor-corrector into unified Eulerian SUPG #689 against that feature branch.

#691 itself remains one focused commit, has no outstanding review comments, and both current CI checks pass. Review is the remaining merge gate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants