Address Issues with Face Selectors

[Thomas Bendall (tommbendall)]

PR Summary

Sci/Tech Reviewer: iboutle
Code Reviewer: Yaswant Pradhan (@yaswant)

This is the Apps part of work to revamp the face_selector infrastructure, which ensure that when looping over faces within columns, faces are only looped over once. This PR fixes a number of flaws and deficiencies with them. It does this by changing how the face_selector fields are computed, and changing the values they can take.

Kernels with this data access pattern that were previously not using the face selectors now are, which is a modest performance improvement.

In particular:

• columns previously always iterated over W and S faces. Now they can just iterate over N, E or no faces if necessary
• every face will now only be iterated over once, irrespective of the MPI decomposition (there were previously decompositions in which faces were iterated over twice)

This fixes a number of problems:

• as recorded in The combination of threading and MPI regions that cross cubed-sphere panel edges is not bit-reproducible #294, the combination of threading with 3-panel cubed-sphere decompositions was not bit-reproducible
• it may now be possible to run safely without annexed DoFs (addressing the above issue was necessary if not sufficient)
• the face_selector fields now take sensible values in halos, which may allow some kernels to used with redundant computation

Linked-To

• LFRic-Core #349

Details

• kernels which previously used the face_selectors are updated to use their new form
• the face_selectors have been added to a series of kernels (mainly physics kernels) that have the same access pattern of writing to a face from only one kernel
• as The combination of threading and MPI regions that cross cubed-sphere panel edges is not bit-reproducible #294 has been fixed, this PR reinstates threading to the baroclinic wave 3-panel test
• KGOs are updated for many tests, but the science change is neutral

Documentation

See the linked core PR for documentation

Results

Here are PDFs containing plots from the test suite showing the neutral scientific impact:

• gungho:
  results_face_selector_gungho.pdf

• lfric_atm:
  results_face_selector_lfric_atm.pdf

There should be a modest reduction in time by using this scheme in the boundary layer. Here are timings from differnt tests in the lfric_atm_weekly suite (just from a single run, so subject to variability) that demonstrate this:

timestepping	vn3.1	Branch
UKV	312.62	308.01
C48 climate	333.72	337.87
C224	221.83	220.03
C896	416.21	415.47

Code Quality Checklist

• I have performed a self-review of my own code
• My code follows the project's style guidelines
• Comments have been included that aid understanding and enhance the readability of the code
• My changes generate no new warnings
• All automated checks in the CI pipeline have completed successfully

Testing

• I have tested this change locally, using the LFRic Apps rose-stem suite
• If any tests fail (rose-stem or CI) the reason is understood and acceptable (e.g. kgo changes)
• I have added tests to cover new functionality as appropriate (e.g. system tests, unit tests, etc.)
• Any new tests have been assigned an appropriate amount of compute resource and have been allocated to an appropriate testing group (i.e. the developer tests are for jobs which use a small amount of compute resource and complete in a matter of minutes)

trac.log

Test Suite Results - lfric_apps - face_selector_apps/run6

Suite Information

Item	Value
Suite Name	face_selector_apps/run6
Suite User	thomas.bendall
Workflow Start	2026-05-18T10:42:54
Groups Run	all

Dependency	Reference	Main Like
casim	MetOffice/casim@2026.03.2	True
jules	MetOffice/jules@2026.03.2	True
lfric_apps	tommbendall/lfric_apps@TBendall/face_selectors	False
lfric_core	tommbendall/lfric_core@TBendall/face_selectors	True
moci	MetOffice/moci@2026.03.2	True
SimSys_Scripts	MetOffice/SimSys_Scripts@2026.03.2	True
socrates	MetOffice/socrates@2026.03.2	True
socrates-spectral	MetOffice/socrates-spectral@2026.03.2	True
ukca	MetOffice/ukca@2026.03.2	True

Task Information

✅ succeeded tasks - 1511

Security Considerations

• I have reviewed my changes for potential security issues
• Sensitive data is properly handled (if applicable)
• Authentication and authorisation are properly implemented (if applicable)

Performance Impact

• Performance of the code has been considered and, if applicable, suitable performance measurements have been conducted

AI Assistance and Attribution

• Some of the content of this change has been produced with the assistance of Generative AI tool name (e.g., Met Office Github Copilot Enterprise, Github Copilot Personal, ChatGPT GPT-4, etc) and I have followed the Simulation Systems AI policy (including attribution labels)

Documentation

• Where appropriate I have updated documentation related to this change and confirmed that it builds correctly

PSyclone Approval

• If you have edited any PSyclone-related code (e.g. PSyKAl-lite, Kernel interface, optimisation scripts, LFRic data structure code) then please contact the TCD Team

Sci/Tech Review

• I understand this area of code and the changes being added
• The proposed changes correspond to the pull request description
• Documentation is sufficient (do documentation papers need updating)
• Sufficient testing has been completed

(Please alert the code reviewer via a tag when you have approved the SR)

Code Review

• All dependencies have been resolved
• Related Issues have been properly linked and addressed
• CLA compliance has been confirmed
• Code quality standards have been met
• Tests are adequate and have passed
• Documentation is complete and accurate
• Security considerations have been addressed
• Performance impact is acceptable

[DanStoneMO]

Tested this with JEDI and all is good. No linked lfric-jedi PR will be needed.

[iboutle]

Just one thing to look at

[iboutle]

I noticed that the checksums had changed for the coma9 and comorph_dev SCM tests, but none of the other SCM tests, which seemed suspicious - these are the 2 jobs which use the version of this kernel on cell-centres.

The code to account for this does not seem to be working properly. For a 2x2 biperiodic domain (although I would assume the same is true for any biperiodic domain), the face selectors pick all 4 faces of the 1st cell, 2 faces of the 2nd and 3rd cells, and no faces of the final cell. This means that the cell centre code for the final cell also gets missed, because total_faces=0 by this line.

I think a suitable replacement would be:
if (nfaces == 1) then
total_faces = 1
else
total_faces = ABS(face_selector_ew(map_w3_2d(1))) + ABS(face_selector_ns(map_w3_2d(1)))
end if
which would give the correct behaviour for the cell face application of the kernel, but ensure that it is always executed for cell centres.

With this, the KGO for all SCM jobs should then remain unchanged.

[Thomas Bendall (tommbendall)]

Thanks for spotting that! Yes, having fixed that the SCM jobs now don't change KGOs.

[iboutle]

Thanks Tom - out of interest, why does the update to fix this have more KGO changes than just the SCM jobs - were the rest just out-of-date and in need of updating anyway, or does the fix affect the KGO of all job (which I wouldn't have expected)?

[Thomas Bendall (tommbendall)]

This was a good point so I've double-checked this. The KGOs were not up-to-date previously -- I have checked the effect of reverting the change in the previous commit, and only the jobs using Comorph change KGOs, which is as it should be!

[thomasmelvin]

Looks good to me

[iboutle]

I'm now on leave until 26th May - I'm happy for this to progress on to code review before the 22nd deadline when the issue with the SCM KGOs has been resolved - everything else is fine.