wetlands_CplHist_377_ne30_BgcCrop_178_[AD, SASU, pSASU]

[wwieder]

Description:

Land model spinup using CPL_HIST data from cesm3_dev run 377.
Branch tag

---

Case directory:
Locally:

/glade/work/wwieder/clm6_cases/wetlands_CplHist_377_ne30_BgcCrop_178_AD
/glade/work/wwieder/clm6_cases/wetlands_CplHist_377_ne30_BgcCrop_178_SASU
/glade/work/wwieder/clm6_cases/wetlands_CplHist_377_ne30_BgcCrop_178_pSASU

working tree created from branch tag with

git worktree add -b wetlands_ctsm5.4.054_cplhist ../wetlands_ctsm5.4.054_cplhist branch_tags/wetlands.n05.ctsm5.4.054
cd /glade/u/home/wwieder/wetlands_ctsm5.4.054_cplhist
./bin/git-fleximod update

git worktree add -b ctsm5.4.054_cplhist_branch ../ctsm5.4.054_cplhist_branch branch_tags/cdeps_updates_for_cplhist.n01.ctsm5.4.054

---

Sandbox:
Locally:
/glade/u/home/wwieder/wetlands_ctsm5.4.054_cplhist

./create_newcase --res ne30pg3_t233 --compset I1850Clm60BgcCropSpinup --case /glade/work/wwieder/clm6_cases/wetlands_CplHist_377_ne30_BgcCrop_178_AD --project CESM0023 --run-unsupported

./create_clone --clone /glade/work/wwieder/clm6_cases/wetlands_CplHist_377_ne30_BgcCrop_178_AD --case /glade/work/wwieder/clm6_cases/wetlands_CplHist_377_ne30_BgcCrop_178_SASU --keepexe

./create_clone --clone /glade/work/wwieder/clm6_cases/wetlands_CplHist_377_ne30_BgcCrop_178_AD --case /glade/work/wwieder/clm6_cases/wetlands_CplHist_377_ne30_BgcCrop_178_pSASU

---

user_nl_ changes:

---

SourceMods:

1. AD

./xmlchange CLM_ACCELERATED_SPINUP=on,MOSART_MODE=NULL
./xmlchange PROJECT=CESM0023,CHARGE_ACCOUNT=P93300041,JOB_PRIORITY=special --force 
./xmlchange DATM_CPLHIST_CASE=b.e30_alpha09d_m.B1850C_MTso_Gris_Marbl.ne30_t233_wgx3.377
./xmlchange DATM_CPLHIST_DIR='/glade/derecho/scratch/hannay/archive/$DATM_CPLHIST_CASE/cpl/hist'
./xmlchange DATM_YR_START=132
./xmlchange DATM_YR_END=151

./xmlchange LND_NTASKS=-24
./xmlchange CPL_NTASKS=-24
./xmlchange ROF_NTASKS=-24
./xmlchange ICE_NTASKS=-24
./xmlchange GLC_NTASKS=-24
./xmlchange WAV_NTASKS=-24
./xmlchange OCN_NTASKS=-24

./xmlchange CONTINUE_RUN=FALSE,STOP_OPTION=ndays,STOP_N=31
./xmlchange --subgroup case.run JOB_WALLCLOCK_TIME=00:20:00

./xmlchange CONTINUE_RUN=FALSE,STOP_OPTION=nyears,STOP_N=100,RESUBMIT=3
./xmlchange --subgroup case.run JOB_WALLCLOCK_TIME=12:00:00

2. SASU

echo 'dribble_crophrv_xsmrpool_2atm = .false.' >> user_nl_clm

./xmlchange MOSART_MODE=NULL
./xmlchange RUN_TYPE=hybrid
./xmlchange STOP_N=1,STOP_OPTION=nmonths
./xmlchange CLM_ACCELERATED_SPINUP=sasu,CLM_FORCE_COLDSTART=off
./xmlchange CONTINUE_RUN=FALSE,RESUBMIT=0

export REFCASE=wetlands_CplHist_377_ne30_BgcCrop_178_AD
./xmlchange GET_REFCASE=TRUE
./xmlchange RUN_REFDATE=0401-01-01
./xmlchange DRV_RESTART_POINTER=rpointer.cpl.0401-01-01-00000
./xmlchange RUN_REFCASE=$REFCASE
./xmlchange RUN_REFDIR=/glade/derecho/scratch/wwieder/archive/$REFCASE/rest/0401-01-01-00000
./xmlchange JOB_PRIORITY=premium
./xmlchange --subgroup case.run JOB_WALLCLOCK_TIME=00:20:00

./xmlchange STOP_N=100,STOP_OPTION=nyears
./xmlchange CONTINUE_RUN=FALSE,RESUBMIT=3
./xmlchange --subgroup case.run JOB_WALLCLOCK_TIME=12:00:00

3. pSASU

echo 'histaux_l2x1yrg = .true.' >> user_nl_cpl
./xmlchange RUN_TYPE=hybrid
./xmlchange CLM_ACCELERATED_SPINUP=off,CLM_FORCE_COLDSTART=off
./xmlchange GET_REFCASE=TRUE
export REFCASE=wetlands_CplHist_377_ne30_BgcCrop_178_SASU
./xmlchange RUN_REFDATE=0401-01-01
./xmlchange DRV_RESTART_POINTER=rpointer.cpl.0401-01-01-00000
./xmlchange RUN_REFCASE=$REFCASE
./xmlchange RUN_REFDIR=/glade/derecho/scratch/wwieder/archive/$REFCASE/rest/0401-01-01-00000
./xmlchange JOB_PRIORITY=premium
./xmlchange MOSART_MODE=NULL
./xmlchange STOP_N=80,STOP_OPTION=nyears
./xmlchange CONTINUE_RUN=FALSE,RESUBMIT=2
./xmlchange --subgroup case.run JOB_WALLCLOCK_TIME=12:00:00

---

Diagnostics:
Diags (if available)

I_CplHist_178 vs. B_377
I_CphHIST_178 vs. I_124, the most recent ne30 case we have pSASU results archived for.

---

Output:
Output (if still available):
/glade/scratch/wwieder/archive/$CASE/

also archived on campaign
/glade/campaign/cgd/tss/common/Land_Only_Simulations/CESM3_CLM6/wetlands_CplHist_377_ne30_BgcCrop_178_pSASU

---

Contacts:
@wwieder

---

Extra details:

[wwieder]

AD equilibrium:

(0)	=======================================================================
(0)	EQUILIBRIUM SUMMARY
(0)	=======================================================================
(0)	TOTECOSYSC is in equilibrium. Eq. Yr. = 202
(0)	TOTSOMC is in equilibrium. Eq. Yr. = 122
(0)	TOTVEGC is in equilibrium. Eq. Yr. = 142
(0)	TLAI is in equilibrium. Eq. Yr. = 42
(0)	GPP is in equilibrium. Eq. Yr. = 122
(0)	TWS is in equilibrium. Eq. Yr. = 202
(0)	H2OSNO is in equilibrium. Eq. Yr. = 202
(0)	At least 97 percent of the land surface is in TOTECOSYSC equilibrium. Eq. Yr. = 302
(0)	Percent of the land surface not in equilibrium (  1.83% )

[wwieder]

SASU run is crashing with a C balance warning after ~18 months. As far as I can tell this is on a crop landunit in the tropics (see below).

The case has failed twice now it the same spot.
Any suggestions @olyson, @slevis-lmwg, @ekluzek?

@dlawrenncar, this will delay the land model spinup a bit.

dec0496.hsn.de.hpc.ucar.edu 1995:  cbalance warning at c =          18  7.114417033740536E-004
dec0496.hsn.de.hpc.ucar.edu 1995:    5702.21044018915
dec0496.hsn.de.hpc.ucar.edu 1995:  column cbalance error    =   7.114417033740536E-004          18
dec0496.hsn.de.hpc.ucar.edu 1995:  is fates column?         =  F
dec0496.hsn.de.hpc.ucar.edu 1995:  Latdeg,Londeg=  -7.37580627510179        280.500000000000
dec0496.hsn.de.hpc.ucar.edu 1995:  begcb                    =    5702.21718595468
dec0496.hsn.de.hpc.ucar.edu 1995:  endcb                    =    5702.21044018915
dec0496.hsn.de.hpc.ucar.edu 1995:  delta store              =  -6.745765530467907E-003
dec0496.hsn.de.hpc.ucar.edu 1995:  --- Inputs ---
dec0496.hsn.de.hpc.ucar.edu 1995:  gpp                      =   0.000000000000000E+000
dec0496.hsn.de.hpc.ucar.edu 1995:  --- Outputs ---
dec0496.hsn.de.hpc.ucar.edu 1995:  er                       =   1.559041457422894E-002
dec0496.hsn.de.hpc.ucar.edu 1995:  col_fire_closs           =   0.000000000000000E+000
dec0496.hsn.de.hpc.ucar.edu 1995:  col_hrv_xsmrpool_to_atm  =   0.000000000000000E+000
dec0496.hsn.de.hpc.ucar.edu 1995:  col_xsmrpool_to_atm      =  -9.556090747057870E-003
dec0496.hsn.de.hpc.ucar.edu 1995:  wood_harvestc            =   0.000000000000000E+000
dec0496.hsn.de.hpc.ucar.edu 1995:  crop_harvestc_to_cropprodc =   0.000000000000000E+000
dec0496.hsn.de.hpc.ucar.edu 1995:  -1*som_c_leached         =   7.722170963320667E-014
dec0496.hsn.de.hpc.ucar.edu 1995: iam = 1867: local  column   index = 18
dec0496.hsn.de.hpc.ucar.edu 1995: iam = 1867: global column   index = 34362
dec0496.hsn.de.hpc.ucar.edu 1995: iam = 1867: global landunit index = 16070
dec0496.hsn.de.hpc.ucar.edu 1995: iam = 1867: global gridcell index = 8012
dec0496.hsn.de.hpc.ucar.edu 1995: iam = 1867: landunit type         = 2
dec0496.hsn.de.hpc.ucar.edu 1995: iam = 1867: column   type         = 267
dec0496.hsn.de.hpc.ucar.edu 1995: iam = 1867: gridcell longitude    =  280.5000000
dec0496.hsn.de.hpc.ucar.edu 1995: iam = 1867: gridcell latitude     =   -7.3758063
dec0496.hsn.de.hpc.ucar.edu 1995: WARNING: Trouble getting the 2D gridcell indices
dec0496.hsn.de.hpc.ucar.edu 1995:  ENDRUN:
dec0496.hsn.de.hpc.ucar.edu 1995:  ERROR in CNBalanceCheckMod.F90 at line 384

[ekluzek]

I think this might be something where it just barely exceeded the error threshold. So increase it in the namelist by 10x and then see if it'll run past the problem for a few months. Possibly then it can go back to the normal value and keep running.

That's a good thing to try as a first step. If you can't get past it that way it'll definitely need more work to diagnose. And these things are tricky if it's a legit problem.

Another thing to try would be to see if a spinup case on the branch tag on master sees the same thing. If not it might be in unique to the wetlands branch which we haven't run as much.

I'd also look in the lnd.log file and see if there are a bunch of warnings about this point (or other points) leading up to the crash.

I can get you the namelist thing to # change if you need it. But need to look a bit for it.

Summary:

• Increase C error tolerance by 10x
• See if you can get a few months past the crash
• if you get a few months past continue with the original tolerance
• See if the spinup with the branch tag to master has the same behavior
• Check lnd.log for warnings leading up to the crash

[wwieder]

Thanks for these ideas @ekluzek I'll give this a try.

Just to clarify, I think I need to adjust cerror in the source code. Is this accurate?
https://github.com/ESCOMP/CTSM/blob/dae706a769a04ff13def0a909930c899a265368a/src/biogeochem/CNBalanceCheckMod.F90#L78

Here's my updates (as of Sunday evening)

• I've started a run with this adjusted tolerance using SourceMods in CNBalanceCheckMod for cerror = 1e-6_r8 here /glade/work/wwieder/clm6_cases/wetlands_CplHist_377_ne30_BgcCrop_178_SASUb
• Spinup with the branch tag to master isn't finished with AD spinup yet (running in the regular queue), but I'll check that when it's ready.
• No evidence for warning leading up to the crash in the lnd.log from the *178_SASU case.

[ekluzek]

Yeah you are right the comment says we could add to namelist but we didn't. It also looks like the error is much higher than the tolerance for Carbon as it's at 1e-4 and the error allowed is 1.e-6.

So it will fail again unless you make it much higher, which probably means higher than we'd think acceptable. But still might be good to try, just to see if it'll work.

Without warnings leading up to the fail this seems more likely to be something missing in the balance check. Or something going wrong in the physics. So probably a thing that really needs to be fixed.

If it doesn't show up in the non wetlands branch that would make it likely to be something in the wetlands changes. Which might make it easier to track down.

Shoot. This is a bummer....

[wwieder]

This run also failed at the same spot, which may not be surprising, because the column balance error is 7e-4... happy to try something different, if there are suggestions.

Otherwise, this namelist flag seems to have been added in the wetlands.n05 case, but was not present in the wetlands.n03 case I was an option in #164 (e.g. wetlands.n03.ctsm5.4.029_f09noCrop_noN_flat_164_SASU)
dribble_crophrv_xsmrpool_2atm = .true.

I'll also try setting this to false to see if that helps, as this is a crop column and there seems to be a resonably large xsmr flux, based on the errors pasted above

[wwieder]

Don't understad why, but setting dribble_crophrv_xsmrpool_2atm = .false. prevents the crash in *178_SASU

[ekluzek]

The description in namelist_definition for that is:

Harvest the XSMR pool at crop harvest time to the atmosphere slowly at an exponential rate

This came in as TRUE for all physics options in: ctsm5.1.dev008

And was for ESCOMP/CTSM#1158. The recommendation at the time was to remove the possibility of turning it off, as we are doing here.

[wwieder]

Regarding dribble_crophrv_xsmrpool_2atm, I made no change, using the default .true. setting in both AD and pSASU cases with no issues.

The only crash came when running with SASU, which makes me wonder if this flux wasn't translated correctly into the matrix code? I'll create a CTSM side issue for this.