enso-characteristic diagnostic fails to copy successful output to results directory

## Describe the bug

The `enso-characteristics` diagnostic fails to copy the successful output from `scratch/` to `results/`, when running REF with a preliminary CMIP7 model dataset. Error message below (and attached).

Note that it runs successfully for a CMIP6 dataset (CanESM5-1 p1)

```

2026-08-18 22:20:59.254 +00:00 e0:03:42.870143s | INFO     | SpawnProcess-1:263332 | climate_ref_core.providers:run:642 - Command execution successful
2026-08-18 22:20:59.258 +00:00 e0:03:42.873952s | ERROR    | SpawnProcess-1:263332 | climate_ref_core.logging:redirect_logs:221 - Execution failed
Traceback (most recent call last):

  File "<string>", line 1, in <module>
  File "/fs/site7/eccc/crd/cccma/users/scrd115/code/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/spawn.py", line 122, in spawn_main
    exitcode = _main(fd, parent_sentinel)
               │     │   └ 4
               │     └ 16
               └ <function _main at 0x14695c32e480>
  File "/fs/site7/eccc/crd/cccma/users/scrd115/code/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/spawn.py", line 135, in _main
    return self._bootstrap(parent_sentinel)
           │    │          └ 4
           │    └ <function BaseProcess._bootstrap at 0x14695c393380>
           └ <SpawnProcess name='SpawnProcess-1' parent=247868 started>
  File "/fs/site7/eccc/crd/cccma/users/scrd115/code/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
    │    └ <function BaseProcess.run at 0x14695c3928e0>
    └ <SpawnProcess name='SpawnProcess-1' parent=247868 started>
  File "/fs/site7/eccc/crd/cccma/users/scrd115/code/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    │    │        │    │        │    └ {}
    │    │        │    │        └ <SpawnProcess name='SpawnProcess-1' parent=247868 started>
    │    │        │    └ (<concurrent.futures.process._SafeQueue object at 0x146921a28500>, <multiprocessing.queues.SimpleQueue object at 0x146923162c...
    │    │        └ <SpawnProcess name='SpawnProcess-1' parent=247868 started>
    │    └ <function _process_worker at 0x146938f84040>
    └ <SpawnProcess name='SpawnProcess-1' parent=247868 started>
  File "/fs/site7/eccc/crd/cccma/users/scrd115/code/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/concurrent/futures/process.py", line 264, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
        │         │   │         │       │         └ {'definition': ExecutionDefinition(diagnostic=ENSOCharacteristics(name='ENSO Characteristics'), key='cmip7_g127_CanESM5-1_r10...
        │         │   │         │       └ <concurrent.futures.process._CallItem object at 0x14692145f0e0>
        │         │   │         └ ()
        │         │   └ <concurrent.futures.process._CallItem object at 0x14692145f0e0>
        │         └ <function _process_run at 0x146920e0a5c0>
        └ <concurrent.futures.process._CallItem object at 0x14692145f0e0>

  File "/fs/site7/eccc/crd/cccma/users/scrd115/REF/REF_v0.16.2/climate-ref/packages/climate-ref/src/climate_ref/executor/local.py", line 52, in _process_run
    return execute_locally(definition=definition, log_level=log_level)
           │                          │                     └ 'ERROR'
           │                          └ ExecutionDefinition(diagnostic=ENSOCharacteristics(name='ENSO Characteristics'), key='cmip7_g127_CanESM5-1_r10i1p2f1', datase...
           └ <function execute_locally at 0x1469215b4540>

  File "/fs/site7/eccc/crd/cccma/users/scrd115/REF/REF_v0.16.2/climate-ref/packages/climate-ref-core/src/climate_ref_core/executor.py", line 65, in execute_locally
    with redirect_logs(definition, log_level):
         │             │           └ 'ERROR'
         │             └ ExecutionDefinition(diagnostic=ENSOCharacteristics(name='ENSO Characteristics'), key='cmip7_g127_CanESM5-1_r10i1p2f1', datase...
         └ <function redirect_logs at 0x14695a35dc60>

  File "/fs/site7/eccc/crd/cccma/users/scrd115/code/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 158, in __exit__
    self.gen.throw(value)
    │    │   │     └ FileNotFoundError(2, 'No such file or directory')
    │    │   └ <method 'throw' of 'generator' objects>
    │    └ <generator object redirect_logs at 0x146920ae9c40>
    └ <contextlib._GeneratorContextManager object at 0x14695acc19d0>

> File "/fs/site7/eccc/crd/cccma/users/scrd115/REF/REF_v0.16.2/climate-ref/packages/climate-ref-core/src/climate_ref_core/logging.py", line 219, in redirect_logs
    yield

  File "/fs/site7/eccc/crd/cccma/users/scrd115/REF/REF_v0.16.2/climate-ref/packages/climate-ref-core/src/climate_ref_core/executor.py", line 66, in execute_locally
    return definition.diagnostic.run(definition=definition)
           │          │                         └ ExecutionDefinition(diagnostic=ENSOCharacteristics(name='ENSO Characteristics'), key='cmip7_g127_CanESM5-1_r10i1p2f1', datase...
           │          └ <member 'diagnostic' of 'ExecutionDefinition' objects>
           └ ExecutionDefinition(diagnostic=ENSOCharacteristics(name='ENSO Characteristics'), key='cmip7_g127_CanESM5-1_r10i1p2f1', datase...

  File "/fs/site7/eccc/crd/cccma/users/scrd115/REF/REF_v0.16.2/climate-ref/packages/climate-ref-core/src/climate_ref_core/diagnostics.py", line 637, in run
    return self.build_execution_result(definition)
           │    │                      └ ExecutionDefinition(diagnostic=ENSOCharacteristics(name='ENSO Characteristics'), key='cmip7_g127_CanESM5-1_r10i1p2f1', datase...
           │    └ <function ESMValToolDiagnostic.build_execution_result at 0x146920c7d940>
           └ ENSOCharacteristics(name='ENSO Characteristics')

  File "/fs/site7/eccc/crd/cccma/users/scrd115/REF/REF_v0.16.2/climate-ref/packages/climate-ref-esmvaltool/src/climate_ref_esmvaltool/diagnostics/base.py", line 455, in build_execution_result
    metric_bundle, output_bundle = self.format_result(
                                   │    └ <staticmethod(<function ENSOCharacteristics.format_result at 0x146920c7e340>)>
                                   └ ENSOCharacteristics(name='ENSO Characteristics')

  File "/fs/site7/eccc/crd/cccma/users/scrd115/REF/REF_v0.16.2/climate-ref/packages/climate-ref-esmvaltool/src/climate_ref_esmvaltool/diagnostics/enso.py", line 449, in format_result
    metrics = pd.read_csv(
              │  └ <function read_csv at 0x146939e16d40>
              └ <module 'pandas' from '/fs/site7/eccc/crd/cccma/users/scrd115/envs/ref_v0.16.2/lib/python3.12/site-packages/pandas/__init__.py'>

  File "/fs/site7/eccc/crd/cccma/users/scrd115/envs/ref_v0.16.2/lib/python3.12/site-packages/pandas/io/parsers/readers.py", line 873, in read_csv
    return _read(filepath_or_buffer, kwds)
           │     │                   └ {'delimiter': ',', 'header': 'infer', 'names': ['dataset', 'metric_name', 'metric_value'], 'index_col': None, 'usecols': None...
           │     └ PosixPath('/fs/site7/eccc/crd/cccma/users/scrd115/REF/REF_v0.16.2/climate-ref/.ref/scratch/esmvaltool/enso-characteristics/g1...
           └ <function _read at 0x146939e168e0>
  File "/fs/site7/eccc/crd/cccma/users/scrd115/envs/ref_v0.16.2/lib/python3.12/site-packages/pandas/io/parsers/readers.py", line 300, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
             │              │                     └ {'delimiter': ',', 'header': 'infer', 'names': ['dataset', 'metric_name', 'metric_value'], 'index_col': None, 'usecols': None...
             │              └ PosixPath('/fs/site7/eccc/crd/cccma/users/scrd115/REF/REF_v0.16.2/climate-ref/.ref/scratch/esmvaltool/enso-characteristics/g1...
             └ <class 'pandas.io.parsers.readers.TextFileReader'>
  File "/fs/site7/eccc/crd/cccma/users/scrd115/envs/ref_v0.16.2/lib/python3.12/site-packages/pandas/io/parsers/readers.py", line 1645, in __init__
    self._engine = self._make_engine(f, self.engine)
    │              │    │            │  │    └ 'c'
    │              │    │            │  └ <pandas.io.parsers.readers.TextFileReader object at 0x146921c9b0e0>
    │              │    │            └ PosixPath('/fs/site7/eccc/crd/cccma/users/scrd115/REF/REF_v0.16.2/climate-ref/.ref/scratch/esmvaltool/enso-characteristics/g1...
    │              │    └ <function TextFileReader._make_engine at 0x146939e17880>
    │              └ <pandas.io.parsers.readers.TextFileReader object at 0x146921c9b0e0>
    └ <pandas.io.parsers.readers.TextFileReader object at 0x146921c9b0e0>
  File "/fs/site7/eccc/crd/cccma/users/scrd115/envs/ref_v0.16.2/lib/python3.12/site-packages/pandas/io/parsers/readers.py", line 1904, in _make_engine
    self.handles = get_handle(
    │    │         └ <function get_handle at 0x14693a6f3240>
    │    └ None
    └ <pandas.io.parsers.readers.TextFileReader object at 0x146921c9b0e0>
  File "/fs/site7/eccc/crd/cccma/users/scrd115/envs/ref_v0.16.2/lib/python3.12/site-packages/pandas/io/common.py", line 930, in get_handle
    handle = open(

FileNotFoundError: [Errno 2] No such file or directory: '/fs/site7/eccc/crd/cccma/users/scrd115/REF/REF_v0.16.2/climate-ref/.ref/scratch/esmvaltool/enso-characteristics/g127_CanESM5-1_r10i1p2f1_g3062_v1_3e18a899/7389/executions/recipe_20260818_221807/work/diagnostic_metrics/plot_script/matrix.csv'
2026-08-18 22:20:59.489 +00:00 e0:03:43.105254s | INFO     | SpawnProcess-1:263332 | climate_ref_core.logging:redirect_logs:224 - Diagnostic execution complete. Results available in esmvaltool/enso-characteristics/g127_CanESM5-1_r10i1p2f1_g3062_v1_3e18a899/7389
2026-08-18 22:20:59.493 +00:00 e0:03:43.109139s | ERROR    | SpawnProcess-1:263332 | climate_ref_core.executor:execute_locally:77 - System error running 'esmvaltool/enso-characteristics/cmip7_g127_CanESM5-1_r10i1p2f1' (will be retried on next solve): [Errno 2] No such file or directory: '/fs/site7/eccc/crd/cccma/users/scrd115/REF/REF_v0.16.2/climate-ref/.ref/scratch/esmvaltool/enso-characteristics/g127_CanESM5-1_r10i1p2f1_g3062_v1_3e18a899/7389/executions/recipe_20260818_221807/work/diagnostic_metrics/plot_script/matrix.csv'

```

## System

 - OS: Linux
 - Python 3.12.13

```
  provider     version   conda environment                                                                           data path                                                                                 
 ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── 
  example      0.16.2                                                                                                                                                                                          
  esmvaltool   0.16.2    /fs/site7/eccc/crd/cccma/users/scrd115/REF/REF_v0.16.2/climate-ref/.ref/software/conda/e…   /fs/site7/eccc/crd/cccma/users/scrd115/REF/REF_v0.16.2/climate-ref/.ref/cache/esmvaltool  
  ilamb        0.16.2                                                                                                /fs/site7/eccc/crd/cccma/users/scrd115/REF/REF_v0.16.2/climate-ref/.ref/cache/ilamb       
  pmp          0.16.2    /fs/site7/eccc/crd/cccma/users/scrd115/REF/REF_v0.16.2/climate-ref/.ref/software/conda/p…   /fs/site7/eccc/crd/cccma/users/scrd115/REF/REF_v0.16.2/climate-ref/.ref/cache/pmp-clima…  
```


[climate-ref_2026-08-18_22-17_SpawnProcess-1.log](https://github.com/user-attachments/files/31200343/climate-ref_2026-08-18_22-17_SpawnProcess-1.log)
[log.txt](https://github.com/user-attachments/files/31200342/log.txt)
[main_log.txt](https://github.com/user-attachments/files/31200344/main_log.txt)


Navigation Menu

Uh oh!

enso-characteristic diagnostic fails to copy successful output to results directory #883

Describe the bug

System

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Uh oh!

enso-characteristic diagnostic fails to copy successful output to results directory #883

Description

Describe the bug

System

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions