Skip to content

Fix missing Weights & Biases metric logging for tracking data#446

Open
marina-aoyama wants to merge 1 commit into
Toni-SM:developfrom
marina-aoyama:isaaclab-custom-wandb
Open

Fix missing Weights & Biases metric logging for tracking data#446
marina-aoyama wants to merge 1 commit into
Toni-SM:developfrom
marina-aoyama:isaaclab-custom-wandb

Conversation

@marina-aoyama

Copy link
Copy Markdown

📝 Description
This PR fixes an issue where certain training metrics (e.g., rewards) were not appearing in Weights & Biases, even when sync_tensorboard=True was enabled.

🔧 Problem
Although W&B was configured to sync TensorBoard logs, some metrics stored in self.tracking_data were not consistently propagated to W&B dashboards. This resulted in incomplete experiment tracking, particularly for key RL metrics such as episode rewards.

💡 Root cause
W&B TensorBoard synchronization does not reliably capture all scalar logs in this setup, especially when metrics are updated outside of immediate TensorBoard flush cycles.

✅ Solution

  • Add explicit W&B logging for self.tracking_data via wandb.log
  • Ensure logging only occurs when 1) W&B is enabled in config, and 2) wandb is installed.

✨ Change summary

  • Added safe import handling for wandb
  • Added runtime check for wandb.run
  • Added explicit metric logging alongside TensorBoard logging
if self.cfg.experiment.wandb and wandb.run is not None:
    wandb.log({k: float(np.mean(v))}, step=timestep)

📊 Impact

  • Restores missing metrics in W&B dashboards (including rewards)
  • Keeps TensorBoard logging unchanged
  • Maintains backward compatibility when W&B is disabled or not installed
  • Improves experiment reproducibility and monitoring reliability

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant