Skip to content

ocsf_json_enabled silently writes nothing with the MicroVM driver (supervisor cannot open /var/log) #3855

Open

Description

@Ash20pk

Version: v0.1.2, macOS 27 (arm64), MicroVM driver, gateway running as a normal user (which is what the Homebrew service does).

What happens

openshell settings set <sandbox> --key ocsf_json_enabled --value true succeeds, and the supervisor logs OCSF JSONL logging toggled. But no openshell-ocsf.YYYY-MM-DD.log ever shows up: not in the host /var/log, not under the driver state dir, not inside the sandbox. The JSON records don't land on stderr either. The only hint is this line when the supervisor starts:

WARN openshell_supervisor: Could not open /var/log for log rotation; using stderr-only logging

The docs (docs/observability/ocsf-json-export.mdx) say sandbox supervisors write to /var/log/openshell-ocsf.YYYY-MM-DD.log, with no driver caveat, so this looks like the feature silently no-ops on the VM driver while reporting success.

Why, as far as I can tell

With the MicroVM driver the supervisor is a host process (host-runtime/openshell-supervisor) running as the gateway user, and the JSONL sink is hard-coded to /var/log (crates/openshell-supervisor/src/main.rs, around lines 308 to 388). When that directory can't be opened, the whole else branch installs no OcsfJsonlLayer at all, so apply_ocsf_json_setting in lib.rs just flips a flag that nothing reads and logs "toggled" anyway. There's no supervisor-side override for the directory. OPENSHELL_OCSF_LOG_DIR only applies to the MXC gateway sink.

The Docker driver on the same machine is fine, because the supervisor container gets a private tmpfs on /var/log.

Expected

Either the file gets written somewhere the supervisor can write (a configurable directory, or the sandbox's driver state dir by default), or the setting is rejected / a clear warning is logged saying there is no sink. Right now it looks like it worked.

Reproduce

Everything below uses the v0.1.2 release binaries, checksum-verified, set up the same way the Homebrew formula does (entitle the VM driver, generate-certs, default config). Takes about 10 to 15 minutes the first time because of the sandbox image pull. Needs port 17670 free.

repro script (sh)
#!/bin/sh
# Reproduce: OpenShell v0.1.2, macOS arm64, MicroVM driver.
# `ocsf_json_enabled=true` is accepted, but no OCSF JSONL file is ever written.
#
# Uses only NVIDIA's release binaries, installed the way the Homebrew formula
# (nvidia/openshell/openshell 0.1.2) installs them. Everything lives in
# ~/openshell-repro plus OpenShell's own default state/config dirs.
# Needs: port 17670 free, no other OpenShell gateway running.
set -eu

P="$HOME/openshell-repro"
R="https://github.com/NVIDIA/OpenShell/releases/download/v0.1.2"
OS="$P/bin/openshell"

echo "== 1. Download the official v0.1.2 binaries and verify NVIDIA's checksums"
mkdir -p "$P/bin" "$P/libexec" "$P/var/openshell" "$P/var/log" "$P/dl"
cd "$P/dl"
for f in openshell-aarch64-apple-darwin.tar.gz \
         openshell-gateway-aarch64-apple-darwin.tar.gz \
         openshell-driver-vm-aarch64-apple-darwin.tar.gz \
         openshell-checksums-sha256.txt openshell-gateway-checksums-sha256.txt; do
  curl -fsSL -O "$R/$f"
done
cat openshell-checksums-sha256.txt openshell-gateway-checksums-sha256.txt \
  | grep -E ' openshell-(gateway-|driver-vm-)?aarch64-apple-darwin.tar.gz$' \
  | shasum -a 256 -c -
tar -xzf openshell-aarch64-apple-darwin.tar.gz -C "$P/bin"
tar -xzf openshell-gateway-aarch64-apple-darwin.tar.gz -C "$P/bin"
tar -xzf openshell-driver-vm-aarch64-apple-darwin.tar.gz -C "$P/libexec"
"$OS" --version

echo "== 2. Same post-install as the Homebrew formula: entitle the VM driver, make certs, default config"
cat > "$P/var/openshell/openshell-driver-vm.entitlements.plist" <<'EOF'
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>com.apple.security.hypervisor</key>
    <true/>
</dict>
</plist>
EOF
/usr/bin/codesign --entitlements "$P/var/openshell/openshell-driver-vm.entitlements.plist" \
  --force -s - "$P/libexec/openshell-driver-vm"
"$P/bin/openshell-gateway" generate-certs --output-dir "$P/var/openshell/tls" \
  --server-san host.openshell.internal >/dev/null 2>&1
# The formula's default config, plus pinning the MicroVM driver.
cat > "$P/var/openshell/gateway.toml" <<'EOF'
[openshell]
version = 2

[openshell.gateway]
compute_driver = "vm"
EOF

echo "== 3. Start the gateway as the Homebrew service would (as your user), register the CLI"
OPENSHELL_LOCAL_TLS_DIR="$P/var/openshell/tls" nohup "$P/bin/openshell-gateway" \
  --config "$P/var/openshell/gateway.toml" \
  >"$P/var/log/gateway.out.log" 2>"$P/var/log/gateway.err.log" &
echo $! >"$P/var/gateway.pid"
i=0; until lsof -nP -iTCP:17670 -sTCP:LISTEN >/dev/null 2>&1; do i=$((i+1)); [ $i -lt 60 ] || { echo "gateway did not start"; exit 1; }; sleep 0.5; done
"$OS" gateway add https://127.0.0.1:17670 --local --name openshell >/dev/null
"$OS" status | grep -E "Status|Version"

echo "== 4. Create a sandbox (default image) with a small policy"
cat > "$P/policy.yaml" <<'EOF'
version: 1
filesystem_policy:
  include_workdir: true
  read_only: [/usr, /lib, /etc, /proc]
  read_write: [/sandbox, /tmp]
landlock:
  compatibility: best_effort
process:
  run_as_user: sandbox
  run_as_group: sandbox
network_policies:
  example:
    name: example
    endpoints:
      - host: example.com
        port: 443
    binaries:
      - path: /usr/bin/curl
EOF
"$OS" sandbox create --name ocsf-repro --policy "$P/policy.yaml" --detach --no-tty --no-auto-providers >/dev/null
i=0; until [ "$("$OS" sandbox get ocsf-repro -o json 2>/dev/null | python3 -c 'import sys,json;print(json.load(sys.stdin).get("phase"))' 2>/dev/null)" = Ready ]; do i=$((i+1)); [ $i -lt 200 ] || { echo "sandbox not ready"; exit 1; }; sleep 3; done
echo "sandbox ready"

echo "== 5. Turn on OCSF JSON export (documented setting); it is accepted"
"$OS" settings set ocsf-repro --key ocsf_json_enabled --value true
sleep 15   # the supervisor polls settings every ~10 s

echo "== 6. Cause an OCSF event after the toggle (bash is not allowed out, so this is denied)"
# Bounded: a denied connection may be refused at once or held until it times out.
"$OS" sandbox exec --name ocsf-repro --timeout 30 -- timeout 5 bash -c 'exec 3<>/dev/tcp/example.com/443' 2>&1 | tail -1 || true
sleep 3

ID=$("$OS" sandbox get ocsf-repro -o json | python3 -c 'import sys,json;print(json.load(sys.stdin)["id"])')
LOG="$HOME/.local/state/openshell/vm-driver/sandboxes/$ID/supervisor.err.log"

echo "== 7. Supervisor log: the warning, the toggle, and the event"
sed 's/\x1b\[[0-9;]*m//g' "$LOG" | grep -E "Could not open /var/log|OCSF JSONL logging toggled|DENIED" | cut -c1-150

echo "== 8. Expected an openshell-ocsf.YYYY-MM-DD.log somewhere. Count of such files:"
echo "  host /var/log:            $(ls /var/log | grep -c openshell-ocsf || true)"
echo "  OpenShell state + repro:  $(find "$HOME/.local/state/openshell" "$P" -name 'openshell-ocsf*' | wc -l | tr -d ' ')"
echo "  inside the sandbox:       $("$OS" sandbox exec --name ocsf-repro -- bash -c 'ls /var/log | grep -c openshell-ocsf' 2>/dev/null || true)"
echo "  supervisor runs as:       $(ps -Ao user,command | grep '[h]ost-runtime/openshell-supervisor' | awk '{print $1}' | head -1)"
echo
echo "Cleanup: $OS sandbox delete ocsf-repro; kill \$(cat $P/var/gateway.pid);"
echo "         rm -rf $P ~/.config/openshell ~/.local/state/openshell"

Output of the relevant steps on my machine:

== 5. Turn on OCSF JSON export (documented setting); it is accepted
✓ Set sandbox setting ocsf_json_enabled=true for ocsf-repro (revision 1)
== 6. Cause an OCSF event after the toggle (bash is not allowed out, so this is denied)
bash: line 1: /dev/tcp/example.com/443: Permission denied
== 7. Supervisor log: the warning, the toggle, and the event
WARN openshell_supervisor: Could not open /var/log for log rotation; using stderr-only logging
INFO openshell_supervisor: OCSF JSONL logging toggled
OCSF NET:OPEN [MED] DENIED /usr/bin/bash(0) -> example.com:443 [reason:transparent_tcp_policy_denied]
== 8. Expected an openshell-ocsf.YYYY-MM-DD.log somewhere. Count of such files:
  host /var/log:            0
  OpenShell state + repro:  0
  inside the sandbox:       0
  supervisor runs as:       ash

I haven't tried Linux or a gateway running as root. Happy to send a PR: my thought is to make the supervisor log directory configurable with a writable fallback for the VM driver, and have the toggle warn when there's no JSONL sink instead of saying "toggled".

Activity

  1. ilaigold commented on Oct 7, 2026

    @ilaigold

    I'm hitting this too, on my Mac with the VM driver. Every sandbox's supervisor.err.log on my local gateway starts with Could not open /var/log for log rotation; using stderr-only logging, so ocsf_json_enabled has nowhere to write.

    @Ash20pk you offered to send a PR. Have you started one? If so, I'll stay out of your way. If not, I have a branch that does roughly what you suggested:

    • The supervisor gets a --log-dir flag (default /var/log, so Docker, Podman and Kubernetes don't change). Both log files use it, and so does the shorthand-log reader behind /v1/denials.
    • The VM driver creates an owner-only logs/ directory in the sandbox's state dir, next to supervisor.log, and passes it in. It goes away when the sandbox is deleted.
    • If the JSONL file couldn't be opened, turning on ocsf_json_enabled logs a warning instead of "toggled".

    I ran the built supervisor as a normal user on macOS. Without the flag it prints the same /var/log warning, and with it both log files show up in the directory I gave it.

    For the maintainers: the docs from #4243 describe the VM case as /var/log on the host "if writable". Is the sandbox state dir the right place for these files with the VM driver, or would you rather have a gateway-level setting? If you accept the issue with this approach, I'll open the PR.

  2. delgadof commented on Oct 16, 2026

    @delgadof

    Same root cause with any host-side supervisor: a supervisor running natively on macOS as a normal user can't open /var/log, falls back to console output, and the console output drops OCSF event messages (for example why an SSH channel or port forward failed). That makes host-side problems hard to diagnose.

    A flag or environment variable for the supervisor's log directory, so drivers can point it at their own state directory, would fix both cases. Still present in v0.1.4 (crates/openshell-supervisor/src/main.rs).

  3. krishicks commented on Oct 26, 2026

    @krishicks
    Collaborator

    Written by Claude Code, reviewed by @krishicks

    Thanks for the detailed repro, and to ilaigold and delgadof for confirming it. I checked the cause against main.

    Confirmed. The supervisor writes its log files to a hard-coded /var/log (crates/openshell-supervisor/src/main.rs, and the shared mode path). There is no flag or env override. If the directory can't be opened, no file sinks are installed, which is what happens for a host-side supervisor running as a normal user, such as the VM driver on macOS. ocsf_json_enabled then has no file to write, and the supervisor still logs OCSF JSONL logging toggled, because that message only reports that the flag changed.

    One part changed in 0.1.3. The issue says the JSON records don't land on stderr either. That was true on 0.1.2. Since #4323, when ocsf_json_enabled is on and there is no file sink, the supervisor writes each record to stderr as a timestamped OCSF-JSON {...} line. For the VM driver that is supervisor.err.log in the sandbox state directory. The repro script filters that log for DENIED, and the JSON record spells it "action":"Denied", so the grep would not match. @Ash20pk, could you check supervisor.err.log for OCSF-JSON lines on a current build?

    Still open:

    • The log directory isn't configurable, so host-side supervisors have no durable, rotated file.
    • The toggle message doesn't say that no file sink exists.
    • With ocsf_json_enabled off, the console shorthand output is gated by the diagnostic log level, so OCSF events can be missing from stderr (delgadof's point).

    I'm keeping this open for those three. I'd re-title it to "Supervisor log directory isn't configurable for host-side supervisors".

  4. added
    state:acceptedA maintainer decided OpenShell should pursue this issue
    topic:observabilityLogging, metrics, and observability work
    and removed
    state:triage-neededOpened without agent diagnostics and needs triage
    on Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    state:acceptedA maintainer decided OpenShell should pursue this issuetopic:observabilityLogging, metrics, and observability work

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions