Python Logging Integration for FINN Build Flow

[tafk7]

Replaces FINN's fragmented subprocess management with unified Python logging. All Xilinx tool invocations (Vivado, Vitis, HLS, xsim, g++) now use direct subprocess calls with structured logging, eliminating bash script wrappers and stdout redirection.

Problems solved:

• Lost subprocess output (warnings/errors captured but discarded)
• No granular verbosity control (single verbose flag for everything)
• Limited library integration (couldn't suppress console output)
• Bash script overhead and inconsistent logging patterns

Key Changes

1. Centralized Subprocess Execution

New launch_process_helper() in src/finn/util/basic.py provides:

• Thread-safe non-blocking I/O capture
• Environment variable expansion ($FINN_ROOT)
• Pattern-based log level detection (ERROR:, WARNING:)
• Flexible error handling

All 10+ subprocess invocation sites now use this unified interface.

2. Hierarchical Logger Design

finn.builder.*     → Build progress (independent)
finn.*             → Tool output (vivado, hls, vitis, gcc, xsim)

Both hierarchies write to file; console output independently controlled.

3. Multi-Level Verbosity Control

New config fields:

show_progress: bool = True              # Builder progress messages
subprocess_log_levels: dict = None      # Per-tool file log levels
subprocess_console_levels: dict = None  # Per-tool console levels (when verbose=True)

Operation modes:

verbose	show_progress	Console	Use Case
False	False	Silent	Library/batch
False	True	Progress only	Default
True	True	Everything	Debug

Example - filter Vivado spam:

cfg = DataflowBuildConfig(
    verbose=True,
    subprocess_console_levels={'vivado': 'WARNING', 'hls': 'ERROR'},
    subprocess_log_levels={'vivado': 'DEBUG', 'hls': 'DEBUG'},
)

Defaults: console ERROR, file DEBUG when not specified.

4. Code Simplification

Before:

# Generate bash wrapper
make_project_sh = proj_dir + "/make_project.sh"
with open(make_project_sh, "w") as f:
    f.write("#!/bin/bash\ncd {}\nvivado -mode batch -source make_project.tcl\n".format(proj_dir))
subprocess.Popen(["bash", make_project_sh], stdout=subprocess.PIPE).communicate()

After:

launch_process_helper(
    ["vivado", "-mode", "batch", "-source", "make_project.tcl"],
    cwd=proj_dir,
    logger=logging.getLogger("finn.vivado.stitch_ip"),
)

Removed:

• StreamToLogger class (stdout redirection)
• use_logging parameter (always enabled)
• Bash script generation throughout codebase
• ~135 lines dead code

File Changes

Core Implementation:

• src/finn/util/basic.py (+257/-22) - launch_process_helper(), CppBuilder refactor
• src/finn/builder/build_dataflow.py (+91/-38) - Logger configuration, removed stdout redirection
• src/finn/builder/build_dataflow_config.py (+15) - New config fields

Subprocess Call Sites (8 files):

• src/finn/util/hls.py, src/finn/util/vivado.py, src/finn/core/rtlsim_exec.py
• src/finn/transformation/fpgadataflow/*.py - IP generation, Zynq synthesis, Vitis build

Tests:

• tests/util/test_launch_process_helper.py (+266) - 26 unit tests

Benefits

1. Complete subprocess coverage - all tool output captured
2. Fine-grained control - per-tool log levels (file + console)
3. Library-friendly - silent mode for external applications
4. Cleaner codebase - no bash wrappers, no stdout redirection
5. Better debugging - structured logs with logger names and timestamps

Backward Compatibility

Breaking:

• verbose=False now silent (was progress visible). Migration: use show_progress=True, verbose=False (new default)
• Log format: [timestamp] [logger] level: message (was [timestamp] message)

Non-breaking:

• verbose=True unchanged
• All existing builds work without modification
• File logging more comprehensive (captures previously lost output)

Testing

Unit Tests (34 total):

• test_launch_process_helper.py (26 tests) - Subprocess execution, streaming, log level detection, environment handling
• test_logging_config.py (8 tests) - Configuration behavior verification:
  • show_progress flag controls progress console output
  • verbose flag controls subprocess console output
  • subprocess_log_levels filters file logging per tool
  • subprocess_console_levels filters console output per tool (when verbose=True)
  • JSON serialization of new config fields
  • Backwards compatibility with old verbose-only configs
  • Fixed bug: Logger level calculation now respects verbose flag (prevents console_level from affecting file logging when console is disabled)

Integration:

• Existing end-to-end tests validate backward compatibility (bnn sanity)