staged-flow 3/6: -c codegen mode for Bluesim and Verilog

DEVELOP.md

@@ -121,6 +121,47 @@ The following is a running list of those writings.
 * [How Bluesim provides implementations for import-BVI](https://groups.io/g/b-lang-discuss/topic/106520424)
 * [How the Bluesim C API is imported into Bluetcl](https://groups.io/g/b-lang-discuss/message/554)
 * There is a template for making Bluesim standalone programs (without Tcl) in `bsc/util/bsim_standalone/`
+* `-c` (codegen mode): per-module byte-identity and object reuse (see below)
+
+#### `-c` (codegen mode): per-module byte-identity
+
+Terminology: a module generated as a `-e` link's *top* is in "top form" (its
+interface methods are fired by the design schedule that the link also
+generates); everywhere else -- under `-c`, or as a submodule of any design --
+it is in "block form".  Block form is the shared, reusable output; top form
+is private to the link that made it.
+
+`-sim -c M` emits `M`'s Bluesim C++ (and only `M`'s) without a runnable top, so
+`M`'s object can be built once and reused wherever `M` appears in a design.
+Reuse *trusts* byte-identity: `isStale` (`SimFileUtils.hs`) reuses an object on a
+version/timestamp/`codeGenOptionDescr` match, never comparing content -- so
+`M`'s `-c` output must equal the C++ it gets as a submodule.  (A link's own
+top is the exception: it is generated in "top form", which the descriptor
+records as `top`, so the two forms are never mixed by reuse.)
+
+It does, because both are generated from the same already-elaborated `M.ba`;
+only two things differ between the runs, and both are neutralized:
+
+* **Interned-`Id` order** varies with what else is compiled, but only *reorders*
+  output, and every per-module emission site is name-sorted (the four
+  "Canonicalize ... order" commits, plus pre-existing sorts).
+* **The schedule** (`M`'s own, firing-suppressed, vs the whole design's) reaches
+  per-module codegen through three channels, each made `M`-intrinsic:
+  *top-ness* (`mkScheduleStmts` drops `M`'s own interface-method firing, so DCE
+  gives submodule form), *VCD clock annotation* (`clk_map`; the emitted domain
+  comes from `M.ba`), and *member-vs-local* (`moveDefsOntoStack`; a top
+  interface method's readiness is reached by the RDY call, as a parent would,
+  not a direct read).
+
+`check_block_codegen_modules` (`testsuite/config/unix.exp`) enforces it: it
+rebuilds every multi-module test's submodules with `-c` and byte-compares.
+
+The Verilog backend has the same mode: `-verilog -c M` regenerates `M.v` from
+`M.ba` (via `vGenMods` in `bsc.hs`), byte-identical to the `.v` that `-g`
+writes for the same elaboration under the same generation flags.  Codegen
+flags are read from the `-c` invocation's command line (elaboration-affecting
+flags are baked into the `.ba`), and foreign-function `.ba` files are found
+via the same `-p`/`-bdir` search path as at link time.
 
 ### Bluetcl
 

doc/user_guide/user_guide.tex

@@ -388,11 +388,17 @@ \subsection{Overview of the BSC process}
 \begin{enumerate}
 \item A designer writes a BSV program, including Verilog, VHDL, and C components as desired.
 \item The BSV program is compiled into a Verilog or Bluesim
-specification. This step is comprised of two distinct stages:
+specification. This step is comprised of three distinct stages:
 \begin{enumerate}
 \item pre-elaboration - parsing and type checking
-\item post-elaboration - code generation
+\item elaboration - evaluation and scheduling, recorded in \te{.ba} files
+\item code generation - producing Verilog or Bluesim C++ from the
+\te{.ba} files
 \end{enumerate}
+These stages are normally run by a single compile command, but code
+generation can also be run separately from saved \te{.ba} files
+(Section~\ref{sec-codegen-mode}), and the linking stage regenerates
+any generated code that is missing or out of date.
 \item  The compilation output is either linked into a simulation
 environment or processed by a synthesis tool.
 \end{enumerate}
@@ -453,10 +459,14 @@ \subsection{Compile}
 Once the type check is complete, you have enough module
 information to use Bluetcl to browse the contents of packages.
 
-The second stage (code generation) generates an elaborated module file
-(\te{.ba}) and, when the target is Verilog, a (\te{.v})  file.  These
-generated files have the same name as the module they implement, not
-the name of the package or file they come from.
+The second stage elaborates each module marked for synthesis, recording
+the result in an elaborated module file (\te{.ba}), and generates code
+from it: a \te{.v} file when the target is Verilog (Bluesim C++ is
+generated when linking).  These generated files have the same name as
+the module they implement, not the name of the package or file they
+come from.  Code generation can also be re-run from a \te{.ba} file
+alone, without the source, as described in
+Section~\ref{sec-codegen-mode}.
 
 \index{attributes!synthesize}
 \index{synthesize attribute@\te{synthesize} attribute}
@@ -787,6 +797,44 @@ \subsubsubsection{Importing C functions}
 
 % -------------------------
 
+\subsubsection{Generating code from elaborated files}
+\label{sec-codegen-mode}
+\index{-c@\te{-c} (compiler flag)}
+\index{code generation!from .ba files}
+
+Code generation can be run separately from compilation, using the
+\te{.ba} file that compilation saves for each synthesized module.  The
+\te{-c} flag names a module to generate code for:
+
+\begin{centerboxverbatim}
+bsc -verilog -c mkFoo
+bsc -sim -c mkFoo
+\end{centerboxverbatim}
+
+The \te{.ba} file is found by module name on the search path (or
+\te{.ba} files may be given on the command line), and the back-end
+flag selects the output: \te{mkFoo.v} for Verilog, or \te{mkFoo.cxx}
+and \te{mkFoo.h} for Bluesim.  The output is identical to what a full
+compile (for Verilog) or link (for Bluesim) would generate for the
+same module, so generated code and compiled objects can be shared: an
+external build system can run \te{-c} once per module, compile the
+results, and let the linking stage reuse whatever is up to date
+(Section~\ref{sec-link}).
+
+The \te{-c} flag generates code only for the named modules; submodules
+are generated by their own \te{-c} commands, or by the linking stage,
+which generates anything that is missing.  Code-generation flags (for
+example \te{-keep-fires}) take effect from the \te{-c} command line;
+elaboration choices are already recorded in the \te{.ba} file.
+
+For Bluesim, the generated module is a reusable block: its C++ does
+not depend on where the module is later instantiated.  The top module
+of a Bluesim simulation is the one exception -- it is always generated
+by the linking stage itself, together with the schedule and model
+files.
+
+% -----
+
 \subsection{Link}
 \index{linking}
 \index{link}
@@ -823,7 +871,11 @@ \subsubsection{Linking with Bluesim}
 at: \verb|${BLUESPECDIR}/Bluesim/|.
 
 Specifically, the linking stage generates a C++ object for each
-elaborated module.  For each module, it generates {\em module}.\te{.h} and
+elaborated module.  Generated files and objects that are already up to
+date -- from an earlier link, or generated with \te{-c}
+(Section~\ref{sec-codegen-mode}) and compiled externally -- are reused
+rather than regenerated; the top module, the schedule, and the model
+files are always generated by the link itself.  For each module, it generates {\em module}.\te{.h} and
 {\em module}\te{.cxx} files which are compiled to a {\bf\tt .o} file.
 The C++ compiler to use is determined from the {\bf\tt CXX}
 environment variable (the default is {\bf\tt c++}) and any flags
@@ -1143,6 +1195,14 @@ \subsubsection{Linking with Verilog}
 detail.  The Verilog simulator is specified by  using the {\bf\tt
 -vsim} flag.
 
+When linking, BSC reads the design's \te{.ba} files (found by module
+name on the search path) and regenerates any module's \te{.v} file
+that is missing or older than its \te{.ba}; up-to-date files are
+reused, and each reuse is reported.  A \te{.v} file listed explicitly
+on the command line is the user's to provide and is never regenerated.
+If no \te{.ba} files can be found for the design, BSC issues a warning
+and links with the \te{.v} files as found.
+
 \index{Verilog simulator!cvc}
 \index{Verilog simulator!cver}
 \index{Verilog simulator!isim}
@@ -1537,15 +1597,18 @@ \subsection{Common compile and linking flags}
 \index{-e@\te{-e} (compiler flag)}
 \index{-o@\te{-o} (compiler flag)}
 \index{-elab@\te{-elab} (compiler flag)}
+\index{-c@\te{-c} (compiler flag)}
 \begin{centerboxverbatim}
 -g module             generate code for `module' (requires -sim or -verilog)
+-c module             generate code for `module' from its elaborated .ba file
 -u                    check and recompile packages that are not up to date
 -sim                  compile BSV generating Bluesim object
 -verilog              compile BSV generating Verilog file
 -vsim simulator       specify which Verilog simulator to use
 -e module             top-level module for simulation
 -o name               name of generated executable
 -elab                 generate a .ba file after elaboration and scheduling
+                      (on by default with -sim and -verilog)
 \end{centerboxverbatim}
 
 
@@ -1598,6 +1661,13 @@ \subsection{Common compile and linking flags}
 bsc -verilog -e mkFoo
 \end{centerboxverbatim}
 
+Code can also be regenerated from a module's \te{.ba} file alone,
+without its source, using the \te{-c} flag
+(Section~\ref{sec-codegen-mode}):
+\begin{centerboxverbatim}
+bsc -verilog -c mkFoo
+\end{centerboxverbatim}
+
 
 The \te{-vsim} flag
 (along with the equivalent
@@ -1622,7 +1692,8 @@ \subsection{Common compile and linking flags}
 
 When using Bluetcl (or applications built on Bluetcl) to analyze a compiled
 design, a \te{.ba} file is required (for each module in the design).
-To generate this file when compiling add the \te{-elab} flag to the compile command. 
+These files are written by default when compiling for either back end;
+the \te{-no-elab} flag suppresses them.
 
 
 % -------------------------

src/comp/ABin.hs

@@ -15,7 +15,6 @@ module ABin (ABin(..),
              abemi_rule_relation_db
              ) where
 
-import Data.Maybe(isNothing)
 import PPrint
 import Id(Id)
 import Position(Position)
@@ -28,7 +27,6 @@ import ADumpScheduleInfo(MethodDumpInfo)
 import VModInfo(VPathInfo)
 import ForeignFunctions(ForeignFunction(..))
 import Flags(Flags(..))
-import Verilog(VProgram)
 
 
 -- ===============
@@ -62,9 +60,7 @@ data ABinModInfo =
         abmi_oqt         :: CQType,
         abmi_method_dump :: MethodDumpInfo,
         abmi_pathinfo :: VPathInfo,
-        abmi_flags       :: Flags,
-        -- the generated Verilog
-        abmi_vprogram :: Maybe VProgram
+        abmi_flags       :: Flags
     }
 
 data ABinForeignFuncInfo =
@@ -113,19 +109,16 @@ instance PPrint ABin where
 
 
 instance PPrint ABinModInfo where
-    pPrint d p (ABinModInfo path srcName apkg aschedinfo pps oqt mi pathinfo flags vprog) =
+    pPrint d p (ABinModInfo path srcName apkg aschedinfo pps oqt mi pathinfo flags) =
         text "path:" <+> text path $+$
         text "package:" <+> text srcName $+$
         pPrint d 0 apkg $+$
         pPrint d 0 aschedinfo $+$
         text "pprop:" <+> pPrint d 0 pps $+$
         text "oqt:" <+> pPrint d 0 oqt $+$
         -- no dump of method info
-        text "pathinfo:" <+> pPrint d 0 pathinfo $+$
+        text "pathinfo:" <+> pPrint d 0 pathinfo
         -- no dump of flags
-        text "vprog:" <+> (if (isNothing vprog)
-                           then text "Nothing"
-                           else text "Just ...") -- no dump of contents
 
 
 instance PPrint ABinForeignFuncInfo where

src/comp/ABinUtil.hs

@@ -4,6 +4,7 @@ module ABinUtil (
                  readAndCheckABin,
                  readAndCheckABinPath,
                  readAndCheckABinPathCatch,
+                 isStaleABinFile,
                  ) where
 
 import Data.List(nub, partition)
@@ -16,7 +17,7 @@ import Control.Monad.State(StateT, runStateT, lift, get, put)
 import Version(bscVersionStr)
 import Backend
 import FileNameUtil(abinSuffix)
-import FileIOUtil(readBinaryFileCatch, readBinFilePath)
+import FileIOUtil(readBinaryFileCatch, readBinFilePath, readBinaryFileMaybe)
 import Util(fromMaybeM)
 
 import Error(internalError, EMsg, EMsgs(..), ErrMsg(..),
@@ -29,7 +30,9 @@ import ASyntaxUtil(getForeignCallNames)
 import VModInfo(vName, getVNameString)
 import ForeignFunctions(ForeignFunction(..), ForeignFuncMap)
 import ABin
-import GenABin(readABinFile)
+import GenABin(readABinFile, readABinFileMaybe)
+
+import qualified Control.Exception as CE
 
 import qualified Data.Map as M
 
@@ -532,4 +535,31 @@ decodeABin errh backend filename contents =
             -- from this point
             (ABinModSchedErr {}) -> Right abi
 
+-- Tolerant counterpart of the decodeABin checks, for deciding whether
+-- an existing .ba file can be used by the current compilation: it must
+-- be readable, in the current format and BSC version, and elaborated
+-- for a compatible backend.  A missing file returns False (not stale);
+-- absence is the timestamp check's concern.  Used by the -u
+-- recompilation check, where an unusable .ba (e.g. one left by
+-- "-verilog -g" when compiling with -sim, or written by another BSC
+-- version) must force re-elaboration rather than be trusted as an
+-- up-to-date generated product.
+isStaleABinFile :: Maybe Backend -> String -> IO Bool
+isStaleABinFile be fname = do
+    mbytes <- readBinaryFileMaybe fname
+    case mbytes of
+      Nothing -> return False
+      Just bytes ->
+          let beOf (ABinMod mi _) = apkg_backend (abmi_apkg mi)
+              beOf (ABinModSchedErr mi _) = apkg_backend (abmsei_apkg mi)
+              beOf (ABinForeignFunc {}) = Nothing
+              stale = case readABinFileMaybe bytes of
+                        Nothing -> True
+                        Just abin ->
+                            (ab_version abin /= bscVersionStr True) ||
+                            not (backendMatches be (beOf abin))
+          in  CE.evaluate stale `CE.catch` handler
+  where handler :: CE.SomeException -> IO Bool
+        handler _ = return True
+
 -- ===============

src/comp/Depend.hs

@@ -28,11 +28,12 @@ import FStringCompat
 import Lex
 import Parse
 import FileNameUtil(hasDotSuf, dropSuf, baseName, dirName,
-                    bscSrcSuffix, bsvSrcSuffix, binSuffix,
+                    bscSrcSuffix, bsvSrcSuffix, binSuffix, abinSuffix,
                     mkAName, mkVName, mkVPIHName, mkVPICName,
                     createEncodedFullFilePath)
 import FileIOUtil(readFilesPath, readBinFilePath, readFileCatch, writeFileCatch,
                   removeFileCatch)
+import ABinUtil(isStaleABinFile)
 import Id
 import PreIds(idPrelude, idPreludeBSV)
 import Parser.Classic(pPackage, errSyntax, classicWarnings)
@@ -263,7 +264,14 @@ chkUpd flags doneMap resultList (pi:pis) = do
         --putStrLn (show genfs)
         genfsClks <- mapM getModTime genfs
         incfsClks <- mapM getModTime incfs
-        let needGenUpd = any (srcMod pi >) genfsClks
+        -- a .ba that exists (and is fresh by timestamp) is still not a
+        -- valid generated product if it cannot be read (another BSC
+        -- version's format) or was elaborated for an incompatible backend
+        -- (a Verilog-tagged .ba left by "-verilog -g" must not satisfy a
+        -- -sim compile, or vice versa); codegen must run again
+        staleAbins <- mapM (isStaleABinFile (backend flags))
+                          (filter (hasDotSuf abinSuffix) genfs)
+        let needGenUpd = any (srcMod pi >) genfsClks || or staleAbins
             needIncUpd = any (lastMod pi <) incfsClks
         --putStrLn (show (fileName pi, genfs, map (srcMod pi >) genfsClks))
         --putStr (ppReadable (pkgName pi, imports pi, DM.keys doneMap))

src/comp/Error.hs

@@ -1154,6 +1154,11 @@ data ErrMsg =
         -- Bluesim-specific errors/warnings
         | EBluesimNoXZ String
 
+        -- Errors for the .ba -> code generation mode (-c)
+        | EGenWithEntry String
+        | EGenWithSrcFile String
+        | EGenWithSystemC
+
         -- Errors/warnings from the SystemC wrapper generator
         | ESystemCWrapperComboPaths String
         | ESystemCWrapperInvalidMethods [String]
@@ -4500,6 +4505,22 @@ getErrorText (EMissingVPIWrapperFile fname is_dpi) =
      in  s2par ("Cannot find the " ++ ifctype ++ " file " ++ ishow fname ++
                 " in the Verilog search path."))
 
+getErrorText (EGenWithEntry entry) =
+    (System 96, empty,
+     s2par ("The flag -c generates code from an elaborated module; " ++
+            "it cannot be combined with -e (linking).  To link " ++
+            ishow entry ++ ", run bsc again with -e."))
+
+getErrorText (EGenWithSrcFile fname) =
+    (System 97, empty,
+     s2par ("The flag -c operates on elaborated (.ba) files, so a " ++
+            "source file (" ++ ishow fname ++ ") cannot be provided.  " ++
+            "To compile and generate from source, use -g."))
+
+getErrorText EGenWithSystemC =
+    (System 98, empty,
+     s2par ("The flag -c is not supported with -systemc; use -sim."))
+
 -- Runtime errors
 getErrorText (EMutuallyExclusiveRulesFire r1 r2) =
     (Runtime 1, empty,