Regenerate missing or stale Verilog when linking

doc/user_guide/user_guide.tex

@@ -388,11 +388,17 @@ \subsection{Overview of the BSC process}
 \begin{enumerate}
 \item A designer writes a BSV program, including Verilog, VHDL, and C components as desired.
 \item The BSV program is compiled into a Verilog or Bluesim
-specification. This step is comprised of two distinct stages:
+specification. This step is comprised of three distinct stages:
 \begin{enumerate}
 \item pre-elaboration - parsing and type checking
-\item post-elaboration - code generation
+\item elaboration - evaluation and scheduling, recorded in \te{.ba} files
+\item code generation - producing Verilog or Bluesim C++ from the
+\te{.ba} files
 \end{enumerate}
+These stages are normally run by a single compile command, but code
+generation can also be run separately from saved \te{.ba} files
+(Section~\ref{sec-codegen-mode}), and the linking stage regenerates
+any generated code that is missing or out of date.
 \item  The compilation output is either linked into a simulation
 environment or processed by a synthesis tool.
 \end{enumerate}
@@ -453,10 +459,14 @@ \subsection{Compile}
 Once the type check is complete, you have enough module
 information to use Bluetcl to browse the contents of packages.
 
-The second stage (code generation) generates an elaborated module file
-(\te{.ba}) and, when the target is Verilog, a (\te{.v})  file.  These
-generated files have the same name as the module they implement, not
-the name of the package or file they come from.
+The second stage elaborates each module marked for synthesis, recording
+the result in an elaborated module file (\te{.ba}), and generates code
+from it: a \te{.v} file when the target is Verilog (Bluesim C++ is
+generated when linking).  These generated files have the same name as
+the module they implement, not the name of the package or file they
+come from.  Code generation can also be re-run from a \te{.ba} file
+alone, without the source, as described in
+Section~\ref{sec-codegen-mode}.
 
 \index{attributes!synthesize}
 \index{synthesize attribute@\te{synthesize} attribute}
@@ -787,6 +797,44 @@ \subsubsubsection{Importing C functions}
 
 % -------------------------
 
+\subsubsection{Generating code from elaborated files}
+\label{sec-codegen-mode}
+\index{-c@\te{-c} (compiler flag)}
+\index{code generation!from .ba files}
+
+Code generation can be run separately from compilation, using the
+\te{.ba} file that compilation saves for each synthesized module.  The
+\te{-c} flag names a module to generate code for:
+
+\begin{centerboxverbatim}
+bsc -verilog -c mkFoo
+bsc -sim -c mkFoo
+\end{centerboxverbatim}
+
+The \te{.ba} file is found by module name on the search path (or
+\te{.ba} files may be given on the command line), and the back-end
+flag selects the output: \te{mkFoo.v} for Verilog, or \te{mkFoo.cxx}
+and \te{mkFoo.h} for Bluesim.  The output is identical to what a full
+compile (for Verilog) or link (for Bluesim) would generate for the
+same module, so generated code and compiled objects can be shared: an
+external build system can run \te{-c} once per module, compile the
+results, and let the linking stage reuse whatever is up to date
+(Section~\ref{sec-link}).
+
+The \te{-c} flag generates code only for the named modules; submodules
+are generated by their own \te{-c} commands, or by the linking stage,
+which generates anything that is missing.  Code-generation flags (for
+example \te{-keep-fires}) take effect from the \te{-c} command line;
+elaboration choices are already recorded in the \te{.ba} file.
+
+For Bluesim, the generated module is a reusable block: its C++ does
+not depend on where the module is later instantiated.  The top module
+of a Bluesim simulation is the one exception -- it is always generated
+by the linking stage itself, together with the schedule and model
+files.
+
+% -----
+
 \subsection{Link}
 \index{linking}
 \index{link}
@@ -823,7 +871,11 @@ \subsubsection{Linking with Bluesim}
 at: \verb|${BLUESPECDIR}/Bluesim/|.
 
 Specifically, the linking stage generates a C++ object for each
-elaborated module.  For each module, it generates {\em module}.\te{.h} and
+elaborated module.  Generated files and objects that are already up to
+date -- from an earlier link, or generated with \te{-c}
+(Section~\ref{sec-codegen-mode}) and compiled externally -- are reused
+rather than regenerated; the top module, the schedule, and the model
+files are always generated by the link itself.  For each module, it generates {\em module}.\te{.h} and
 {\em module}\te{.cxx} files which are compiled to a {\bf\tt .o} file.
 The C++ compiler to use is determined from the {\bf\tt CXX}
 environment variable (the default is {\bf\tt c++}) and any flags
@@ -1143,6 +1195,14 @@ \subsubsection{Linking with Verilog}
 detail.  The Verilog simulator is specified by  using the {\bf\tt
 -vsim} flag.
 
+When linking, BSC reads the design's \te{.ba} files (found by module
+name on the search path) and regenerates any module's \te{.v} file
+that is missing or older than its \te{.ba}; up-to-date files are
+reused, and each reuse is reported.  A \te{.v} file listed explicitly
+on the command line is the user's to provide and is never regenerated.
+If no \te{.ba} files can be found for the design, BSC issues a warning
+and links with the \te{.v} files as found.
+
 \index{Verilog simulator!cvc}
 \index{Verilog simulator!cver}
 \index{Verilog simulator!isim}
@@ -1537,15 +1597,18 @@ \subsection{Common compile and linking flags}
 \index{-e@\te{-e} (compiler flag)}
 \index{-o@\te{-o} (compiler flag)}
 \index{-elab@\te{-elab} (compiler flag)}
+\index{-c@\te{-c} (compiler flag)}
 \begin{centerboxverbatim}
 -g module             generate code for `module' (requires -sim or -verilog)
+-c module             generate code for `module' from its elaborated .ba file
 -u                    check and recompile packages that are not up to date
 -sim                  compile BSV generating Bluesim object
 -verilog              compile BSV generating Verilog file
 -vsim simulator       specify which Verilog simulator to use
 -e module             top-level module for simulation
 -o name               name of generated executable
 -elab                 generate a .ba file after elaboration and scheduling
+                      (on by default with -sim and -verilog)
 \end{centerboxverbatim}
 
 
@@ -1598,6 +1661,13 @@ \subsection{Common compile and linking flags}
 bsc -verilog -e mkFoo
 \end{centerboxverbatim}
 
+Code can also be regenerated from a module's \te{.ba} file alone,
+without its source, using the \te{-c} flag
+(Section~\ref{sec-codegen-mode}):
+\begin{centerboxverbatim}
+bsc -verilog -c mkFoo
+\end{centerboxverbatim}
+
 
 The \te{-vsim} flag
 (along with the equivalent
@@ -1622,7 +1692,8 @@ \subsection{Common compile and linking flags}
 
 When using Bluetcl (or applications built on Bluetcl) to analyze a compiled
 design, a \te{.ba} file is required (for each module in the design).
-To generate this file when compiling add the \te{-elab} flag to the compile command. 
+These files are written by default when compiling for either back end;
+the \te{-no-elab} flag suppresses them.
 
 
 % -------------------------

src/comp/Error.hs

@@ -1141,6 +1141,7 @@ data ErrMsg =
 
         -- ABin (.ba) file issues
         | WExtraABinFiles [String]
+        | WNoABinForVerilogRegen String
         | EMissingABinModFile String (Maybe String)
         | EMissingABinForeignFuncFile String String
         | EMultipleABinFilesForName String [String]
@@ -4143,6 +4144,12 @@ getErrorText (WExtraABinFiles filenames) =
     (System 39, empty,
      s2par ("The following elaboration files were not used in the design:") $$
      nest 2 (sepList (map text filenames) comma))
+getErrorText (WNoABinForVerilogRegen topmod) =
+    (System 99, empty,
+     s2par ("No elaboration file (.ba) was found for module " ++
+            ishow topmod ++ " or one of its submodules, so BSC cannot " ++
+            "check whether the generated Verilog is up to date; linking " ++
+            "will use the Verilog files found on the search path."))
 getErrorText (EMissingABinModFile module_name mparent) =
     (System 40, empty,
      case (mparent) of

src/comp/SimFileUtils.hs

@@ -13,8 +13,9 @@ import Version(bscVersionStr)
 import FileNameUtil
 import ErrorUtil(internalError)
 
+import StaleUtils(allFreshVs)
+
 import System.Posix.Files
-import System.Posix.Types(EpochTime)
 import System.IO(openFile, hGetContents, hClose, IOMode(..))
 import Control.Monad(filterM)
 import Control.Exception(bracketOnError)
@@ -23,14 +24,6 @@ import qualified Data.Map as M
 
 -- import Debug.Trace(traceM)
 
-getModTime :: FilePath -> IO (Maybe EpochTime)
-getModTime f =
-    do ok <- fileExist f
-       if ok
-        then do s <- getFileStatus f
-                return $ Just (modificationTime s)
-        else return Nothing
-
 -- Settings that make an object unreusable under a different setting (isStale
 -- checks these before reuse).  blockCodegen (-c mode) is omitted: its output
 -- equals the submodule form (see DEVELOP.md).  "top" marks top form (the
@@ -66,18 +59,12 @@ isStale flags prefix ba_map top_pkg pkg =
               Nothing        -> internalError $ "isStale: unknown package " ++ name
               (Just ba_file) -> do h_file <- genFileName mkHName (cdir flags) "" name
                                    o_file <- genFileName mkObjName (cdir flags) "" name
-                                   ba_time <- getModTime ba_file
-                                   h_time <- getModTime h_file
-                                   obj_time <- getModTime o_file
-                                   let stale_time =
-                                        case (ba_time, h_time, obj_time) of
-                                          (Just t1, Just t2, Just t3) -> (t2 < t1) || (t3 < t1)
-                                          _                           -> True
+                                   fresh_time <- allFreshVs ba_file [h_file, o_file]
                                    cg_opt <- readCodeGenOptionDescr h_file
                                    let is_top = (sp_name pkg) == top_pkg
                                        cg_tgt = Just ("/* " ++ (codeGenOptionDescr flags is_top) ++ " */")
                                        stale_options = cg_opt /= cg_tgt
-                                   return $ stale_time || stale_options
+                                   return $ (not fresh_time) || stale_options
 
 remove_stale :: M.Map String [String] -> [String] -> [String] -> [String]
 remove_stale _     []   _      = []

src/comp/StaleUtils.hs

@@ -0,0 +1,30 @@
+module StaleUtils ( getModTime, allFreshVs ) where
+
+import System.Posix.Files
+import System.Posix.Types(EpochTime)
+
+-- Shared primitives for the generated-file staleness checks: the -u
+-- recompilation check (Depend), Bluesim object reuse (SimFileUtils) and
+-- Verilog reuse at link (VFileUtils) all decide "regenerate or reuse"
+-- by comparing a source artifact's time against its products'.  The
+-- conventions live here so the checks agree: a missing product is never
+-- fresh, and equal timestamps are fresh.
+
+-- modification time of a file, or Nothing if it does not exist
+getModTime :: FilePath -> IO (Maybe EpochTime)
+getModTime f =
+    do ok <- fileExist f
+       if ok
+        then do s <- getFileStatus f
+                return $ Just (modificationTime s)
+        else return Nothing
+
+-- are all the products at least as new as the source?
+-- (a missing product, or a missing source, is not fresh)
+allFreshVs :: FilePath -> [FilePath] -> IO Bool
+allFreshVs source products = do
+    msrc <- getModTime source
+    mprods <- mapM getModTime products
+    let fresh (Just st) (Just pt) = pt >= st
+        fresh _         _         = False
+    return $ all (fresh msrc) mprods

src/comp/VFileUtils.hs

@@ -0,0 +1,42 @@
+module VFileUtils ( partitionStaleVerilogMods ) where
+
+import Flags(Flags(..))
+import ABin(ABinModInfo(..))
+import ASyntax(apkg_name)
+import Id(unQualId, getIdString)
+import FileNameUtil(genFileName, mkVName, getFullFilePath, getRelativeFilePath)
+import StaleUtils(allFreshVs)
+import VFileName
+
+import Data.Either(partitionEithers)
+
+-- Check whether generated Verilog files are up to date with respect to
+-- their elaborated (.ba) files.  The Verilog analogue of SimFileUtils,
+-- built on the same StaleUtils conventions, with deliberate differences:
+--   * no transitive invalidation: a parent's .v refers to child modules
+--     by name only, so a stale child never invalidates a fresh parent
+--   * no version check: a .ba that loads is current-version by
+--     construction (decodeABin rejects other versions when it is read)
+--   * no options descriptor (for now): a .v generated under different
+--     codegen flags is reused as long as it is newer than the .ba
+
+-- Split modules into (stale: regenerate, fresh: reuse the .v).  A
+-- module's .v is stale when it is missing from the location BSC would
+-- write it (per -vdir and the prefix) or is older than its .ba.  Fresh
+-- modules are returned as (full path for the simulator, relative name
+-- for reporting), mirroring writeVerilog's two uses of the file name.
+partitionStaleVerilogMods :: Flags -> String
+                          -> [(String, ABinModInfo)]
+                          -> IO ([(String, ABinModInfo)], [(VFileName, String)])
+partitionStaleVerilogMods flags prefix abmis = do
+    let checkOne (bafile, abmi) = do
+          let modstr = getIdString (unQualId (apkg_name (abmi_apkg abmi)))
+          vName_init <- genFileName mkVName (vdir flags) prefix modstr
+          let vName = getFullFilePath vName_init
+              vNameRel = getRelativeFilePath vName_init
+          fresh <- allFreshVs bafile [vName]
+          return $ if fresh
+                   then Right (VFileName vName, vNameRel)
+                   else Left (bafile, abmi)
+    results <- mapM checkOne abmis
+    return (partitionEithers results)

src/comp/bsc.hs

@@ -15,7 +15,7 @@ import System.Directory(getDirectoryContents, doesFileExist, getCurrentDirectory
 import System.Time(getClockTime, ClockTime(TOD)) -- XXX: from old-time package
 import Data.Char(isSpace, toLower, ord)
 import Data.List(intersect, nub, partition, intersperse, sort,
-            isPrefixOf, isSuffixOf, unzip5, intercalate, foldl')
+            isPrefixOf, isSuffixOf, unzip5, intercalate, foldl', tails)
 import Data.Time.Clock.POSIX(getPOSIXTime)
 import Data.Maybe(isJust, isNothing)
 import Numeric(showOct)
@@ -143,6 +143,7 @@ import SimCOpt(simCOpt)
 import SimBlocksToC(simBlocksToC)
 import SystemCWrapper(checkSystemCIfc, wrapSystemC)
 import SimFileUtils(analyzeBluesimDependencies)
+import VFileUtils(partitionStaleVerilogMods)
 import Verilog(VProgram(..), vGetMainModName, getVeriInsts)
 import Depend
 import Version(bscVersionStr, copyright, buildnum)
@@ -1576,6 +1577,11 @@ simLink errh flags toplevel afilenames cfilenames = do
 
 -- Reuse a Bluesim generated object file
 -- returns the name of the object file being reused
+-- report that a generated Verilog file was reused rather than regenerated
+reuseVerilogFile :: Flags -> String -> IO ()
+reuseVerilogFile flags vNameRel =
+    unless (quiet flags) $ putStrLnF ("Verilog file reused: " ++ vNameRel)
+
 reuseBluesimCFile :: Flags -> String -> IO String
 reuseBluesimCFile flags oName = do
     -- show is used for quoting
@@ -2025,11 +2031,12 @@ vLink errh flags topmod_name vfilenames0 afilenames cfilenames = do
                           [(cmdPosition,
                             EMultipleABinFilesForName link_name file_names)]
 
-            -- XXX Until we allow re-generation of Verilog files,
-            -- XXX the module .ba files are unused
-            when (not (null (mod_abis))) $
-                bsWarning errh
-                    [(cmdPosition, WExtraABinFiles (map fst mod_abis))]
+            -- without the design's .ba hierarchy, the staleness of
+            -- generated Verilog cannot be checked; warn and use the .v
+            -- files as found (module .ba files given explicitly on the
+            -- command line are still regenerated from, below)
+            bsWarning errh
+                [(cmdPosition, WNoABinForVerilogRegen topmod_name)]
 
             let ffuncs = [ abffi_foreign_func abfi
                            | (_, (ABinForeignFunc abfi _)) <- ffunc_abis ]
@@ -2052,17 +2059,32 @@ vLink errh flags topmod_name vfilenames0 afilenames cfilenames = do
     t <- dump errh flags t DFreadelab dumpnames
              (map (pfpString . ff_name) ffuncs ++ map fst mod_abmis)
 
-{-
-    -- generate Verilog for the module abis
-    -- XXX only if the verilog doesn't exist or is older
-    -- XXX print a message about reusing a file?
-    (t, gen_vfilenames) <-
-        if (updCheck flags)
-        then vGenMods t flags mod_abmis
-        else return (t, [])
-    let vfilenames = vfilenames0  ++ gen_vfilenames
--}
-    let vfilenames = vfilenames0
+    -- Regenerate any module .v that is missing or older than its .ba, and
+    -- reuse the rest, so a link is self-sufficient regardless of which
+    -- subset was pre-generated (with -c or an earlier compile).  A module
+    -- whose .v was given explicitly on the command line is the user's to
+    -- provide and is never regenerated.  File arguments can be glob
+    -- patterns (bsc forwards them to the simulator's shell unexpanded, so
+    -- "*.v" has always meant "the user provides every module"); match
+    -- module names against the patterns rather than comparing literally.
+    let vfile_pats = [ baseName (dropSuf (vfnString vfn))
+                     | vfn <- vfilenames0 ]
+        -- glob match with * and ? (the shell metacharacters that can
+        -- appear in a forwarded filename argument)
+        globMatch ('*':ps) s = any (globMatch ps) (tails s)
+        globMatch ('?':ps) (_:cs) = globMatch ps cs
+        globMatch (p:ps) (c:cs) = (p == c) && globMatch ps cs
+        globMatch [] [] = True
+        globMatch _ _ = False
+        abmiModName abmi = getIdString (unQualId (apkg_name (abmi_apkg abmi)))
+        checked_abmis = [ p | p@(_, abmi) <- mod_abmis
+                            , not (any (`globMatch` abmiModName abmi)
+                                       vfile_pats) ]
+    (stale_abmis, reused_vs) <-
+        partitionStaleVerilogMods flags prefix checked_abmis
+    mapM_ (reuseVerilogFile flags . snd) reused_vs
+    (t, gen_vfilenames) <- vGenMods errh flags t stale_abmis
+    let vfilenames = vfilenames0 ++ map fst reused_vs ++ gen_vfilenames
 
     -- generate files for the foreign functions
     start flags DFcompileVPI

testsuite/bsc.driver/no_filenames/vlog_build.out.expected

@@ -1 +1,7 @@
+Verilog file reused: vd/mkSub3.v
+Verilog file reused: vd/mkMid2.v
+Verilog file reused: vd/mkSub2.v
+Verilog file reused: vd/mkSub1.v
+Verilog file reused: vd/mkMid1.v
+Verilog file reused: vd/mkTop.v
 Verilog binary file created: mkTop.vexe