mdcmd

Execute commands in Markdown files, embed output, generate TOCs

e.g.:

seq 3
# 1
# 2
# 3

☝️ This block is updated programmatically by mdcmd (and verified in CI; see raw README.md).

(formerly: bmdf)

• Overview
• Install
• mdcmd: execute commands in Markdown files, embed output
  • bmdf example
  • HTML example
• bmd: format bash command and output as Markdown
  • bmdf (bmd -f): command+output mode
  • bmdff (bmd -ff): two-fence mode
  • bmdfff (bmd -fff): <details> mode
  • Piping
  • Env vars
  • -w/--workdir / $BMDF_WORKDIR
• toc: Markdown Table of Contents
• Examples

☝️ This TOC is generated programmatically by mdcmd and toc (and verified in CI; see raw README.md).

Overview

This package provides 3 CLIs:

• mdcmd: execute shell commands in Markdown files, embed output
• bmd: run Bash commands, wrap output for Markdown embedding
  • Useful in conjunction with mdcmd
  • bmdf, bmdff, bmdfff provide different types of "fencing" for command output
• toc: generate Markdown table of contents (with custom "id"s for sections)
• mktoc: convenience wrapper for mdcmd -x '^toc$'

Install

Global install via pipx or uv (recommended):

pipx install mdcmd
# or: uv tool install mdcmd

You can also install in the current (v)env:

pip install mdcmd

mdcmd: execute commands in Markdown files, embed output

mdcmd --help

Usage: mdcmd [OPTIONS] [PATH] [OUT_PATH]

  Parse a Markdown file, updating blocks preceded by <!-- `[cmd...]` -->
  delimiters.

  If no paths are provided, will look for a README.md, and operate "in-place"
  (same as ``mdcmd -i README.md``).

Options:
  -a, --amend                     Squash changes onto the previous Git commit;
                                  suitable for use with `git rebase -x`
  -C, --no-concurrent             Run commands in sequence (by default, they
                                  are run concurrently)
  -i, --inplace / -I, --no-inplace
                                  Edit the file in-place
  -n, --dry-run                   Print the commands that would be run, but
                                  don't execute them
  -T, --no-cwd-tmpdir             In in-place mode, use a system temporary-
                                  directory (instead of the current workdir,
                                  which is the default)
  -x, --execute TEXT              Only execute commands that match these
                                  regular expressions
  -X, --exclude TEXT              Only execute commands that don't match these
                                  regular expressions
  --help                          Show this message and exit.

# Modify README.md in-place
mdcmd -i README.md
# Same as above; no args defaults to `-i README.md`
mdcmd

That's how the various command examples in this file are generated / updated!

bmdf example

The example at the top of this file is generated by a line like:

<!-- `bmdf seq 3` -->

mdcmd transforms that into:

<!-- `bmdf seq 3` -->
```bash
seq 3
# 1
# 2
# 3
```

Notes:

• HTML comments (<!-- ... -->) are hidden in rendered markdown, so all the user sees is the output of bmdf seq 3
  • bmdf formats output as a "Bash fence" block
  • bmd (and variants) are useful for displaying commands (and their output) in Markdown (especially in conjunction with mdcmd).
• mdcmd is idempotent:
  • It looks for the block immediately following the <!-- `[cmd...]` --> line, and replaces that with the output of running [cmd...].
  • If there's already output there, it will be replaced with new/current output.

HTML example

Scripts that output raw HTML also work, e.g. print-table.py generates this table:

header 1	header 2
cell 1	cell 2

That table is generated by a line like:

<!-- `python test/print-table.py` -->

mdcmd maintains an output block immediately after it:

<!-- `python test/print-table.py` -->
<table>
  <tr>
    <th>header 1</th>
    <th>header 2</th>
  </tr>
  <tr>
    <td>cell 1</td>
    <td>cell 2</td>
  </tr>
</table>

bmd: format bash command and output as Markdown

bmd --help

Usage: bmd [OPTIONS] COMMAND...

  Format a command and its output to markdown, either in a `bash`-fence or
  <details> block, and copy it to the clipboard.

Options:
  -A, --strip-ansi                Strip ANSI escape sequences from output
  -C, --no-copy                   Disable copying output to clipboard
                                  (normally uses first available executable
                                  from ['pbcopy', 'xclip', 'clip']
  -e, --error-fmt TEXT            If the wrapped command exits non-zero,
                                  append a line of output formatted with this
                                  string. One "%d" placeholder may be used,
                                  for the returncode. Defaults to
                                  $BMDF_ERR_FMT
  -E, --env TEXT                  k=v env vars to set, for the wrapped command
  -f, --fence                     Pass 0-3x to configure output style: 0x:
                                  print output lines, prepended by "# "; 1x:
                                  print a "```bash" fence block including the
                                  <command> and commented output lines; 2x:
                                  print a bash-fenced command followed by
                                  plain-fenced output lines; 3x: print a
                                  <details/> block, with command <summary/>
                                  and collapsed output lines in a plain fence.
  -i, --include-stderr / -I, --no-include-stderr
                                  Capture and interleave both stdout and
                                  stderr streams; falls back to
                                  $BMDF_INCLUDE_STDERR
  -s, --shell / -S, --no-shell    Disable "shell" mode for the command; falls
                                  back to $BMDF_SHELL, but defaults to True if
                                  neither is set
  -t, --fence-type TEXT           When -f/--fence is 2 or 3, this customizes
                                  the fence syntax type that the output is
                                  wrapped in
  -u, --expanduser / -U, --no-expanduser
                                  Pass commands through `os.path.expanduser`
                                  before `subprocess`; falls back to
                                  $BMDF_EXPANDUSER
  -v, --expandvars / -V, --no-expandvars
                                  Pass commands through `os.path.expandvars`
                                  before `subprocess`; falls back to
                                  $BMDF_EXPANDVARS
  -w, --workdir TEXT              `cd` to this directory before executing
                                  (falls back to $BMDF_WORKDIR
  -x, --executable TEXT           `shell_executable` to pass to Popen
                                  pipelines (default: $SHELL)
  --help                          Show this message and exit.

bmd (and aliases bmdf, bmdff, bmdfff) takes a bash command as input, and renders the command and/or its output in various Markdown-friendly formats:

bmdf (bmd -f): command+output mode

Suppose you want to embed a command and its output in a README.md, like this:

seq 3
# 1
# 2
# 3

(Note how the command is bash-highlighted, and output lines are rendered as comments)

Put a placeholder like this in your README.md:

<!-- `bmdf seq 3` -->

then run mdcmd to update your README containing this embedded command block.

bmdff (bmd -ff): two-fence mode

bmdff (alias for bmd -ff) renders two code fences, one with the Bash command (syntax-highlighted appropriately), and a second (non-highlighted) block with the output, e.g.:

<!-- `bmdff seq 5` -->

becomes:

seq 5

1
2
3
4
5

bmdfff (bmd -fff): <details> mode

When a command's output is large, rendering it as a <details><summary> (with the output collapsed, by default) may be preferable.

bmdfff (3 fs, alias for bmd -fff) transforms placeholders like this:

<!-- `bmdfff seq 10` -->

to:

seq 10

1
2
3
4
5
6
7
8
9
10

Piping

Piping works too, e.g.:

<!-- `bmdf -- seq 10 | wc -l` -->

will become:

seq 10 | wc -l
# 10

(the -- is needed so that that -l isn't parsed as an opt to bmdf)

Env vars

By default, shell=True is passed to subprocess calls (but can be disabled via -S).

This means env vars are expanded; they can also be set via -E, e.g.:

<!-- `bmdf -E FOO=bar echo $FOO` -->

yields:

FOO=bar echo '$FOO'
# bar

More examples of quoting/splitting behavior

Quoting "$FOO":

<!-- `bmdf -E FOO=bar echo "$FOO"` -->

yields:

FOO=bar echo '$FOO'
# bar

Arg with spaces:

<!-- `bmdf -E FOO=bar echo "FOO: $FOO"` -->

yields:

FOO=bar echo 'FOO: $FOO'
# FOO: bar

Escaping $:

<!-- `bmdf -E FOO=bar echo "\$FOO=$FOO"` -->

yields:

FOO=bar echo '\$FOO=$FOO'
# $FOO=bar

-w/--workdir / $BMDF_WORKDIR

By default, bmdf runs in the current working directory. This can be overridden with -w:

<!-- `bmdf -w .github ls` -->

ls
# workflows

toc: Markdown Table of Contents

toc --help

Usage: toc [OPTIONS] [PATH]

  Generate a table of contents from a markdown file.

  If no PATH is provided, will try to use $MDCMD_FILE (set by mdcmd), or
  default to README.md if that's not set.

Options:
  -n, --indent-size INTEGER  Indent size (spaces)
  --help                     Show this message and exit.

toc generates a table of contents from markdown headings, and pairs well with mdcmd for maintaining TOCs in markdown files.

1. Put a line like this in your README.md:

   <!-- `toc` -->
   
   

   (the trailing blank line is important, don't put other content immediately under a <!-- ... --> line)

2. Put empty <a> tags next to headings to includ them in the TOC (and specify an id):

   ## My section heading <a id="my-section"></a>

   This allows for custom/short ids, as well as skipping sections.

3. Run mdcmd as usual:

   # Update all command blocks in README.md, including the TOC
   mdcmd

   mdcmd will see the <!-- toc -->, and embed the TOC generated by toc under it.

A mktoc script is also provided, which just wraps mdcmd -x '^toc$' (mktoc was implemented separately, in previous versions, before being decomposed into mdcmd and toc in 0.7.0).

Examples

• The examples in this file are all rendered by bmdf and mdcmd.
• The TOC above is rendered by toc.
• The ci.yml GitHub Action verifies the examples and TOC.

These repos' READMEs also use bmdf / mdcmd / toc to execute example commands (and in some cases also verify them with a GitHub Action):

• runsascoded/juq
• runsascoded/utz
• runsascoded/dvc-utils
• ryan-williams/dvc-helpers (GHA)
• ryan-williams/git-helpers
• ryan-williams/parquet-helpers (GHA)
• ryan-williams/tdbs-dask
• TileDB-Inc/scverse-ml-workshop-2024