Skip to content

feat: display inline images from tool results in chat buffer - #221

Merged
dnouri merged 3 commits into
dnouri:masterfrom
conornash:conornash/inline-images
Aug 30, 2026
Merged

feat: display inline images from tool results in chat buffer#221
dnouri merged 3 commits into
dnouri:masterfrom
conornash:conornash/inline-images

Conversation

@conornash

Copy link
Copy Markdown
Contributor

When a tool (e.g. read) returns image content blocks alongside text, the images are now rendered inline inside the tool block overlay.

In GUI Emacs: decoded and displayed via create-image/insert-image with max-width/max-height scaling to fit the window.

In terminal Emacs: a text placeholder showing MIME type and size.

Implementation:

  • New struct slot: tool-block.image-blocks stores content for toggle
  • pi-coding-agent--mime-to-image-type: MIME string to Emacs type symbol
  • pi-coding-agent--insert-inline-image: decode + insert or placeholder
  • pi-coding-agent--insert-tool-images: iterate content, insert images
  • display-tool-end: calls insert-tool-images after text rendering
  • toggle-tool-output: re-inserts images from stored blocks after toggle

Handles: corrupt base64, unknown MIME types, empty data, vectors and lists, multiple image blocks, image-only results (no text).

12 new tests covering all paths. 942/942 full suite passes.

@conornash
conornash force-pushed the conornash/inline-images branch from 3174ef2 to fb9aa34 Compare June 8, 2026 18:23
@conornash
conornash force-pushed the conornash/inline-images branch 2 times, most recently from 5c62560 to bb62753 Compare June 17, 2026 15:47
dnouri pushed a commit to conornash/pilish that referenced this pull request Aug 30, 2026
Render completed tool-result image blocks inline in graphical Emacs and as
type-and-size placeholders in terminals. Bound each returned source and the
number shown per tool, and preserve rendered previews through toggling and
cooling.
@dnouri
dnouri force-pushed the conornash/inline-images branch from bb62753 to 5a12525 Compare August 30, 2026 08:44
@dnouri

dnouri commented Aug 30, 2026

Copy link
Copy Markdown
Owner

Thanks @conornash for the original #221 work, and @bobrowadam for the #211 implementation consolidated here.

I’ve refreshed this PR onto current master and kept it focused on displaying image results:

  • Completed tool-result images appear inline in graphical Emacs.
  • Terminals show a useful image type-and-size placeholder.
  • Raster images returned by Pi’s built-in read use Pi’s persisted image content; replay never reopens the original path.
  • Complete, standalone SVG text returned by read can also be previewed. SVGs with obvious scripts or external resources remain ordinary text.
  • Previews survive history replay, output toggling, and cooled tool history.
  • Returned image sources are capped per image, and at most eight previews are shown per tool result.

The branch also keeps test dependencies project-local and passes the selected Emacs/package directory to child batch, GUI, and benchmark processes. This fixes the package lookup mismatch behind the old CI failures without changing normal compile behavior.

Verification is green: full make check, 18/18 headless GUI tests, native GUI coverage, fake integration, Emacs 29 focused coverage, empty-HOME package propagation, tool-update/reload smoke benchmarks, and a terminal smoke with PNG and SVG placeholders.

A few limits are intentional:

  • Image blocks from partial tool updates are shown when the final result arrives.
  • Pi’s built-in read resizes raster images, but unusual custom-tool images may still cost more decoder memory than their compressed size suggests.
  • A chat first rendered in a terminal may need a toggle or history reload after moving to a GUI.
  • SVG support is deliberately conservative rather than a general SVG sanitizer.

This remains display-only. Pasting an image or typing its path does not attach image data to an outgoing prompt; prompt image attachments remain separate #261 work.

Lugh and others added 2 commits August 30, 2026 10:50
Keep dependency installs project-local per Emacs major and pass the selected
Emacs and package directory to child batch, GUI, and benchmark processes.
Preserve the existing in-tree compile behavior.

Co-authored-by: Daniel Nouri <daniel.nouri@gmail.com>
Render completed tool-result image blocks inline in graphical Emacs and as
type-and-size placeholders in terminals. Bound each returned source and the
number shown per tool, and preserve rendered previews through toggling and
cooling.
@dnouri
dnouri force-pushed the conornash/inline-images branch from 5a12525 to bec5ecf Compare August 30, 2026 08:54
Preview complete standalone SVG text returned by read without reopening the
tool argument path. Leave text with obvious scripts or external resources
undecoded, and keep previews display-only.

Co-authored-by: Adam Bobrow <adam@bradwell.ai>
@dnouri
dnouri force-pushed the conornash/inline-images branch from bec5ecf to c81812f Compare August 30, 2026 10:56
@dnouri

dnouri commented Aug 30, 2026

Copy link
Copy Markdown
Owner

A final protocol-shaped review caught one small mismatch before merge: ordinary read results encode truncation as JSON null, and the SVG preview gate was treating Emacs’s :null sentinel as a real truncation record.

That is now corrected. Nil/JSON-null means the returned SVG is complete, while non-null truncation metadata still leaves it as text. Regression coverage exercises both cases, and all 11 replacement CI checks pass.

@dnouri
dnouri merged commit 78ed28a into dnouri:master Aug 30, 2026
11 checks passed
@conornash
conornash deleted the conornash/inline-images branch August 31, 2026 12:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants