Markdown editor with Rich Text preview via MDAST - with cleanup

[Soxasora]

Description

Based on #2657
This is the first stage of the transition to a new Lexical-based editor.

Introduces a Lexical plain text editor with a rich text preview, using MDAST to map and transform Lexical nodes to markdown, aiming to provide lossless transformations.

This PR focuses on providing the same editor experience as before, while transitioning to Lexical.

cleanup: Removes unused text input code and legacy markdown-related code.

Screenshots

Refreshed reply editor style, same background as comments

Refreshed top-level editor style

Editor workings

Informations can be found here:

• Lexical: lib/lexical/README.md
• SN Extensions: lib/lexical/exts/README.md
• Server: lib/lexical/server/README.md -> TODO
• Components: components/editor/README.md -> TODO

Removed

• LegacyText (ReactMarkdown) -> Text (SNReader)
  • Mention -> UserMentionNode
  • Sub -> TerritoryMentionNode
  • Item -> ItemMentionNode
  • Footnote -> Reference, Definition, Section, Backref nodes
  • MediaLink -> MediaNode
  • Table -> TableNode (handled by lexical)
  • CodeSkeleton
  • Code -> CodeNode (handled by lexical with Shiki)
  • P
• remarkToc -> handled by MDAST and Lexical
• remarkUnicode -> not needed under KaTeX
• remarkMath -> extracted mdast-util-math
• rehypeSN -> feature parity achieved via MDAST visitors and transforms
• rehypeMathjax -> replaced with KaTeX
• rehypeSNStyled -> superseded by MDAST
• remarkGfm -> not needed
• React Syntax Highlighter -> replaced with Shiki

Bundle analysis

tmi: Lexical post-removals is 0.03 MB heavier than master on first load (what matters for the client). I was expecting a much higher first-load and overall bundle from Lexical, considering all the things it handles and can handle, but it's instead marginal.

Master

_app

+ First Load JS shared by all              1.01 MB
  ├ chunks/framework-c6ee5607585ef091.js   44.9 kB
  ├ chunks/main-ea7afcb22284b775.js        37.3 kB
  ├ chunks/pages/_app-99d52cc32cadfb73.js  883 kB
  ├ css/554058231c942e24.css               46.5 kB
  └ other shared chunks (total)            2.13 kB

Lexical pre-removals

_app

+ First Load JS shared by all              1.09 MB
  ├ chunks/framework-c6ee5607585ef091.js   44.9 kB
  ├ chunks/main-ea7afcb22284b775.js        37.3 kB
  ├ chunks/pages/_app-e6dec95ecd8fe254.js  948 kB
  ├ css/198ebdadd5386d4b.css               55.4 kB
  └ other shared chunks (total)            6.2 kB

Lexical post-removals

_app

+ First Load JS shared by all              1.04 MB
  ├ chunks/framework-c6ee5607585ef091.js   44.9 kB
  ├ chunks/main-ea7afcb22284b775.js        37.3 kB
  ├ chunks/pages/_app-db96a9243c75fba9.js  897 kB
  ├ css/087336682c6d686c.css               55.9 kB
  └ other shared chunks (total)            6.14 kB

Feature parity

parity list

• write/preview: Y
• bold shortcut: Y
• italic shortcut: Y
• link shortcut: Y
• upload shortcut: Y
• tab insert shortcut: NO
• submit shortcut: Y
• paste images: Y
• file DnD: Y
• upload button: Y
• upload fees: Y
• user/territory/item mentions and autocomplete: Y
• local drafts: Y
• formik: Y
• max length: Y
• undo/redo: Y (by Lexical HistoryPlugin)
• markdown help link: NO
• textarea autosize: Y (by Lexical)
• rich preview: Y (LexicalReader)
• footnotes: Y

new features

• preview toggle shortcut (meta+P)
• undo/redo via Lexical
• Shiki syntax highlighting
• KaTeX math support
• MDAST-based markdown transformations

MDAST

Lexical already supports markdown transformations via @lexical/markdown, the problem with Lexical's native approach is that we're forced to do workarounds for proper markdown bi-directional transformations.

By treating the Lexical tree as an MDAST tree we can be confident in creating lossless bi-directional markdown transformations. More informations can be found in lib/lexical/mdast/README.md

Additional Context

lib/lexical/server/headless.js
Creating a fake DOM for Lexical operations server-side pollutes the global environment

global.window = newWindow
global.document = newDocument

It is safely cleaned at the end of an operation with the finally block, but it's still something we should keep an eye on.

---

CLEANUP

The following removals have been postponed:

• useDualAutocomplete
  • provides user and territory mentions to inputs, should be replaced with Lexical but this raises questions about avoidable overhead on simple text inputs
• remove-markdown
  • removes markdown syntax from given text, its performance has to be evaluated to justify migration to our own MDAST system

Checklist

Are your changes backward compatible? Please answer below:

For example, a change is not backward compatible if you removed a GraphQL field or dropped a database column.

On a scale of 1-10 how well and how have you QA'd this change and any features it might affect? Please answer below:

For frontend changes: Tested on mobile, light and dark mode? Please answer below:

Did you introduce any new environment variables? If so, call them out explicitly here:

Did you use AI for this? If so, how much did it assist you?

---

Note

Replaces legacy markdown pipeline with a Lexical-based editor/reader, adds SSR lexical processing and HTML generation, updates GraphQL schema/resolvers, and refactors UI to use the new system.

• Editor/Reader (Frontend):
  • Introduces components/editor/* (Lexical editor/reader, toolbar, plugins: upload, mentions, preview, code theme, etc.).
  • Replaces MarkdownInput with SNInput across forms and switches rendering to Text using SNReader.
  • Adds KaTeX renderer, gallery/media nodes handling, ToC via Lexical, and new styles in styles/text.scss.
  • Updates item/comment/bio/territory components to consume item.lexicalState/item.html and pass readerRef.
• API/SSR:
  • Extends GraphQL Item and Sub types with lexicalState and html fields.
  • Adds resolvers to load lexical state and generate HTML (lexicalStateLoader, lexicalHTMLGenerator); wires context in SSR Apollo and API.
• Lexical Infra:
  • Adds MDAST import/export utilities, custom nodes (mentions, media, gallery, footnotes, math, ToC), extensions (autolink, item-context, gallery, shiki, shortcuts, md-commands), and server helpers (headless, loader).
• Other:
  • Updates MediaOrLink, ToC, and tooltips; adds use-callback-ref.
  • Config: Next.js alias for canvas.
  • Dependencies: Adds Lexical, Shiki, KaTeX, DOMPurify, LinkeDOM, DataLoader, mdast/micromark libs; removes legacy markdown/rendering deps.

^{Written by Cursor Bugbot for commit 4fac5bb. This will update automatically on new commits. Configure here.}

[huumn]

Cannot update a component (SNReader) while rendering a different component (LexicalExtensionComposer). To locate the bad setState() call inside LexicalExtensionComposer, follow the stack trace as described in https://reactjs.org/link/setstate-in-render Error Component Stack

alternative pattern using context:

import dynamic from "next/dynamic"
import { createContext, useContext } from "react"
import { useRouter } from "next/router"

const HtmlContext = createContext("")

function HtmlFallback() {
  const html = useContext(HtmlContext)
  return <div dangerouslySetInnerHTML={{ __html: html }} />
}

const Reader = dynamic(() => import("./reader"), {
  ssr: false,
  loading: () => <HtmlFallback />,
})

export function SNReader({ html, ...props }) {
  const router = useRouter()

  // optional: avoid query flicker on first render in some cases
  const showDebugHtml = router.isReady && router.query.html != null

  if (showDebugHtml) {
    return <div dangerouslySetInnerHTML={{ __html: html }} />
  }

  return (
    <HtmlContext.Provider value={html}>
      <Reader {...props} />
    </HtmlContext.Provider>
  )
}

[Soxasora]

This fits perfectly, thank you!

[Soxasora]

this ? was conflicting with the new changes

[Soxasora]

This PR can be merged without the previous one as QA is being done here, and new changes and fixes are being pushed here.

If it's better to do the original PR and then the cleanup PR, I can un-draft