The system prompt generated by catalog.prompt() is approximately 4,300 tokens for our catalog of 18 components.
This creates problems for small language models and models running locally on browsers, mobile devices, and other edge environments. Many of these models support relatively small context windows (at most ~4k tokens). A 4,300-token system prompt, alone, cannot fit inside this context window.
For now, we manually reduced the system prompt to approximately 2,000 tokens, so json-render can work on a SLM.
Desired outcome
Provide a way in PromptOptions to enable the caller of catalog.prompt() to generate a much smaller system prompt.
Implementation Ideas:
size option: a rough t-shirt size (sm, md, lg) for high level prompt length control, (where sm targets the prompt to ~1k tokens, md 2k tokens, and lg the current size)sections options + size option: grandular control over the specificity at each sectiontemplate option (with sections + size hooks): allow user to add a handlebars-like template with preset helpers for sections. This is the most robust and flexible but could potentially introduce a new dependency.
The system prompt generated by
catalog.prompt()is approximately 4,300 tokens for our catalog of 18 components.This creates problems for small language models and models running locally on browsers, mobile devices, and other edge environments. Many of these models support relatively small context windows (at most ~4k tokens). A 4,300-token system prompt, alone, cannot fit inside this context window.
For now, we manually reduced the system prompt to approximately 2,000 tokens, so json-render can work on a SLM.
Desired outcome
Provide a way in
PromptOptionsto enable the caller ofcatalog.prompt()to generate a much smaller system prompt.Implementation Ideas:
sizeoption: a rough t-shirt size (sm,md,lg) for high level prompt length control, (wheresmtargets the prompt to ~1k tokens,md2k tokens, andlgthe current size)sectionsoptions +sizeoption: grandular control over the specificity at each sectiontemplateoption (with sections + size hooks): allow user to add a handlebars-like template with preset helpers for sections. This is the most robust and flexible but could potentially introduce a new dependency.