Skip to content

getting_started.md

Latest commit

 

History

History
207 lines (153 loc) · 9.56 KB

getting_started.md

File metadata and controls

207 lines (153 loc) · 9.56 KB
Raw

Getting Started

Read this to walk through every feature on one concrete step for the first time.

The README shows a minimal SummarizeArticle step. This guide walks through the features you reach for as production requirements grow: budget caps so runaway inputs don't drain your LLM provider budget, evals so you catch regressions in CI, and CI gating so a merge that lowers accuracy gets blocked.

The walkthrough

Start with the README example, then add features one layer at a time. Each is optional — use what you need.

The prompt body uses a small template syntax — {X} placeholders are replaced with values from the input at call time. With the default input_type String, the entire article_text you pass to SummarizeArticle.run(article_text) lands in {input}. For input_type Hash, each top-level key becomes its own placeholder (e.g. {summary}, {article}). See Prompt AST for the full DSL surface.

class SummarizeArticle < RubyLLM::Contract::Step::Base
  # 1. Prompt (required)
  prompt <<~PROMPT
    Summarize this article for a UI card. Return a short TL;DR,
    3 to 5 key takeaways, and a tone label.

    {input}
  PROMPT

  # 2. Schema — sent to the provider via with_schema, validated client-side
  output_schema do
    string :tldr
    array  :takeaways, of: :string, min_items: 3, max_items: 5
    string :tone, enum: %w[neutral positive negative analytical]
  end

  # 3. Business rules — things JSON Schema cannot express
  validate("TL;DR fits the card")  { |o, _| o[:tldr].length <= 200 }
  validate("takeaways are unique") { |o, _| o[:takeaways] == o[:takeaways].uniq }

  # 4. Retry with model fallback on validation_failed / parse_error
  retry_policy models: %w[gpt-4.1-nano gpt-4.1-mini gpt-4.1]

  # 5. Refuse before calling the LLM if input is too large or estimated cost exceeds the cap
  max_input  2_000
  max_output 4_000
  max_cost   0.01
end

Each validate block receives (parsed_output, context) — the convention |o, _| reads "output, ignore context". Use o[:key] to assert on any field your output_schema declared. The block returns truthy/falsy: false → status :validation_failed, retry fires if retry_policy was set.

Validation and retry behavior

When the cheap model returns output that fails a validate block (status :validation_failed) or can't be parsed as JSON (status :parse_error), retry falls back to the next model in models: and tries again. Transport errors (timeouts, 5xx) retry independently via Faraday, not via retry_policy. See Optimizing retry_policy for tuning the chain.

result = SummarizeArticle.run(article_text)

result.status           # => :ok
result.parsed_output    # => { tldr: "...", takeaways: [...], tone: "analytical" }
result.trace[:model]    # => "gpt-4.1-mini"  (first model that passed)
result.trace[:cost]     # => 0.00052  (sum of all attempts)
result.trace[:attempts]
# => [
#   { attempt: 1, model: "gpt-4.1-nano", status: :validation_failed,
#     cost: 0.00010, latency_ms: 45, ... },
#   { attempt: 2, model: "gpt-4.1-mini", status: :ok,
#     cost: 0.00042, latency_ms: 92, ... }
# ]

If the whole chain exhausts, result.status is the status of the last attempt (:validation_failed or :parse_error) and result.parsed_output is the last attempt's output. The caller decides what to do — ship it anyway, fall back to a template, or raise.

Per-attempt reasoning effort

models: accepts config hashes as well as model-name strings, so a fallback can "try harder" (more reasoning) on retry, not just switch model:

retry_policy models: [
  "gpt-5-nano",                                      # attempt 1: cheap + fast
  { model: "gpt-5-mini", reasoning_effort: "high" }  # attempt 2: stronger + more reasoning
]

reasoning_effort is a gpt-5 family feature (gpt-4.1 is not a reasoning model). The per-attempt value is forwarded via with_thinking (provider-agnostic — OpenAI reasoning_effort and Anthropic extended-thinking budget both supported). Passing it alongside a non-reasoning model is forwarded unchanged to the provider, which will either ignore it or reject the request — the gem does not guard against this.

Evals and CI gates

An eval is a named scenario you can run to verify the step still works. sample_response makes it offline — zero API calls — so CI can run it on every merge without burning budget.

SummarizeArticle.define_eval("smoke") do
  default_input <<~ARTICLE
    Ruby 3.4 ships with frozen string literals on by default, measurable YJIT
    speedups on Rails workloads, and tightened Warning.warn category filtering.
    The release notes also mention several parser fixes and faster keyword
    argument handling.
  ARTICLE

  sample_response({
    tldr: "Ruby 3.4 brings frozen string literals by default, YJIT speedups, and parser fixes.",
    takeaways: [
      "Frozen string literals are the default",
      "YJIT adds measurable speedups on Rails workloads",
      "Warning.warn category filtering is tighter"
    ],
    tone: "analytical"
  })
end

report = SummarizeArticle.run_eval("smoke")
report.passed?  # => true — schema + validates pass on the canned response
report.score    # => 1.0
report.print_summary

default_input sets the input once for every case in this eval — useful for smoke evals where you reuse one synthetic article. For regression evals where each case needs its own input, use add_case input: "..." (shown next).

For real regression testing, define cases with expected output (online — calls the LLM):

SummarizeArticle.define_eval("regression") do
  add_case "ruby release",
           input: "Ruby 3.4 was released...",
           expected: { tone: "analytical" }  # partial match

  add_case "critical review",
           input: "The new mesh networking hardware failed under load...",
           expected: { tone: "negative" }
end

expected: is treated as a partial match against parsed_output — extra output keys are ignored, listed keys must equal. So this case passes whenever the model produces tone: "analytical", regardless of tldr content. To assert on multiple keys, list them all in expected.

Gate CI on score and cost thresholds:

# RSpec — blocks merge if accuracy drops or cost spikes
expect(SummarizeArticle).to pass_eval("regression")
  .with_minimum_score(0.8)
  .with_maximum_cost(0.01)

Save a baseline once, then block regressions automatically:

report = SummarizeArticle.run_eval("regression")
report.save_baseline!

# In CI:
expect(SummarizeArticle).to pass_eval("regression").without_regressions

without_regressions fails the build only if a previously-passing case now fails — a new model version, a prompt tweak, or an upstream change that silently lowered quality.

Budget caps

max_input, max_output, and max_cost are preflight checks — the LLM is never called if an estimate exceeds the limit. Zero tokens spent, zero cost.

result = SummarizeArticle.run(giant_10mb_document)
result.status            # => :limit_exceeded
result.validation_errors
# => ["Input token limit exceeded: estimated 32000 tokens (heuristic ±30%), max 2000"]

max_cost fails closed when the model's pricing isn't known — register custom or fine-tuned models explicitly:

RubyLLM::Contract::CostCalculator.register_model("ft:gpt-4o-custom",
  input_per_1m: 3.0, output_per_1m: 6.0)

Or opt into a soft warning instead of a refusal when pricing is missing:

max_cost 0.01, on_unknown_pricing: :warn

Default is :refuse. Use :warn only when you accept running without a cost ceiling (fine-tuned models you trust, private endpoints).

Preflight cost estimates

Check what a call is likely to cost before invoking it:

SummarizeArticle.estimate_cost(input: article_text)
# => {
#      model: "gpt-4.1-mini",
#      input_tokens: 812, output_tokens_estimate: 4000,
#      estimated_cost: 0.00243
#    }

# Estimate what a full eval would cost across candidate models
SummarizeArticle.estimate_eval_cost("regression",
  models: %w[gpt-4.1-nano gpt-4.1-mini gpt-4.1])
# => { "gpt-4.1-nano" => 0.00041, "gpt-4.1-mini" => 0.0018, "gpt-4.1" => 0.0092 }

estimate_cost returns nil when pricing isn't registered. estimate_eval_cost silently treats unknown-pricing cases as $0.00 and sums the rest — it does not fail closed the way max_cost does. Treat its output as a floor, not a guarantee; register pricing via CostCalculator.register_model before relying on it for budget decisions.

output_schema vs with_schema

with_schema in ruby_llm tells the provider to force a specific JSON structure. output_schema in this gem does the same thing (calls with_schema under the hood) plus validates the response client-side. Cheaper models sometimes ignore schema constraints — with_schema is a request; output_schema is a request plus verification.

See also

  • Prompt AST — prompt DSL variants: system, rule, section, example, user, and dynamic prompts with |input|.
  • Eval-First — datasets, baselines, A/B gates, the workflow that makes the above evals useful.
  • Optimizing retry_policy — find the cheapest viable fallback list with compare_models and optimize_retry_policy.
  • Testing — test adapter, stub_step, full RSpec + Minitest matcher reference.
  • Output Schema — nested objects in arrays, constraints, pattern reference.
  • Rails integration — where step classes live, initializer, jobs, logging, specs, CI gate.