[adityamparikh]

Part of #256. Adds the harness for the opt-in LLM tool-selection evals and the ./gradlew eval task. The first scenario is #265.

Stacked on #269 (./gradlew build runs buildSrc's tests) and #266 (google-java-format): the first two commits here are those PRs, and only the top commit (test(eval): add the harness and Gradle task for LLM tool-selection evals) is new.

How it works

A Spring AI ChatClient on OpenAI, set up as an MCP client of solr-mcp, gets natural-language prompts and calls the server's tools. An advisor records every tool call so a scenario can check which tools were called and with what arguments.

• ./gradlew eval runs tests tagged llm-eval. Every other Test task excludes that tag, so ./gradlew build never calls a model. eval is never up to date, doesn't finalize jacocoTestReport (which would pull in test), and passes the bootJar path as solr.mcp.jar.
• EvalConfiguration is a @TestConfiguration with @EnableAutoConfiguration, so Main's component scan never picks it up. Auto-configuration provides everything else:
  • Model: spring-ai-starter-model-openai with Spring AI's defaults (gpt-5-mini, key from OPENAI_API_KEY). Nothing is configured.
  • MCP client: spring-ai-starter-mcp-client starts the built jar over STDIO. The connection is in application-integration-test.properties: ${java.home}/bin/java, -jar ${solr.mcp.jar}, and SOLR_URL=${solr.url} from TestcontainersConfiguration. The profile also sets spring.ai.mcp.server.enabled=false, so auto-configuration doesn't start an MCP server on the test JVM's stdin and stdout.
  • ChatClient bean: the server's instructions (defaultSystem), its tools (defaultTools), and ToolCallAuditAdvisor (defaultAdvisors).
  • Data: an ApplicationRunner loads mydata/devnexus-2026.json (116 sessions) into devnexus through the server's own tools.
• ToolCallAuditAdvisor observes the tool-calling loop from inside, as the Spring AI reference describes under Observing the tool-calling loop. Ordered after ToolCallingAdvisor, it runs on every iteration, and each iteration's request carries the turn so far. When the model answers without asking for another tool, that request holds every tool call and the ToolResponseMessage that answered it. The advisor records them as spring-ai-typesafe JevJudgeInput.ToolCall (name, arguments, result), and ToolCallAuditAdvisor.toolCalls(response) reads them back. No tool is wrapped.
• ToolCallAuditAdvisorTest (in build) runs a real ChatClient tool loop over a scripted ChatModel and checks what the advisor recorded.

The three new dependencies are test-only. The shipped classpath, the SBOM (141 components) and the generated LICENSE are unaffected.

Verification

• ./gradlew build: 467 tests, 0 skipped, 0 failed (with JDK 25 first on PATH for McpClientStdioIntegrationTest), plus buildSrc's 12 tests, which run in the same build via build: run buildSrc's tests as part of the build #269.
• ./gradlew nativeTest -Pnative: 306 passed, 0 failed.
• The eval context itself is exercised by test(eval): check the search arguments for a workshops prompt #265's scenario.

🤖 Generated with Claude Code

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud