Agent evals
The only eval material read here for CAP Java is the Java half of SAP's own skill eval file, which asks what a coding agent writes. The question an application asks of its own model output is answered in the ordinary test support capire's Java testing guide documents, with the fixtures and the judging written by hand.
Showing artifacts, examples and content with confirmed Java applicability. View All CAP
Overview
The same cap-developer eval file marks cases "stack": "java", so SAP's own skill evals ask the Java questions too, and capire's Java testing guide covers running tests against a CAP Java application.
Not covered: At the coding agent. @saptarishi/cds-plugin-llm@saptarishi/cds-plugin-llmA CAP service kind that puts eleven model providers behind one chat, stream, embed and batch interface, so which backend a service calls is a profile in package.json rather than a change to the handler.independent, cap, which runs the regression inside the application, is a CDS plugin and states Node.js, so a CAP Java application assembles its own fixtures, its own judge and its own run on top of the test support capire documents.
Core-only is the reading of the providers that state a flavor: a provider that states a Java answer for an application's own model output would change it.
No community artifact is documented here.
Artifacts
Tasks
Catch a change in what the application's AI feature answers
A model, prompt or schema change is caught before it ships, by sending the application's own prompts over fixtures and judging the answers that come back.
Covers the model output the application itself produces. Checking a coding agent that writes the application is a separate task, and comparing two models before one is picked is a third.
Check that a coding agent still writes correct CAP code
After a model, a prompt or a skill set changes, the agent is given prompts whose expected answer is written down, and what it produced is checked against them.
Covers checking what a coding agent writes for CAP work. Checking the model output an application ships to its own users is a separate task, and so is writing the skills themselves.
capire/skillsTwo skill directories read here carry an eval file. cap-add-remote-service holds three prompts, the expected answer written out for each, and a CAP fixture project at evals/fixture to run them against; cap-developer holds prompts whose assertions are named one by one, and marks every case "stack": "node" or "stack": "java", so the same file asks the question in both CAP flavors.
Remaining work: Run them. No page read here documents a runner, a command or what counts as a pass, so the prompt, the fixture and the comparison are driven by whatever the project already has. The cases are about SAP's own skills, so a project that has written its own skills has written no cases for them.
Compare two models for a CAP feature before choosing one
Two models, or two versions of a prompt, answer the same cases and the results are put side by side before one of them is wired into the service.
Covers picking between models or prompt versions on evidence. Catching a later change in what the chosen one answers is a separate task, and versioning the prompt itself is prompt management.
Details
Coverage
Designs that place it
Comparison properties
These are the questions used to compare what the artifacts do. Values come from their documentation and may apply to a particular ecosystem or runtime. Each recorded value in the comparison opens its statement and sources.
For your agent
- Ref
capability:agent-evals- JSON
- /capabilities/agent-evals.json
- MCP
- Connecting an agent