Agent evals
Knowing whether an AI feature still behaves after a model, prompt or schema change. Regression testing for non-deterministic output, against fixtures that reflect real business data.
Showing artifacts, examples and content with confirmed Node.js applicability. View All CAP
Overview
cap-developer's eval file marks each case with a stack, and "stack": "node" is one of the two it uses, so SAP's own skill evals ask the Node.js questions. cds.test runs whatever test the project writes around them.
Not covered: Where the door's answer stops. Nothing read here is specific to Node.js beyond the marker: the eval files arrive without a runner, and the regression question for an application's own model output is answered by a plugin rather than by the runtime.
The relationship is unassessed here for the same reason it is on the door: how the plugin came to stand beside capire/skills is not documented in any page retrieved.
@saptarishi/cds-plugin-llm@saptarishi/cds-plugin-llmA CAP service kind that puts eleven model providers behind one chat, stream, embed and batch interface, so which backend a service calls is a profile in package.json rather than a change to the handler.independent, cap runs the regression inside the application, and it is a CDS plugin, which the graph reads as Node.js.
Artifacts
Tasks
Catch a change in what the application's AI feature answers
A model, prompt or schema change is caught before it ships, by sending the application's own prompts over fixtures and judging the answers that come back.
Covers the model output the application itself produces. Checking a coding agent that writes the application is a separate task, and comparing two models before one is picked is a third.
@saptarishi/cds-plugin-llmpromptRegression runs a fixture file the project wrote through the configured provider and formatRegressionReport prints what came back; llmJudge and judgeMany score an answer against a written rubric with a model. Because the backend is a profile under cds.requires.llm, the run and the feature it guards can sit on different models.
Remaining work: Write the fixtures and the rubric, and decide the threshold. A judge is itself a model call, so what it costs and how steady its own answers are belong to the project's run.
Check that a coding agent still writes correct CAP code
After a model, a prompt or a skill set changes, the agent is given prompts whose expected answer is written down, and what it produced is checked against them.
Covers checking what a coding agent writes for CAP work. Checking the model output an application ships to its own users is a separate task, and so is writing the skills themselves.
capire/skillsTwo skill directories read here carry an eval file. cap-add-remote-service holds three prompts, the expected answer written out for each, and a CAP fixture project at evals/fixture to run them against; cap-developer holds prompts whose assertions are named one by one, and marks every case "stack": "node" or "stack": "java", so the same file asks the question in both CAP flavors.
Remaining work: Run them. No page read here documents a runner, a command or what counts as a pass, so the prompt, the fixture and the comparison are driven by whatever the project already has. The cases are about SAP's own skills, so a project that has written its own skills has written no cases for them.
Compare two models for a CAP feature before choosing one
Two models, or two versions of a prompt, answer the same cases and the results are put side by side before one of them is wired into the service.
Covers picking between models or prompt versions on evidence. Catching a later change in what the chosen one answers is a separate task, and versioning the prompt itself is prompt management.
@saptarishi/cds-plugin-llmpromptExperiment runs two variants against live traffic and reports which one won at a 95 per cent confidence interval. Because each backend is a profile under cds.requires.llm, the same handler answers from a different model by configuration, so the two sides of a comparison need no second code path.
Remaining work: Decide what winning means and which traffic the experiment may touch. Running variants against live traffic puts the comparison in front of real users, which is a choice the project makes rather than a setting the package decides.
Details
Coverage
Designs that place it
Comparison properties
These are the questions used to compare what the artifacts do. Values come from their documentation and may apply to a particular ecosystem or runtime. Each recorded value in the comparison opens its statement and sources.
For your agent
- Ref
capability:agent-evals- JSON
- /capabilities/agent-evals.json
- MCP
- Connecting an agent