Agent evals
Knowing whether an AI feature still behaves after a model, prompt or schema change. Regression testing for non-deterministic output, against fixtures that reflect real business data.
Overview
capire/skills SAP's own skill set for coding agents working on CAP — cap-developer, cap-add-remote-service, cap-upgrade and cap-trivia, each a SKILL.md with references beside it.
capire/skills SAP's own skill set for coding agents working on CAP — cap-developer, cap-add-remote-service, cap-upgrade and cap-trivia, each a SKILL.md with references beside it.
Not covered: It stops at the skill, and at the run. The evals check a coding agent writing CAP code rather than an AI feature the application ships: capire's guide to @cap-js/agents@cap-js/agentsSAP's own CDS plugin for building agents inside a CAP service, wired to audit logging and telemetry through peer dependencies rather than leaving observability to the author.sap official, cap still lists evals-based agent testing among the features to come, and the cds.test page says nothing about output that varies between runs, advising instead against comparing whole response objects. No page read here documents a runner for the eval files, so the prompt, the fixture and the comparison are driven by whatever harness the project already has.
The relationship is unassessed: how the independent providers came to stand beside capire/skills is not documented in any page retrieved.
@saptarishi/cds-plugin-llm@saptarishi/cds-plugin-llmA CAP service kind that puts eleven model providers behind one chat, stream, embed and batch interface, so which backend a service calls is a profile in package.json rather than a change to the handler.independent, cap moves the question inside the application: promptRegression runs a fixture file the project writes through the configured provider, and llmJudge scores a response against a rubric with a model. DanKromp/skillOverflowDanKromp/skillOverflowA collection of 41 agent skills, 37 for CAP, Fiori, BTP, UI5, ABAP, Integration Suite and HANA work and four for creating, reviewing and retiring skills, each with its own eval.independent, cap answers the other half of SAP's, one eval per shipped skill, each a prompt with the assertions its output is checked against.
Artifacts
Tasks
Catch a change in what the application's AI feature answers
A model, prompt or schema change is caught before it ships, by sending the application's own prompts over fixtures and judging the answers that come back.
Covers the model output the application itself produces. Checking a coding agent that writes the application is a separate task, and comparing two models before one is picked is a third.
@saptarishi/cds-plugin-llmpromptRegression runs a fixture file the project wrote through the configured provider and formatRegressionReport prints what came back; llmJudge and judgeMany score an answer against a written rubric with a model. Because the backend is a profile under cds.requires.llm, the run and the feature it guards can sit on different models.
Remaining work: Write the fixtures and the rubric, and decide the threshold. A judge is itself a model call, so what it costs and how steady its own answers are belong to the project's run.
Check that a coding agent still writes correct CAP code
After a model, a prompt or a skill set changes, the agent is given prompts whose expected answer is written down, and what it produced is checked against them.
Covers checking what a coding agent writes for CAP work. Checking the model output an application ships to its own users is a separate task, and so is writing the skills themselves.
capire/skillsTwo skill directories read here carry an eval file. cap-add-remote-service holds three prompts, the expected answer written out for each, and a CAP fixture project at evals/fixture to run them against; cap-developer holds prompts whose assertions are named one by one, and marks every case "stack": "node" or "stack": "java", so the same file asks the question in both CAP flavors.
Remaining work: Run them. No page read here documents a runner, a command or what counts as a pass, so the prompt, the fixture and the comparison are driven by whatever the project already has. The cases are about SAP's own skills, so a project that has written its own skills has written no cases for them.
DanKromp/skillOverflowtests/skillOverflow-evals.json states what it is for and what each case holds: a prompt a developer would write, and the assertions the answer is checked against, named one by one. The repository counts one eval for each of its 41 skills, and its CAP cases carry assertions about CDS and handler code rather than an expected file.
Remaining work: Run them, and decide what a failed assertion means. The assertions are sentences for a reader or another agent to check, so comparing an answer against them is the project's step. The cases are about this repository's skills.
Compare two models for a CAP feature before choosing one
Two models, or two versions of a prompt, answer the same cases and the results are put side by side before one of them is wired into the service.
Covers picking between models or prompt versions on evidence. Catching a later change in what the chosen one answers is a separate task, and versioning the prompt itself is prompt management.
@saptarishi/cds-plugin-llmpromptExperiment runs two variants against live traffic and reports which one won at a 95 per cent confidence interval. Because each backend is a profile under cds.requires.llm, the same handler answers from a different model by configuration, so the two sides of a comparison need no second code path.
Remaining work: Decide what winning means and which traffic the experiment may touch. Running variants against live traffic puts the comparison in front of real users, which is a choice the project makes rather than a setting the package decides.
Details
Coverage
Designs that place it
Comparison properties
These are the questions used to compare what the artifacts do. Values come from their documentation and may apply to a particular ecosystem or runtime. Each recorded value in the comparison opens its statement and sources.
For your agent
- Ref
capability:agent-evals- JSON
- /capabilities/agent-evals.json
- MCP
- Connecting an agent