Agent evals
Knowing whether an AI feature still behaves after a model, prompt or schema change. Regression testing for non-deterministic output, against fixtures that reflect real business data.
Overview
capire/skills SAP's own skill set for coding agents working on CAP — cap-developer, cap-add-remote-service, cap-upgrade and cap-trivia, each a SKILL.md with references beside it.
capire/skills SAP's own skill set for coding agents working on CAP — cap-developer, cap-add-remote-service, cap-upgrade and cap-trivia, each a SKILL.md with references beside it.
Not covered: It stops at the skill, and at the run. The evals check a coding agent writing CAP code rather than an AI feature the application ships: capire's guide to @cap-js/agents@cap-js/agentsSAP's own CDS plugin for building agents inside a CAP service, wired to audit logging and telemetry through peer dependencies rather than leaving observability to the author.sap official, cap still lists evals-based agent testing among the features to come, and the cds.test page says nothing about output that varies between runs, advising instead against comparing whole response objects. No page read here documents a runner for the eval files, so the prompt, the fixture and the comparison are driven by whatever harness the project already has.
The relationship is unassessed: how the independent providers came to stand beside capire/skills is not documented in any page retrieved.
@saptarishi/cds-plugin-llm@saptarishi/cds-plugin-llmA CAP service kind that puts eleven model providers behind one chat, stream, embed and batch interface, so which backend a service calls is a profile in package.json rather than a change to the handler.independent, cap moves the question inside the application: promptRegression runs a fixture file the project writes through the configured provider, and llmJudge scores a response against a rubric with a model. DanKromp/skillOverflowDanKromp/skillOverflowA collection of 41 agent skills, 37 for CAP, Fiori, BTP, UI5, ABAP, Integration Suite and HANA work and four for creating, reviewing and retiring skills, each with its own eval.independent, cap answers the other half of SAP's, one eval per shipped skill, each a prompt with the assertions its output is checked against.
On BTP evaluation sits beside the model: SAP AI Core's generative AI hub compares prompts and models in SAP AI Launchpad before a prompt reaches application code.
The generative AI hub in SAP AI Core has an Evaluation Service that benchmarks a use case across models with defined metrics against test datasets, a prompt optimizer that adapts prompts to a target model, and a prompt registry that versions the templates.
Not covered: At the prompt and the model. An evaluation runs a registered prompt against a dataset somebody prepares, so fixtures drawn from the application's own data, and a run on every change to the application, remain the project's to build.
SAP's TechEd workshop teched2025-AI161teched2025-AI161An SAP TechEd 2025 hands-on session that benchmarks, evaluates and optimises prompts across models with the generative AI hub capability in SAP AI Core.sap official, btp demonstrates and does not provide; the SAP side rests on the AI Core services as the SAP Architecture Center describes them. How the independent skill collection came to stand beside them is not documented in any page retrieved.
DanKromp/skillOverflowDanKromp/skillOverflowA collection of 41 agent skills, 37 for CAP, Fiori, BTP, UI5, ABAP, Integration Suite and HANA work and four for creating, reviewing and retiring skills, each with its own eval.independent, cap lists BTP among the stacks it serves and ships an eval with each skill, so it brings the coding-agent kind of eval to this door; the AI Core services above compare prompts and models.
Here the judge is the system. LLM-Benchmark-ABAP-Code-GenerationLLM-Benchmark-ABAP-Code-GenerationA benchmark that has a model write ABAP for 180 tasks ten times each, then feeds the SAP system's own syntax check, activation and ABAP Unit results back for up to five correction rounds.independent, abap creates each generated class in a running ABAP instance and lets the syntax check, the activation and ABAP Unit decide whether it holds, then feeds the error back for another round, so what the numbers report is code that ran rather than code that read well.
No core answer is documented.
Not covered: At one benchmark's own dataset. LLM-Benchmark-ABAP-Code-GenerationLLM-Benchmark-ABAP-Code-GenerationA benchmark that has a model write ABAP for 180 tasks ten times each, then feeds the SAP system's own syntax check, activation and ABAP Unit results back for up to five correction rounds.independent, abap measures 180 fixed tasks against one trial image, so a team that wants to know whether its own prompts still hold after a model change assembles its own tasks, its own ABAP Unit fixtures and its own run.
Community-only rests on independent providers of two kinds: the benchmark above, which measures models writing ABAP, and agentic-abap-evalsagentic-abap-evalsSix ABAP navigation tasks for a coding agent, each with needles and anti-needles that grade only the final answer, and a Node grader that applies them whatever agent stack produced the reply.independent, abap by Albert MinkAlbert Minkmaintains 1 entry, contributed to 8 entries, ABAP navigation tasks and a grader for a coding agent. SAP's part here is the oracle rather than the harness: the syntax check, the activation and ABAP Unit that decide whether generated ABAP holds are the system's own, and SAP's sap--abap-1 is one of the models measured. No SAP page has been checked that states a position on evaluating generated ABAP, so what_core_does is left empty.
Artifacts
Tasks
Catch a change in what the application's AI feature answers
A model, prompt or schema change is caught before it ships, by sending the application's own prompts over fixtures and judging the answers that come back.
Covers the model output the application itself produces. Checking a coding agent that writes the application is a separate task, and comparing two models before one is picked is a third.
@saptarishi/cds-plugin-llmpromptRegression runs a fixture file the project wrote through the configured provider and formatRegressionReport prints what came back; llmJudge and judgeMany score an answer against a written rubric with a model. Because the backend is a profile under cds.requires.llm, the run and the feature it guards can sit on different models.
Remaining work: Write the fixtures and the rubric, and decide the threshold. A judge is itself a model call, so what it costs and how steady its own answers are belong to the project's run.
Check that a coding agent still writes correct CAP code
After a model, a prompt or a skill set changes, the agent is given prompts whose expected answer is written down, and what it produced is checked against them.
Covers checking what a coding agent writes for CAP work. Checking the model output an application ships to its own users is a separate task, and so is writing the skills themselves.
capire/skillsTwo skill directories read here carry an eval file. cap-add-remote-service holds three prompts, the expected answer written out for each, and a CAP fixture project at evals/fixture to run them against; cap-developer holds prompts whose assertions are named one by one, and marks every case "stack": "node" or "stack": "java", so the same file asks the question in both CAP flavors.
Remaining work: Run them. No page read here documents a runner, a command or what counts as a pass, so the prompt, the fixture and the comparison are driven by whatever the project already has. The cases are about SAP's own skills, so a project that has written its own skills has written no cases for them.
DanKromp/skillOverflowtests/skillOverflow-evals.json states what it is for and what each case holds: a prompt a developer would write, and the assertions the answer is checked against, named one by one. The repository counts one eval for each of its 41 skills, and its CAP cases carry assertions about CDS and handler code rather than an expected file.
Remaining work: Run them, and decide what a failed assertion means. The assertions are sentences for a reader or another agent to check, so comparing an answer against them is the project's step. The cases are about this repository's skills.
Compare two models for a CAP feature before choosing one
Two models, or two versions of a prompt, answer the same cases and the results are put side by side before one of them is wired into the service.
Covers picking between models or prompt versions on evidence. Catching a later change in what the chosen one answers is a separate task, and versioning the prompt itself is prompt management.
@saptarishi/cds-plugin-llmpromptExperiment runs two variants against live traffic and reports which one won at a 95 per cent confidence interval. Because each backend is a profile under cds.requires.llm, the same handler answers from a different model by configuration, so the two sides of a comparison need no second code path.
Remaining work: Decide what winning means and which traffic the experiment may touch. Running variants against live traffic puts the comparison in front of real users, which is a choice the project makes rather than a setting the package decides.
Content
Seen working
Details
Coverage
Designs that place it
Comparison properties
These are the questions used to compare what the artifacts do. Values come from their documentation and may apply to a particular ecosystem or runtime. Each recorded value in the comparison opens its statement and sources.
For your agent
- Ref
capability:agent-evals- JSON
- /capabilities/agent-evals.json
- MCP
- Connecting an agent