{"ref":"capability:agent-evals","page":"/capabilities/agent-evals","json":"/capabilities/agent-evals.json","generated":"2026-10-26","drafted_with_ai":true,"entity":{"id":"agent-evals","name":"Agent evals","domain":"ai-and-agents","ecosystems":["cap","btp","abap"],"universal":true,"description":"Knowing whether an AI feature still behaves after a model, prompt or schema change. Regression testing for non-deterministic output, against fixtures that reflect real business data.","short":"Checking an AI feature still behaves after a model or prompt change","core_coverage":{"state":"core-and-community","assessed_at":"2026-10-15","what_core_does":"[capire/skills](artifact:capire-skills) keeps eval files beside its skills. The two read here are cap-add-remote-service, three prompts with the expected output written out and a CAP fixture project at `evals/fixture` to run them against, and cap-developer, prompts whose assertions are named one by one and whose cases are marked `\"stack\": \"node\"` or `\"stack\": \"java\"`. cds.test, the core's part of [testing](capability:testing), writes and runs tests against CAP services, and capire's Java testing guide covers the same ground for CAP Java.","where_it_stops":"It stops at the skill, and at the run. The evals check a coding agent writing CAP code rather than an AI feature the application ships: capire's guide to [@cap-js/agents](artifact:cap-js-agents) still lists evals-based agent testing among the features to come, and the cds.test page says nothing about output that varies between runs, advising instead against comparing whole response objects. No page read here documents a runner for the eval files, so the prompt, the fixture and the comparison are driven by whatever harness the project already has.","community_adds":"[@saptarishi/cds-plugin-llm](artifact:saptarishi-cds-plugin-llm) moves the question inside the application: `promptRegression` runs a fixture file the project writes through the configured provider, and `llmJudge` scores a response against a rubric with a model. [DanKromp/skillOverflow](artifact:dankromp-skilloverflow) answers the other half of SAP's, one eval per shipped skill, each a prompt with the assertions its output is checked against.","note":"The relationship is unassessed: how the independent providers came to stand beside capire/skills is not documented in any page retrieved.","sources":[{"url":"https://cap.cloud.sap/docs/guides/ai/cap-agents","title":"CAP-level Agents, More to come — capire","checked":"2026-10-15","quote":"Evals-based Agent Testing – using best practice evaluation frameworks."},{"url":"https://cap.cloud.sap/docs/node.js/cds-test","title":"Testing with cds.test — capire","checked":"2026-10-15","quote":"cds.test, CAP's testing utility for writing and running tests against CAP services."},{"url":"https://cap.cloud.sap/docs/node.js/cds-test","title":"Testing with cds.test, Best Practices — capire","checked":"2026-10-15","quote":"Don't Test Snapshots: Same applies to using equal with whole response objects, which might contain additional properties like timestamps, ids, etc., that are not relevant for the test and might change without actually breaking the functionality."},{"url":"https://cap.cloud.sap/docs/java/developing-applications/testing","title":"Testing Applications (CAP Java) — capire","checked":"2026-10-15","quote":"This section describes some best practices and recommendations for testing CAP Java applications."},{"url":"https://github.com/capire/skills/blob/main/skills/cap-add-remote-service/evals/evals.json","title":"capire/skills — cap-add-remote-service evals.json","checked":"2026-10-15","quote":"\"files\": [\"skills/cap-add-remote-service/evals/fixture\"]"},{"url":"https://github.com/capire/skills/blob/main/skills/cap-developer/evals/evals.json","title":"capire/skills — cap-developer evals.json","checked":"2026-10-15","quote":"\"expected_output\": \"Correct cds init command, no cds add sample, namespace defined, cuid/managed aspects used, db entity exposed via service projection, PascalCase/camelCase naming conventions followed.\""},{"url":"https://github.com/capire/skills/blob/main/skills/cap-developer/evals/evals.json","title":"capire/skills — cap-developer evals.json, stack marker","checked":"2026-10-15","quote":"\"stack\": \"node\""}],"relationship":"unassessed"},"coverage_by_flavor":{"node":{"state":"core-and-community","assessed_at":"2026-10-15","what_core_does":"cap-developer's eval file marks each case with a stack, and `\"stack\": \"node\"` is one of the two it uses, so SAP's own skill evals ask the Node.js questions. cds.test runs whatever test the project writes around them.","where_it_stops":"Where the door's answer stops. Nothing read here is specific to Node.js beyond the marker: the eval files arrive without a runner, and the regression question for an application's own model output is answered by a plugin rather than by the runtime.","community_adds":"[@saptarishi/cds-plugin-llm](artifact:saptarishi-cds-plugin-llm) runs the regression inside the application, and it is a CDS plugin, which the graph reads as Node.js.","note":"The relationship is unassessed here for the same reason it is on the door: how the plugin came to stand beside capire/skills is not documented in any page retrieved.","relationship":"unassessed","sources":[{"url":"https://github.com/capire/skills/blob/main/skills/cap-developer/evals/evals.json","title":"capire/skills — cap-developer evals.json, stack marker","checked":"2026-10-15","quote":"\"stack\": \"node\""},{"url":"https://cap.cloud.sap/docs/node.js/cds-test","title":"Testing with cds.test — capire","checked":"2026-10-15","quote":"cds.test, CAP's testing utility for writing and running tests against CAP services."}]},"java":{"state":"core-only","assessed_at":"2026-10-15","what_core_does":"The same cap-developer eval file marks cases `\"stack\": \"java\"`, so SAP's own skill evals ask the Java questions too, and capire's Java testing guide covers running tests against a CAP Java application.","where_it_stops":"At the coding agent. [@saptarishi/cds-plugin-llm](artifact:saptarishi-cds-plugin-llm), which runs the regression inside the application, is a CDS plugin and states Node.js, so a CAP Java application assembles its own fixtures, its own judge and its own run on top of the test support capire documents.","note":"Core-only is the reading of the providers that state a flavor: a provider that states a Java answer for an application's own model output would change it.","sources":[{"url":"https://github.com/capire/skills/blob/main/skills/cap-developer/evals/evals.json","title":"capire/skills — cap-developer evals.json, stack marker","checked":"2026-10-15","quote":"\"stack\": \"java\""},{"url":"https://raw.githubusercontent.com/capire/skills/main/skills/cap-developer/SKILL.md","title":"capire/skills — cap-developer SKILL.md","checked":"2026-10-15","quote":"Expert guidance for building and extending CAP (Cloud Application Programming Model) applications in either Node.js or Java."},{"url":"https://cap.cloud.sap/docs/java/developing-applications/testing","title":"Testing Applications (CAP Java) — capire","checked":"2026-10-15","quote":"This section describes some best practices and recommendations for testing CAP Java applications."}]}},"framing_by_flavor":{"java":{"description":"The only eval material read here for CAP Java is the Java half of SAP's own skill eval file, which asks what a coding agent writes. The question an application asks of its own model output is answered in the ordinary test support capire's Java testing guide documents, with the fixtures and the judging written by hand."}},"facets":[{"key":"case_source","label":"Where the cases come from","kind":"one","values":["Shipped with the package","Written by the project"],"ecosystems":["cap"]},{"key":"judged_by","label":"How the output is judged","kind":"many","values":["Expected output","Written assertions","Model as judge"],"ecosystems":["cap"]}],"gap":{"open_since":"2026-03-01","note":"The eval material that exists is written for coding agents, or for comparing models before one is chosen. For the AI feature an application ships, the fixtures, the judge and the run on every change are still the project's own."},"history":[{"date":"2026-09-14","field":"description","from":"Knowing whether an AI feature built on CAP still behaves after a model, prompt or schema change. Regression testing for non-deterministic output, against fixtures that reflect real business data.","to":"Knowing whether an AI feature still behaves after a model, prompt or schema change. Regression testing for non-deterministic output, against fixtures that reflect real business data.","reason":"The description placed the AI feature on CAP, which is false for the BTP door; the words \"built on CAP\" are removed and nothing else changes."},{"date":"2026-09-14","field":"ecosystems","from":"cap","to":"cap, btp","reason":"SAP-samples' teched2025-AI161 benchmarks, evaluates and optimises prompts across models with the generative AI hub's registry and evaluation tools in SAP AI Launchpad, which is this regression question on BTP without CAP."},{"date":"2026-09-14","field":"core_coverage","from":"unmapped","to":"core-only","reason":"The validator now rejects an unmapped reading on a door with its own providers, and capire/skills (SAP-official) provides this capability."},{"date":"2026-09-20","field":"core_coverage","from":"core-only","to":"core-and-community","reason":"Two independent artifacts provide this capability on CAP beside capire/skills: @saptarishi/cds-plugin-llm and DanKromp/skillOverflow, both entered on 2026-09-18, after the core-only reading."},{"date":"2026-09-25","field":"ecosystems","from":"cap, btp","to":"cap, btp, abap","reason":"[LLM-Benchmark-ABAP-Code-Generation](artifact:marianfoo-llm-benchmark-abap-code-generation), proposed in the same batch, measures generated ABAP across models by creating each class in an SAP system and reading its syntax check, activation and unit-test result. The concern is one an ABAP developer decides behind their own door, and the door had no tile for it to appear under."},{"date":"2026-10-15","field":"core_coverage","from":"assessed 2026-09-20","to":"assessed 2026-10-15, community_adds added","reason":"Re-read against capire and the two eval files. The CAP-level Agents guide still lists evals-based agent testing among the features to come and the cds.test page still says nothing about output that varies between runs, so the boundary holds; what_core_does now says what the eval files contain, including cap-developer's named assertions and its per-stack marker, and community_adds says what the two independent options answer on top."},{"date":"2026-10-15","field":"coverage_by_flavor","from":null,"to":"node: core-and-community, java: core-only","reason":"cap-developer's eval file marks its cases node or java, so SAP's own material covers both flavors. The one option that runs the regression inside the application states Node.js, which leaves CAP Java with the core's answer alone."},{"date":"2026-10-15","field":"facets","from":null,"to":"case_source, judged_by","reason":"The three CAP options differ on where their eval cases come from and on how an answer is judged, and both differences are written in the maintainers' own files. Scoped to CAP: the other two doors hold one entry each, where a column would compare nothing."},{"date":"2026-10-15","field":"gap.note","from":"Everyone ships AI features; nobody has a way to tell when one silently degrades.","to":"The eval material that exists is written for coding agents, or for comparing models before one is chosen. For the AI feature an application ships, the fixtures, the judge and the run on every change are still the project's own.","reason":"The note rested on \"everyone\" and \"nobody\", which nothing here sources. The replacement says what the material read on the three doors covers and what it leaves to the project, which is checkable."},{"date":"2026-10-26","field":"core_coverage.note, coverage_by_flavor.node.community_adds, coverage_by_flavor.node.note, coverage_by_flavor.java.where_it_stops, coverage_by_flavor.java.note","from":"\"no retrieved page says how the independent providers came to stand beside capire/skills\"; \"the one artifact that runs the regression inside the application\"; \"no entry behind this door states a Java answer\"","to":"\"how the independent providers came to stand beside capire/skills is not documented in any page retrieved\"; \"runs the regression inside the application\"; \"a provider that states a Java answer ... would change it\"","reason":"Reassessed against the graph (check:capabilities): the wording flagged as counting providers or claiming none exist is rephrased; states unchanged."},{"date":"2026-10-26","field":"coverage_by_ecosystem.btp.state, coverage_by_ecosystem.btp.relationship, coverage_by_ecosystem.btp.community_adds, coverage_by_ecosystem.btp.note","from":"core-only; \"The one entry behind this door is an SAP TechEd workshop...\"","to":"core-and-community, relationship unassessed; community_adds names DanKromp/skillOverflow; note on the workshop kept without a count","reason":"Reassessed against the graph (check:capabilities): DanKromp/skillOverflow is independent and lists btp in serves_ecosystems, so the community is present beside the AI Core services the block's sourced what_core_does describes."},{"date":"2026-10-26","field":"coverage_by_ecosystem.abap.note","from":"One provider and it is independent, so community-only is a assessment of the corpus. SAP's part here is the oracle ...","to":"Community-only rests on independent providers: the benchmark and agentic-abap-evals ... SAP's part here is the oracle ...","reason":"Reassessed against the graph (check:capabilities): albertmink-agentic-abap-evals, an independent set of ABAP tasks and a grader for coding agents, was entered after the block was written; the count is dropped and the second kind named. State unchanged."}],"coverage_by_ecosystem":{"btp":{"state":"core-and-community","assessed_at":"2026-09-14","relationship":"unassessed","what_core_does":"The generative AI hub in SAP AI Core has an Evaluation Service that benchmarks a use case across models with defined metrics against test datasets, a prompt optimizer that adapts prompts to a target model, and a prompt registry that versions the templates.","where_it_stops":"At the prompt and the model. An evaluation runs a registered prompt against a dataset somebody prepares, so fixtures drawn from the application's own data, and a run on every change to the application, remain the project's to build.","community_adds":"[DanKromp/skillOverflow](artifact:dankromp-skilloverflow) lists BTP among the stacks it serves and ships an eval with each skill, so it brings the coding-agent kind of eval to this door; the AI Core services above compare prompts and models.","note":"SAP's TechEd workshop [teched2025-AI161](artifact:sap-samples-teched2025-ai161) demonstrates and does not provide; the SAP side rests on the AI Core services as the SAP Architecture Center describes them. How the independent skill collection came to stand beside them is not documented in any page retrieved.","sources":[{"url":"https://architecture.learning.sap.com/docs/golden-path/ai-golden-path/build-and-deliver/genai-applications","title":"GenAI Applications — SAP Architecture Center","checked":"2026-09-14","quote":"The Evaluation Service allows users to test and benchmark AI use cases across different models using defined metrics against test datasets."},{"url":"https://github.com/sap-tutorials/Tutorials/blob/master/tutorials/ai-core-genaihub-evaluation/ai-core-genaihub-evaluation.md","title":"Generative AI Custom Evaluation - Quickstart — SAP Tutorials","checked":"2026-09-14","quote":"This tutorial demonstrates how to use SAP AI Core Custom Evaluation to benchmark Large Language Models (LLMs) using **Prompt Registry**."}]},"abap":{"state":"community-only","assessed_at":"2026-09-25","where_it_stops":"At one benchmark's own dataset. [LLM-Benchmark-ABAP-Code-Generation](artifact:marianfoo-llm-benchmark-abap-code-generation) measures 180 fixed tasks against one trial image, so a team that wants to know whether its own prompts still hold after a model change assembles its own tasks, its own ABAP Unit fixtures and its own run.","note":"Community-only rests on independent providers of two kinds: the benchmark above, which measures models writing ABAP, and [agentic-abap-evals](artifact:albertmink-agentic-abap-evals) by [Albert Mink](actor:albertmink), ABAP navigation tasks and a grader for a coding agent. SAP's part here is the oracle rather than the harness: the syntax check, the activation and ABAP Unit that decide whether generated ABAP holds are the system's own, and SAP's sap--abap-1 is one of the models measured. No SAP page has been checked that states a position on evaluating generated ABAP, so what_core_does is left empty."}},"framing_by_ecosystem":{"btp":{"description":"On BTP evaluation sits beside the model: SAP AI Core's generative AI hub compares prompts and models in SAP AI Launchpad before a prompt reaches application code."},"abap":{"description":"Here the judge is the system. [LLM-Benchmark-ABAP-Code-Generation](artifact:marianfoo-llm-benchmark-abap-code-generation) creates each generated class in a running ABAP instance and lets the syntax check, the activation and ABAP Unit decide whether it holds, then feeds the error back for another round, so what the numbers report is code that ran rather than code that read well."}},"drafted_with_ai":true},"sap_delivery":{"door":"cap","all":{"built_in":[],"documented_pattern":[],"announced":[],"packages":[{"ref":"artifact:capire-skills","name":"capire/skills","json":"/artifacts/capire-skills.json"}]},"by_flavor":{"node":{"built_in":[],"documented_pattern":[],"announced":[],"packages":[{"ref":"artifact:capire-skills","name":"capire/skills","json":"/artifacts/capire-skills.json"}]},"java":{"built_in":[],"documented_pattern":[],"announced":[],"packages":[{"ref":"artifact:capire-skills","name":"capire/skills","json":"/artifacts/capire-skills.json"}]}}},"edges":{"use_cases":[{"id":"catch-a-change-in-what-a-cap-ai-feature-answers","name":"Catch a change in what the application's AI feature answers","capability":"capability:agent-evals","ecosystem":"cap","flavors":["node","java"],"outcome":"A model, prompt or schema change is caught before it ships, by sending the application's own prompts over fixtures and judging the answers that come back.","boundary":"Covers the model output the application itself produces. Checking a coding agent that writes the application is a separate task, and comparing two models before one is picked is a third.","origin":"Editorial hypothesis, from the regression and judge helpers in @saptarishi/cds-plugin-llm and the evals entry on capire's CAP-level Agents roadmap.","requirement_dimensions":["Whether the application is CAP Node.js or CAP Java","Whether the fixtures may hold real business data","Whether the run may call a model, which a judge does and a fixed comparison does not","How much variation between runs counts as a change"],"needs":[{"key":"case_source","value":"Written by the project"}],"paths":[{"key":"node-test-suite","name":"Fixtures run from the CAP Node.js test suite"}],"contributions":[{"ref":"artifact:saptarishi-cds-plugin-llm","role":"the regression run and the judge","state":"documented","paths":["node-test-suite"],"flavors":["node"],"description":"`promptRegression` runs a fixture file the project wrote through the configured provider and `formatRegressionReport` prints what came back; `llmJudge` and `judgeMany` score an answer against a written rubric with a model. Because the backend is a profile under `cds.requires.llm`, the run and the feature it guards can sit on different models.","remaining_work":"Write the fixtures and the rubric, and decide the threshold. A judge is itself a model call, so what it costs and how steady its own answers are belong to the project's run.","remaining_kinds":["application-code"],"sources":[{"url":"https://raw.githubusercontent.com/kalyanjanumpally/sap-joule-ai-plugin/main/cds-plugin-llm/README.md","title":"kalyanjanumpally/sap-joule-ai-plugin — cds-plugin-llm README","checked":"2026-10-15","quote":"The package also ships **first-class helpers** for RAG orchestration (`ragChain`), LLM-as-judge scoring (`llmJudge`), CI eval harnesses (`promptRegression`, `lintPrompt`), and offline batch workflows (`runBatch`, `waitForBatch`)"},{"url":"https://raw.githubusercontent.com/kalyanjanumpally/sap-joule-ai-plugin/main/cds-plugin-llm/README.md","title":"kalyanjanumpally/sap-joule-ai-plugin — cds-plugin-llm README, RAG + Eval","checked":"2026-10-15","quote":"`promptRegression` / `loadFixtures` / `formatRegressionReport`"},{"url":"https://raw.githubusercontent.com/kalyanjanumpally/sap-joule-ai-plugin/main/cds-plugin-llm/README.md","title":"kalyanjanumpally/sap-joule-ai-plugin — cds-plugin-llm README","checked":"2026-09-18","quote":"A CAP service kind that turns `cds.connect.to('llm')` into a working LLM client — with one unified interface (`chat`, `stream`, `embed`, `batch`) that speaks to any of eleven backends. Swapping backends is a config change, not a code change."}]},{"ref":"artifact:sap-cds","role":"the test the project already runs","state":"documented","paths":["node-test-suite"],"flavors":["node"],"description":"cds.test writes and runs tests against CAP services, so the call under test is made the way every other CAP test makes it. The page says nothing about output that varies between runs, and under Best Practices it advises against comparing whole response objects at all.","remaining_work":"Everything the outcome needs past running the call: the fixtures, something that judges an answer which is different every time, and the threshold. The snapshot advice is written for responses whose timestamps and ids move, not for model output, so a project reading it as guidance here is extending it.","remaining_kinds":["application-code"],"sources":[{"url":"https://cap.cloud.sap/docs/node.js/cds-test","title":"Testing with cds.test — capire","checked":"2026-10-15","quote":"cds.test, CAP's testing utility for writing and running tests against CAP services."},{"url":"https://cap.cloud.sap/docs/node.js/cds-test","title":"Testing with cds.test, Best Practices — capire","checked":"2026-10-15","quote":"Don't Test Snapshots: Same applies to using equal with whole response objects, which might contain additional properties like timestamps, ids, etc., that are not relevant for the test and might change without actually breaking the functionality."}]},{"ref":"artifact:cap-java-cds-services-bom","role":"the test the project already runs, on CAP Java","state":"documented","flavors":["java"],"description":"capire's Java testing guide covers testing a CAP Java application across the layers it names, so the service call under test runs inside the suite the project already has.","remaining_work":"The whole eval half: the fixtures, something that judges an answer which varies, and the threshold. No option read here states a CAP Java answer for that, so it is written by hand.","sources":[{"url":"https://cap.cloud.sap/docs/java/developing-applications/testing","title":"Testing Applications (CAP Java) — capire","checked":"2026-10-15","quote":"This section describes some best practices and recommendations for testing CAP Java applications."},{"url":"https://cap.cloud.sap/docs/java/developing-applications/testing","title":"Testing Applications (CAP Java) — capire","checked":"2026-09-27","quote":"As described in Modular Architecture, a CAP Java application consists of weakly coupled components, which enables you to define your test scope precisely and focus on parts that need a high test coverage."}]},{"ref":"artifact:cap-js-agents","role":"the agent the application serves","state":"explicitly-unsupported","description":"capire's CAP-level Agents guide lists evals-based agent testing under More to come, so the released plugin does not test the agent it builds.","remaining_work":"Watch the page. A released feature would change this from a roadmap line to an answer, and nothing else in the record would move.","sources":[{"url":"https://cap.cloud.sap/docs/guides/ai/cap-agents","title":"CAP-level Agents, More to come — capire","checked":"2026-10-15","quote":"Evals-based Agent Testing – using best practice evaluation frameworks."}]}],"unknowns":["No published scenario read here shows the plugin's regression call and cds.test in one project, which is why the path carries no source.","Whether the Node.js plugin's regression helpers can be driven from a CAP Java project through any documented route was not established.","No page read here states what a fixture file for `promptRegression` contains, so its shape is taken from the helper's name and not from a sentence."],"assessed_at":"2026-10-15","drafted_with_ai":true,"readings":[],"omitted_contributions":0,"assessment_kind":"documented-contributions"},{"id":"check-a-coding-agent-still-writes-correct-cap-code","name":"Check that a coding agent still writes correct CAP code","capability":"capability:agent-evals","ecosystem":"cap","flavors":["node","java"],"also_in":["abap"],"outcome":"After a model, a prompt or a skill set changes, the agent is given prompts whose expected answer is written down, and what it produced is checked against them.","boundary":"Covers checking what a coding agent writes for CAP work. Checking the model output an application ships to its own users is a separate task, and so is writing the skills themselves.","origin":"Editorial hypothesis, from the eval files that capire/skills and DanKromp/skillOverflow keep beside their skills.","requirement_dimensions":["Whether the project is CAP Node.js or CAP Java","Whether a case has to run against a project fixture or against the prompt alone","Who drives the run: a person reading the answer, a script in the repository, or a pipeline","Whether the skills under test are SAP's own or a third party's"],"contributions":[{"ref":"artifact:capire-skills","role":"eval cases for SAP's own CAP skills","state":"documented","description":"Two skill directories read here carry an eval file. cap-add-remote-service holds three prompts, the expected answer written out for each, and a CAP fixture project at `evals/fixture` to run them against; cap-developer holds prompts whose assertions are named one by one, and marks every case `\"stack\": \"node\"` or `\"stack\": \"java\"`, so the same file asks the question in both CAP flavors.","remaining_work":"Run them. No page read here documents a runner, a command or what counts as a pass, so the prompt, the fixture and the comparison are driven by whatever the project already has. The cases are about SAP's own skills, so a project that has written its own skills has written no cases for them.","sources":[{"url":"https://github.com/capire/skills/blob/main/skills/cap-add-remote-service/evals/evals.json","title":"capire/skills — cap-add-remote-service evals.json","checked":"2026-10-15","quote":"\"files\": [\"skills/cap-add-remote-service/evals/fixture\"]"},{"url":"https://github.com/capire/skills/blob/main/skills/cap-developer/evals/evals.json","title":"capire/skills — cap-developer evals.json","checked":"2026-10-15","quote":"\"expected_output\": \"Correct cds init command, no cds add sample, namespace defined, cuid/managed aspects used, db entity exposed via service projection, PascalCase/camelCase naming conventions followed.\""},{"url":"https://github.com/capire/skills/blob/main/skills/cap-developer/evals/evals.json","title":"capire/skills — cap-developer evals.json, stack marker","checked":"2026-10-15","quote":"\"stack\": \"java\""},{"url":"https://raw.githubusercontent.com/capire/skills/main/skills/cap-developer/SKILL.md","title":"capire/skills — cap-developer SKILL.md","checked":"2026-10-15","quote":"Expert guidance for building and extending CAP (Cloud Application Programming Model) applications in either Node.js or Java."}]},{"ref":"artifact:dankromp-skilloverflow","role":"one eval per shipped skill","state":"documented","description":"tests/skillOverflow-evals.json states what it is for and what each case holds: a prompt a developer would write, and the assertions the answer is checked against, named one by one. The repository counts one eval for each of its 41 skills, and its CAP cases carry assertions about CDS and handler code rather than an expected file.","remaining_work":"Run them, and decide what a failed assertion means. The assertions are sentences for a reader or another agent to check, so comparing an answer against them is the project's step. The cases are about this repository's skills.","sources":[{"url":"https://github.com/DanKromp/skillOverflow/blob/main/tests/skillOverflow-evals.json","title":"SkillOverflow — tests/skillOverflow-evals.json","checked":"2026-10-15","quote":"Eval test cases for all skills. Each eval has a realistic prompt and assertions to check the output against."},{"url":"https://github.com/DanKromp/skillOverflow/blob/main/tests/skillOverflow-evals.json","title":"SkillOverflow — tests/skillOverflow-evals.json, an assertion","checked":"2026-10-15","quote":"Uses Composition of many Contracts on suppliers.contracts"},{"url":"https://github.com/DanKromp/skillOverflow","title":"SkillOverflow — README","checked":"2026-10-15","quote":"Total: 41 skills | 41 evals"}]}],"unknowns":["No page read on 2026-10-15 documents how either eval file is run, or what counts as a pass.","Whether capire/skills keeps eval files beside skills other than cap-add-remote-service and cap-developer was not established: only those two directories were read.","Neither eval file states which model or agent it was written against, so what a result says about a different agent is not established."],"assessed_at":"2026-10-15","drafted_with_ai":true,"readings":[],"omitted_contributions":0,"assessment_kind":"documented-contributions"},{"id":"compare-two-models-for-a-cap-feature-before-choosing-one","name":"Compare two models for a CAP feature before choosing one","capability":"capability:agent-evals","ecosystem":"cap","flavors":["node","java"],"also_in":["btp"],"outcome":"Two models, or two versions of a prompt, answer the same cases and the results are put side by side before one of them is wired into the service.","boundary":"Covers picking between models or prompt versions on evidence. Catching a later change in what the chosen one answers is a separate task, and versioning the prompt itself is [prompt management](capability:prompt-management).","origin":"Editorial hypothesis, from the Evaluation Service in SAP's AI golden path and the experiment helper in @saptarishi/cds-plugin-llm.","requirement_dimensions":["Whether the models compared are reached through SAP AI Core or directly","Whether the comparison may run against live traffic or only against a prepared dataset","Whether the test dataset may hold the application's own records","Whether the result has to be reproducible for an audit"],"contributions":[{"ref":"artifact:btp-ai-core","role":"the platform's comparison, beside the model","state":"conditional","conditions":["The models compared are reached through the generative AI hub in SAP AI Core","A test dataset and the metrics are prepared before the run"],"description":"The Evaluation Service benchmarks a use case across models against test datasets with metrics somebody defines, and SAP AI Launchpad is where the results are compared and a model picked. The comparison sits beside the model rather than inside the CAP service, so what it measures is the prompt and the model, not the handler around them.","remaining_work":"Prepare the dataset and the metrics, then wire the chosen model back into the CAP service. Nothing read here says an evaluation can read the CAP application's own data, so the dataset is assembled separately.","remaining_kinds":["application-code"],"sources":[{"url":"https://architecture.learning.sap.com/docs/golden-path/ai-golden-path/build-and-deliver/genai-applications","title":"GenAI Applications — SAP Architecture Center","checked":"2026-10-15","quote":"The Evaluation Service allows users to test and benchmark AI use cases across different models using defined metrics against test datasets."},{"url":"https://architecture.learning.sap.com/docs/golden-path/ai-golden-path/build-and-deliver/genai-applications","title":"GenAI Applications — SAP Architecture Center","checked":"2026-10-15","quote":"Access this service through AI Launchpad to compare model performance across your specific use cases and select the most suitable model for your requirements."}]},{"ref":"artifact:saptarishi-cds-plugin-llm","role":"the comparison inside the application","state":"documented","flavors":["node"],"description":"`promptExperiment` runs two variants against live traffic and reports which one won at a 95 per cent confidence interval. Because each backend is a profile under `cds.requires.llm`, the same handler answers from a different model by configuration, so the two sides of a comparison need no second code path.","remaining_work":"Decide what winning means and which traffic the experiment may touch. Running variants against live traffic puts the comparison in front of real users, which is a choice the project makes rather than a setting the package decides.","sources":[{"url":"https://raw.githubusercontent.com/kalyanjanumpally/sap-joule-ai-plugin/main/cds-plugin-llm/README.md","title":"kalyanjanumpally/sap-joule-ai-plugin — cds-plugin-llm README","checked":"2026-10-15","quote":"`promptExperiment` (v2.20.0 — live A/B testing with 95% CI winner detection)"},{"url":"https://raw.githubusercontent.com/kalyanjanumpally/sap-joule-ai-plugin/main/cds-plugin-llm/README.md","title":"kalyanjanumpally/sap-joule-ai-plugin — cds-plugin-llm README","checked":"2026-09-18","quote":"LLM-agnostic AI platform for SAP CAP. **11 providers** behind one unified interface — Anthropic, Google Gemini, AWS Bedrock, Azure OpenAI, Ollama, Groq, Fireworks, DeepSeek, Mistral, any OpenAI-compatible endpoint, and SAP Generative AI Hub"}]}],"unknowns":["Whether an evaluation in SAP AI Launchpad can read a CAP service's own records as its test dataset is not established by the pages read here.","SAP AI Core is listed as a contributor on the strength of the SAP Architecture Center's description of the service. It carries no provides edge to this capability, and whether one belongs there is a question for the BTP door rather than this assessment.","No CAP Java option read here states an answer, so the Java side of this outcome rests on the platform contribution alone."],"assessed_at":"2026-10-15","drafted_with_ai":true,"readings":[],"omitted_contributions":0,"assessment_kind":"documented-contributions"}],"shared_tasks":[],"related_tasks":[],"options":[{"ref":"artifact:saptarishi-cds-plugin-llm","name":"@saptarishi/cds-plugin-llm","json":"/artifacts/saptarishi-cds-plugin-llm.json","ecosystem":"cap","origin_ring":"independent"},{"ref":"artifact:albertmink-agentic-abap-evals","name":"agentic-abap-evals","json":"/artifacts/albertmink-agentic-abap-evals.json","ecosystem":"abap","origin_ring":"independent"},{"ref":"artifact:capire-skills","name":"capire/skills","json":"/artifacts/capire-skills.json","ecosystem":"cap","origin_ring":"sap-official"},{"ref":"artifact:dankromp-skilloverflow","name":"DanKromp/skillOverflow","json":"/artifacts/dankromp-skilloverflow.json","ecosystem":"cap","origin_ring":"independent"},{"ref":"artifact:marianfoo-llm-benchmark-abap-code-generation","name":"LLM-Benchmark-ABAP-Code-Generation","json":"/artifacts/marianfoo-llm-benchmark-abap-code-generation.json","ecosystem":"abap","origin_ring":"independent"}],"samples":[{"ref":"artifact:sap-samples-teched2025-ai161","name":"teched2025-AI161","json":"/artifacts/sap-samples-teched2025-ai161.json"}],"explained_by":[{"ref":"content:devtoberfest-2026-ai21-multiple-models","name":"🟣 Enable Your AI application to Run on Multiple Models to Ensure Resilience and Compliance","json":"/content/devtoberfest-2026-ai21-multiple-models.json"}],"normative_context":[]}}