Agent evals

Here the judge is the system. LLM-Benchmark-ABAP-Code-Generation creates each generated class in a running ABAP instance and lets the syntax check, the activation and ABAP Unit decide whether it holds, then feeds the error back for another round, so what the numbers report is code that ran rather than code that read well.

Overview

SAP SHIPS

No core answer is documented.

Not covered: At one benchmark's own dataset. LLM-Benchmark-ABAP-Code-Generation measures 180 fixed tasks against one trial image, so a team that wants to know whether its own prompts still hold after a model change assembles its own tasks, its own ABAP Unit fixtures and its own run.

Community-only rests on independent providers of two kinds: the benchmark above, which measures models writing ABAP, and agentic-abap-evals by Albert Mink, ABAP navigation tasks and a grader for a coding agent. SAP's part here is the oracle rather than the harness: the syntax check, the activation and ABAP Unit that decide whether generated ABAP holds are the system's own, and SAP's sap--abap-1 is one of the models measured. No SAP page has been checked that states a position on evaluating generated ABAP, so what_core_does is left empty.