{"ref":"artifact:albertmink-agentic-abap-evals","page":"/artifacts/albertmink-agentic-abap-evals","json":"/artifacts/albertmink-agentic-abap-evals.json","generated":"2026-10-26","drafted_with_ai":true,"entity":{"id":"albertmink-agentic-abap-evals","name":"agentic-abap-evals","artifact_type":"cli","ecosystem":"abap","domain":"ai-and-agents","origin_ring":"independent","use_as":"starting-point","description":{"one_line":"Six ABAP navigation tasks for a coding agent, each with needles and anti-needles that grade only the final answer, and a Node grader that applies them whatever agent stack produced the reply.","short":"ABAP navigation tasks and a grader for coding agents","what_it_solves":"Six natural-language tasks an ABAP developer might hand to an agent: walk two inheritance chains to the ancestor they share, find the framework class that dispatches a redefined method, get from an OO repository method to the DDIC table behind it. Each is a JSON spec holding the prompt, the substring needles a correct answer contains, and `antiNeedles` regexes that fail a run which fabricates line numbers or reference counts. The suite mandates nothing about the agent: you hand it the prompt text, then pass the spec and the reply to `grading/grade.mjs`, as a command or as an imported `grade` function. What a spec does constrain is capabilities rather than tool names, so `defaults.json` forbids `localFilesystemAccess`, `shellExecution`, `webSearch`, `subAgents` and `destinationMutation` on every prompt and the runner translates each term into its own denylist, which is what lets one eval run against more than one agent stack. All six prompts target SAP-standard objects, so a system with a standard ABAP stack can answer them; nothing here grades the trajectory and nothing here runs the agent.","to_build_on_it":"Adding prompts is a pull request against the checklist in `PROMPT_QUALITY.md`: phrased by purpose so the answer identifier is not in the prompt, one verifiable answer, needles that do not also appear on a defensible wrong path, a fixture in `grading/grade.test.mjs`, and one real run pasted into the pull request. What a prompt may ask is bounded by the suite being read-only, and by output grading, which misses the runs where the agent guessed correctly without doing real work; trajectory-aware metrics are left to the runner you build. Capability tags on prompts, a reference runner and aggregated benchmark dashboards are named as later work, the runner once two or more agent stacks have informally validated the suite."},"links":{"repo":"https://github.com/albertmink/agentic-abap-evals"},"maintainers":["actor:albertmink"],"compat":{"runtimes":["node"]},"provides":[{"capability":"capability:agent-evals","what_it_does":"Puts six ABAP tasks in front of whatever agent stack a project already has and grades only the reply that comes back: substring needles for the identifier the agent should land on, and `antiNeedles` regexes that fail a run showing fabricated line numbers or reference counts. One of the six is a negative path, where the correct behaviour is one search, no result and no answer invented.","sources":[{"url":"https://github.com/albertmink/agentic-abap-evals","title":"albertmink/agentic-abap-evals — README","checked":"2026-10-19","quote":"Each eval is a natural-language task an ABAP developer might hand to an agent (find an override, walk a type hierarchy, locate a framework dispatcher) plus a rubric that grades the agent's final answer **without inspecting how it got there**."},{"url":"https://github.com/albertmink/agentic-abap-evals","title":"albertmink/agentic-abap-evals — README, How to use them","checked":"2026-10-19","quote":"**Grade the agent's final reply.** Pass the spec and the answer text to `grading/grade.mjs`."},{"url":"https://github.com/albertmink/agentic-abap-evals","title":"albertmink/agentic-abap-evals — README, Why grade only the answer","checked":"2026-10-19","quote":"The compensation is `antiNeedles` (catches lying patterns: fabricated line numbers, \"Found N references\" on non-existent classes) and prompts that are hard enough that guessing usually fails."},{"url":"https://github.com/albertmink/agentic-abap-evals","title":"albertmink/agentic-abap-evals — README, The six showcase evals","checked":"2026-10-19","quote":"Restraint test. The agent should run one search, see nothing, and stop — no fabrications."}]}],"requires":[{"ref":"external:abap-platform","requirement":"required","basis":"binding","role":"evaluation-target","for":"The system the prompts are about: all six ask after SAP-standard objects such as `cl_aff_*`, `cl_abap_compiler` and `if_ci_test`, and the source the agent has to read lives there rather than on the runner's disk.","sources":[{"url":"https://github.com/albertmink/agentic-abap-evals","title":"albertmink/agentic-abap-evals — README, The six showcase evals","checked":"2026-10-19","quote":"All six target SAP-standard objects (`cl_aff_*`, `cl_abap_compiler`, `if_ci_test`) so any system with a standard ABAP stack can run them."},{"url":"https://raw.githubusercontent.com/albertmink/agentic-abap-evals/main/CAPABILITIES.md","title":"albertmink/agentic-abap-evals — CAPABILITIES.md","checked":"2026-10-19","quote":"ABAP source lives as virtual files on the SAP system."}]}],"interfaces":[{"key":"grade","kind":"api","direction":"in","for":"Scoring a reply in process: a Node-based runner imports `grade`, passes the spec and the answer, and reads back `pass` and `reasons`.","sources":[{"url":"https://github.com/albertmink/agentic-abap-evals","title":"albertmink/agentic-abap-evals — README, Library use","checked":"2026-10-19","quote":"// Library use, from any Node-based runner"}]}],"provenance":{"discovered_via":[{"source":"source:github"}],"first_seen":"2026-10-19"},"drafted_with_ai":true,"contributors":["actor:albertmink"]},"effective_normative_relations":[],"readings":{"liveness":[{"label":"Maintenance","tier":"unknown","evidence":"no release data"},{"label":"Continuity","tier":"fair","evidence":"one contributor, personally owned"},{"label":"Repository","tier":"good","evidence":"pushed 2026-06, not archived"}],"compatibility":[{"label":"Declared for","tier":"fair","evidence":"nothing declared"},{"label":"Survived majors","tier":"unknown","evidence":"not read on ABAP: its lines run side by side"},{"label":"Maintainer's CI","tier":"fair","evidence":"no CI configured"}],"practice":[{"label":"Tests","tier":"good","evidence":"1 test file"},{"label":"Docs","tier":"good","evidence":"README 6 KB"},{"label":"Changelog","tier":"fair","evidence":"none in the repo"}],"reach":[{"value":null,"unit":"installs/week"},{"value":7,"unit":"watching"}]},"signals":{"crawled_at":"2026-10-24","npm":"not published under this name","stars":7,"forks":0,"open_issues":2,"archived":false,"pushed_at":"2026-06-10","license_spdx":"MIT","org_owned":false,"default_branch":"main","created_at":"2026-06-01T17:19:17Z","open_prs":0,"tree_truncated":false,"test_files":1,"workflows":0,"other_ci":false,"readme_bytes":6137,"docs_files":0,"changelog":false,"license_file":true,"dts_files":0,"first_commit":"2026-06-01","last_commit":"2026-06-10","commits":4,"contributors":1,"readme_path":"README.md","readme_sha":"c5bc2a4737143f4b6a020f29866b17096db90424","repo_read":"2026-10-24"},"edges":{"maintainers":[{"ref":"actor:albertmink","name":"Albert Mink","json":"/actors/albertmink.json"}],"contributors":[{"ref":"actor:albertmink","name":"Albert Mink","json":"/actors/albertmink.json"}],"explained_by":[],"alternative_to_this":[],"needed_by":[],"available_via":[],"parts":[],"add_ons":[],"answered_through_parts":[]}}