{"ref":"artifact:marianfoo-llm-benchmark-abap-code-generation","page":"/artifacts/marianfoo-llm-benchmark-abap-code-generation","json":"/artifacts/marianfoo-llm-benchmark-abap-code-generation.json","generated":"2026-10-26","drafted_with_ai":true,"entity":{"id":"marianfoo-llm-benchmark-abap-code-generation","name":"LLM-Benchmark-ABAP-Code-Generation","artifact_type":"cli","ecosystem":"abap","domain":"ai-and-agents","origin_ring":"independent","use_as":"starting-point","description":{"one_line":"A benchmark that has a model write ABAP for 180 tasks ten times each, then feeds the SAP system's own syntax check, activation and ABAP Unit results back for up to five correction rounds.","short":"Benchmark of LLM ABAP code generation against a real SAP system","what_it_solves":"Whether a model can write ABAP that activates and passes its tests is a different question from whether the code reads well, and answering it needs a real system. This harness generates a class per prompt, creates it in an `sapse/abap-cloud-developer-trial:2023` container over ADT, and records class creation, syntax check, activation and unit tests in that order; a failure goes back to the model as feedback for up to five further rounds, and a conversation stops as soon as the tests pass. The dataset is 180 prompts with canonical solutions, tables and ABAP Unit fixtures beside them, run ten times per model at temperature 0.2, except for GPT-5 models, which OpenAI restricts to 1; the method follows [Benchmarking Large Language Models for ABAP Code Generation](https://arxiv.org/abs/2601.15188). Models are reached through OpenAI, Anthropic, Groq and Mistral, and SAP's own `sap--abap-1` through SAP AI Core orchestration. It measures that one dataset: the conversation logs under `data/` are the source of truth, every step is restartable, and the consolidated results, the leaderboard and the plots are regenerated from them by one script."},"links":{"repo":"https://github.com/marianfoo/LLM-Benchmark-ABAP-Code-Generation"},"type_properties":{"distribution":{"value":["container","pypi"],"sources":[{"url":"https://github.com/marianfoo/LLM-Benchmark-ABAP-Code-Generation","title":"LLM-Benchmark-ABAP-Code-Generation — README","checked":"2026-10-18","quote":"docker run --stop-timeout 3600 -i --name a4h -h vhcala4hci \\"},{"url":"https://github.com/marianfoo/LLM-Benchmark-ABAP-Code-Generation","title":"LLM-Benchmark-ABAP-Code-Generation — README","checked":"2026-10-18","quote":".venv/bin/pip install -e ."}]}},"identity":{"license":"MIT"},"source_available":true,"provides":[{"capability":"capability:agent-evals","what_it_does":"Runs one ABAP task set across models and records what held: 180 prompts, ten repetitions each, and up to five rounds in which the SAP system's compiler and unit-test output goes back to the model before its next attempt.","sources":[{"url":"https://raw.githubusercontent.com/marianfoo/LLM-Benchmark-ABAP-Code-Generation/main/README.md","title":"marianfoo/LLM-Benchmark-ABAP-Code-Generation — README","checked":"2026-09-25","quote":"The benchmark runs **180 tasks x 10 repetitions** per model."},{"url":"https://raw.githubusercontent.com/marianfoo/LLM-Benchmark-ABAP-Code-Generation/main/README.md","title":"marianfoo/LLM-Benchmark-ABAP-Code-Generation — README, Round definition","checked":"2026-09-25","quote":"System feeds SAP compiler / unit-test error back to LLM; LLM generates a corrected version."},{"url":"https://raw.githubusercontent.com/marianfoo/LLM-Benchmark-ABAP-Code-Generation/main/README.md","title":"marianfoo/LLM-Benchmark-ABAP-Code-Generation — README, Comparability checklist","checked":"2026-09-25","quote":"Same validation order: class creation → syntax check → activation → unit tests"}]}],"requires":[{"ref":"artifact:btp-ai-core","requirement":"conditional","condition":"the model being benchmarked is SAP's sap--abap-1","for":"the orchestration endpoint, client credentials and resource group the run reads from the environment to reach SAP's own ABAP model; every other provider is called directly","sources":[{"url":"https://raw.githubusercontent.com/marianfoo/LLM-Benchmark-ABAP-Code-Generation/main/README.md","title":"marianfoo/LLM-Benchmark-ABAP-Code-Generation — README, API keys","checked":"2026-09-25","quote":"# SAP AI Core (for ABAP-1 via orchestration)"}]}],"maintainers":["actor:marianfoo"],"drafted_with_ai":true,"provenance":{"discovered_via":[{"source":"source:github-orgs","channel":"marianfoo"}],"first_seen":"2026-09-25"},"contributors":["actor:marianfoo","actor:pokrakam"]},"effective_normative_relations":[],"readings":{"liveness":[{"label":"Maintenance","tier":"unknown","evidence":"no release data"},{"label":"Continuity","tier":"fair","evidence":"2 contributors"},{"label":"Repository","tier":"good","evidence":"pushed 2026-04, not archived"}],"compatibility":[{"label":"Declared for","tier":"fair","evidence":"nothing declared"},{"label":"Survived majors","tier":"unknown","evidence":"not read on ABAP: its lines run side by side"},{"label":"Maintainer's CI","tier":"good","evidence":"CI green, 2026-04-08"}],"practice":[{"label":"Tests","tier":"fair","evidence":"none found in the tree"},{"label":"Docs","tier":"good","evidence":"docs/ with 6 pages, README 16 KB"},{"label":"Changelog","tier":"fair","evidence":"none in the repo"}],"reach":[{"value":null,"unit":"installs/week"},{"value":13,"unit":"watching"}]},"signals":{"crawled_at":"2026-10-26","npm":"not published under this name","stars":13,"forks":4,"open_issues":0,"archived":false,"pushed_at":"2026-04-08","license_spdx":"MIT","org_owned":false,"default_branch":"main","created_at":"2026-01-23T10:39:48Z","tree_truncated":false,"test_files":0,"workflows":2,"other_ci":false,"readme_bytes":16734,"docs_files":6,"changelog":false,"license_file":true,"dts_files":0,"first_commit":"2025-07-07","last_commit":"2026-04-08","commits":68,"contributors":2,"contributors_since":"2026-01-23","ci_status":"success","ci_run_at":"2026-04-08","ci_workflow":"CI","ci_event":"push","readme_path":"README.md","readme_sha":"449b593ffa41d4ca57e499fc22f9bc5c389bfe4b","open_prs":0,"repo_read":"2026-10-26"},"edges":{"maintainers":[{"ref":"actor:marianfoo","name":"Marian Zeis","json":"/actors/marianfoo.json"}],"contributors":[{"ref":"actor:marianfoo","name":"Marian Zeis","json":"/actors/marianfoo.json"},{"ref":"actor:pokrakam","name":"Mike Pokraka","json":"/actors/pokrakam.json"}],"explained_by":[],"alternative_to_this":[],"needed_by":[],"available_via":[],"parts":[],"add_ons":[],"answered_through_parts":[]}}