@saptarishi/cds-plugin-llm

cds-plugin-llm

LLM-agnostic AI platform for SAP CAP. 11 providers behind one unified interface — Anthropic, Google Gemini, AWS Bedrock, Azure OpenAI, Ollama, Groq, Fireworks, DeepSeek, Mistral, any OpenAI-compatible endpoint, and SAP Generative AI Hub — plus 30+ middleware primitives spanning cost, resilience, security, observability, compliance, RAG, and CI eval.

Status: stable (v2.0.0). Formal API stability contract in force for the 2.x line — see MIGRATION.md. 11 providers implemented; 2188 automated tests on Node 20 + 22 in CI; zero known bugs, zero deprecated APIs; TypeScript definitions ship in the package.

What it is

A CAP service kind that turns cds.connect.to('llm') into a working LLM client — with one unified interface (chat, stream, embed, batch) that speaks to any of eleven backends. Swapping backends is a config change, not a code change.

Beyond the provider abstraction, the package ships a comprehensive middleware platform: cost + budget guards, cost forecasting, circuit breakers, bulkheads (with latency- and quota-driven adaptive tuners), retries, deadlines, provider fallback, region failover, prompt-injection detection, PII redaction, content-safety classifiers, structured-output validators, response scoring, multi-model consensus voting, prompt-cache metrics, response caches (exact + semantic + pluggable-vector-store), embedding dedup, JSON logs, OTel spans, Prometheus metrics, tenant isolation, distributed locks, compliance audit trails, and more. See the Middleware catalog below.

The package also ships first-class helpers for RAG orchestration (ragChain), LLM-as-judge scoring (llmJudge), CI eval harnesses (promptRegression, lintPrompt), and offline batch workflows (runBatch, waitForBatch) — plus a saptarishi-llm CLI with subcommands for chat/stream/embed/verify/init/mcp/chain-visualize/doctor/export-dashboard/lint-prompts and more.

Complementary to @cap-js/ai, which focuses on value-help recommendations and SAP AI Core integration. This plugin fills the more general "I need a CAP-idiomatic way to call LLMs, with a local development story and multiple provider options" gap.

Middleware catalog

Every primitive below is a shipping middleware (or top-level helper) with dedicated per-section documentation, tests, TypeScript types, and a config://<primitive> MCP resource for live introspection. All are stable in the 2.x line — see MIGRATION.md.

Group	Primitives
Cost	usageMetering · usageMeteringToCap · costBudget · costGuard · costForecast · quotaManager (v2.23.0 — per-user USD quota with warnings) · inMemoryQuotaStore · costOverrunPredictor (v2.33.0 — calendar-window spend projection) · startOfMonth / endOfMonth (+day / quarter helpers) · adaptiveMaxTokens · estimateCost · promptCacheStats
Resilience	retryOnRateLimit · circuitBreaker · bulkhead · adaptiveBulkhead · adaptiveRateLimit · clientSideRateLimit (v2.26.0 — proactive N-per-window throttle) · deadline · gracePeriod (v2.28.0 — soft-deadline warnings + optional hard timeout) · chatWithFallback · regionFailover · autoRetry · providerHealthProbe · autoContinue · idempotency · distributedLock · speculativeHedge (v2.12.0 — staggered parallel hedges for tail latency) · retryBudget (v2.15.0 — SRE-style global retry cap)
Security	guardrails · promptInjectionGuard · piiRedact · reversibleTokenization (v2.13.0 — round-trip PII replacement) · tokenizePII / detokenizePII · PII_PATTERNS · safetyClassifier · sensitiveDataAudit · requestSigning (v2.21.0 — HMAC receipts + verifyReceiptChain) · responseSigning (v2.32.0 — HMAC responses + verifyResponseSignature) · emptyResponseDetector (v2.37.0 — catch empty/refusal responses with auto-retry)
Observability	jsonLog · otel · otelSpans · promMetrics · prometheusHandler · traceCorrelation · healthHandler · replayBuffer · retryAfterPropagation · providerHealthAggregate (v2.31.0 — unified provider health score) · latencyHistogram (v2.35.0 — per-dimension p50/p95/p99 with Prometheus export)
Testing	chaosInjector (v2.11.0 — deterministic seeded fault injection; test-only, refuses to construct without opt-in)
Routing	modelRouter · tenantIsolate · costAwareRouter (v2.10.0 — cheap-first with quality escalation) · fairShareScheduler (v2.14.0 — per-tenant WRR admission control) · semanticRouter (v2.16.0 — embedding-based route selection) · providerLoadBalancer (v2.17.0 — rotate across N credentials of same kind) · multimodalRouter (v2.25.0 — capability-aware routing by attachment type)
Caching	responseCache (exact + semantic) · semanticCache (pluggable vector store) · cosineSimilarity · embeddingDedup · requestCoalescer (in-flight dedup) · fuzzyDedup (v2.38.0 — near-duplicate detection via Jaccard-trigram / Levenshtein, no embedder required) · inMemoryFuzzyStore
Contract / GitOps	structuredOutputValidator · structuredOutputRepair (v2.9.0 — multi-strategy repair) · jsonAutoFix · functionCallArbitrator (v2.18.0 — tool-call allowlist + validation) · normalizeToolShape · normalizeToolList · contentLengthGate (v2.36.0 — pre-flight size validation) · defaultTokenEstimator · schemas (Invoice, PurchaseOrder, SupplierRisk, ContractSummary, ExpenseReport, EmailDraft) · validateMiddlewareOrder · chainSnapshot · chainDiff · preflight · capabilities (+ PROVIDER_CAPABILITY_MATRIX, MODEL_CAPABILITY_OVERRIDES)
Prompts	PromptRegistry · builtInPrompts · gitPromptRegistry (v2.1.0 — Git-backed prompt-as-code) · promptVersionPin (v2.30.0 — canary/rollback for prompt templates) · PromptVersionRegistry
Long-context	compactHistory · sessionContextStore (v2.19.0 — per-session history with prune / summarize) · inMemorySessionStore
Streaming	wrapStreamCompletion · hasStreamCompletion · streamThrottle · streamAggregator (v2.24.0 — coalesce per-token chunks for smoother UI)
RAG + Eval	ragChain · llmJudge / judgeMany · promptRegression / loadFixtures / formatRegressionReport · lintPrompt / lintPrompts / formatLintReport · scoreResponse · consensusVoting · promptExperiment (v2.20.0 — live A/B testing with 95% CI winner detection) · responseRevision (v2.29.0 — quality-driven re-ask loop) · userFeedbackAggregator (v2.34.0 — human thumbs/stars aggregation)
Bulk workflows	runBatch · waitForBatch · batchAggregator (v2.27.0 — window-based pooling of concurrent LLM calls)
Multimodal helpers	imageFromFile / pdfFromUrl / audioFromBase64 / uploadPdfFromUrl / etc.
Agent orchestration	runTools · streamTools · Agent · runAgents · streamAgents · autoToolChain (v2.22.0 — cascading tool loop with cycle detection)
Error taxonomy	LLMError base + errorRegistry with 18 stable codes (CIRCUIT_OPEN, BUDGET_EXCEEDED, PROMPT_INJECTION, SAFETY_CLASSIFIER_BLOCKED, ALL_REGIONS_FAILED, ...)

Every primitive is composable via llm.use(mw) Koa-style — outermost first. resilience.bundle() wires the full resilience stack (retry → breaker → bulkhead → deadline → probes → tuner) with one call. chainSnapshot(llm) extracts the live config for GitOps drift detection; chainDiff(baseline, live) diffs; validateMiddlewareOrder(chain) warns on suspicious orderings.

Architecture

    Your CAP handler / OData action
          │
          │  cds.connect.to('llm')  →  { chat, stream, embed, batch }
          ↓
    ┌─────────────────────────────────────────────────────────────┐
    │  Middleware chain (llm.use(...), Koa-style, outer → inner)  │
    │                                                             │
    │   otelSpans / traceCorrelation / jsonLog                    │
    │     → promptCacheStats                                      │
    │       → modelRouter                                         │
    │         → embeddingDedup                                    │
    │           → safetyClassifier                                │
    │             → autoContinue                                  │
    │               → promptInjectionGuard                        │
    │                 → guardrails                                │
    │                   → piiRedact                               │
    │                     → sensitiveDataAudit                    │
    │                       → costGuard / costBudget              │
    │                         → adaptiveMaxTokens                 │
    │                           → idempotency / distributedLock   │
    │                             → circuitBreaker                │
    │                               → bulkhead                    │
    │                                 → retryOnRateLimit          │
    │                                   → responseCache           │
    │                                     → structuredOutputValidator
    │                                       → replayBuffer        │
    │                                         → (provider)        │
    └─────────────────────────────────────────────────────────────┘
                 │
                 ▼
    ┌─────────────────────────────────────────────────────────────┐
    │  LLMService (base class): retries · structured-output       │
    │  parsing · unified chunk shape · batch API dispatch         │
    └───────────────────┬─────────────────────────────────────────┘
                        │
                        ▼
    ┌─── AnthropicLLM ────┬─── AzureOpenAILLM ──┬─── GeminiLLM ─┐
    │    OpenAICompatible │    GroqLLM          │    BedrockLLM │
    │    FireworksLLM     │    DeepSeekLLM      │    MistralLLM │
    │    OllamaLLM        │    GenAIHubLLM      │               │
    └─────────────────────┴─────────────────────┴───────────────┘

The middleware sandwich is fully composable — every layer is optional, layer order is validated by validateMiddlewareOrder, and the full snapshot is exportable for GitOps drift detection via chainSnapshot. Each layer exposes an asMcpResource() handler for live observability.

• No CDS entities or served OData surface — this is a client library, not an OData service. cds.connect.to('llm') returns the provider instance directly.
• Provider selection at connect time via cds.requires.llm.kind — profile-aware, so [development], [production], [genai-hub], etc. can each point at a different backend.
• Provider inheritance: GroqLLMService and GenAIHubLLMService both extend OpenAICompatibleLLMService (they speak the OpenAI /chat/completions shape); the latter adds OAuth + resource-group headers on top.

Install

npm install @saptarishi/cds-plugin-llm

Optional peer dep for the Anthropic path: @anthropic-ai/sdk (installed automatically as a dependency).

TypeScript: full type definitions ship in the package (lib/index.d.ts). No @types/* package needed.

CLI: a saptarishi-llm executable ships with the package. Use it via npx without installing globally, or install once with npm i -g @saptarishi/cds-plugin-llm. See the CLI section.

Configure

Add to your CAP app's package.json under cds.requires:

{
  "cds": {
    "requires": {
      "llm": {
        "[development]": { "kind": "llm-groq",       "modelId": "llama-3.3-70b-versatile" },
        "[ollama]":      { "kind": "llm-ollama",     "modelId": "qwen2.5:14b"             },
        "[production]":  { "kind": "llm-genai-hub", "credentials": { "deploymentId": "..." } }
      }
    }
  }
}

Set the appropriate env var (see .env.example):

• ANTHROPIC_API_KEY for llm-anthropic
• OLLAMA_BASE_URL for llm-ollama (defaults to http://localhost:11434)
• GROQ_API_KEY for llm-groq
• OPENAI_API_KEY + OPENAI_BASE_URL for llm-openai-compatible

Providers

Kind	Backend	Cost to test	Status
llm-anthropic	Claude via Anthropic API	Pennies per call	Working
llm-ollama	Local Ollama daemon	Free	Working
llm-groq	Groq's hosted Llama/Mixtral/Qwen (sub-second inference)	Generous free tier	Working
llm-openai-compatible	Any endpoint speaking OpenAI's /chat/completions (OpenAI, Together, Fireworks, DeepSeek direct, LM Studio, LocalAI...)	Varies	Working
llm-azure-openai	Azure OpenAI Service (per-deployment URL + api-key header)	Paid via Azure subscription	Working (mock-verified end-to-end)
llm-gemini (new in 1.19.0)	Google Gemini via Google AI Studio (chat, streaming, tools, vision, embeddings)	Free tier + paid per token	Working (mock-verified end-to-end)
llm-bedrock (new in 1.19.0)	AWS Bedrock Converse API (chat, streaming, tools, vision) + Titan/Cohere embed. Uses @aws-sdk/client-bedrock-runtime (optional peer).	Paid via AWS Bedrock	Working (mock-verified end-to-end)
llm-fireworks (new in 1.23.0)	Fireworks AI — hosted OSS models (Llama, Qwen, Mixtral, DeepSeek) behind an OpenAI-compatible endpoint	Pay-per-token via Fireworks	Working (mock-verified end-to-end)
llm-deepseek (new in 1.23.0)	DeepSeek — direct API access to DeepSeek-V3 (chat) and DeepSeek-R1 (reasoning)	Very cheap per-token	Working (mock-verified end-to-end)
llm-mistral (new in 1.23.0)	Mistral AI — Mistral Large, Codestral, and the open-weights family	Pay-per-token via Mistral	Working (mock-verified end-to-end)
llm-genai-hub	SAP AI Core / Generative AI Hub	Paid (extended plan)	Spec-compliant · mock-verified end-to-end · live verify open (community help welcome)

Stability

• Semantic Versioning. 1.x will preserve the public API contract documented in lib/index.d.ts. Breaking changes require a major version bump.
• What's covered: all exported symbols in lib/index.js — LLMService and 5 provider subclasses, chat / stream / embed shapes, ContentBlock union (text / image / document), message shapes, tool call shapes, stream chunk shapes, image / PDF helpers, and provider option fields (kind, modelId, credentials, retries, responseCache).
• What's not covered: provider-native raw response objects (shape follows the upstream API), CDS kind config field names (which follow CAP conventions), and behavior of individual model IDs (upstream provider concern).
• Deprecation policy: any deprecated field or method will be marked in the JSDoc + CHANGELOG for at least one minor release before removal, and only removed in a subsequent major version.
• Full history in CHANGELOG.md.

Use

Standard CAP idiom — cds.connect.to():

const cds = require('@sap/cds');

module.exports = class ProcurementService extends cds.ApplicationService {
  async init() {
    const llm = await cds.connect.to('llm');

    this.on('summarizePO', async (req) => {
      const { poId } = req.data;
      const po = await SELECT.one.from('PurchaseOrders').where({ ID: poId });

      const { text } = await llm.chat({
        system: 'You summarize purchase orders for approvers in 2 sentences.',
        messages: [{ role: 'user', content: JSON.stringify(po) }],
        cache: true,  // Anthropic-only: caches the system prompt
      });

      return text;
    });

    return super.init();
  }
};

Structured outputs (new in v0.3.0)

Pass a JSON schema via format and get a parsed .data field back:

const { data, usage } = await llm.chat({
  system: 'Assess supplier invoice risk for AP triage.',
  messages: [{ role: 'user', content: invoiceJson }],
  format: {
    type: 'object',
    properties: {
      risk: { type: 'string', enum: ['low', 'medium', 'high'] },
      rationale: { type: 'string' },
    },
    required: ['risk', 'rationale'],
    additionalProperties: false,
  },
});

console.log(data.risk);       // 'high'
console.log(data.rationale);  // 'Amount over 100k EUR without matched PO...'

Under the hood:

• Anthropic: uses output_config.format (native JSON schema)
• OpenAI-compatible / Groq: uses response_format: { type: 'json_object' } and prepends the schema to the system prompt for broadest model coverage
• Ollama: uses native format field (schema-strict on recent Ollama versions)
• Base class post-parses .text into .data uniformly; falls back to first-{...}-block extraction if the model wrapped the JSON in prose

Tool use / function calling (new in v0.3.0)

Pass a unified tool schema; get normalized toolCalls back:

const turn1 = await llm.chat({
  system: 'Help procurement approvers. Use tools to fetch data.',
  messages: [{ role: 'user', content: 'Fetch PO 4500000123' }],
  tools: [{
    name: 'get_purchase_order',
    description: 'Fetch a purchase order by its 10-digit ID',
    input_schema: {
      type: 'object',
      properties: { purchaseOrderId: { type: 'string' } },
      required: ['purchaseOrderId'],
    },
  }],
});

if (turn1.toolCalls?.length) {
  const call = turn1.toolCalls[0];  // { id, name, input }
  const result = await fetchPO(call.input.purchaseOrderId);  // your app logic

  // Feed the result back for turn 2
  const turn2 = await llm.chat({
    system: '...',
    messages: [
      { role: 'user',      content: 'Fetch PO 4500000123' },
      { role: 'assistant', toolCalls: turn1.toolCalls },
      { role: 'tool',      tool_call_id: call.id, content: JSON.stringify(result) },
    ],
    tools: [...],
  });
  console.log(turn2.text);  // model's final answer
}

Works across providers with matching { id, name, input } shape. Individual model quality varies for multi-tool scenarios — llama-3.3-70b on Groq is solid for single-tool cases; Claude and qwen2.5 are more reliable for chained tool use.

Tool runner — automatic multi-turn loop (new in v1.1.0)

For agent-style code that would otherwise write the "call → execute → feed back → repeat" loop by hand, runTools() wraps the pattern:

const { runTools } = require('@saptarishi/cds-plugin-llm');

const result = await runTools({
  llm,
  system: 'You help procurement approvers.',
  messages: [{ role: 'user', content: 'Fetch PO 4500000123 and summarize it.' }],
  tools: [{
    name: 'get_purchase_order',
    description: 'Fetch a PO by 10-digit ID',
    input_schema: {
      type: 'object',
      properties: { purchaseOrderId: { type: 'string' } },
      required: ['purchaseOrderId'],
    },
    run: async ({ purchaseOrderId }) =>
      await SELECT.one.from('PurchaseOrders').where({ ID: purchaseOrderId }),
  }],
  maxSteps: 10,
});

result.text        // final assistant answer
result.steps       // number of chat() calls made
result.toolCalls   // [{ id, name, input, result, isError }, ...] — every call executed
result.usage       // aggregated tokens across all turns

What it handles for you:

• Executes every tool call in every turn (multiple in one turn all run)
• Appends assistant + tool messages correctly (matches the format both Anthropic and OpenAI-compat expect)
• Catches tool exceptions and surfaces them as tool_result with is_error: true so the model can recover
• Rejects with a clear message when an unknown tool name is called
• maxSteps safety cap (default 10) — throws if the model loops forever
• Optional onStep({ step, response }) callback to observe each turn

Streaming (new in v0.6.0)

Get tokens as they arrive from the model instead of waiting for the full response:

for await (const chunk of llm.stream({
  system: 'You are a poet.',
  messages: [{ role: 'user', content: 'Write a haiku about SAP procurement.' }],
})) {
  if (chunk.type === 'text_delta') {
    process.stdout.write(chunk.text);
  }
  if (chunk.type === 'done') {
    console.log(`\n[${chunk.usage.output_tokens} tokens, stopReason: ${chunk.stopReason}]`);
  }
}

Chunk types:

type	payload	when
text_delta	{ text } — incremental piece of text	fires per token/token-group as the model generates
done	{ text, usage, stopReason, model } — accumulated text + final metadata	once at the end

Wire-shape parsing per provider:

• Anthropic: uses the SDK's messages.stream() — SSE under the hood, events adapted to unified chunks
• OpenAI-compatible / Groq / GenAI Hub: parses SSE (data: {json}\n\n) from the /chat/completions streaming endpoint, adds stream_options: {include_usage: true} so usage populates on the done chunk
• Ollama: parses NDJSON from /api/chat, emits done when the stream's final message carries done:true

Retries are not applied to streams (partial-response semantics are unclear). If a stream fails mid-way, the caller sees the error thrown from the generator.

Try the demo:

node scripts/stream-demo.js "Explain streaming LLM responses in 3 sentences."

Embeddings (expanded in v0.7.0)

const { embeddings } = await llm.embed({
  input: ['first document', 'second document', 'third'],
  model: 'text-embedding-3-small',  // optional; falls back to configured modelId
});
// embeddings is number[][] — one vector per input string

Supported providers:

• Ollama — mxbai-embed-large, nomic-embed-text, all-minilm, any embedding model you've pulled
• OpenAI-compatible (including Groq, Together AI, DeepSeek, LM Studio) — text-embedding-3-small, text-embedding-3-large, text-embedding-ada-002, provider-specific models
• Anthropic: not supported (no first-party embeddings)
• GenAI Hub: needs a separate embedding-model deployment; not yet plumbed (planned for 0.9)

Single string or array of strings both work. Returns { embeddings: number[][], model: string } — the outer array always matches the input length.

Vision / multimodal input (new in v0.5.0)

Pass images inline as content blocks. Works across all providers with vision-capable models (Claude 3.5+, GPT-4o, Groq's llama-3.2-*-vision, Ollama's llava / moondream / llama3.2-vision).

const { imageFromFile, imageFromUrl, imageFromBase64 } = require('@saptarishi/cds-plugin-llm');

// Load from disk
const image = await imageFromFile('/tmp/scanned-invoice.png');

// Or from a URL (Anthropic + OpenAI-compat; Ollama needs base64)
const image = imageFromUrl('https://example.com/invoice.png');

// Or from base64 data you already have
const image = imageFromBase64(base64Data, 'image/png');

const { data } = await llm.chat({
  model: 'gpt-4o',  // or claude-opus-4-7, llama-3.2-11b-vision-preview, llava, ...
  system: 'Extract structured data from scanned invoices.',
  messages: [{
    role: 'user',
    content: [
      image,
      { type: 'text', text: 'Return the vendor, invoice number, and line items.' },
    ],
  }],
  format: {
    type: 'object',
    properties: {
      vendor: { type: 'string' },
      invoiceNumber: { type: 'string' },
      lineItems: {
        type: 'array',
        items: {
          type: 'object',
          properties: {
            description: { type: 'string' },
            quantity: { type: 'number' },
            unitPrice: { type: 'number' },
          },
        },
      },
    },
  },
});

Wire-shape translation is provider-aware:

• Anthropic: native content blocks (source can be url or base64)
• OpenAI-compatible / Groq: image_url blocks with data URLs for base64
• Ollama: text goes in content, images extracted to images: [base64, ...] (Ollama does not accept URLs — use imageFromFile() or imageFromBase64())

PDF documents (v0.8.0 · expanded in v0.9.0)

Pass PDF documents inline. Full native support on Anthropic (Claude 3.5+ parses text + visuals in one pass). Since v0.9.0 OpenAI-compat providers accept base64 PDFs too via the file content-block shape — works on GPT-4o and newer OpenAI models. Groq and other OpenAI-compat providers that don't accept files will 400 upstream.

const { pdfFromFile, pdfFromUrl, pdfFromBase64 } = require('@saptarishi/cds-plugin-llm');

const pdf = await pdfFromFile('/tmp/scanned-invoice.pdf');
// or: const pdf = pdfFromUrl('https://example.com/invoice.pdf');
// or: const pdf = pdfFromBase64(base64Data);

const { data } = await llm.chat({
  model: 'claude-opus-4-7',
  system: 'Extract structured data from scanned invoices.',
  messages: [{
    role: 'user',
    content: [
      pdf,
      { type: 'text', text: 'Return vendor, invoice number, line items.' },
    ],
  }],
  format: { /* JSON schema */ },
});

Provider notes:

• Anthropic — native, both base64 and URL sources
• OpenAI-compat (GPT-4o+) — base64 works out of the box. URL PDFs (new in v1.14.0) via the Files API: call uploadPdfFromUrl(url, {apiKey}) first — it fetches, uploads to /v1/files, and returns a document block with source.type='file_id' that the provider translates to {type:'file', file:{file_id}}.
• Groq / other OpenAI-compat that don't accept files — base64 will 400 upstream; /v1/files isn't exposed so uploadPdfFromUrl will also 404 there
• Ollama — no PDF support; render pages to images via pdftoppm (poppler) and pass to a vision model like llava or llama3.2-vision

URL PDFs on OpenAI (v1.14.0+)

const { uploadPdfFromUrl } = require('@saptarishi/cds-plugin-llm');

const doc = await uploadPdfFromUrl('https://example.com/contract.pdf', {
  apiKey: process.env.OPENAI_API_KEY,
  // baseUrl:  'https://api.openai.com/v1'  (default)
  // purpose:  'user_data'                  (default)
  // filename: <inferred from URL basename, .pdf appended if missing>
});

await openai.chat({
  messages: [{ role: 'user', content: [doc, { type: 'text', text: 'Summarize' }] }],
});

The returned document block has source: { type: 'file_id', file_id: 'file-xxx' }. Reuse the block across multiple chat calls to avoid re-uploading. OpenAI retains uploaded files by account — bring your own retention policy.

SAP Generative AI Hub setup

The llm-genai-hub kind targets a deployment in your BTP AI Core instance. Prerequisites:

1. Provision AI Core — BTP Cockpit → Service Marketplace → AI Core → extended plan (free plan does not include Generative AI Hub).
2. Create a resource group (or use default).
3. Deploy a model via SAP AI Launchpad, ai-api-cli, or the SDK — e.g. gpt-4o, mistral-large-instruct, claude-3-5-sonnet. Note the deployment ID.
4. Configure the plugin — three ways depending on where your CAP app runs.

On BTP Cloud Foundry (recommended)

Bind the AI Core service instance to your CAP app:

cf bind-service <your-app> <ai-core-instance>
cf restage <your-app>

Then set only the deployment ID (credentials auto-discovered from VCAP_SERVICES):

cf set-env <your-app> AICORE_DEPLOYMENT_ID <deployment-id>
cf set-env <your-app> AICORE_RESOURCE_GROUP default   # optional; defaults to 'default'

In package.json:

{
  "cds": { "requires": { "llm": {
    "[production]": { "kind": "llm-genai-hub", "modelId": "gpt-4o" }
  }}}
}

On Kyma

Attach the service binding manifest, then set the same env vars via a ConfigMap or Secret. The VCAP_SERVICES layout is preserved by the SBO (Service Binding Operator).

Local dev pointing at a BTP-hosted AI Core

Extract the service key JSON from BTP Cockpit (Service Instance → Service Keys → View). Put values in .env:

AICORE_API_URL=https://api.ai.prod.eu-central-1.aws.ml.hana.ondemand.com
AICORE_AUTH_URL=https://<subaccount>.authentication.<region>.hana.ondemand.com
AICORE_CLIENT_ID=sb-...
AICORE_CLIENT_SECRET=...
AICORE_DEPLOYMENT_ID=abc123                # chat model deployment
AICORE_EMBEDDING_DEPLOYMENT_ID=def456      # optional; enables llm.embed()
AICORE_MODEL=gpt-4o

Or pass explicitly in package.json:

{
  "cds": { "requires": { "llm": {
    "[genai-hub]": {
      "kind": "llm-genai-hub",
      "modelId": "gpt-4o",
      "credentials": {
        "aiCoreUrl":     "https://api.ai.prod.eu-central-1.aws.ml.hana.ondemand.com",
        "tokenUrl":      "https://<subaccount>.authentication.<region>.hana.ondemand.com",
        "clientId":      "sb-...",
        "clientSecret":  "...",
        "deploymentId":  "abc123",
        "resourceGroup": "default"
      }
    }
  }}}
}

What it handles for you

• OAuth2 client-credentials flow against XSUAA
• Token caching + refresh (60s before expiry)
• AI-Resource-Group header
• Deployment-based inference endpoint construction
• VCAP_SERVICES.aicore auto-discovery when the service is bound

Known limitations (v0.4.0)

• OpenAI-shape only. Deployments that expose the OpenAI /chat/completions shape (GPT, Mistral, Llama, Gemini, and Anthropic-via-shim) work. Native Anthropic-shape deployments (Claude via /invoke) are not yet supported — use the llm-anthropic kind directly for Claude.
• Not yet live-verified. Built to the SAP-documented API contract and unit-tested against mocks. Live verification against an AI Core extended deployment is the next contribution wanted.

Automatic retries (new in v0.3.0)

Every chat() and embed() call is wrapped with exponential-backoff retry on 429 / 5xx responses. Honors Retry-After headers. Configurable per-call or globally:

// Per-call override
await llm.chat({ messages: [...], retries: { max: 5, baseMs: 1000, maxMs: 30000 } });

// Or via cds.requires.llm config:
{ "cds": { "requires": { "llm": {
  "kind": "llm-groq",
  "retries": { "max": 5 }
}}}}

Middleware / interceptors (new in v1.2.0)

Register hooks around every chat / stream / embed call. Koa-style compose — outermost first, next() returns the next middleware's result (or the provider's response). Middleware may inspect or transform the request AND the response, share state via ctx.meta, or short-circuit by returning without calling next().

const llm = await cds.connect.to('llm');

// 1. Logging + duration
llm.use(async (ctx, next) => {
  const start = Date.now();
  const res = await next();
  console.log(`[${ctx.method}] ${Date.now() - start}ms`);
  return res;
});

// 2. Cost tracking (aggregate tokens across every call)
const totals = { in: 0, out: 0 };
llm.use(async (ctx, next) => {
  const res = await next();
  totals.in  += res?.usage?.input_tokens  ?? 0;
  totals.out += res?.usage?.output_tokens ?? 0;
  return res;
});

// 3. Auto-injected system prompt suffix
llm.use(async (ctx, next) => {
  if (ctx.method === 'chat' && ctx.request.system) {
    ctx.request.system += '\n\nBe concise. If uncertain, say so.';
  }
  return next();
});

// 4. Streams: wrap the iterator to observe each chunk
llm.use(async (ctx, next) => {
  if (ctx.method !== 'stream') return next();
  const inner = await next();
  return (async function* () {
    for await (const chunk of inner) {
      if (chunk.type === 'text_delta') myLiveUI.append(chunk.text);
      yield chunk;
    }
  })();
});

Context object:

• ctx.method — 'chat' | 'stream' | 'embed'
• ctx.request — mutable request options (modify before next() to affect the provider call)
• ctx.meta — scratchpad for cross-middleware state (e.g. timing marks, request IDs)

Notes:

• Middleware runs around the response cache, retries, and format-parsing — those are internal concerns your middleware can observe. Cache hits arrive at your middleware with cached: true set.
• Calling next() more than once from the same middleware throws.
• Errors propagate up the chain.
• For streams, next() returns an async iterable. To observe/transform chunks, wrap it into a new async generator.

Built-in middleware (new in v1.3.0)

Two production-oriented middlewares ship in the box: rate limiting and OpenTelemetry tracing.

rateLimit — token-bucket limiter

const { rateLimit } = require('@saptarishi/cds-plugin-llm');

const llm = await cds.connect.to('llm');

// Global: 60 requests burst, refill at 1/s
llm.use(rateLimit({ capacity: 60, refillPerSecond: 1 }));

// Per-user: keyed off ctx.meta.user (populate this from an earlier middleware
// that inspects your CAP request)
llm.use(rateLimit({
  capacity: 10,
  refillPerSecond: 0.2,
  keyFn: (ctx) => ctx.meta.user ?? 'anon',
  mode: 'wait',   // 'throw' (default) or 'wait' — pause instead of erroring
}));

When mode: 'throw' and the bucket is empty, the middleware throws an Error with code: 'RATE_LIMITED' and retryAfterMs so you can surface a proper 429 to your caller. Buckets are in-process — for multi-instance CF apps that need a shared counter, back with Redis via your own middleware.

otel — OpenTelemetry spans

const { trace } = require('@opentelemetry/api');
const { otel } = require('@saptarishi/cds-plugin-llm');

llm.use(otel({
  tracer: trace.getTracer('cap-app'),
  systemAttribute: 'anthropic',   // sets gen_ai.system on every span
}));

Emits one span per chat / stream / embed call. Attributes follow the emerging GenAI semantic conventions where possible: gen_ai.system, gen_ai.operation.name, gen_ai.request.model, gen_ai.response.model, gen_ai.usage.input_tokens / output_tokens, gen_ai.response.stop_reason. Plus a few plugin-specific ones: llm.cached, llm.tool_calls.count, llm.stream.chunks, llm.embed.count. Duck-typed against @opentelemetry/api — no hard dependency, works with any object exposing startSpan().

Stream spans stay open through the whole iterator (span ends on the done chunk, on early break, or on error — never leaks).

redisRateLimit — shared bucket across CF instances (new in v1.4.0)

The in-process rateLimit is fine for single-instance apps. For multi-instance CF deployments where a shared quota must be enforced globally, back the bucket with Redis.

const Redis = require('ioredis');
const { redisRateLimit } = require('@saptarishi/cds-plugin-llm');

llm.use(redisRateLimit({
  redis: new Redis(process.env.REDIS_URL),
  capacity: 60,
  refillPerSecond: 1,
  keyFn: (ctx) => ctx.meta.user ?? 'anon',
  keyPrefix: 'ratelimit:llm:',   // default 'saptarishi:llm:rl:'
  mode: 'throw',                  // 'throw' | 'wait'
}));

Uses an atomic Lua EVAL so two instances checking the bucket at the same time cannot both succeed when only one token is left. Duck-typed client — any object with an eval(script, numKeys, ...args) promise API satisfies (works with ioredis and node-redis v4+ out of the box). On BTP, bind a Redis service to your CF app and pull the URL from VCAP_SERVICES.

CLI (new in v1.5.0)

A saptarishi-llm executable ships with the package. Handy for provider health checks in CI, quick prompt experiments from the shell, pipelining embeddings into a downstream tool, or scaffolding a fresh CAP project pre-wired to the plugin (v1.6.0).

npx @saptarishi/cds-plugin-llm --help

# or install globally
npm install -g @saptarishi/cds-plugin-llm
saptarishi-llm --help

Commands

saptarishi-llm chat -p "explain SAP CAP in one sentence"
saptarishi-llm stream -p "write a haiku about procurement"
saptarishi-llm embed -p "purchase order for steel coils" --json
saptarishi-llm verify --provider anthropic
saptarishi-llm providers

Provider selection

--provider <kind> or SAPTARISHI_LLM_PROVIDER env var. Same five kinds as the CAP plugin: anthropic, ollama, groq, openai-compatible, genai-hub.

Credentials come from env vars (never CLI flags — avoids leaking secrets into shell history):

ANTHROPIC_API_KEY=sk-ant-... saptarishi-llm chat -p "hello"
OLLAMA_URL=http://192.168.5.13:11434 saptarishi-llm chat --provider ollama -p "hello"
GROQ_API_KEY=gsk-... saptarishi-llm verify --provider groq

Input sources

Prompt can come from --prompt / -p, --file / -f, stdin, or a positional arg. Multiple sources concatenate with a blank line between them.

echo "summarize this" | saptarishi-llm chat -f contract.pdf.txt
saptarishi-llm embed -p "one\n---\ntwo\n---\nthree"      # 3 vectors from 1 call

CI health checks

verify connects, runs a tiny probe, reports latency, and exits 0 on success / 1 on unexpected reply / 1 on error. Drop it in a nightly workflow to catch expired credentials or endpoint outages before your CAP app does.

saptarishi-llm verify --provider genai-hub --json

Expose as an MCP server (new in v1.7.0)

saptarishi-llm mcp runs a Model Context Protocol server over stdio that exposes the configured provider as tools any MCP client can call. Register it in Claude Desktop, Cursor, Zed, or any other MCP-capable client and those clients gain a chat / embed / verify / list_providers tool backed by your provider config — with all its middleware, caching, rate limits, and OTel tracing.

The point: one MCP server = "the sanctioned way to call an LLM at MyCompany". Centralized credentials, centralized policy, developer productivity everywhere.

Claude Desktop (~/Library/Application Support/Claude/claude_desktop_config.json on macOS):

{
  "mcpServers": {
    "saptarishi-llm": {
      "command": "npx",
      "args": ["-y", "@saptarishi/cds-plugin-llm", "mcp"],
      "env": {
        "SAPTARISHI_LLM_PROVIDER": "groq",
        "GROQ_API_KEY": "gsk-..."
      }
    }
  }
}

Tools exposed:

Tool	Purpose
chat	Send a prompt, return text. { prompt, system?, maxTokens? }
embed	Embed input(s) into vectors. { input: string | string[] }
verify	Tiny probe against the provider. Returns { ok, latencyMs, model, text }.
list_providers	Enumerate every supported provider kind with default models.

Uses a hand-rolled MCP implementation (2024-11-05 spec) over stdio JSON-RPC 2.0 — zero new dependencies. Full protocol coverage: initialize, tools/list, tools/call, resources/list, resources/read, resources/subscribe, resources/unsubscribe, prompts/list, prompts/get, ping, notifications. Tool errors surface as result.isError: true per spec so the model can recover, not as JSON-RPC errors.

Resources exposed (readable via resources/read):

URI	Description
config://active-provider	Current provider kind + model + middleware count.
config://supported-providers	Every provider kind + default model.

Prompts registered (invokable via prompts/get) — see Prompt-template registry below for the full built-in list.

Resource templates (new in v1.9.0; parametrized URIs discoverable via resources/templates/list):

URI template	What clients can read
provider://{kind}	Default model for a specific provider kind (provider://groq → { kind: 'groq', defaultModel: 'llama-3.3-70b-versatile' }).
prompt://{name}	Metadata (arguments, description) for a registered prompt template. To render, use prompts/get.

Scaffold a fresh CAP project (new in v1.6.0)

init creates a fully-wired CAP app in seconds — no manual package.json editing, no CDS boilerplate:

npx @saptarishi/cds-plugin-llm init joule-demo --provider groq
cd joule-demo
cp .env.example .env               # then fill in real credentials
npm install
cds watch

Then:

curl 'http://localhost:4004/ai/chat(prompt='"'"'hello'"'"')'

Generated:

joule-demo/
├── package.json          # cds.requires.llm pointing at chosen provider
├── srv/
│   ├── ai-service.cds    # service AIService { chat(prompt), summarize(text) }
│   └── ai-service.js     # handlers using cds.connect.to('llm')
│                         # + SSE streaming endpoint at POST /stream/chat
├── .env.example          # provider-specific env vars
├── .gitignore            # excludes .env, node_modules/, gen/
└── README.md             # how to run

Streaming out of the box (v1.9.0+): the generated srv/ai-service.js registers POST /stream/chat inline, streaming tokens as they arrive:

curl -N -X POST http://localhost:4004/stream/chat \
  -H 'content-type: application/json' \
  -d '{"prompt":"write a haiku about SAP CAP"}'

Flags:

• --provider <kind> — anthropic (default) | ollama | groq | openai-compatible | genai-hub
• --model <id> — override the default model for the chosen provider
• --force — overwrite a non-empty target directory
• --dry-run — print the file list without writing anything

Prompt-template registry (new in v1.8.0)

Register named prompt templates once, invoke them by name from any CAP handler, and automatically expose them over MCP so external clients (Claude Desktop, Cursor, Zed) can invoke them too. Same registry, three surfaces.

const { PromptRegistry, builtInPrompts } = require('@saptarishi/cds-plugin-llm');

const prompts = new PromptRegistry()
  .registerAll(builtInPrompts())                    // 5 built-ins
  .register({                                       // your own
    name: 'invoice_dispute_response',
    description: 'Draft a courteous response to an invoice dispute.',
    arguments: [
      { name: 'dispute', required: true, description: 'The dispute text' },
      { name: 'tone',    required: false, description: 'formal | friendly' },
    ],
    render: ({ dispute, tone = 'formal' }) => ({
      system: `You are AP support. Reply in a ${tone} tone. Never admit liability.`,
      messages: [{ role: 'user', content: dispute }],
    }),
  });

// From a CAP handler:
const req = prompts.render('invoice_dispute_response', { dispute });
const res = await llm.chat({ ...req, maxTokens: 512 });

Built-ins (builtInPrompts()):

Name	Purpose
summarize	Condense text to N sentences (text, sentences?).
extract_json	Extract structured JSON against a schema (text, schema).
classify	Assign one label from a set (text, labels).
translate	Translate to a target language (text, targetLanguage).
procurement_risk_scorer	SAP-flavored risk analyst prompt (text).

Templates auto-appear as MCP prompts when you run saptarishi-llm mcp. Claude Desktop shows them as slash-commands the user can pick.

Load templates from a folder (new in v1.9.0)

Instead of registering templates inline, drop *.mjs or *.js files into a directory and load them at boot:

prompts/
├── invoice_dispute.mjs        # export default { name, render, ... }
├── kpi_extractor.mjs          # export default { ... }
└── shared.mjs                 # export const foo = ...; export const bar = ...

await registry.loadFromDir('./prompts');

Or expose the whole folder via MCP without writing any code:

saptarishi-llm mcp --prompts-dir ./prompts
# or: SAPTARISHI_LLM_PROMPTS_DIR=./prompts saptarishi-llm mcp

Three export conventions handled in one scan: export default <template>, export default [<t1>, <t2>], or named exports.

Add --watch-prompts and the server hot-reloads templates when files change — iterate without restart (new in v1.10.0):

saptarishi-llm mcp --prompts-dir ./prompts --watch-prompts

Git-backed prompt registry (new in v2.1.0)

gitPromptRegistry extends PromptRegistry with a loader that pulls templates from a Git repository, caches them locally, and optionally polls for updates. Prompt changes go through PR review separately from code deploys — a real prompt-as-code workflow.

const { gitPromptRegistry } = require('@saptarishi/cds-plugin-llm');

const registry = await gitPromptRegistry({
  url:     'https://github.com/your-org/prompts.git',
  branch:  'main',
  subdir:  'templates',      // optional — load only a subfolder
  pollMs:  60_000,           // poll every 60s for new commits
  onUpdate: (info) => cds.log('llm:prompts').info('reloaded', info),
});

// Same API as PromptRegistry:
const req = registry.render('invoice_dispute_response', { dispute });

• Exposes config://git-prompt-registry (URL, branch, current SHA, refresh timestamp, counters) — live-visible in your MCP client.
• Pin to a specific ref (SHA / tag) for immutable prompts in prod; use branch for rolling updates in staging.
• Works with any prompt directory the shipped loadFromDir scanner accepts.

HTTP+SSE transport (new in v1.10.0)

By default saptarishi-llm mcp speaks stdio (for Claude Desktop / Cursor / Zed as a local subprocess). Add --http and it runs as a network service instead — the same MCP server, addressable over HTTP. Deploy to CF, put behind an auth proxy, share with a team.

saptarishi-llm mcp --http --port 3333 --host 0.0.0.0
# → MCP HTTP+SSE listening on http://0.0.0.0:3333/sse
# → GET /health for liveness/session count

Wire protocol (MCP 2024-11-05 SSE spec):

Method	Path	Purpose
GET	/sse	Client opens SSE stream. Server sends event: endpoint\ndata: /messages?sessionId=<uuid>. Server replies to that session flow back on this stream.
POST	/messages?sessionId=<uuid>	Client sends a single JSON-RPC message. Server acknowledges 202; reply arrives on the SSE stream.
GET	/health	{ server, version, transport, sessions } — plug into monitoring.

Multi-session — N concurrent clients each get their own session and stream. Graceful shutdown on SIGINT/SIGTERM.

Bearer-token auth (new in v1.11.0)

Deploying --http to anything beyond 127.0.0.1 should require a credential. Set --auth-token (or SAPTARISHI_LLM_MCP_TOKEN env) and every /sse and /messages request must send Authorization: Bearer <token>:

saptarishi-llm mcp --http --host 0.0.0.0 --port 3333 \
  --auth-token "$(openssl rand -hex 32)"

Behavior:

• Missing / wrong token → 401 Unauthorized with WWW-Authenticate: Bearer realm="mcp".
• /health stays public so load balancers can probe without credentials. Its response now includes authRequired: <bool> so clients know whether they need a token.
• Constant-time comparison on the byte-by-byte match to avoid trivial timing side-channels.
• Binding to a non-loopback host without a token prints a loud stderr warning — no hard failure (upstream proxies may legitimately terminate auth), but the intent is impossible to miss.

Client-side example (curl):

curl -N -H 'Authorization: Bearer <token>' http://host:3333/sse

JWT / JWKS validation (new in v1.16.0)

For real enterprise deployments, validate signed JWTs against your IdP's JWKS endpoint. Works with SAP XSUAA, Auth0, Okta, Azure AD, Google, Keycloak, AWS Cognito, and any other standards-compliant IdP.

npm install jose   # optional peer dep, only needed for JWT auth

saptarishi-llm mcp --http --host 0.0.0.0 \
  --jwks-url https://tenant.authentication.us10.hana.ondemand.com/token_keys \
  --jwt-issuer https://tenant.authentication.us10.hana.ondemand.com \
  --jwt-audience sb-my-cap-app!t12345

Behavior:

• Each request's Authorization: Bearer <jwt> is verified against the JWKS (signature, exp, iat, nbf, optional iss / aud).
• JWKS cached + auto-refreshed on unknown kid by jose.
• Any failure — bad signature, expired, wrong issuer, wrong audience, unknown key — collapses to a uniform 401. Never leaks which check failed (avoids oracle attacks; same rationale as bearer-token constant-time compare).

On BTP with an XSUAA service binding, use the values from VCAP_SERVICES.xsuaa[0].credentials:

• --jwks-url <url>/token_keys (where <url> = credentials.url)
• --jwt-issuer <url>
• --jwt-audience credentials.xsappname

Programmatic use — build your own verifier for custom flows (introspection endpoints, mTLS metadata → role mapping, JWT + local RBAC check):

const { createHttpTransport, createJwtVerifier } = require('@saptarishi/cds-plugin-llm');

const jwtVerifier = createJwtVerifier({ jwksUrl, issuer, audience });
createHttpTransport({
  server,
  authTokenVerifier: async (token) => {
    const claims = await jwtVerifier(token);
    if (!claims) return null;
    // Extra check: reject unless the client has the 'llm-user' scope
    if (!claims.scope?.includes('llm-user')) return null;
    return claims;
  },
});

Progress notifications (new in v1.12.0)

Long-running tools can push notifications/progress back to the client instead of leaving them hanging silently. Tool handlers grow an optional second argument — a ctx object with reportProgress(current, total?):

server.registerTool({
  name: 'batch_summarize',
  description: 'Summarize N documents',
  inputSchema: { type: 'object', properties: { docs: { type: 'array', items: { type: 'string' } } } },
  handler: async ({ docs }, ctx) => {
    const summaries = [];
    for (let i = 0; i < docs.length; i++) {
      summaries.push(await llm.chat({ /* ... */ }));
      ctx.reportProgress(i + 1, docs.length);
    }
    return summaries;
  },
});

When the client calls this tool with _meta.progressToken: 'x', each reportProgress(...) becomes a server-pushed notification carrying that token. Delivered on both transports — stdio writes to the same stdout; HTTP+SSE pushes on the requesting session's SSE stream. Existing tools that ignore the second arg keep working (backwards-compatible).

Resource subscriptions (new in v1.17.0)

Complements the list-changed notifications from v1.13.0 with per-URI push updates. Use list_changed when the set of resources shifts, resources/subscribe when a specific resource's content changes.

// client -> server
{ "jsonrpc": "2.0", "id": 1, "method": "resources/subscribe",
  "params": { "uri": "prompt://summarize" } }

// server -> client (later, when that prompt is hot-reloaded)
{ "jsonrpc": "2.0", "method": "notifications/resources/updated",
  "params": { "uri": "prompt://summarize" } }

• Per-connection state — session A subscribing never affects session B. Cleaned up when the SSE stream / stdio pipe closes.
• Both transports — stdio writes updates to stdout on the reply stream; HTTP+SSE pushes to the subscribing session's stream only.
• Unknown URIs rejected at subscribe time so client typos surface immediately. Matches both static resources and template URIs (prompt://{name}, provider://{kind}).
• resources/unsubscribe is idempotent — safe to call for URIs you never subscribed to.

--watch-prompts now fires resources/updated for every subscribed prompt://<name> URI on hot-reload (in addition to the existing prompts/list_changed broadcast). Pin a client to a specific prompt and every edit refreshes without a resources/read.

For third-party transports / programmatic MCP setups, MCPServer exposes:

• notifyResourceUpdated(uri) — broadcast to every subscriber of uri. Silent no-op with none.
• subscribedUris(prefix?) — distinct URIs any connected client subscribed to, optionally prefix-filtered.

Per-session provider overrides (new in v1.18.0)

One saptarishi-llm mcp process, multiple named provider configs behind a single MCP endpoint. Clients pick which one to use per session or per tool call. Real enterprise use: one authenticated endpoint, different agents hit different backends — DevOps on cheap, compliance reviewer on smart, local dev on local. Credentials centralized server-side; agents never touch API keys.

saptarishi-llm mcp --http --host 0.0.0.0 \
  --providers-config ./providers.json

// providers.json  —  chmod 600 and keep out of git
{
  "cheap": {
    "kind": "groq",
    "model": "llama-3.1-8b-instant",
    "credentials": { "apiKey": "gsk_..." }
  },
  "smart": {
    "kind": "anthropic",
    "model": "claude-opus-4-7",
    "credentials": { "apiKey": "sk-ant-..." }
  },
  "local": {
    "kind": "ollama",
    "model": "qwen2.5:14b",
    "credentials": { "baseUrl": "http://localhost:11434" }
  }
}

Resolution order per tool call: arguments.provider → sessionState.provider (from initialize._meta.provider) → top-level default.

Session default — client declares once at initialize; every subsequent call on that session uses it:

{ "jsonrpc": "2.0", "id": 1, "method": "initialize",
  "params": { "protocolVersion": "2024-11-05",
              "_meta": { "provider": "smart" } } }

Per-call override — bypass the session default one call at a time:

{ "jsonrpc": "2.0", "id": 2, "method": "tools/call",
  "params": { "name": "chat",
              "arguments": { "prompt": "dispatch this", "provider": "cheap" } } }

Discovery — list_providers tool and config://providers resource both dump the alias list (kind + model per alias; credentials never returned). Unknown aliases surface as tool-result errors with the configured list so the model can self-correct on the next turn.

Boot-time validation — every alias is instantiated + init()'d at server startup. Bad credentials or unknown kinds die on saptarishi-llm mcp, not on the first client call.

Response caching (new in v0.9.0)

Opt-in per-instance LRU cache with TTL. Skips tool-use calls (side-effects) and streaming (partial responses). Hits return the same ChatResponse shape with cached: true set.

{
  "cds": { "requires": { "llm": {
    "kind": "llm-groq",
    "modelId": "llama-3.3-70b-versatile",
    "responseCache": true                              // defaults: 5min TTL, 100 entries
    // or: "responseCache": { "ttlMs": 600000, "maxEntries": 500 }
  }}}
}

Cache key is a SHA-1 of the request's stable JSON representation (model + maxTokens + system + messages + tools + format + thinking). Requests with any of those fields differing miss the cache.

const r1 = await llm.chat({ messages: [{ role: 'user', content: 'hi' }] });
r1.cached  // undefined  (miss + fresh call)

const r2 = await llm.chat({ messages: [{ role: 'user', content: 'hi' }] });
r2.cached  // true        (hit — no upstream call)

llm.responseCache.hits    // 1
llm.responseCache.misses  // 1
llm.responseCache.size()  // 1

Common wins:

• Same PO summarized twice (approver reopens the review) — instant, zero tokens
• Batch classification with duplicate inputs — deduplicates automatically
• Load-testing — dev-env queries hit cache after the first pass

Semantic response cache (new in v2.7.0)

semanticCache is a middleware-form cache keyed by embedding similarity, not byte-equality — useful when the same question keeps showing up in slightly different wording. It's separate from the built-in responseCache (which is exact-match + LRU). Bring your own embedder and vector store; an in-memory linear-scan store is provided so you can wire it up with zero infra.

const { semanticCache, inMemorySemanticStore } = require('@saptarishi/cds-plugin-llm');

llm.use(semanticCache({
  embedder: async (text) => (await llm.embed({ input: text })).embeddings[0],
  store:    inMemorySemanticStore({ maxEntries: 1000, ttlMs: 3600_000 }),
  threshold: 0.92,          // cosine similarity → hit
  keyPrefix: `tenant:${tenantId}:`,
  shouldCache: (_ctx, r) => r?.status !== 'error',
  onHit:  (i) => cds.log('llm:sc').info('hit', i),
}));

Highlights:

• Fail-open. Any exception from the embedder or store falls through to next() — a broken cache never takes the request path down.
• Multi-tenant safe. keyPrefix is post-filtered by the middleware even if the store ignores the hint, so hits cannot leak across tenants.
• Swap the store for prod. The interface is three async methods (get, put, findSimilar) — implement against pgvector, Redis, Pinecone, Weaviate, etc.
• MCP resource at config://semantic-cache exposes hitRate, hit/miss/store counts, threshold, and last similarity for live dashboards.

Composition rule: wrap semanticCache outside bulkhead/retry (cache hits shouldn't burn concurrency slots or retry budget) and inside guardrails/promptInjectionGuard (don't cache answers whose inputs were rejected).

Fuzzy dedup (new in v2.38.0)

fuzzyDedup is near-duplicate request detection using cheap character-level similarity — Jaccard on character trigrams (default, no library deps) or normalized Levenshtein. Cache-like semantics: on match, returns the prior response without calling the provider. No embedder required, so it complements (rather than competes with) semanticCache (v2.7.0). Ideal for support-ticket dedup + spam prevention where users retype the same question with typos or slight rewording.

const { fuzzyDedup, inMemoryFuzzyStore } = require('@saptarishi/cds-plugin-llm');

llm.use(fuzzyDedup({
  similarityKind: 'jaccard-trigram',   // or 'levenshtein'
  threshold:      0.85,
  store:          inMemoryFuzzyStore({ maxEntries: 5000, ttlMs: 24 * 3600_000 }),
  minKeyLength:   8,
  onHit:          (i) => cds.log('llm:fuzzy').info('near-dup', i),
}));

Fills the third slot in the dedup story alongside existing primitives:

• requestCoalescer (v2.8.0) — BYTE-IDENTICAL, IN-FLIGHT
• responseCache (v0.9.0) — BYTE-IDENTICAL, AT-REST
• semanticCache (v2.7.0) — EMBEDDING-SIMILAR (requires embedder)
• fuzzyDedup (v2.38.0, this) — CHARACTER-SIMILAR (no embedder)

Highlights:

• Two-phase matching — exact-key hit via store.get first (fast), then store.findSimilar fuzzy scan only as needed. stats.exactHits and stats.fuzzyHits are broken out separately so operators can see how much lift comes from typo-level matching vs plain repeats.
• Two algorithms — jaccard-trigram (O(n+m), scales to a few thousand chars) or levenshtein (O(n*m), catches char-level typos more precisely — cap prompt length in practice).
• In-memory reference store ships with LRU eviction (maxEntries) and optional TTL — same { get, put, findSimilar } contract used by any custom store (Redis, PostgreSQL, etc.).
• Multi-tenant safe via keyPrefix — namespace one shared store across many dedup instances; post-filter enforces isolation even if the store ignored the hint.
• minKeyLength: 8 default skips trivially-short prompts ("hi", "ok") where every prompt would collide.
• Fail-open on store errors — increments storeErrors, passes through; cache degrades to no-op rather than blocking.
• Standalone jaccardTrigram(a, b), normalizedLevenshtein(a, b), levenshteinDistance(a, b), trigrams(text) exported for custom similarity strategies.
• MCP resource: config://fuzzy-dedup.

Composition rules: compose OUTSIDE requestCoalescer (v2.8.0) so byte-identical concurrent requests coalesce first, and OUTSIDE responseCache (v0.9.0) so exact repeats short-circuit before the fuzzy scan runs at all. Compose with semanticCache (v2.7.0) when both are available — fuzzyDedup catches typos + trivial rewordings cheaply; the semantic layer catches genuine paraphrases (higher recall, embedder cost). Order dedup → quota/cost primitives so fuzzy hits never accrue cost against quotaManager (v2.23.0) or costOverrunPredictor (v2.33.0).

Empty-response detector (new in v2.37.0)

emptyResponseDetector catches broken model responses (empty string, whitespace-only, single-char replies, refusal patterns like "I can't help with that") BEFORE they reach the caller. Common failure modes caught: provider hiccups (empty payload), safety-filter or guardrail evasion that returned "", or soft-refusals on legitimate requests.

const { emptyResponseDetector } = require('@saptarishi/cds-plugin-llm');

llm.use(emptyResponseDetector({
  minChars:   10,
  onEmpty:    'retry',       // 'throw' | 'retry' | 'log'
  maxRetries: 1,
  onDetected: (i) => cds.log('llm:empty').warn('empty response', i),
}));

Distinct from other quality primitives:

• structuredOutputRepair (v2.9.0) — SCHEMA-driven repair
• responseRevision (v2.29.0) — RUBRIC-driven re-ask
• emptyResponseDetector (v2.37.0) — EMPTY / REFUSAL detection

Built-in refusal patterns (6 anchored-to-start regexes): I can't/cannot/won't help/assist, I'm unable/not able/sorry to, Sorry, but I can't, I must decline/refuse, This request cannot be, As an AI assistant, I can't. Anchored to start so mid-response mentions don't false-positive.

Highlights:

• Custom detection: pass detectEmpty(result) for full override; return true/false or { empty, reason }. Return null to fall back to defaults.
• Reason-aware retry prompt: refusal detection asks the model to explain WHY rather than refuse outright; too-short asks for a substantive answer.
• Per-reason stats (byReason) — dashboard breakdowns across too-short / whitespace / refusal-pattern / custom.
• Fail-safe custom detector — exceptions fall back to default detection.
• MCP resource: config://empty-response-detector.

Compose with responseRevision (v2.29.0) OUTSIDE this detector — first check for emptiness; if substantive but low-scoring, then revise. Two-stage quality gate.

Content-length gate (new in v2.36.0)

contentLengthGate is pre-flight size validation: reject (or truncate) prompts that exceed a per-model token budget BEFORE they hit the provider. Prevents 400 errors on over-limit contexts + saves tokens on obviously-too-large inputs. Distinct from time-based limiters (deadline, gracePeriod) — this is size-based.

const { contentLengthGate } = require('@saptarishi/cds-plugin-llm');

llm.use(contentLengthGate({
  modelLimits: {
    'gpt-4o':          128_000,
    'gpt-4o-mini':     128_000,
    'claude-opus-4-7': 200_000,
    default:            64_000,
  },
  overageMode: 'truncate-oldest',   // 'throw' | 'truncate-oldest' | 'log'
  onOverage:   (i) => cds.log('llm:size').warn('over-limit', i),
}));

Three-way overage policy:

• 'throw' (default) — raises ContentLengthExceededError (code CONTENT_LENGTH_EXCEEDED) with .tokens, .chars, .limitTokens, .model. Fail-fast.
• 'truncate-oldest' — drops oldest messages until under limit, preserving system + latest user message by default (preserveSystem / preserveLatestUser opt-out).
• 'log' — pass through unmodified with onOverage observability; let the provider decide (useful for shadow deployments before enforcing).

Highlights:

• GPT-family token heuristic by default (char / 4); users with real tokenizers pass tokenEstimator(text) → tokens.
• Per-model limits with default fallback; unknown models pass through unchecked with unknownModelCount counter for observability.
• Original request restored after truncation-mode calls — caller state unmodified, only the downstream provider saw the trimmed version.
• overageRate() returns overageCount / totalCalls for real-time tuning.
• MCP resource: config://content-length-gate.

Composition rule: put contentLengthGate outside providers so the check runs before any network work. Compose with compactHistory (v1.91.0) as complementary primitives — compact keeps multi-turn conversations bounded via LLM summarization; gate catches single-turn over-limit as the hard backstop.

Latency histogram (new in v2.35.0)

latencyHistogram tracks per-dimension latency distributions using Prometheus-style bucketed counts (bucket counts + sum + count — not raw samples). Reports p50/p95/p99 percentiles cheaply. O(1) memory per dimension regardless of sample count. Complements providerHealthAggregate (v2.31.0 — single p95 verdict) with the full percentile shape + exportable Prometheus format.

const { latencyHistogram } = require('@saptarishi/cds-plugin-llm');

const hist = latencyHistogram({
  dimensionsOf:    (ctx, result) => ({ model: result?.model ?? 'unknown' }),
  overThresholdMs: 15_000,             // p95 threshold
  onOverThreshold: (i) => cds.log('llm:slo').warn('p95 breach', i),
});
llm.use(hist);

// Query percentiles:
const p = hist.getPercentiles({ model: 'gpt-4o' });
// → { count: 145, sum: 92500, mean: 638, p50: 500, p95: 2500, p99: 5000 }

// Export for Prometheus scraping:
const prom = hist.prometheusHistograms('llm_latency_ms');
// llm_latency_ms_bucket{model="gpt-4o",le="500"} 87
// llm_latency_ms_bucket{model="gpt-4o",le="1000"} 120
// ...
// llm_latency_ms_sum{model="gpt-4o"} 92500
// llm_latency_ms_count{model="gpt-4o"} 145

Highlights:

• Prometheus-canonical bucket layout by default: [5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000, 10000] ms. Users with different SLOs pass a custom buckets array.
• Percentile computation via cumulative-count walk — O(buckets) regardless of sample count.
• Filtered percentiles — getPercentiles({ model: 'gpt-4o' }) aggregates matching dimensions.
• Custom percentile arrays — getPercentiles(filter, [10, 90, 99.9]) for tail-latency analysis.
• Rising-edge over-threshold callback rearms on recovery so subsequent breaches also fire.
• Fail-safe: dimensionsOf errors captured via onError; latency still recorded on downstream error paths.
• MCP resource: config://latency-histogram.

Compose with providerHealthAggregate (v2.31.0) — aggregate gives the healthy/degraded verdict; histogram gives the full percentile shape for diagnostics. Compose with promMetrics (v1.x) — expose prometheusHistograms() output alongside the shipped llm_* metrics for unified Grafana dashboards.

User feedback aggregator (new in v2.34.0)

userFeedbackAggregator collects human ratings (thumbs up/down or 1-5 stars) per response and aggregates them by dimensions (prompt template, model, tenant) for real-time dashboards. Two-part primitive: middleware attaches feedbackId for later attribution; submitFeedback(id, rating, meta?) records the human rating once the user actually reacts.

const { userFeedbackAggregator } = require('@saptarishi/cds-plugin-llm');

const feedback = userFeedbackAggregator({
  dimensionsOf: (ctx, result) => ({
    template: ctx.meta?.promptVersion?.name,
    model:    result?.model,
    tenant:   ctx.request.tenantId,
  }),
  ratingKind:   'binary',      // 'binary' | 'scale' | 'custom'
  windowMs:     7 * 24 * 3600_000,
});
llm.use(feedback);

// Later, when the user clicks 👍:
feedback.submitFeedback(response.feedbackId, 'up');

// Dashboard query:
feedback.getAggregate({ template: 'summarize' });
// → { totalRatings: 145, positive: 123, positiveRate: 0.848, ... }

Distinct from the mechanical eval primitives:

• scoreResponse (v2.4.0) — mechanical rubric scoring
• llmJudge (v1.84.0) — LLM-as-judge scoring
• promptExperiment (v2.20.0) — A/B test with automatic scoring
• userFeedbackAggregator (v2.34.0) — HUMAN feedback

Highlights:

• Three rating kinds: 'binary' (thumbs), 'scale' (1-N stars), 'custom' (any number + user-supplied positivityOf).
• Per-dimension aggregation — getAggregate(filter?) for filtered stats; snapshotByDimension('model') for pivot-table dashboards.
• Rolling window (default 30 days) with lazy pruning. O(1) memory over long-running deployments.
• Validation — invalid ratings + unknown IDs rejected with { accepted: false, reason } status object; never throws.
• Multiple ratings per response supported (thumbs vote + written comment via meta).
• MCP resource: config://user-feedback.

Compose with promptVersionPin (v2.30.0) — the pin annotates ctx.meta.promptVersion with { name, version, source }; use those as dimensionsOf fields to attribute feedback per version. Roll out v4, see v4's positive rate lagging v3's, roll back via pinFor.

Cost overrun predictor (new in v2.33.0)

costOverrunPredictor tracks spend against a fixed calendar billing window (typically end-of-month) and projects end-of-window spend based on the current burn rate. Warns rising-edge when the projected spend would exceed targetUsd. Distinct from costForecast (v2.1.0) — that projects a rolling window; this projects to a fixed calendar boundary.

const { costOverrunPredictor, startOfMonth, endOfMonth } = require('@saptarishi/cds-plugin-llm');

llm.use(costOverrunPredictor({
  windowStart: () => startOfMonth(new Date()),
  windowEnd:   () => endOfMonth(new Date()),
  targetUsd:   1000,
  costOf:      (ctx, result) => {
    const inTok  = result.usage?.input_tokens  ?? 0;
    const outTok = result.usage?.output_tokens ?? 0;
    return (inTok / 1e6) * 0.15 + (outTok / 1e6) * 0.60;
  },
  warnAtRatio: 0.85,
  onWarn:      (i) => cds.log('llm:cost').warn('overrun-projected', i),
  onExhausted: (i) => cds.log('llm:cost').error('overrun-imminent', i),
}));

Fills the cost stack:

• costBudget (v1.x) — GLOBAL cap (hard block)
• quotaManager (v2.23.0) — PER-USER cap
• costForecast (v2.1.0) — ROLLING-window projection
• costOverrunPredictor (v2.33.0) — CALENDAR-window projection
• costAwareRouter (v2.10.0) — cheap-first with quality escalation

Highlights:

• Linear projection — projected = (spent / elapsed) × fullWindow. Simple + effective for steady-state workloads.
• Rising-edge warn at warnAtRatio × targetUsd (default 0.85). Separate rising-edge exhausted at targetUsd so alerting distinguishes "on track to overspend" from "definitely overspending."
• Auto window rollover — when windowStart() returns a new timestamp, spend + rising-edge state reset automatically. No cron jobs needed.
• minSampleSize gate (default 20) — prevents projection noise during the first minutes of a window.
• projection() returns full live state for dashboards + ops UIs.
• Six calendar helpers exported: startOfMonth, endOfMonth, startOfDay, endOfDay, startOfQuarter, endOfQuarter.
• MCP resource: config://cost-overrun-predictor.

Compose with quotaManager (v2.23.0) — predictor gives early warning ("you'll blow the monthly budget"); quota gives hard enforcement ("this user hit their cap"). Both signals feed the same alerting pipeline.

Response signing (new in v2.32.0)

responseSigning HMAC-signs the LLM response before delivering to the caller. Downstream consumers (SAP Event Mesh, message queues, service mesh hops) can verify authenticity + integrity offline with the shared secret. Complements requestSigning (v2.21.0):

• requestSigning — signs OUTBOUND requests + emits per-call receipts (audit trail)
• responseSigning — signs the RETURNED response so it can be verified wherever it later travels

const { responseSigning, verifyResponseSignature } = require('@saptarishi/cds-plugin-llm');

llm.use(responseSigning({
  secret:    process.env.LLM_RESPONSE_HMAC_KEY,
  algorithm: 'sha256',
}));

// Downstream (possibly in a different process/service):
const { valid, reason } = verifyResponseSignature(result, secret);
// { valid: true, reason: null }   or   { valid: false, reason: 'hash-mismatch' }

The signature envelope attaches as result.signature = { hash, sig, algorithm, ts }. Consumers who don't care about signing just ignore the field; it doesn't interfere with result.text / result.data / result.toolCalls.

Highlights:

• Deterministic — canonical serialization uses a stable-key-order JSON stringifier so identical results always produce identical hashes across restarts + processes.
• Standalone verifier — verifyResponseSignature(result, secret, options?) usable in any downstream service without the middleware runtime.
• Fail-safe — signing errors are captured via onError; the underlying response is still returned unsigned. Non-object results (plain strings) skip signing without error.
• Three algorithms: sha256 / sha384 / sha512. Frozen list: RESPONSE_SIGNING_ALGORITHMS.
• MCP resource: config://response-signing.

Composition rule: place responseSigning at the outermost layer so the signed result is what actually returns to the caller (post any repair, cache, hedge). Any middleware wrapping around it would either re-sign or invalidate the signature.

Provider health aggregate (new in v2.31.0)

providerHealthAggregate combines multiple health signals per provider into a unified score — routing decisions can consult one metric instead of stitching together five. Instruments every call, records latency + success/error per provider, computes a rolling error rate + p95 latency, emits onDegraded / onRecovered transitions with hysteresis via minSampleSize.

const { providerHealthAggregate } = require('@saptarishi/cds-plugin-llm');

const health = providerHealthAggregate({
  providerOf:          (ctx, result) => result?.model?.split('/')[0] ?? 'unknown',
  windowMs:            60_000,
  errorRateThreshold:  0.10,           // >10% errors → degraded
  latencyP95Threshold: 15_000,          // >15s p95 → degraded
  minSampleSize:       10,             // don't judge until enough data
  onDegraded:  (i) => cds.log('llm:health').error('degraded', i),
  onRecovered: (i) => cds.log('llm:health').info('recovered', i),
});
llm.use(health);

// Later, in routing logic:
const s = health.getHealth('openai');
// → { healthy: true, score: 0.92, errorRate: 0.02, latencyP95Ms: 3200, samples: 145, ... }

Complements the four other health primitives:

• providerHealthProbe (v1.62.0) — PROACTIVE liveness pings
• circuitBreaker (v1.x) — REACTIVE per-error trip
• adaptiveBulkhead (v1.61.0) — latency-driven concurrency tuner
• providerHealthAggregate (v2.31.0) — UNIFIED score across all signals

Highlights:

• Weighted composite (default: 60% error rate + 40% p95 latency; optional 20% breaker-state weight when breakerFor is wired). Score in [0, 1] where 1 = fully healthy.
• Sample-size gate prevents cold-start false-degrades — verdict is not computed until minSampleSize samples accumulate.
• Transition callbacks fire on state changes only (rising / falling edges), not per-call — perfect for alerting.
• Rolling window with lazy pruning — O(1) memory per provider over long-running processes.
• snapshotAll() / listProviders() for dashboards.
• MCP resource: config://provider-health-aggregate.

Composition rule: put providerHealthAggregate outside providers so it observes every call. It's a passive instrumentation layer — never modifies the request path. Use getHealth() in your chatWithFallback / regionFailover selector to skip degraded providers proactively.

Prompt version pinning (new in v2.30.0)

promptVersionPin is version selection for prompt templates: rollback-friendly for canary deployments where 5% of traffic pins to v3 while 95% runs v4, or per-session rollback for a user who reported a bad response on the new version. Ships with a lightweight PromptVersionRegistry class; bring-your-own resolver if you want to back the registry with Git commit SHAs, a database, or gitPromptRegistry (v2.1.0).

const { promptVersionPin, PromptVersionRegistry } = require('@saptarishi/cds-plugin-llm');

const registry = new PromptVersionRegistry();
registry.register('summarize', 1, (vars) => ({
  messages: [{ role: 'user', content: `Summarize (v1): ${vars.text}` }],
}));
registry.register('summarize', 2, (vars) => ({
  messages: [{ role: 'user', content: `Summarize concisely (v2): ${vars.text}` }],
}));

llm.use(promptVersionPin({
  resolveTemplate: (name, version) => registry.getVersion(name, version),
  latestVersionOf: (name) => registry.latestVersion(name),
  templateRefOf:   (ctx) => ctx.request.templateRef,
  pinFor:          (ctx, name) => sessionPins[ctx.request.sessionId]?.[name],
  applyTemplate:   (req, tpl, ctx) => ({ ...req, ...tpl(req.vars ?? {}) }),
  onPin: (i) => cds.log('llm:pin').info(i),
}));

// Callers set:
llm.chat({ templateRef: { name: 'summarize' }, vars: { text: '...' } });

Resolution priority (highest first):

1. pinFor(ctx, name) — per-request pin (canary / rollback)
2. templateRef.version — explicit caller-supplied version
3. latestVersionOf(name) — falls back to registry's latest

Audit metadata written to ctx.meta.promptVersion = { name, version, source } — downstream logs, MCP resources, and dashboards see exactly which template version was used per call.

Highlights:

• Bring-your-own registry — the middleware just needs resolveTemplate + latestVersionOf functions. Wire in gitPromptRegistry (v2.1.0) with commit SHAs as versions, a database, whatever.
• Fail-open — missing templates or unknown versions fall through to next() without modifying the request. Never blocks the request path.
• Per-template usage counts (stats.byTemplate.summarize = { '1': 12, '2': 45 }) + pinRate() for canary-progress monitoring.
• MCP resource: config://prompt-version-pin.

Compose with promptExperiment (v2.20.0) — experiment picks the variant (name), pin locks a version of that variant. Rollback while A/B testing.

Response revision loop (new in v2.29.0)

responseRevision is a quality-driven revision loop. When the response scores below threshold, this middleware automatically re-asks the same model with the low-scoring response + rubric feedback inline, up to maxRevisions times. Returns the best-scoring response across all attempts even if none pass the threshold.

const { responseRevision, scoreResponse } = require('@saptarishi/cds-plugin-llm');

llm.use(responseRevision({
  scorer: (result) => {
    const report = scoreResponse(result, { rubric: myRubric });
    return {
      score: report.score,
      feedback: report.results
        .filter((r) => !r.pass)
        .map((r) => `- ${r.check}: ${r.reason}`)
        .join('\n'),
    };
  },
  scoreThreshold: 0.8,
  maxRevisions:   2,
  onRevision: (i) => cds.log('llm:revise').info('revision', i),
}));

Distinct from the three other quality primitives:

• structuredOutputRepair (v2.9.0) — SCHEMA-driven repair
• costAwareRouter (v2.10.0) — escalates to a DIFFERENT model
• consensusVoting (v2.5.0) — fans out to N models in parallel
• responseRevision (v2.29.0) — iterates on the SAME model with rubric feedback

Highlights:

• Flexible scorer: return number in [0, 1] OR { score, feedback }. Feedback (if present) is injected into the revision prompt.
• Best-effort fallback: even when all revisions fail, returns the highest-scoring attempt (not the last). Prevents the "kept re-asking and got progressively worse" tail.
• Fail-safe scorer: exceptions or non-numeric returns treated as unscorable → return the first result unmodified. Never blocks the request path.
• avgRevisions() and passRate() for real-world tuning of maxRevisions / scoreThreshold.
• Default applyRevision appends previous assistant reply + user revision request — matches the standard multi-turn revision pattern most models understand.
• MCP resource: config://response-revision.

Composition rule: place responseRevision outside responseCache / semanticCache (revisions shouldn't be cached mid-loop). Compose with costAwareRouter (v2.10.0) underneath — revise on the cheap model first; if still low, escalate to premium as the fallback path.

Grace period / soft deadline (new in v2.28.0)

gracePeriod complements the shipped deadline (v1.x hard timeout) with a soft deadline that fires a warning callback while the request keeps running. Useful for SLO monitoring: catch tail-latency spikes early, before they become full timeouts.

const { gracePeriod } = require('@saptarishi/cds-plugin-llm');

llm.use(gracePeriod({
  softMs: 5000,          // warn at 5s
  hardMs: 15000,          // kill at 15s
  onSoftDeadline: (i) => cds.log('llm:slo').warn('slow', i),
  onHardDeadline: (i) => cds.log('llm:slo').error('killed', i),
}));

Three modes:

• softMs alone → warning only, no kill (pure SLO monitoring)
• hardMs alone → equivalent to the shipped deadline
• softMs + hardMs → warn at soft, kill at hard (SLO + safety net)

Highlights:

• AbortSignal integration — fresh AbortController per call on ctx.signal (opt-out via attachAbortSignal: false). Aborts on hard deadline so cancellation-aware providers early-exit instead of continuing to burn tokens after the timeout fires. Original ctx.signal restored after the call.
• GracePeriodExhaustedError (code GRACE_PERIOD_EXHAUSTED) with .elapsedMs, .hardMs, .softMs.
• Buckets — completedUnderSoft vs completedOverSoft for real-world SLO tuning. softDeadlineRate() returns the fraction of calls that crossed the soft line.
• MCP resource: config://grace-period.

Compose with speculativeHedge (v2.12.0) — use onSoftDeadline as a signal to fire a late hedge ("primary is probably slow, start a backup"). Compose with promMetrics — feed onSoftDeadline into a histogram bucket for real-time p95/p99 tracking.

Batch aggregator (new in v2.27.0)

batchAggregator is window-based pooling of concurrent LLM calls. Waits up to batchWindowMs for a batch to fill (or hits maxBatchSize), fires one upstream call, then splits the response back to individual callers. Real cost saver for high-throughput fan-out patterns with uniform structure (classification, embedding, translation) where N similar prompts can be concatenated into 1 request and the response is an ordered array.

const { batchAggregator } = require('@saptarishi/cds-plugin-llm');

llm.use(batchAggregator({
  batchWindowMs: 50,
  maxBatchSize:  10,
  batchable:     (ctx) => ctx.request.batchKey === 'classify-support-ticket',

  // User owns the concatenation logic:
  aggregateRequests: (batch) => ({
    messages: [{ role: 'user', content:
      'Classify each ticket (respond as a JSON array of labels):\n' +
      batch.map((b, i) => `${i+1}. ${b.ctx.request.messages[0].content}`).join('\n')
    }],
    format: { type: 'array', items: { enum: ['billing', 'bug', 'question'] } },
  }),
  // User owns the split logic:
  splitResponse: (result, batch) => {
    const labels = result.data;
    return batch.map((b, i) => ({ ...result, data: labels[i], text: labels[i] }));
  },

  onBatch: (i) => cds.log('llm:batch').info('flushed', i),
}));

Distinct from other batching primitives:

• requestCoalescer (v2.8.0) — dedups IDENTICAL requests
• runBatch / waitForBatch (v1.79.0) — offline batch-API workflows (24-hour turnaround)
• batchAggregator (v2.27.0) — ONLINE pooling of DIFFERENT requests across a short window

Highlights:

• batchable(ctx) predicate opts calls in — non-uniform prompts pass straight through.
• batchKeyOf(ctx) partitions pending batches (e.g., per model, per user, per tenant) — members with different keys batch independently.
• Fail-loud contract — if splitResponse returns a wrong-count or non-array, all batch members fail with BatchAggregationError (code BATCH_AGGREGATION_FAILED) so bugs surface immediately.
• pendingCount(), pendingKeys(), avgBatchSize() for tuning.
• MCP resource: config://batch-aggregator.

Composition rule: put batchAggregator outside semanticCache / responseCache / requestCoalescer (batching only happens on cache misses) and inside bulkhead (one batch = one slot).

Client-side rate limiter (new in v2.26.0)

clientSideRateLimit is a proactive N-per-window throttler. Blocks/queues LLM calls to stay under a configured rate rather than letting the provider 429 you and relying on after-the-fact recovery. Closes the reactive-vs-proactive rate-limit story:

• retryOnRateLimit (v1.x) — REACTS to 429 by waiting + retrying
• adaptiveRateLimit (v2.6.0) — REACTS to headers by shrinking bulkhead
• clientSideRateLimit (v2.26.0) — PROACTIVELY shapes traffic to never hit the cap

const { clientSideRateLimit } = require('@saptarishi/cds-plugin-llm');

llm.use(clientSideRateLimit({
  strategy:       'token-bucket',
  rate:           10,          // requests per second
  burst:          20,          // bucket capacity
  keyOf:          (ctx) => ctx.request.userId ?? 'global',
  queueTimeoutMs: 30_000,
  onTimeout:      (i) => cds.log('llm:rate').warn('timeout', i),
}));

Two strategies:

• 'token-bucket' — classic Nginx-style: refill at rate per second up to burst capacity; consume 1 token per call. Best for bursty traffic where short spikes are OK.
• 'sliding-window' — strict N-per-windowMs count. Best when you have a documented provider quota like "100 requests/min" and want exact enforcement.

Highlights:

• Per-key limits via keyOf(ctx) — separate token buckets/windows per tenant, per model, per user. Null/empty key falls to the 'global' bucket.
• FIFO queue with timeout — over-limit calls wait up to queueTimeoutMs; then reject with RateLimitTimeoutError (code RATE_LIMIT_QUEUE_TIMEOUT) carrying .rateLimitKey, .waitedMs, .queueTimeoutMs.
• Per-key drain scheduler ticks only when there are waiters — idle keys consume zero CPU.
• snapshotKeys() returns per-key state (tokens or timestamp count, queue depth) for real-time dashboards. avgWaitMs() for tuning burst/window/timeout.
• MCP resource: config://client-rate-limit.

Composition rule: put clientSideRateLimit outside providers so throttling happens before any network work. Compose with retryOnRateLimit (inside) as defense in depth — client-side shapes traffic; retry recovers if the provider's actual quota drifted lower than expected.

Multimodal attachment router (new in v2.25.0)

multimodalRouter inspects the request's message content blocks, detects which media types are present (text, vision, pdf, audio, video), and routes to the model configured for that capability set. Fills the "how do I automatically pick the right model for a vision request vs. a text request?" gap that the four other routers don't cover.

const { multimodalRouter } = require('@saptarishi/cds-plugin-llm');

llm.use(multimodalRouter({
  routes: {
    'text':          { model: 'openai/gpt-4o-mini' },
    'vision':        { model: 'openai/gpt-4o' },
    'text+vision':   { model: 'openai/gpt-4o' },
    'pdf':           { model: 'anthropic/claude-opus-4-7' },
    'audio':         { model: 'openai/whisper-1' },
    'pdf+vision':    { model: 'anthropic/claude-opus-4-7' },
  },
  fallbackKey: 'text',
  onRoute:    (i) => cds.log('llm:mm').info('routed', i.key),
  onFallback: (i) => cds.log('llm:mm').warn('no route for', i.detectedKey),
}));

Detection recognizes both Anthropic and OpenAI shapes:

• { type: 'image' } / { type: 'image_url' } / { type: 'image_base64' } → vision
• { type: 'pdf' } / { type: 'document', mimeType: 'application/pdf' } → pdf
• { type: 'audio' } / { type: 'input_audio' } → audio
• { type: 'video' } → video
• string content or { type: 'text' } blocks → text
• MIME sniffing on document/file blocks (image/* → vision, audio/* → audio, etc.)

Complements the four other routers:

• modelRouter — static keyword rules
• semanticRouter — embedding-based routing
• costAwareRouter — quality-driven escalation
• multimodalRouter — capability-aware routing

Compose them: multimodal picks the capability tier upfront; cost-aware escalates within that tier if the response scores low.

Highlights:

• Canonical sorted-set keys ('pdf+vision', not 'vision+pdf'). Middleware validates + rejects unsorted keys at construction so config mistakes fail loud.
• Respects capability matrix — routes are just config; if you route vision to a text-only model, capabilities(llm) (v2.3.0) flags the mismatch and preflight refuses to boot.
• Fail-open on detection error — a throwing detectAttachments falls through unmodified.
• MCP resource: config://multimodal-router (route table + distribution + full stats).

Streaming chunk aggregator (new in v2.24.0)

streamAggregator buffers per-token stream chunks and emits aggregated chunks either when the buffer reaches minChars (default 20) OR when it's been idle for maxIdleMs (default 100ms). Reduces UI cursor jitter when the provider emits per-character chunks — a common quirk of some OpenAI-compat endpoints and local models via Ollama.

const { streamAggregator, streamThrottle } = require('@saptarishi/cds-plugin-llm');

llm.use(streamThrottle({ maxTokensPerSecond: 40 }));   // outer — paces cadence
llm.use(streamAggregator({
  minChars:  20,      // flush when buffer >= 20 chars
  maxIdleMs: 100,     // OR after 100ms idle
  onFlush:   (i) => cds.log('llm:agg').debug('flush', i),
}));

Distinct from the shipped streamThrottle (v1.97.0):

• streamThrottle — PACES existing chunks with a delay (throttle bursts)
• streamAggregator — COMBINES chunks (fewer to render)

Use both together: aggregator produces fewer chunks; throttle paces them at a smooth cursor speed.

Highlights:

• Promise.race between source chunk and idle timer — flushes as soon as EITHER condition fires. Timer is unref()'d so it doesn't hold the event loop.
• Terminal chunk detection — done, isDone, type: 'done', finish_reason, stopReason all recognized as end-of-stream markers. Buffer flushed first, then terminal chunk passes through untouched so the wrapStreamCompletion (v1.72.0) tracker fires with the correct doneChunk.
• Non-text chunks pass through — tool call deltas, metadata chunks, etc. bypass the buffer. Ordering preserved: buffer flushes BEFORE the non-text chunk emits.
• Preserves stream completion — hasStreamCompletion still detects the wrapped stream; .onComplete(), .completedInfo, .isCompleted all re-exposed.
• Custom shapes — extractText(chunk) for provider-specific chunk formats (e.g., OpenAI's delta.content); makeChunk(text) to change the emitted-chunk shape.
• reductionRatio() returns 1 - (emitted / source). Higher = more coalescing happening — instrumented for tuning minChars / maxIdleMs.
• MCP resource: config://stream-aggregator.

Per-user quota manager (new in v2.23.0)

quotaManager is per-user (or per-tenant) USD quota with sliding-window tracking, rising-edge warning thresholds, and hard cap with configurable grace period. Real ask for SaaS deployments that need to bill or throttle by usage. Blocks calls before any real work with QuotaExhaustedError, so an over-quota user doesn't consume a slot or a retry budget.

const { quotaManager, inMemoryQuotaStore } = require('@saptarishi/cds-plugin-llm');

llm.use(quotaManager({
  keyOf:  (ctx) => ctx.request.userId ?? 'anon',
  store:  inMemoryQuotaStore({ maxKeys: 10_000 }),
  costOf: (ctx, result) => {
    // Compute USD from result.usage using your provider pricing.
    const inputCost  = (result.usage?.input_tokens  ?? 0) * 0.15 / 1e6;
    const outputCost = (result.usage?.output_tokens ?? 0) * 0.60 / 1e6;
    return inputCost + outputCost;
  },
  quotas: {
    default:      { limitUsd: 10 },      // most users
    'user-vip':   { limitUsd: 1000 },     // VIP tier
    'user-power': { limitUsd: 100 },
  },
  windowMs: 30 * 24 * 3600_000,           // 30-day rolling window
  warnThresholds: [0.5, 0.8, 0.95],
  gracePeriodRatio: 0.02,                 // allow 2% overshoot
  onWarn:      (i) => cds.log('llm:quota').warn('threshold', i),
  onExhausted: (i) => cds.log('llm:quota').error('exhausted', i),
}));

Complements the shipped cost primitives — pick the right layer for the job:

• costBudget — GLOBAL cap across all users
• costGuard — per-call cap (single-call safety net)
• costForecast (v2.1.0) — burn-rate projection (early warning)
• costAwareRouter (v2.10.0) — quality-driven tier escalation
• quotaManager (v2.23.0) — per-user fairness enforcement

Highlights:

• Pluggable store — three async methods (get, add, optional setLastWarnLevel / reset). Ships with inMemoryQuotaStore (LRU eviction) as zero-infra default; swap for Redis/DynamoDB/HANA for persistence across restarts.
• Grace period (default 2%) — prevents "you were charged $10.01 for a$10 quota" complaints while keeping the cap meaningful. Set to 0 for a strict cap.
• Rising-edge warnings — each threshold fires once as usage crosses it upward. Reset naturally as samples age out of the sliding window.
• getUsage(key) returns { usageUsd, limitUsd, utilization, samplesInWindow } for real-time dashboards and admin UIs.
• resetKey(key) manually clears one user's usage — useful for ops overrides (compensation, quota rebase).
• Fail-open — a broken store never takes the request path down.
• MCP resource: config://quota-manager.

Composition rule: put quotaManager outside bulkhead / retryOnRateLimit / providers — the quota check should short-circuit BEFORE any real work. Compose with fairShareScheduler (v2.14.0) for the full multi-tenant story: fair-share enforces per-tenant concurrency, quota manager enforces per-tenant spend.

Auto tool chain (new in v2.22.0)

autoToolChain is the cascading tool-call loop that every agentic workflow needs. When the model returns tool_calls, this middleware automatically runs the tools via registered handlers, feeds tool results back as tool messages, re-invokes the chain, and loops until the model returns a final answer — or hits a safety cap. Frees callers from writing the tool-invocation-loop boilerplate for every agentic workflow.

const { autoToolChain, functionCallArbitrator } = require('@saptarishi/cds-plugin-llm');

// Compose with functionCallArbitrator (v2.18.0) for allowlist + schema safety:
llm.use(functionCallArbitrator({ tools: [
  { name: 'lookup_customer', input_schema: {...} },
  { name: 'send_email',      input_schema: {...} },
], onInvalid: 'strip' }));

llm.use(autoToolChain({
  handlers: {
    lookup_customer: async (input) => ({ id: input.id, name: 'Alice', email: 'alice@x.com' }),
    send_email:      async (input) => ({ sent: true, to: input.to }),
  },
  maxDepth:          10,          // safety cap
  detectCycles:      true,        // reject same tool + same input twice
  handleUnknownTool: 'throw',     // 'throw' | 'skip' | 'error-back'
  onToolCall:        (i) => cds.log('llm:chain').info('tool', i),
  onChainComplete:   (i) => cds.log('llm:chain').info('done at depth', i.depth),
}));

Highlights:

• maxDepth (default 10) bounds chain depth. Exceeding throws MaxDepthExceededError (code MAX_TOOL_DEPTH_EXCEEDED).
• Cycle detection (default on) — stable-JSON hash of (toolName, input) catches the model calling the same tool with same input twice in a row. Throws ToolChainCycleError (code TOOL_CHAIN_CYCLE).
• Three-way policy on unknown tools: 'throw' (default UnknownToolError), 'skip' (drop), 'error-back' (send {error: "unknown tool"} to model so it can recover).
• Handler errors sent back to model — a handler that throws emits an {error: message} tool result instead of propagating. Matches the OpenAI / Anthropic pattern for agentic loops.
• Correct multi-turn construction — assistant tool_use turn appended before tool results, one {role: 'tool', tool_call_id, content} message per call. Handles the message-stack accumulation across hops automatically.
• avgChainDepth() + maxObservedDepth — instrumented for real-world tuning.
• MCP resource: config://auto-tool-chain.

Composition rule: put functionCallArbitrator (v2.18.0) outside this middleware — the arbitrator validates + allowlists tool calls; this middleware runs them. Together: only vetted + schema-validated calls trigger a handler. Place outside structuredOutputRepair (v2.9.0) so repair operates on the FINAL response, not intermediate tool-request responses.

Request signing / HMAC receipts (new in v2.21.0)

requestSigning cryptographically signs each outbound LLM call with HMAC-SHA256/384/512 and emits a hash-chained receipt on the response. Offline auditors can later verify with a shared secret that request/response pairs weren't tampered with between capture and audit. Strong compliance differentiator for SAP financial / healthcare / regulated workloads.

const { requestSigning, verifyReceiptChain } = require('@saptarishi/cds-plugin-llm');

const receipts = [];
llm.use(requestSigning({
  secret:     process.env.LLM_HMAC_KEY,   // Buffer or hex string
  algorithm:  'sha256',                    // 'sha256' | 'sha384' | 'sha512'
  onReceipt:  (r) => receipts.push(r),     // persist to HANA/S3/etc
  includeRequestBody: false,               // opt-in — receipts stay small
}));

// Later, offline:
const { valid, brokenIdx, reason } = verifyReceiptChain(receipts, secret);
// { valid: true, brokenIdx: -1, reason: null }

Complements the shipped sensitiveDataAudit (v1.96.0):

• sensitiveDataAudit — immutable hash-chained audit LOGS (what happened)
• requestSigning — cryptographically-signed RECEIPTS keyed to a secret only the auditor can verify (non-repudiation)

Together they give the full compliance story.

Highlights:

• Chain continues across errors — failed calls emit receipts with isError: true. Compliance requires audit of attempts, not just successes.
• Detects tampering + reordering — each receipt is signed over (requestHash + responseHash + prevReceiptHash), and prevReceiptHash is the SHA hash of the prior receipt's sig. Tampering with any field breaks the chain from that point forward; reordering breaks it too.
• Canonical serialization excludes volatile fields (credentials, retries config) so the same logical call produces the same signature across restarts. Users can supply custom canonicalizeRequest / canonicalizeResponse if needed.
• chainCursor() returns { index, prevSigHash } — useful for live observability + persisting chain state across restarts (survives reset()).
• MCP resource: config://request-signing.

Composition rule: place requestSigning outside providerLoadBalancer (v2.17.0) — the signature should be over the canonical request, not the credential-applied one (the canonical serializer already excludes credentials so this is defensive).

Prompt A/B experiment framework (new in v2.20.0)

promptExperiment runs N variants of a prompt against each other in production traffic, captures per-variant score + latency + cost distributions, and declares a statistically-defensible winner via 95% CI overlap check. Complements the shipped offline eval primitives (llmJudge, promptRegression, scoreResponse, consensusVoting) with the online production-traffic half of the eval story.

const { promptExperiment, scoreResponse } = require('@saptarishi/cds-plugin-llm');

llm.use(promptExperiment({
  name: 'invoice-summarization-2026Q3',
  variants: [
    { name: 'control', weight: 5 },
    { name: 'terse',   weight: 3, apply: (req) => ({ ...req, system: 'Be terse. Bullets only.' }) },
    { name: 'chatty',  weight: 2, apply: (req) => ({ ...req, system: 'Be friendly + detailed.' }) },
  ],
  splitKeyOf:    (ctx) => ctx.request.userId ?? ctx.request.sessionId,
  scorer:        (result) => scoreResponse(result, { rubric: myRubric }).score,
  costEstimator: (result) => (result.usage?.output_tokens ?? 0) * 0.00001,
  onWinner: (i) => cds.log('llm:exp').warn('winner!', i),
}));

// Later, check the current winner:
const w = experiment.getWinner();
// { winner: 'terse', status: 'confident',
//   top: 'terse',   topScore: 0.87, topCI: [0.82, 0.92],
//   runnerUp: 'control', runnerUpScore: 0.71, runnerUpCI: [0.66, 0.76],
//   variants: [...] }

Highlights:

• Weighted consistent-hash assignment — same splitKey always maps to the same variant, so users experience the same variant across restarts, in-flight requests, and session reloads. Distinct experiments assign independently (a user in "control" of Experiment A may be in "v2" of Experiment B).
• Welford's online algorithm for score / latency / cost mean + variance. O(1) memory regardless of sample count — safe to run indefinitely without accumulating storage.
• 95% CI winner detection with three status values:
  • 'insufficient-samples' — fewer than minSampleSize (default 30) per variant
  • 'inconclusive-overlap' — CIs of top two variants overlap; difference is within noise
  • 'confident' — top CI doesn't overlap runner-up; statistically defensible
• Multiple experiments compose — just declare each with a different name and stack them in the chain. Each has its own MCP resource (config://prompt-experiment/${name}).
• Standalone helpers hash32(str), welfordUpdate, welfordStats, ciAroundMean exported for custom analytics outside the middleware.

Composition rule: put promptExperiment outside semanticCache/responseCache (variants must not share cache entries; otherwise treatment leaks across arms). Use the session ID as splitKeyOf when composed with sessionContextStore so the same session always sees the same variant across turns (avoids mixed-treatment noise within a conversation).

Session context store (new in v2.19.0)

sessionContextStore is multi-turn memory management for chat apps. On each call it prepends per-session message history to the request, appends the new user + assistant turns to the store after the response, and prunes to a sliding window (oldest-drop or synthetic-summary replacement). Solves the "how do I remember what the user said 3 turns ago without blowing the context budget" problem without hand-rolling per-app state.

const { sessionContextStore, inMemorySessionStore } = require('@saptarishi/cds-plugin-llm');

llm.use(sessionContextStore({
  sessionOf:   (ctx) => ctx.request.sessionId,
  store:       inMemorySessionStore({ maxSessions: 10_000, ttlMs: 3600_000 }),
  maxMessages: 20,
  // Optional: summarize dropped messages instead of just dropping them.
  pruneStrategy: 'summarize',
  summarizer: async (dropped) => (await llm.chat({
    messages: [{ role: 'user', content: 'Summarize briefly:\n' + JSON.stringify(dropped) }],
  })).text,
  onSessionHit:  (i) => cds.log('llm:session').debug('hit',  i),
  onSessionMiss: (i) => cds.log('llm:session').debug('miss', i),
}));

Highlights:

• Pluggable store — three async methods (get, put, append) + optional delete + size. Ships with inMemorySessionStore (LRU eviction, optional TTL) as a zero-infra default; swap for pgvector/Redis/DynamoDB in prod using the same interface.
• System-prompt preservation — a leading system message on the caller's request stays at position 0; prior turns are spliced in AFTER, so history doesn't override session instructions.
• Tool-call turns persisted — an assistant response with toolCalls[] is stored with those toolCalls attached, so the next turn's context includes the tool-call trace.
• Fail-open — a broken store never takes the request path down. Store errors are captured via onError and the middleware falls through without prepending history.
• Compose with compactHistory (v1.91.0) as the summarizer when using pruneStrategy: 'summarize' — LLM-summarized context compression without hand-rolled logic.
• MCP resource: config://session-context-store (hitRate, turns-appended / prunes / summarizations / store-errors).

Composition rule: put sessionContextStore outside guardrails / promptInjectionGuard (those need to see the merged history — prior turns can contain injected content too) and outside semanticCache / responseCache (a stateful chat is context-dependent and shouldn't share cache entries with other sessions).

Function-call arbitrator (new in v2.18.0)

functionCallArbitrator is a policy + validation layer for LLM tool calls. Each shipped provider already normalizes tool schemas to { name, description?, input_schema } on outbound and tool-call responses to { id, name, input } on inbound. This middleware sits on top to enforce cross-provider consistency + safety.

const { functionCallArbitrator } = require('@saptarishi/cds-plugin-llm');

llm.use(functionCallArbitrator({
  tools: [
    {
      name: 'lookup_customer',
      description: 'Look up a customer by ID',
      input_schema: {
        type: 'object',
        required: ['customerId'],
        properties: { customerId: { type: 'string' } },
      },
    },
    // ... more tools
  ],
  onInvalid:         'strip',       // 'throw' | 'strip' | 'log'
  allowUnregistered:  false,
  onCall:        (i) => cds.log('llm:tools').info('called', i.call.name),
  onInvalidCall: (i) => cds.log('llm:tools').warn('rejected', i),
}));

What it enforces:

• Registered-tool allowlist — tool calls to names not in the registry are rejected by default. Prevents the model from calling tools you never intended to expose.
• Per-call JSON Schema validation — every result.toolCalls[].input is validated against the tool's input_schema using the shipped validateBuiltIn from structuredOutputValidator. Catches missing required fields, wrong types, unexpected props.
• Three-way policy on invalid calls:
  • 'throw' → raise InvalidToolCallError (code: 'INVALID_TOOL_CALL') with .callId, .name, .errors[]
  • 'strip' → silently drop the invalid call from result.toolCalls (default — fail-soft)
  • 'log' → keep the call but tag it with .invalid = true + .invalidErrors[] for downstream inspection

Cross-provider tool shape normalization on outbound (opt-out via normalizeOutbound: false) accepts Anthropic (input_schema), OpenAI (parameters), aliases (schema), and Gemini (functionDeclarations wrapper) forms — all normalized to the canonical input_schema shape before the provider sees the request.

Standalone helpers normalizeToolShape(tool) and normalizeToolList(tools) are exported for one-off use outside the middleware chain (offline shape migration, tests, config validation).

MCP resource: config://function-call-arbitrator (registered tool names + policy + stats + invalidRate()).

Placement rule: put functionCallArbitrator inside prompt-safety middleware (guardrails, promptInjectionGuard) so those still guard the raw text; outside the tool runner (runTools, streamTools, Agent) so the runner only ever sees validated + allowlisted calls.

Provider load balancer (new in v2.17.0)

providerLoadBalancer rotates across N credential sets of the same provider kind — multiple OpenAI accounts to work around per-account rate limits, multiple Azure regions for geographic spread, multiple Bedrock cross-account roles, etc. Complements the shipped 11-provider abstraction (which supports one credential set per provider kind) with per-account rotation on top.

const { providerLoadBalancer } = require('@saptarishi/cds-plugin-llm');

llm.use(providerLoadBalancer({
  credentials: [
    { name: 'openai-org-a', apiKey: process.env.OPENAI_KEY_A, weight: 3 },
    { name: 'openai-org-b', apiKey: process.env.OPENAI_KEY_B, weight: 1 },
    { name: 'openai-org-c', apiKey: process.env.OPENAI_KEY_C, weight: 1 },
  ],
  strategy: 'least-loaded',                                       // round-robin | least-loaded | weighted-random | sticky
  applyCredential: (req, cred) => ({ ...req, credentials: { apiKey: cred.apiKey } }),
  unhealthyThreshold:  3,                                         // 3 consecutive failures → mark unhealthy
  unhealthyCooldownMs: 30_000,
  onHealthChange: (i) => cds.log('llm:lb').warn('health', i),
}));

Four strategies:

• round-robin — cycle through credentials in order (default)
• least-loaded — pick the credential with the fewest in-flight calls
• weighted-random — probability ∝ weight (higher weight = more picks)
• sticky — same stickyKeyOf(ctx) value always picks the same credential (session affinity, tenant isolation across accounts)

Health tracking is opt-in via unhealthyThreshold. After N consecutive failures, a credential is marked unhealthy and skipped by the strategy until unhealthyCooldownMs elapses. A successful call resets the consecutive-error counter. Composes with providerHealthProbe (v1.62.0) — that primitive tests liveness proactively; this one reacts to real failures.

• snapshotCredentials() returns per-credential state (weight, inFlight, totalPicks, totalErrors, healthy, unhealthySince) for real-time SaaS dashboards.
• markUnhealthy(name) / markHealthy(name) for ops overrides.
• AllCredentialsUnhealthyError (code ALL_CREDENTIALS_UNHEALTHY) when the strategy has no healthy credential to pick — fail loud rather than silently proceed with no credential applied.
• MCP resource: config://provider-load-balancer.

Placement rule: put providerLoadBalancer outside providers, inside routing (semanticRouter v2.16.0, costAwareRouter v2.10.0). The routing decides which model to use; this middleware decides which credential to use.

Semantic router (new in v2.16.0)

semanticRouter picks the model + system prompt + params by embedding the user's request and matching it against per-route centroids (cosine similarity). Each route is defined by a bucket of example prompts; the middleware averages their embeddings to form a centroid, then classifies incoming queries by nearest neighbor.

const { semanticRouter } = require('@saptarishi/cds-plugin-llm');

llm.use(semanticRouter({
  embedder: async (text) => (await llm.embed({ input: text })).embeddings[0],
  routes: [
    {
      name:     'code',
      model:    'anthropic/claude-opus-4-7',
      system:   'You are an expert engineer. Return runnable code.',
      examples: ['Write a Python function to reverse a string', 'Debug this TS error', 'Refactor this Go handler'],
    },
    {
      name:     'procurement',
      model:    'openai/gpt-4o',
      system:   'You are an SAP procurement analyst.',
      examples: ['Analyze this vendor quote', 'Draft a PO for 500 units', 'What are the payment terms?'],
    },
    {
      name:        'chit-chat',
      model:       'openai/gpt-4o-mini',
      temperature: 0.9,
      examples:    ['Hi', 'How are you?', 'Tell me a joke'],
    },
  ],
  threshold:    0.75,
  defaultRoute: 'chit-chat',
  onRoute: (i) => cds.log('llm:router').info('routed', i),
}));

Complements the two other routing primitives:

• modelRouter (1.x) — static keyword/predicate routing. Deterministic, easy to reason about, but requires you to enumerate rules.
• costAwareRouter (v2.10.0) — reactive escalation: cheap tier first, escalate on low score.
• semanticRouter (v2.16.0) — predictive first pick: classify by meaning, get the right tier from the start.

Compose them: semantic router picks the tier upfront; cost-aware router escalates when the prediction was wrong.

Highlights:

• Lazy centroids — construction is zero-latency. First call computes example embeddings and caches them. warmup() pre-computes eagerly if you want deterministic startup.
• Precomputed centroids — ship centroid: [1, 0, 0, ...] on a route to skip embedding entirely. Useful for offline-computed centroids that stay stable across deploys.
• Fail-open — embedder or extractKey errors fall through to next() unmodified. The router never takes the request path down.
• Threshold + fallback — below-threshold queries use defaultRoute (or passthrough if null). Prevents forcing a bad route onto ambiguous queries.
• MCP resource config://semantic-router exposes route table (names, models, example counts, centroid readiness), threshold, default route, and route distribution.

Retry budget (new in v2.15.0)

retryBudget is a Google-SRE-style global cap on the retries-to-requests ratio in a rolling window. It prevents retry storms during partial outages: with retryOnRateLimit + regionFailover + speculativeHedge + costAwareRouter's escalate-on-error combined, one call can trigger up to N×M×K upstream attempts. If 10k concurrent requests each retry 5x, you fire 50k requests at an already-failing upstream — a self-inflicted DoS. The retry budget refuses further retries when the ratio breach is real.

const { retryBudget, retryOnRateLimit } = require('@saptarishi/cds-plugin-llm');

const budget = retryBudget({
  retryRatio:    0.10,     // SRE-canonical: max 10% retries
  windowMs:      60_000,
  minSampleSize: 100,      // don't trip on cold-start noise
  onExhausted: (i) => cds.log('llm:budget').warn('exhausted', i),
  onLowBudget: (i) => cds.log('llm:budget').info('near-cap', i),
});

llm.use(retryOnRateLimit());   // outer — retries re-enter budget
llm.use(budget);               // inner — sees each retry

How retries are detected: a WeakMap tracks per-ctx call counts. The first pass for a given ctx is a request; every subsequent pass with the same ctx reference is a retry. Works because retry primitives (retryOnRateLimit, autoRetry, regionFailover, costAwareRouter's escalation) re-invoke next() with the same ctx. Middlewares that create fresh ctx per attempt (speculativeHedge) look like separate requests — correct, because each hedge IS a separate upstream call.

Highlights:

• SRE-canonical default of 10% retries matches the Google SRE book.
• minSampleSize gate (default 100) prevents tripping on cold-start noise. Ratio check is skipped until enough requests accumulate.
• Rising-edge callbacks at 50% and 80% of the retry budget (configurable). Level is only re-fired after ratio drops below the lowest threshold — no spam.
• RetryBudgetExhaustedError (code RETRY_BUDGET_EXHAUSTED) with .currentRatio, .retryRatio, .requests, .retries, .windowMs.
• MCP resource config://retry-budget exposes live counts, current ratio, budget fraction, per-threshold fire counts.

Composition rule: place retryBudget inside the retry primitives (outer: retry; inner: budget) so retries re-enter the budget — that's how it observes them. Test-drive the whole stack with chaosInjector (v2.11.0) to verify budget behavior under simulated outages before hitting prod.

Multi-tenant fair-share scheduler (new in v2.14.0)

fairShareScheduler sits above the shipped concurrency primitives and admits requests in weighted round-robin (WRR) order across tenants. No single tenant can starve the others under load — solves the noisy-neighbor problem in SaaS deployments of the plugin.

const { fairShareScheduler } = require('@saptarishi/cds-plugin-llm');

llm.use(fairShareScheduler({
  tenantOf:          (ctx) => ctx.request.tenantId ?? 'anon',
  maxConcurrent:     20,
  weights:           { gold: 5, silver: 2, free: 1 },
  defaultWeight:     1,
  maxPerTenantQueue: 100,
  onReject: (i) => cds.log('llm:fair-share').warn('backpressure', i),
}));

Distinct from the shipped primitives:

• tenantIsolate just TAGS requests for observability. fairShareScheduler schedules.
• bulkhead caps concurrency globally without fairness. fairShareScheduler cap + share fairly.
• distributedLock is per-key exclusivity across instances. fairShareScheduler is per-tenant queueing within an instance.

Highlights:

• WRR: weights: { gold: 3, silver: 1 } gives gold 3× more slots per cycle. When all tenants with pending work run out of credit, credits refill from weight — but only for tenants with pending work, so idle tenants don't accumulate unfair credit debt.
• Backpressure: per-tenant queue depth cap (maxPerTenantQueue). Exceeding throws FairShareRejectedError (code FAIR_SHARE_QUEUE_FULL) with .tenant/.queueDepth/.queueLimit. One over-subscribed tenant cannot block others — separate per-tenant queues.
• Live snapshot: snapshotTenants() returns per-tenant weight, credits, active, queued, admitted, rejected. Great for real-time SaaS dashboards. MCP resource: config://fair-share-scheduler.

Composition rule: place fairShareScheduler outside the shipped bulkhead (the scheduler manages its own ceiling; wrapping bulkhead underneath would double-count slots). Place retryOnRateLimit outside the scheduler so per-tenant retries consume per-tenant queue slots (respect fairness); placing it inside means retries bypass fairness — appropriate only for short internal loops.

Reversible PII tokenization (new in v2.13.0)

reversibleTokenization strips PII from the outbound request (replaces with opaque <TYPE_N> tokens like <EMAIL_1>, <SSN_1>), sends the tokenized prompt to the model, then restores the original values in the response before returning. The model never sees raw customer data but callers get useful answers back. Ideal for GDPR / HIPAA / financial compliance workloads where sending PII to an external LLM is legally or contractually restricted.

const { reversibleTokenization } = require('@saptarishi/cds-plugin-llm');

llm.use(reversibleTokenization({
  patterns: {                              // extends the built-in set
    BADGE_ID: /\bE\d{6}\b/g,
  },
  onTokenize: (i) => cds.log('llm:pii').info('scrubbed', i.byType),
  onUnknownToken: () => '[REDACTED]',     // rewrite hallucinated tokens
}));

// Then:
const r = await llm.chat({ prompt: 'Send invoice to alice@x.com about SSN 123-45-6789.' });
// Model saw:  'Send invoice to <EMAIL_1> about SSN <SSN_1>.'
// Model said: 'I sent the invoice to <EMAIL_1> regarding <SSN_1>.'
// Caller gets:'I sent the invoice to alice@x.com regarding 123-45-6789.'

Built-in patterns (low false-positive taxonomy): EMAIL, CREDIT_CARD, SSN, PHONE, IPV4, IBAN. PERSON / ADDRESS are intentionally NOT included — regex-based name detection is too noisy; compose with a proper NER model instead.

Highlights:

• Dedup per request — same value → same token everywhere. alice@x.com in both system and messages[] becomes the same <EMAIL_1>. Reduces token cost and makes downstream detokenization exact.
• Hallucinated tokens — if the model returns <EMAIL_99> (no such token was ever emitted), it's left as-is by default; override via onUnknownToken(token) => replacement to redact.
• Standalone helpers — tokenizePII(text) and detokenizePII(text, mapping) exported for one-off scrubbing (offline scripts, audit-log preparation, unit tests of pattern sets).
• MCP resource config://reversible-tokenization exposes pattern types, restorationRate, tokens-created / restored / unknown, per-type counts.

Composition rule: place reversibleTokenization inside guardrails / promptInjectionGuard (those need raw text) and outside any cache (semanticCache, responseCache) so tokenized prompts become the cache keys and PII never leaks into shared caches.

Speculative hedging (new in v2.12.0)

speculativeHedge fires the same request to N candidates with staggered delays; the first successful reply wins, losers are signalled to abort. Trades $ for tail latency — pay for a small amount of duplicate work to guarantee p99 when your SLO is p99 (not p50).

const { speculativeHedge } = require('@saptarishi/cds-plugin-llm');

llm.use(speculativeHedge({
  candidates: [
    { name: 'primary' /* fires immediately */ },
    { name: 'us-east', hedgeDelayMs: 200, modifyRequest: (r) => ({ ...r, model: 'us-east/gpt-4o' }) },
    { name: 'eu-west', hedgeDelayMs: 500, modifyRequest: (r) => ({ ...r, model: 'eu-west/gpt-4o' }) },
  ],
  isSuccess: (r) => !!r?.text,
  onWin:  (i) => cds.log('llm:hedge').info('win', i),
  onLoss: (i) => cds.log('llm:hedge').info('lost', i),
}));

Highlights:

• Per-candidate config — each entry can carry its own hedgeDelayMs + modifyRequest. Common patterns: swap model, route to different regions, or hit different provider endpoints.
• Loser cancellation via Web-standard AbortSignal on ctx.signal. Providers that respect it early-exit; providers that don't keep running but their result is discarded (token cost may still be billed — choose hedgeDelayMs conservatively).
• isSuccess(result) — treat a returned result as a losing hedge (still racing). Useful for "keep waiting for a better one" against low-confidence responses.
• Cost visibility — hedgeRatio() = hedgesLaunched / totalCalls; per-candidate winsByCandidate + launchesByCandidate counters identify the reliable candidate. MCP resource: config://speculative-hedge.
• AllHedgesFailedError (code: 'ALL_HEDGES_FAILED') — thrown when every hedge errors or returns non-success. Carries .errors[] and .candidateNames[].

Placement rule: put speculativeHedge outside any per-attempt retry / bulkhead / breaker. Each hedge should consume its own retry / concurrency budget — otherwise a shared retry counter is drained by hedges the winner is racing against.

Chaos injector (new in v2.11.0 — test-only)

chaosInjector gives you a way to verify that the shipped resilience stack — retryOnRateLimit, circuitBreaker, bulkhead, adaptiveRateLimit, requestCoalescer, costAwareRouter — actually behaves correctly when the provider misbehaves. Given a seed, it reproduces the exact same fault sequence, so a red CI run can be re-run locally byte-identical.

const { chaosInjector, retryOnRateLimit, circuitBreaker } = require('@saptarishi/cds-plugin-llm');

llm.use(retryOnRateLimit());
llm.use(circuitBreaker());
// Injector sits INNERMOST — closest to the provider — so all outer
// resilience middleware sees the fault and can react.
llm.use(chaosInjector({
  seed: 42,
  faults: {
    rate429:     0.30,
    rateTimeout: 0.10,
    rateGarbage: 0.05,
  },
}));

Fault types (stable priority order — networkError → timeout → 500 → 503 → 429 → garbage → slow; first roll that hits wins):

• rateNetworkError → ECONNRESET
• rateTimeout → ETIMEDOUT
• rate500 / rate503 / rate429 → HTTP status errors with .status/.statusCode set
• rateGarbage → calls next() but replaces result.text with malformed content (empty string, truncated JSON, HTML error pages) so downstream parsers/validators see a real broken response
• rateSlow → adds a configurable delay before calling next()

Safety guard: chaosInjector refuses to construct unless one of NODE_ENV=test, CHAOS_INJECTOR_ENABLED=1, or iKnowThisIsChaos: true is present. That way an accidental prod deploy of a fault-injecting middleware throws loudly at boot instead of silently degrading requests. MCP resource: config://chaos-injector (exposes seed + rates + lastFault + lastSeedRoll for debugging non-reproducible failures).

Cost-aware model router (new in v2.10.0)

costAwareRouter tries a cheap-tier model first; if the response scores below your quality threshold, it escalates to the premium tier. Ties together three previously-shipped primitives — modelRouter (static routing), scoreResponse (mechanical scoring), and costForecast (budgeting) — into an economic win pattern: most calls run cheap; only the hard ones escalate.

const { costAwareRouter, scoreResponse } = require('@saptarishi/cds-plugin-llm');

llm.use(costAwareRouter({
  tiers: [
    { model: 'gpt-4o-mini', pricePerMtokIn: 0.15, pricePerMtokOut: 0.60 },
    { model: 'gpt-4o',      pricePerMtokIn: 2.50, pricePerMtokOut: 10.00 },
  ],
  scorer:         (result) => scoreResponse(result, { rubric: myRubric }).score,
  scoreThreshold: 0.75,
  onEscalate: (i) => cds.log('llm:router').info('escalate', i),
  onFinal:    (i) => cds.log('llm:router').info('final',    i),
}));

Highlights:

• Multi-tier: 2, 3, or N tiers. maxEscalations caps how far the router will escalate.
• escalateOnError (default true): a downstream throw on the cheap tier promotes to the next tier instead of surfacing the error — this is a quality-driven re-attempt on a different model, distinct from retry (transport-error re-attempt on the same one).
• Fail-safe scorer: exceptions or non-numeric returns count as failing scores. Safer to escalate than to fail loudly.
• Cost accounting: when tier pricing is set, tokensSpentUsd / tokensSavedUsd / savingsRatio() are computed from the shipped result.usage. Live-visible via config://cost-aware-router.

Placement rule: put costAwareRouter outside any middleware that reads request.model (retry, bulkhead, providers). The router mutates request.model per attempt so downstream sees each tier as a fresh call.

Structured-output auto-repair (new in v2.9.0)

structuredOutputRepair upgrades the "response failed schema validation" story from fail (v1.x structuredOutputValidator in throw mode) or fail-then-re-ask (validator with onInvalid: 'retry') to a strategy chain: cheap deterministic local fixes are tried first, and the LLM is only re-asked when local repair fails. Most fixable outputs cost zero extra tokens.

const { structuredOutputRepair, schemas } = require('@saptarishi/cds-plugin-llm');

llm.use(structuredOutputRepair({
  schemaFrom: (ctx) => ctx.request.format,
  strategies: ['json-fix', 're-ask'],   // order matters
  maxLlmRetries: 1,
  onRepair: (info) => cds.log('llm:repair').info(info),
}));

Built-in strategies:

• json-fix — strips ```json fences, narrows to the outer {...} block, removes // and /* */ comments (respecting quoted regions), removes trailing commas, quotes unquoted keys, converts single-quoted strings to double-quoted. Deterministic, idempotent for already-valid JSON.
• re-ask — sends a corrective user message with the schema violations inline, bounded by maxLlmRetries. Retry-result is propagated so callers see the corrected text/data, not just the parsed object.

Custom strategies plug in as { name, apply(rawText, { errors, schema, parsed }) }. Throwing custom strategies are skipped silently and the chain continues. All strategies share the same StructuredOutputInvalidError on exhaustion → unified downstream error handling with the 1.x validator. MCP resource: config://structured-output-repair (strategy list + per-strategy hit counts + LLM retries used).

Standalone jsonAutoFix(text) is also exported for one-off cleanup outside a middleware chain (useful in offline scripts).

Request coalescer (new in v2.8.0)

requestCoalescer is the in-flight sibling of semanticCache. When N concurrent requests arrive with the same key (same prompt + model + system by default), only ONE upstream call fires — all N callers await the same shared Promise and receive the same result. Fixes the classic cache-stampede on cold caches: the first miss no longer triggers N parallel misses racing to populate.

const { requestCoalescer, semanticCache } = require('@saptarishi/cds-plugin-llm');

llm.use(requestCoalescer({
  ttlMs: 500,               // also coalesce briefly-following requests
  onCoalesce: (i) => cds.log('llm:coalesce').info(i),
}));
llm.use(semanticCache({ embedder, store }));

• Streaming methods are skipped by default (a consumed stream can't be fanned out). Overridable via skipMethods.
• cloneResult: true deep-clones the shared result per caller if downstream code mutates it.
• keyPrefix namespaces coalescing across tenants; maxInFlightKeys caps the map size as a safety valve during pathological fan-out.
• MCP resource config://request-coalescer exposes savingsRatio, in-flight + recently-settled counts, peakInFlight, and lead/coalesce/ttl-hit counters.

Composition rule: wrap requestCoalescer outside semanticCache/responseCache to absorb the burst before it hits the cache, and outside bulkhead/retry — one shared upstream call should only consume one slot and one retry budget, not N.

Adaptive rate-limit tuner (new in v2.6.0)

adaptiveRateLimit watches the rate-limit headers the provider sends back on every call (parsed by the shipped OpenAI / Anthropic / Gemini / Bedrock providers into result._rateLimit) and dynamically adjusts the paired bulkhead()'s maxConcurrent to stay within remaining quota. On a 429 or 503 it immediately halves concurrency; on a healthy sample it grows back toward (remainingRatio − headroom) × maxConcurrent.

const { bulkhead, adaptiveRateLimit } = require('@saptarishi/cds-plugin-llm');

const bh = bulkhead({ maxConcurrent: 20 });
llm.use(bh);
llm.use(adaptiveRateLimit({
  bulkhead:      bh,
  headroom:      0.20,       // stay 20% below remaining quota
  minConcurrent: 2,
  maxConcurrent: 50,
  onAdjust: (i) => cds.log('llm:arl').info(i),
  on429:    (i) => cds.log('llm:arl').warn('429', i),
}));

Composes with adaptiveBulkhead (1.61) — that one is latency-driven (AIMD on p95), this one is quota-driven (headroom against remaining requests/tokens). Both can call setMaxConcurrent on the same bulkhead; they naturally settle on the tighter of the two signals. MCP resource: config://adaptive-rate-limit.

Response scoring + consensus voting (new in v2.4.0 · v2.5.0)

Two composable primitives for output quality:

• scoreResponse (v2.4.0) — checks-based grader with a frozen CHECKS catalog (no-hallucinated-numbers, min-length, contains, matches, no-refusal, ...). Returns a score in [0, 1] plus per-check details. Wire it as a soft signal for logging + dashboards, or as a hard gate via shouldCache: (_c, r) => scoreResponse(r).score >= 0.8 in semanticCache.
• consensusVoting (v2.5.0) — fan out to N models in parallel; return the majority winner with configurable quorum, comparators (equalityIgnoreCase, jsonDeepEqual, numeric, firstToken, ...), and confidence. Useful for compliance-sensitive answers where a single-model reply is not defensible.

Cost forecasting (new in v2.1.0)

costForecast projects your rolling burn rate against a monthly budget, raising warn and critical events before you hit the wall (unlike costBudget, which fires when spend has already crossed a threshold). Straight-to-critical transitions correctly fire the warn callback first. MCP resource: config://cost-forecast.

Provider capability probe (new in v2.3.0)

capabilities(llm) returns a live capability report for the configured provider + model — what actually works, not what a hand-maintained matrix claims. Complements the static Provider capability matrix below with runtime evidence: what tools work, whether structured outputs land, whether the model actually streams, prompt-cache support, and vision / PDF handling.

const { capabilities } = require('@saptarishi/cds-plugin-llm');

const report = await capabilities(llm, { live: false });
// {
//   provider: 'llm-anthropic', model: 'claude-sonnet-4-6',
//   chat: true, stream: true, tools: true, structuredOutputs: true,
//   embed: false, promptCache: true, vision: true, pdf: true, ...
// }

Pass { live: true } to actually round-trip a probe request per capability (~5s) — useful for CI verification against a real endpoint. PROVIDER_CAPABILITY_MATRIX + MODEL_CAPABILITY_OVERRIDES exports let you consume the static matrix directly for gating in code.

Full API

llm.chat({
  messages: [{ role: 'user' | 'assistant' | 'tool', content: string | ContentBlock[], toolCalls?, tool_call_id? }],
  system?: string,
  model?: string,          // overrides configured default
  maxTokens?: number,      // default 16000 (lower for Groq free tier — TPM: 12k)
  format?: JSONSchema,     // unified structured output; returns parsed data
  tools?: Tool[],          // [{ name, description, input_schema }]
  thinking?: { type: 'adaptive' } | false,  // Anthropic-only; default adaptive
  cache?: boolean,         // Anthropic-only; caches the system prompt
  retries?: { max, baseMs, maxMs },
}) => Promise<{
  text: string,
  data?: unknown,          // populated when format was set
  toolCalls?: [{ id, name, input }],  // populated when model called tools
  raw: unknown,            // provider-native response
  usage: { input_tokens, output_tokens, ... },
  stopReason: string,      // 'end_turn' | 'tool_use' | 'max_tokens' | ...
  model: string,
}>

llm.embed({ input: string | string[], model? })
  => Promise<{ embeddings: number[][], model: string }>

Provider capability matrix

	llm-anthropic	llm-ollama	llm-groq	llm-openai-compatible	llm-genai-hub
chat	✓	✓	✓	✓	✓
stream	✓ (SDK)	✓ (NDJSON)	✓ (SSE)	✓ (SSE)	✓ (SSE)
structured output (format)	✓ (output_config)	✓ (native format)	✓ (json_object mode)	✓ (json_object)	✓ (json_object)
tool use (tools)	✓ (native)	✓ (qwen2.5, llama3.1+)	✓ (function-calling models)	✓	✓
vision (images)	✓ (Claude 3.5+)	✓ (llava, moondream, llama3.2-vision)	✓ (llama-4-scout, etc.)	✓ (gpt-4o, etc.)	✓ (deployment-dependent)
PDF (documents)	✓ (Claude 3.5+ native)	— (render pages to images first)	— (Groq doesn't accept files)	✓ (base64, on models that support file blocks — GPT-4o+)	✓ (deployment-dependent)
embeddings	— (no first-party embeddings)	✓ (mxbai-embed-large, etc.)	✓ (when model available)	✓ (text-embedding-3-*, ada-002, etc.)	✓ (needs embeddingDeploymentId)
response cache (opt-in)	✓	✓	✓	✓	✓
prompt caching (cache)	✓ (Anthropic system prompt ephemeral)	—	—	—	—
adaptive thinking (thinking)	✓ (Opus 4.7 native)	—	—	—	—

FAQ

How does this relate to @cap-js/ai? @cap-js/ai is scoped to value-help recommendations and SAP-RPT-1 with SAP AI Core integration. This plugin is a general-purpose LLM client with multi-provider support and a broader feature surface (streaming, tool use, vision, structured output). The two can coexist: one CAP app can cds.connect.to('ai') for value-help features and cds.connect.to('llm') for direct LLM calls.

Can I use this without SAP BTP? Yes. Only the llm-genai-hub kind requires BTP (specifically AI Core). The other four kinds (Anthropic, Ollama, Groq, OpenAI-compatible) work in any Node.js environment. This is deliberate — the plugin is useful for CAP apps that don't run on BTP, and useful for prototyping before a BTP deployment.

Is this production-ready? As of 1.0.0: the public API surface is committed under semver (breaking changes require a major bump). The core is functional and unit-tested (64 tests + wire-protocol E2E). The plugin is deployed live on SAP BTP Cloud Foundry through this repo's joule-project-api. The one remaining honesty note: the GenAI Hub provider is spec-compliant and mock-verified end-to-end but not yet live-verified against a real AI Core extended deployment — happy to accept a PR from anyone with access. Everything else has been live-verified against its target service.

How do I add a new provider? Extend LLMService (or OpenAICompatibleLLMService if the target speaks the OpenAI shape), implement _chat (required), plus _stream and _embed if applicable. Register a kind in your package.json under cds.requires.kinds.<my-provider> with impl pointing at the new class file and external: true.

Which model do you recommend for common tasks?

• Structured extraction / classification: any 7B+ instruction-tuned model. Groq's llama-3.3-70b-versatile is a good default (fast + free tier).
• Vision (invoice OCR, chart reading): Claude Opus 4.7 or GPT-4o for accuracy; Groq's meta-llama/llama-4-scout-17b-16e-instruct or Ollama's llava for local/cheap.
• Long-context summarization: Claude Opus 4.7 (1M context) or a GenAI Hub deployment of the same.
• Tool use / agentic loops: Claude 3.5+ or qwen2.5 on Ollama for multi-step reliability. Groq's llama models work for single-tool cases.
• Embeddings: Ollama with mxbai-embed-large or nomic-embed-text.

Why not just use @anthropic-ai/sdk or openai directly? Three reasons: (1) CAP idiom — cds.connect.to('llm') is more natural in a CAP handler than importing an SDK class. (2) Provider swap without code change — flip a config value from llm-groq to llm-anthropic and the same handler works. (3) Unified interface for tools + structured output + streaming across all providers, so you don't rewrite the message-translation code five times.

How do I do RAG (retrieval-augmented generation)? The plugin gives you embed() — combine with a vector store to complete the loop. Companion package @saptarishi/cds-plugin-vector-hana provides a SAP HANA Cloud vector store (native REAL_VECTOR + COSINE_SIMILARITY) with a SQLite fallback for local dev, so you can build semantic-search features without HANA access and swap backends at deploy time.

What happens if the underlying provider's API changes? Each provider adapter is a thin file (~150 lines). Provider API changes are localized to one file. The plugin's public surface (chat, stream, embed) is stable across provider changes.

Contributing

PRs and issues welcome. The repo has the plugin as cds-plugin-llm/. Standard workflow:

git clone https://github.com/kalyanjanumpally/sap-joule-ai-plugin
cd sap-joule-ai-plugin/cds-plugin-llm
npm install
npm test              # 39 unit tests, no external deps
npm run typecheck     # TypeScript definition check
node ../scripts/verify-genai-hub.js   # E2E mock verification

CI runs the same checks on every push (Node 20 + 22 matrix).

Highest-value contributions right now:

• Live-verification of the GenAI Hub provider against a real AI Core extended deployment
• Embeddings support for OpenAI-compatible providers
• Additional structured-output modes (JSON schema strict on models that support it)

Roadmap

• 0.7: embeddings on OpenAI-compat / Groq ✓ shipped in v0.7.0
• 0.8: PDF content blocks (Anthropic native; other providers explicit-reject) ✓ shipped in v0.8.0
• 0.9: OpenAI-compat inline PDF (via file content-block) + GenAI Hub embeddings + response caching layer ✓ shipped in v0.9.0
• 1.0: API stability commitment ✓ shipped in v1.0.0 (live verification of GenAI Hub open — see FAQ)
• 1.1: runTools() — automatic multi-turn tool-use loop ✓ shipped in v1.1.0
• 1.2: middleware / interceptor pattern ✓ shipped in v1.2.0
• 1.3: built-in rateLimit + otel middlewares; vector store upsertMany ✓ shipped in v1.3.0 (llm) / v0.2.0 (vector-hana)
• 1.4: redisRateLimit middleware; HANA HNSW index config ✓ shipped in v1.4.0 (llm) / v0.3.0 (vector-hana)
• 1.5: saptarishi-llm CLI ✓ shipped in v1.5.0
• 1.6: saptarishi-llm init CAP-app scaffolder ✓ shipped in v1.6.0
• 1.7: saptarishi-llm mcp server (MCP over stdio) ✓ shipped in v1.7.0
• 1.8: PromptRegistry + MCP resources + MCP prompts ✓ shipped in v1.8.0
• 1.9: loadFromDir prompt loader + --prompts-dir on MCP + scaffold SSE endpoint + MCP resource templates ✓ shipped in v1.9.0
• 1.10: MCP HTTP+SSE transport (--http) + hot-reload prompts (--watch-prompts) ✓ shipped in v1.10.0
• 1.11: MCP bearer-token auth (--auth-token) ✓ shipped in v1.11.0
• 1.12: MCP progress notifications on tools/call ✓ shipped in v1.12.0
• 1.13: MCP list-changed notifications (broadcasts on hot-reload) ✓ shipped in v1.13.0
• 1.14: OpenAI Files API for URL PDFs (uploadPdfFromUrl) ✓ shipped in v1.14.0
• 1.15: Azure OpenAI provider ✓ shipped in v1.15.0
• 1.16: MCP OAuth2 / JWKS-based JWT auth + pluggable authTokenVerifier ✓ shipped in v1.16.0
• 1.17: MCP resource subscriptions (resources/subscribe + notifications/resources/updated) ✓ shipped in v1.17.0
• 1.18: per-session provider overrides (--providers-config, initialize._meta.provider, per-call provider arg) ✓ shipped in v1.18.0
• 1.19+: CAP model registry (providers in .cds files), Bedrock + Vertex provider paths, mTLS auth for MCP HTTP, Anthropic prompt caching
• Companion package: @saptarishi/cds-plugin-vector-hana — HANA Cloud vector store + SQLite fallback for RAG

License

Apache-2.0