@saptools/cf-otel
A read-only command line that queries the OpenTelemetry trace spans SAP Cloud LoggingSAP Cloud LoggingAn instance-based observability service built on OpenSearch that stores and analyses logs, metrics and traces from Cloud Foundry, Kyma, Kubernetes and other runtime environments.sap official, btp has already ingested and ranks the spans of a trace by self-time.
Finding where the time in a slow request went means reading spans out of the OpenSearch index behind SAP Cloud LoggingSAP Cloud LoggingAn instance-based observability service built on OpenSearch that stores and analyses logs, metrics and traces from Cloud Foundry, Kyma, Kubernetes and other runtime environments.sap official, btp, otel-v1-apm-span-*. This command line logs in with the cf CLI, finds a dashboards basic-auth credential on the Cloud Logging instance from existing service keys or an app binding made before SAML was switched on, and then answers queries: find locates traces by service, time or attribute, or from the vcapRequestId on a @saptools/cf-logs@saptools/cf-logsA command line and typed Node.js library that fetches and streams the logs of one Cloud Foundry app and parses them into structured rows.independent, btp row, selftime ranks a trace's spans by self-time, and diff compares two traces. It never instruments an app, so it reads only spans already exported to Cloud Logging, and those reach the index a few seconds after the request. Minting a new credential means switching SAML off on the instance for a moment, which it does only behind --allow-mint-credential.
Facts
Where it sits
Who is behind it
In the atlas: maintains 18, contributes to 18@dongitranmaintainer
What it declares
Every artifact
- Type
- CLI
- Origin
- Independently maintained
- Runs on
- Cloud Foundry
As a CLI
- Distributed through
- npm
Connections
Traces, metrics and correlation for a running service
OpenSearch-based store for logs, metrics and traces
Download
Install
npm add @saptools/cf-otelDerived from the package name. No recipe recorded; the README has the configuration.
Architecture
Download
README
quoted from dongitran/saptools under MIT, read@saptools/cf-otel
Query and analyze OpenTelemetry trace spans already ingested into SAP Cloud Logging's
OpenSearch backend (index pattern otel-v1-apm-span-*). This is a read-only, post-hoc
tool: it never instruments a running process and never mutates application data. For live
request/response capture or breakpoint debugging on a running app, see @saptools/cf-live-trace
and @saptools/cf-inspector instead — cf-otel picks up after a trace has already been
exported, to find the real bottleneck in it.
Install
npm install -g @saptools/cf-otel
Updates
Every command first checks npm for a newer @saptools/cf-otel (at most once an hour, one small request
with a 2-second timeout) and, when one exists, installs that exact version with the package manager
that owns the running binary and re-runs the command you typed on the new version. Both steps are
announced on stderr; nothing is printed when the install is already current:
cf-otel: updating 0.2.0 -> 0.3.0 ...
cf-otel: updated to 0.3.0; re-running the command
If the install cannot complete, one stderr line gives the manual command and the command runs on the
installed version; that version is not retried for a day. cf-otel self-update forces the check and
install now; cf-otel self-update --check only reports.
| Control | Effect |
|---|---|
SAPTOOLS_AUTO_UPDATE=on|notify|off | on (default) installs and re-runs; notify prints the manual command once per version; off never checks. Applies to every @saptools CLI. |
CF_OTEL_AUTO_UPDATE | same values, this CLI only; wins over the global variable |
SAPTOOLS_UPDATE_INTERVAL_MINUTES | minutes between checks (default 60; 0 checks on every run) |
SAPTOOLS_NPM_REGISTRY | registry to check and install from (default: npm's configured registry, then npmjs) |
SAPTOOLS_UPDATE_DEBUG=1 | explain on stderr why nothing happened |
The updater switches itself off in CI (CI set), under NODE_ENV=test or NO_UPDATE_NOTIFIER, when
the binary runs from a source checkout, an npm link or an npx cache, and inside the re-run itself.
It never writes to stdout, never asks for input, never uses sudo, and never moves onto a prerelease.
Its state lives in ~/.saptools/updates/.
Auth
Every command reads SAP_EMAIL / SAP_PASSWORD from the environment for the underlying
cf api / cf auth / cf target login — never pass them as flags. There is no separate
login step: each command is a complete, one-shot operation.
export SAP_EMAIL=you@example.com
export SAP_PASSWORD=your-password
Targeting
Pass --region, --org, and --space explicitly, or omit any of them to fall back to the
currently targeted cf target session. Whichever way it resolves, the CLI prints a one-line
notice to stderr naming the resolved target:
cf-otel: target eu10/example-org/space-demo (resolved from ambient 'cf target'; pass --region/--org/--space to pin)
--service <name> is a plain query filter on serviceName — it never targets or connects to
a running app the way cf-inspector/cf-hana do.
Credential discovery
Reaching OpenSearch requires a Cloud Logging dashboards basic-auth credential, which is
harder to get than it sounds once SAML is enabled on the instance's dashboards (a known,
current SAP gap: new bindings and service keys on a SAML-enabled instance stop returning a
username/password, only an endpoint). cf-otel tries, in order:
- Existing service keys on the instance (
--service-instance,--service-key, repeatable). - A pre-existing app binding created before SAML was enabled (
--fallback-binding-app, repeatable) — such a binding keeps its original basic-auth credential forever. - Only behind
--allow-mint-credential: temporarily disable SAML, mint a new key, restore SAML immediately after. This is disruptive (breaks SSO dashboards login for everyone during the window) and is never attempted by default. If the minted key turns out to be unusable, it is deleted again once SAML has been restored, so a retry never leaves a trail ofcf-otel-*keys on the instance; when the deletion itself fails, the error names the key and the exactcf delete-service-keycommand to run.
Both service-key payload shapes are read: the fields nested under credentials, which is what
CF CLI v8 prints, and the flat fields v7 printed. The same applies to cf service-keys, whose
table gained two columns in v8.
Pass --verbose to see exactly which step succeeded and why. If every step fails, the error
names every key and binding that was tried.
Commands
| Command | Purpose |
|---|---|
sample | Dump the N most recent full documents, unfiltered — start here when you know nothing yet. |
mapping | Field-type discovery (keyword vs text) before aggregating on any field. |
find | Locate trace(s) matching a service/name/time/attribute filter, or resolve one request id from a log row. |
top | Outlier hunting across a time range without a starting traceId. |
count | Fast existence/frequency check — the trust-but-verify companion to selftime. |
spans <traceId> | Fetch every span in one trace, paginated past 10000 automatically. |
span <traceId> [spanId] | Fetch one span's full, unfiltered document, by ID or by name/kind. |
fields <traceId> [spanId] | List every flat attribute key on a sample span. |
selftime <traceId> | Rank spans by self-time descending — the core, highest-value command. |
gaps <traceId> <spanId> | Analyze timing gaps between one parent span's direct children. |
detached <traceId> | Find likely detached/orphaned trace continuations in the same window. |
diff <traceIdA> <traceIdB> | Compare two traces' self-time breakdowns before/after a fix. |
result show|list|prune|clear | Inspect results saved via --save. |
Every command supports --format table|json|json-compact|csv (default table); most support
--save, which prints ref=<id> instead of the result and stores it under
~/.saptools/cf-otel/results/<ref>/ for later inspection with cf-otel result show <ref>.
Saved results
A saved result is kept for 7 days, then removed by the next cf-otel command that touches the
store (or immediately by cf-otel result prune). Nothing else expires it and nothing caps how many
accumulate, so result list is worth checking if you save often; result clear removes them all.
Files are written 0600 inside 0700 directories.
Pruning only ever removes a result that has expired, or a ref directory it has verified is empty. One whose manifest cannot be read — a permission
error, a partial write, or a format a newer cf-otel wrote — is deliberately left on disk and
reported by result prune on stderr, so a downgrade or a stale global install cannot destroy saved
results it merely fails to understand. result show says which of those happened rather than
reporting a readable file as missing.
--save checks that the store is writable while it validates the rest of the arguments, so a broken
store costs milliseconds instead of a full credential discovery. If the save fails anyway — a full
disk, or a result past the per-result byte cap — the rows are printed in the requested --format
with a diagnostic on stderr and the exit code is non-zero, rather than a completed query being
thrown away. Scripts of the shape ref=$(cf-otel find --save …) therefore fail loudly instead of
binding a table row as if it were a ref.
Attribute filters
--attr <key><op><value> (repeatable) supports >=, <=, >, <, = and ~ (contains). A bare
key is resolved against span.attributes. and then resource.attributes., so
--attr 'http@status_code=200' finds the real field.
= matches the whole value, in either encoding it may be stored in. An OTel attribute whose
value is an array reaches this index as the JSON array rendered to text — the stored keyword for
an HTTP request header is literally ["aaaaaaaa-bbbb-4ccc-8ddd-eeeeeeeeeeee"], brackets and quotes
included, and the whole span.attributes.http@request@header@* family (46 fields) is like this. =
sends both forms, so you write the plain value and it matches either way. The array form is never
sent at a numeric, date or ip field, where an unparseable extra term would fail the whole search
rather than merely not match.
Two limits worth knowing. ~ compiles to a leading-wildcard scan over every backing index, so
prefer = where it will do. Matching is case-sensitive, and stored ids are lower-case hex — the
--vcap-request-id flag folds case for you, --attr does not. And every attribute field has an ignore_above ceiling that varies by
field (256 to 2048 on a real tenant): a longer value is kept in _source but never indexed, so no
operator can find it. Check a specific field with cf-otel mapping --field <name>.
Joining a log row to a trace
@saptools/cf-logs reports two identifiers on every parsed row, and only one of them can pin a
trace:
| cf-logs field | span attribute | Grain |
|---|---|---|
vcapRequestId | span.attributes.http@request@header@x-vcap-request-id | One trace. Measured 1:1 over 1,504 server spans |
correlationId | …@x-correlationid from a router row, …@x-correlation-id from a JSON application row | One business transaction — measured 11.5 and 29.1 traces per value, and up to 778. Read it from a full cf-logs snapshot --json; compact rows do not carry it |
The two correlation spellings are not synonyms. A router line's x_correlationid reaches the span
as …@x-correlationid, while a value the caller supplied in an X-Correlation-Id header reaches it
as …@x-correlation-id; measured over 551 server spans carrying both, they differed on 69 of them.
Query whichever one the row you started from actually holds, or try both.
Pass the hop id straight through. It needs neither --service nor --since, because the value is
unique across the whole retention window — and the trace it finds often has its root in a
different service than the one that served the request:
cf-logs snapshot --app demo-app --compact --json | jq -r '.rows[] | select(.vcapRequestId) | .vcapRequestId'
cf-otel find --vcap-request-id <id>
cf-otel selftime <traceId>
A log line is queryable before its trace is. Spans reach the index a few seconds behind the request
(measured at 6-10s on a live tenant), so running the lookup the instant a row appears can report
(no rows) for a trace that does exist. Wait and retry before concluding the id is wrong.
These headers are captured at ingress, so they exist only on SPAN_KIND_SERVER spans — find one
with cf-otel span <traceId> --name '*' --kind SPAN_KIND_SERVER. To go the other way, from a span
back to its logs, read the id off that span and pass it to
cf-logs snapshot --app <app> --search <id>. Strip the brackets and quotes first: span prints the
raw stored value, so the id shows as ["<id>"] and pasting that verbatim matches nothing.
Time bounds
find, top, count and sample take --since/--until, each accepting either a relative
duration — 30m, 24h, 7d, units s/m/h/d — or an absolute ISO-8601 timestamp such as
2026-08-28T03:00:00Z. A date alone (2026-08-28), a ±HH:MM offset, and fractional seconds of any
width are all accepted; nanosecond precision is passed through untouched, since startTime is a
date_nanos field.
Anything else is rejected before the CF login runs, so a typo costs milliseconds rather than a full
credential discovery. A bare number is rejected too: --since 24 reads just as easily as a 24h
missing its unit as it does epoch milliseconds, and the two are decades apart. A --since that
resolves later than its --until is also rejected, because an inverted range would otherwise return
zero rows indistinguishably from "no data".
top and count have no default bound, so omitting --since scans the whole retention window —
around 5.4M spans on a busy tenant versus ~11k for the last hour. Pass one.
One limit is not checked locally: startTime is date_nanos, which stores non-negative nanoseconds
since the epoch, so a bound before 1970-01-01 (or at/after roughly 2262-04-11) is rejected by
OpenSearch itself rather than by cf-otel, and currently surfaces as a raw HTTP 400. Its reason field
does say so plainly. Use 1970-01-01 when you mean "everything".
Request timeouts
Each Dashboards console-proxy request gets 60 seconds, overridable with
CF_OTEL_HTTP_TIMEOUT_MS. The ceiling is per request, so a paginated spans fetch of 50 pages is
allowed 50 × the value. A malformed override is ignored rather than fatal. This exists to stop an
endpoint that accepts the connection and never answers; raise it if a genuinely wide aggregation
needs longer.
See .skills/cf-otel/SKILL.md (or the installed ~/.claude/skills/cf-otel/SKILL.md) for the
full command reference with worked examples.
Example
cf-otel selftime 1c870cd78e4e88e89c3bca8ee76867ea --top 5
Root span: cds.spawn - run task (duration: 118.203s)
Clamped spans (children-sum > own duration): 0
NAME COUNT SELF_TOTAL SELF_AVG PCT_OF_ROOT
POST 178 60.56s 340.2ms 51.23%
HEAD 176 40.64s 230.9ms 34.38%
Development
pnpm --filter @saptools/cf-otel lint
pnpm --filter @saptools/cf-otel typecheck
pnpm --filter @saptools/cf-otel test:unit
pnpm --filter @saptools/cf-otel test:e2e
pnpm --filter @saptools/cf-otel build
Unit tests mock the OpenSearch console-proxy and cf CLI layers directly; e2e tests spawn the
built CLI against a fake cf binary and an in-memory fake OpenSearch Dashboards console-proxy
server — no real network or SAP credentials are used in either suite.
History
Fixes 1
Notes
Fixed
--servicenow reads "(required)" in--helpfortop— the flag was already enforced at runtime, but its description never said so.
Removed 1, Features 1, Changed 1, Fixes 2
Notes
Changed
- Dashboards-credential discovery now uses the Cloud Foundry v3 API (matching
@saptools/cf-metrics) instead of parsingcf services/cf service-keys/cf envoutput, and reuses the currentcf targetsession when it already matches instead of always performing an isolated login.
Added
- An on-disk dashboards-credential cache (
CF_OTEL_CREDENTIAL_CACHE=0to opt out), matching@saptools/cf-metrics. Discovery previously cost 10-40s on every command; a cached credential is now reused until it expires or OpenSearch rejects it.
Fixed
saml-toggle.ts's temporary SAML-params directory (holding a full instance params blob, secrets included) is now tracked for cleanup on SIGINT/SIGTERM, closing the leak described in the completeness audit.cfApi/cfAuth/cfTargetSpacenow refuse to run against the user's own ambientcfsession (matching@saptools/cf-metrics's existing guard), closing a latent gap where a programming error could have silently re-pointed the user's realcf target.
Removed
- The pre-migration
cf services/cf service-keys/cf envtable-parsing discovery primitives (cfServices,cfServiceKeys,cfEnv,extractVcapServices,parseServicesTable,parseServiceKeyNames), superseded by the v3 API discovery above.
- Dashboards-credential discovery now uses the Cloud Foundry v3 API (matching
Fixes 10
Notes
Fixed
-
--attron a container field lost the only warning it had. Reporting implicit objects instead of calling them absent — the fix above — also took them out of the "matches no field" notice, so--attr 'resource=x'resolved tomappedType: "object"and emitted a baretermwith no notice at all. Measured live, both atermfilter and atermsaggregation on any of the twelve containers return nothing with zero shard failures, and the new field listing prints every one of them as a row a caller can copy straight into--attr. Containers andnestedparents now say the clause cannot match, before the numeric guard rather than after it. -
resolveAggregatableFieldaccepted three targets that aggregate to silence: a container, anestedparent, and a field inside one. It also chose the.keywordtarget from the first index'sfieldsblock while consensus-checking everything else, so withtextin both indices and.keywordin only one the answer flipped on_mappingkey order — and whichever way it fell, the other index's shards contributed empty buckets with no error. -
The deeper listing surfaced a second silent dead end, so it now names it.
eventsandlinksarenested, which stores their children as separate hidden documents: thirteen listed fields sit inside them,events.attributes.exception@typeamong them — exactly what someone chasing an error would filter on, and exactly what a plaintermfails to match while reporting nothing. A newNESTED_INcolumn names the parent, and--attron such a key now says the filter cannot reach it instead of resolving cleanly and returning an empty result. -
Every implicit-object field was reported as absent. OpenSearch never writes
"type": "object"— it is implied by apropertiesblock — and a missing type was read as "no answer", somapping --field spananswered "was not found in the mapping" for a field present in all fifteen backing indices, and the listing printedunknown, the sentinel reserved for a field this version cannot read. Twelve containers are declared that way on the live span index,span,span.attributes,resourceandresource.attributesamong them.@saptools/cf-metricshas handled this since it hit the same thing; the check was missed when the rest of the lookup was unified in 0.8.0. The same fix removes an order-dependence that answered either "was not found" or "mapped inconsistently (keyword)" — a disagreement naming one type — depending on_mappingkey order. -
--attrtreated "no reliable type" as "not in this attribute bag". A key whose backing indices type it differently fell through to the bare name, which is never a real document field: the filter matched nothing at exit 0, the numeric-type guard was skipped, and the notice told the caller to check a spelling that was already correct. A disagreement now keeps the resolved key, refuses a numeric comparison outright, and says the field exists but is typed inconsistently. -
resolveAggregatableFieldtook the first index's opinion where the rest of the module requires consensus, so for a field mappedtextin one index andkeywordin another the aggregation target flipped between<field>.keywordand<field>on nothing but_mappingkey order — and either way the disagreeing indices' shards returned empty buckets with no error. It also substituted an alias's target for non-text types, added in 0.8.0: OpenSearch resolves an alias in aggregations exactly as in queries (measured: identical buckets and hit counts), so naming the target bought nothing, while a target some index does not map would silently contribute empty buckets. The alias is now kept as written except for the.keywordcase, which genuinely needs the target name and is refused when the indices point the alias at different targets. -
mapping --field <name>.keywordreported "was not found" for a multi-field the module itself hands out:resolveAggregatableFieldreturnsdescription.keyword, and looking that name up failed. Six exist on the live span index (traceState.keyword,derived.*.keyword). -
Hour-24 timestamps, a tenth fractional digit, and offsets past ±18:00 were forwarded verbatim and rejected by the backend after a full credential discovery. Hour 24 is legal end-of-day ISO-8601 and
Date.parseeven rolls it forward, butjava.timeresolvesHOUR_OF_DAYstrictly 0-23. All three boundaries were measured against a live instance and now match@saptools/cf-metrics. -
--since/--untilwritten without a timezone were compared in the operator's local zone while OpenSearch reads them as UTC, which both refused genuinely forward windows and let inverted ones through to return nothing at exit 0. The skew was previously documented here as an accepted trade against rejecting the date-only form; normalizing gives up neither. An inversion below millisecond precision also escaped the check, thoughstartTimeisdate_nanosand stores it. -
mappingwith no--fieldlisted only top-level keys: 32 names for 173 leaf fields on the live index, with all 111span.attributes.*and 15resource.attributes.*fields — the two bags--attrresolves a key against — missing entirely. The command whose stated job is field discovery showed none of the fields the package's main filter flag can use.
-
Changed 1, Fixes 4
Notes
Fixed
-
mapping --fieldon a field alias reportedaliasrather than the type of the field it points at.otel-v1-apm-span-*carries 135 of them (measured, nine per backing index): short names likeapp_namepointing atresource.attributes.sap@cf@app_name. Since the same lookup also feeds--attrresolution,alias— which is not a numeric type — made a perfectly valid comparison like--attr <alias>>=400fail with "is mapped as alias, not a numeric type". The resolved type is now reported, with the target named in a newALIAS_OFcolumn.Measured while fixing this, and worth recording because it contradicts the obvious reading:
aliasis not inTEXTUAL_MAPPING_TYPESeither, so=fell back to a plainterminstead of the array-rendered disjunction — but against a real tenant both encodings returned identical counts, for the alias and for an array-shaped span attribute alike. The disjunction matters for attributes that really are stored array-rendered; an alias onto one still needs the resolved type to reach it. -
resolveAggregatableFieldbuilt<alias>.keyword, a name that does not exist. A field alias registers only its own full name; the target's multi-fields are not reachable through it, and atermsaggregation on an unmapped field returns empty buckets with no error — the exact silent failure that function exists to prevent. It now names the target's sub-field. -
A divergent alias target, and a divergent
ignore_above, are now reported alongside the agreed type rather than instead of it, as(varies)in the relevant column. Withholding the type looked safer and was not: three callers read "no type" as "field absent", and each then does something worse than reporting a type with a caveat — the--attrnumeric guard is skipped entirely (so>=on a keyword becomes a silently lexicographicrange),assertFieldExistsblames the tenant's collector configuration, andmapping --fieldcalls a field present in every index missing. A blankignore_abovewas its own trap: it reads as "no cap", the safe interpretation, while divergence is the hazardous one. -
A field present in every backing index but mapped inconsistently is no longer reported as "was not found in the mapping", in
mapping --fieldor inassertFieldExists; the listing showsambiguousrather thanunknown.
Changed
mappingoutput carries a fourth column,ALIAS_OF, empty for a concrete field.
-
Fixes 1
Notes
Fixed
- The log-to-trace guide named only
…@x-correlation-idfor a cf-logscorrelationId. The two spellings are not synonyms: a router line'sx_correlationidarrives as…@x-correlationid, while a caller-suppliedX-Correlation-Idheader arrives as…@x-correlation-id. Measured over 551 live server spans carrying both, they held different values on 69. Following the old wording with an id taken from a router row would silently match nothing. Both are documented now, with which row type produces which.
- The log-to-trace guide named only
Features 2, Changed 3, Fixes 6
Notes
Fixed
--attr <key>=<value>never matched an array-valued attribute. An OTel attribute whose value is an array reachesotel-v1-apm-span-*as the JSON array rendered to text, so the stored keyword for an HTTP request header is literally["<id>"]— brackets and quotes included — and a plaintermon the bare value matched none of thespan.attributes.http@request@header@*family. The command printed(no rows)and exited 0, indistinguishable from "that value never occurred".=now sends both encodings. Measured against a live tenant, 0 of 60 real request ids matched before and 60 of 60 match now, each resolving to exactly one trace. This is the sequel to the 0.1.1 fix, which resolved the attribute key but left the value encoding wrong. The array form is withheld at numeric, date and ip fields, where an extra unparseable term fails the entire search rather than simply not matching.- A query that failed on only some shards was reported as a complete result.
otel-v1-apm-span-*spans 14 backing indices over 28 shards, and OpenSearch answers a partial failure with HTTP 200, the count in_shards.failed, and whatever the surviving shards found — which this client read as the whole answer. The same applies to a search OpenSearch markstimed_out. Every command now fails loudly instead of returning a short result at exit 0. SetCF_OTEL_ALLOW_PARTIAL_SHARDS=1to accept a partial answer when a shard is persistently down. - A field's mapping type was taken from whichever backing index reported it first, even though the
query runs against all of them. Dynamic mapping can give one path different types in different
indices after an ingest change, and the
=encoding above depends on that type being right — a wrong answer there would send a term that some shards reject. The type is now reported only when every index holding the field agrees, and treated as unknown otherwise. --attron a key that matches no field in the index printed an empty result with no explanation. It now says so on stderr. Top-level fields (status.code,kind,serviceName) also resolve their real mapping type now instead of none._mappingwas re-fetched once per attribute-prefix candidate and again per command. It is now fetched at most once per client, which is the whole lifetime of one command. Measured: four--attrfilters went from five HTTP round trips to one.- The
--attrhelp text's example,http@status_code>=400, names a keyword-mapped field, so copying it verbatim was rejected as a numeric comparison against text.
Added
find --vcap-request-id <id>andcount --vcap-request-id <id>resolve one Cloud Foundry gorouter request id — what@saptools/cf-logs0.8.0 reports asParsedLogRow.vcapRequestId— to its trace. Measured 1:1 with a trace over 1,504 server spans. A hex id is trimmed and lower-cased, because keyword matching is exact and every stored id sampled was lower case. If the tenant's collector does not export request headers the command says so by name rather than returning nothing.AttrFiltergainsmappedType, the resolved OpenSearch mapping type, andresolveAndValidateAttrFiltersandVCAP_REQUEST_ID_FIELDare now exported, so a library consumer gets the same=behaviour as the CLI rather than the pre-0.7.0 broken path.
Changed
find --format json-compactnow emits the trace id column instead of silently falling back to full JSON. It returns one entry per matching span, so a trace with several matching spans appears more than once; pipe throughsort -uwhen feeding it to another command.- Commands can now fail where they previously returned a short result at exit 0. That is the point of the shard and timeout checks above, but it is a behaviour change for any script that treated exit 0 as "the query ran".
find --serviceis no longer required. A request id is unique on its own, and the trace it finds frequently has its root in a different service than the one that served the request, so demanding a service both blocked the lookup and pointed the investigation the wrong way. As withcount, an unfilteredfindnow reaches across the whole retention window; pass--since. (topstill requires--service, because its aggregation is not meaningful unscoped.)
Changed 1, Fixes 6, Other 3
Notes
Fixed
result prune's retained notice said the sessions' "manifest could not be read or recognized", butretainedalso counts a ref directory whose manifest is absent and whose emptiness could not be established. It now says this version could not read them, which covers all three cases. Same correction applied to thePruneOutcomedoc comment, and tocf-metricsandcf-hana, so the three CLIs report this identically.- A unit test asserting that
result listomits an expired session prune could not delete passed for the wrong reason: the injected clock also reached the implicit prune, which deleted the fixture, so the empty result came from the absent-directory path rather than from the list-side expiry filter. The test now holds the results directory read-only, and fails if that filter is removed. - Pruning no longer deletes a saved result whose expiry it cannot establish.
resolveExpiryMillispreviously fell back to "treat as expired", so a manifest with damaged timestamps had its rows deleted even though they were perfectly readable — and a version that changed only the timestamp encoding (a TemporalZonedDateTime, an ISO week date, epoch millis as a string, or canonical ISO with surrounding whitespace, whichDate.parsedoes not trim) would have had its data destroyed by an older binary. Such a session is now retained and reported, like any other manifest this version cannot fully interpret. One rule now covers every case: only an expired session with a resolvable date, or a ref directory verified to be empty, is ever deleted. - A
ttlMinutestoo large to date no longer produces an immortal session. 0.5.0 guardedNumber.isFinite(ttlMinutes), which accepted1e308whose product overflows toInfinity; and finiteness was the wrong bound anyway, sinceNumber.isSafeIntegeradmits a value whose product reaches 5.4e20 while aDateholds at most ±8.64e15 ms. Either way the session survived every prune and was not even counted. A derived expiry beyond theDaterange is now unresolvable, hence retained. result prunenames the refs it left in place instead of only counting them. A retained session is omitted fromresult listand no command removes one, so the ref is the only way to find the file.result pruneexits non-zero when it could not delete an expired session. It is the only machine-readable health signal the store has, and reporting a partial sweep as success meantif cf-otel result prune; then …could never detect a store it had failed to clean.
Changed
PruneOutcomeis now{ removed, failed, retainedRefs }.retainedwas a count the user could not act on; the refs replace it. This is why 0.6.0 is a minor rather than a patch.
Known limitations
- A leftover
<ref>.tmp-<pid>directory from an interrupted save is invisible toresult list,result pruneandresult clear, and no TTL reaches it. Reclaim it by hand. - A retained session is reported by
result prunebut not listed, and there is noresult rm;result clearis the only in-tool removal and it removes everything it can see. result listomits a session it could not read without saying so. Onlyresult prunereports those.
Changed 3, Fixes 12
Notes
Fixed
-
Pruning no longer deletes saved results it cannot read.
pruneResultSessionsruns at the head of every save, read and list, and it treated four distinct outcomes — file absent, I/O error, invalid JSON, unrecognized shape — as one "missing" answer, thenrm -rf'd the directory. A single nominally read-onlyresult listwas measured destroying a valid unexpired manifest whose only difference wasversion: 2, and a manifest that was merely unreadable at that moment (chmod 000) with its data fully intact. Because the self-updater makes mixed versions on one machine normal, that meant an older cf-otel silently wiped a store a newer one had written.Reads are now classified, and only an expired session or an empty ref directory is removed. A manifest that cannot be read or recognized is left exactly where it is and counted as retained; a ref directory holding files under names this version does not know is retained too, so a future manifest filename cannot be mistaken for a crashed save and reclaimed.
-
A saved result with a damaged
expiresAtis no longer immortal.Date.parsereturnsNaNfor it andNaN <= nowisfalse, so such a session survived every prune for ever and still read back — the TTL failing in the direction that keeps production span data past its retention window. Expiry now falls back tocreatedAtplusttlMinutes, and a session whose age cannot be established by either route is treated as expired. -
One undeletable directory no longer takes the whole store offline. Deletion is now attempted per session, so a results directory that cannot be written no longer fails every save, read and list outright; and because the housekeeping prune is best-effort,
result showandresult listkeep working on intact sessions even when the directory cannot be listed at all.result showalso enforces the TTL itself instead of relying on that prune having succeeded. -
result shownow says which of the three things went wrong. An unreadable manifest and one written in a newer format used to report "Saved result not found or expired", which was simply untrue; both now name the file and, for a newer format, say it was left in place. Two error codes were added:RESULT_UNREADABLEandRESULT_STORE_NOT_WRITABLE. -
result listprints the ref that actually resolves. The listed ref came from inside the manifest while lookup uses the directory name; when those disagreed,listadvertised a ref thatresult showthen rejected as invalid. Expired sessions that prune could not remove are omitted rather than displayed. -
--saveno longer discards a completed query when the store cannot be written. The store is now probed while arguments are validated — before the CF login and credential discovery every--savecommand otherwise pays for first — so a store that cannot be written fails in ~120ms instead of after ~20s of tenant work. One hook on the root program covers all eleven--savecommands, so the check cannot drift out of one of them.For the failures that cannot be known in advance (a full disk, the per-result byte cap), the rows are printed in the requested
--formatwith a diagnostic on stderr, and the exit code is non-zero. Exiting 0 was not an option:ref=is a machine contract, and inref=$(cf-otel find --save …)a zero exit with a table on stdout would bind a table row as if it were a ref. -
The unit test suite no longer writes into the developer's own
~/.saptools.mapping --savein the CLI tests fakes the OpenSearch client but not the store, and the vitest config set no store override — 69 of the 70 sessions found in one real store had been written by that one test, one per run. The config now pointsCF_OTEL_RESULTS_ROOTat a throwaway directory that is cleared as it loads. -
Pruning no longer deletes a saved result whose expiry it cannot establish.
resolveExpiryMillispreviously fell back to "treat as expired", so a manifest with damaged timestamps had its rows deleted even though they were perfectly readable — and a version that changed only the timestamp encoding (a TemporalZonedDateTime, an ISO week date, epoch millis as a string, or canonical ISO with surrounding whitespace, whichDate.parsedoes not trim) would have had its data destroyed by an older binary. Such a session is now retained and reported, like any other manifest this version cannot fully interpret. One rule now covers every case: only an expired session with a resolvable date, or a ref directory verified to be empty, is ever deleted. -
A
ttlMinutestoo large to date no longer produces an immortal session. The bound is what aDatecan hold, not what a float can hold:Number.isSafeIntegeradmits a value whose product with 60000 reaches 5.4e20, while aDateholds at most ±8.64e15 ms — socreatedAt + ttlMinutescould yield a finite expiry no clock will ever reach, which survived every prune and was not even counted. A derived expiry beyond theDaterange is now treated as unresolvable, hence retained. -
result prunenames the refs it left in place instead of only counting them. A retained session is omitted fromresult listand no command removes one, so the ref is the only way to find the file. -
result pruneexits non-zero when it could not delete an expired session. It is the only machine-readable health signal the store has, and reporting a partial sweep as success meantif cf-otel result prune; then …could never detect a store it had failed to clean. -
PruneOutcomeis now{ removed, failed, retainedRefs }.retainedwas a count the user could not act on; the refs replace it.
Changed
pruneResultSessionsreturns{ removed, retained, failed }instead of a bare count.result prunestill printsremoved=Nas its only stdout line and reports retained and undeletable counts on stderr.tryCreateResultSessionwas removed. It had no caller, and it could not become one: it discards the error a diagnostic needs, and itsResultSession | undefinedreturn makes the caller decide what to print — the decision now made explicitly inemitRows.assertResultStoreWritableand thePruneOutcometype are exported.
-
Fixes 3
Notes
Fixed
-
--since/--untilare validated instead of forwarded blind. Previously anything that did not match<digits><s|m|h|d>was passed to OpenSearch verbatim, so--since yesterdaycame back as a rawHTTP 400 parse_exceptiondump, and--since 999999999doverflowed theDaterange and surfaced as a barecf-otel: Invalid time valuethat named no flag at all. Both now fail with a message that names the flag and spells out the accepted grammar.The check also catches dates
Date.parseaccepts but OpenSearch'sstrict_date_optional_timerejects: it silently rolls2026-02-30over to March 2 and2026-04-31to May 1, so those are now reported as "not a real calendar date" rather than quietly querying the wrong day. Month length is probed against a fixed leap year and corrected for February rather than passing the caller's year toDate.UTC, which maps a year below 100 into 1900-1999 and would therefore reject the real date0000-02-29.A
--sinceresolving later than its--untilis rejected too — an inverted range returned zero rows, which on a read-only tool is indistinguishable from "no data".Validation runs alongside the other argument checks rather than inside the query builder, which sits behind
withOpenSearchClient. A malformed bound now fails in ~125ms instead of after a full CF login and credential discovery (~23s measured).Behavior change: a bare number is no longer accepted. The index's
startTimeformat isstrict_date_optional_time||epoch_millis, so a number that was never checked used to reach OpenSearch and work.--since 24reads just as easily as a24hmissing its unit, and as an epoch-millis lower bound it would match the entire index instead of the last day, so the error names both readings. -
spans --fieldsrejects a name no column builder knows.--fields duration(a plausible typo fordurationInNanos) used to print one empty{}per span at exit 0 — and because the list is also sent as an OpenSearch_sourcefilter, the documents genuinely came back without the field, leaving nothing downstream able to notice. The valid-name list and the default selection are now derived from the column builders instead of being maintained as a second hand-written list, which is what allowed the two to drift. -
Dashboards console-proxy requests now carry a deadline. Node's
fetchapplies none of its own, so an endpoint that accepted the connection and never answered hung the CLI indefinitely with no output; the e2e suite reproduces this, taking 5 minutes and a harness kill before the fix versus 1.5s after. The default is 60s per request, overridable withCF_OTEL_HTTP_TIMEOUT_MS, and a timeout is reported distinctly from a transport failure so the message points at the right cause. Ported from cf-metrics so both CLIs behave the same way; unlike cf-metrics, no caller-suppliedAbortSignalis threaded through the client, because cf-otel has no long-running command that needs one.The deadline covers the response body as well as the request. Headers can arrive well before the payload finishes streaming — the normal shape for a wide aggregation — so an abort often lands on the body read; left unhandled it escaped as a bare
The operation was aborted due to timeoutwith no path, no ceiling and no hint.The configured value is also normalized before it reaches
AbortSignal.timeout, which throws aRangeErrorfor a fractional or negative delay and, above 2^31-1, silently reduces the timer to 1ms with only a process warning — handing anyone who raised the ceiling the exact opposite. A fractional, negative or non-numeric value now falls back to the default, and an over-large one is clamped to 2^31-1 rather than defaulted, since it still expresses "wait a long time".
-
Other 5
Notes
cf servicesran twice on every auto-discovered command, and now runs once. It is the most expensive command in the credential path: the CF CLI implements it as one request per instance in the space, and a traced coldcounton a real tenant spent 11.8s and 15.9s in its two calls, 27.7s of a 40.6s total. The redundancy was structural rather than accidental. Instance discovery filtered the listing to thecloud-loggingoffering, required exactly one row, and then returned only that row's name; the fallback-binding step later re-fetched the entire listing purely to readboundAppsback off that same row. Discovery now returns the row it selected, and the fallback step uses the bound apps it already carries. Cloud Foundry enforces unique instance names within a space, so this is the same row the second listing would have found: the value is identical, not merely equivalent.- Call counts per path, so nothing regressed for anyone: auto-discovery with no usable service key
(the common case on a SAML-enabled instance) goes from two listings to one; auto-discovery where
a service key works stays at one; a pinned
--service-instancewith unpinned fallback apps stays at one, since nothing has listed the services on that path; and pinning both--service-instanceand--fallback-binding-appstays at zero. Measured from the other direction on the same tenant, a run with an explicit--service-instance(which already skipped the first listing) took 21.4s against 34.3s without it. - An auto-discovered instance with no bound apps is still a real answer, not a cache miss: it reports "no apps are bound to instance X" as before rather than triggering a second listing.
- API change for library consumers:
discoverServiceInstancenow resolves to theCfServiceRowit selected instead of a bare instance name, since that row is what carriesboundApps.CfServiceRowwas already exported and was already the element type oflistCloudLoggingInstances, so no new type is introduced.findBoundAppsandlistCloudLoggingInstanceskeep their signatures;findBoundAppsis now only reached when the caller pinned--service-instance. - Regression tests at both levels: unit tests pin the exact
cf servicescall count for all four paths and for the empty-bound-apps case, and an e2e test asserts a singleservicesentry in the fakecf's call trace for a run that falls through to the binding path. Three of them fail against the previous release.
Other 6
Notes
Three correctness fixes for CF CLI v8, whose output shapes differ from v7's in ways this package never handled. All three were verified against the CF CLI v8.18.0 source and, where possible, against a real tenant.
--service-keynever worked, and--allow-mint-credentialfailed after disabling SAML.cf service-keynests the credential fields under acredentialsobject on CLI v8, where v7 printed them flat, and only the top level was read. Confirmed live: a realcf service-keyon a current CLI returns exactly one top-level key,credentials. Both shapes are now accepted, with the top level preferred when it carries the fields, so the already-unwrapped VCAP payload keeps working. The minting path readscf service-keydirectly, so the same bug made every freshly minted key look empty — after SAML had already been switched off on a shared instance, and with a message ("did not contain dashboards-username/dashboards-password") that pointed at SAML rather than at the parser.cf service-keyswas never parsed at all on v8. v8 renders a three-column table (name,last operation,message) throughDisplayTableWithHeader, and the parser required the header line to equalname, so it always returned no keys. Every run therefore reported "no service keys exist on instance X" for instances that have them, and fell through to the far slower per-appcf envscan (measured at 16s on a real tenant). Both header shapes are now read, and cells are sliced by column position rather than split on whitespace —messageis routinely blank andlast operationcontains a space, so splitting took the wrong field. The column boundary is found by scanning for the next column in the header instead of matching the literal textlast operation, so a renamed column cannot silently reintroduce whole-row key names.- A failed mint no longer leaves its service key behind.
--allow-mint-credentialcreatedcf-otel-<hex>and, on any later failure, discarded the name along with the error, leaving an unusable key on a shared instance and adding another on every retry. The key is now deleted (cf delete-service-key … -f;-fis required, or v8 prompts on stdin and the command hangs until the exec timeout). The delete runs after the SAML restore, never before, so cleanup can never extend the window in which SSO is disabled for everyone, and it is attempted even whencf create-service-keyitself failed, since a create killed by a timeout may still have been applied by the broker. A cleanup failure never replaces the error being reported: the orphaned key and the exact recovery command are appended to it instead. A successful mint keeps its key — that key is the returned credential — and the one branch where a working key is deliberately kept despite a failure (mint succeeded, restore did not) now names it rather than leaving it silently. - A styled table header silently broke both parsers, and now cannot. Measured against a real
tenant on cf 8.18.0: with
CF_COLOR=trueexported,cfstyles each table header cell with ANSI escapes even when stdout is a pipe, which shifts every column index —cf serviceswent from 42 parsed rows to 0 (so instance discovery reported that the space has no Cloud Logging instance at all) andcf service-keysfrom 54 parsed key names to 0. Two independent guards now: everycfinvocation runs withCF_COLOR=false, and the parsers strip escape sequences themselves, which is provably safe because CF pads its columns by visible width — stripping reproduced the uncolored output byte for byte.cf targetwas measured not to colorize at all; it gets the same treatment for uniformity, not because it was broken. - Regression tests for every item above, at both levels: the unit tests cover both service-key
payload shapes, both
service-keysheader shapes, styled headers using the exact escape sequences a realcfemits, and each cleanup branch; and the fakecfused by the e2e suite now emits the v8 shapes by default with switches for the v7/v6 ones, so the end-to-end tests exercise the real shapes rather than pinning the old ones. - Every fix above was also verified against a real tenant rather than only against fixtures: the
patched extractor pulls a working credential out of a real binding's
{"credentials": {...}}payload that the previous code returned nothing for, and that credential authenticates against the real OpenSearch; the patched parser returns exactly the 54 key names the Cloud Controller reports for a real 54-key instance (including a 132-character key name, against a 135-wide name column) where the previous code returned none; andcf delete-service-key … -fwas confirmed to exit 0 for a key that does not exist, while the same call without-fprompts and fails.
Other 1
Notes
- Self-updating. Every command now checks npm for a newer
@saptools/cf-otel(at most once an hour) and, when one exists, installs that exact version and re-runs the command on it, announcing both steps on stderr.SAPTOOLS_AUTO_UPDATE=on|notify|offcontrols it (see the README's Updates section); it is off by itself in CI, in tests, from a source checkout and inside the re-run. Newcf-otel self-update [--check]forces the check and install now.--versionnow readspackage.jsonat runtime. Both come from the private, build-time-bundled@saptools/corepackage; the published tarball gains no runtime dependency.
- Self-updating. Every command now checks npm for a newer
Other 4
Notes
- Fixed
searchAfterAll's pagination tiebreaker: it sorted by OpenSearch's_idmeta-field, which is documented as restricted from sorting (falls back to fielddata, a deprecated path outside this tool's control). Now sorts byspanId, a mandatory OpenTelemetry field mapped as a plainkeywordwith doc_values on.spanandfieldsnow share the same tiebreaker constant instead of duplicating the sort array. - Fixed
mintDashboardsCredential's last-resort SAML restore (--allow-mint-credential): it used to hard-codesaml.enabled=trueon restore regardless of the instance's actual original value, permanently turning SSO on for an instance that never had it. Now writes the original params blob back verbatim, and the critical-failure error message's "broken for ALL users" framing only appears when SAML was genuinely on beforehand. - Fixed
top/detached'sby_traceterms aggregation: it had no explicitorder, defaulting to OpenSearch's_count-desc bucket selection even when sorting by duration — on a window with more than 10,000 distinct traces, a long-but-low-span-count trace could be silently dropped before the client-side sort ever saw it. Both commands now order by the same metric they sort by, and warn when that 10,000-bucket cap is actually hit. - Added regression tests covering every fix above.
- Fixed
Other 8
Notes
- Fixed
--attrfilters (e.g.--attr 'http@status_code>=400'): bare attribute keys were never resolved against their realspan.attributes.*/resource.attributes.*mapping paths, so every--attrfilter silently matched nothing. AddedresolveAndValidateAttrFiltersto resolve keys against the live index mapping before querying. - Fixed
mapping --fieldfor nested attribute paths: lookups only checked one level ofproperties, so dotted paths likeurl@pathnever resolved against the real nested mapping tree (properties.span.properties.attributes.properties.*). Field lookups now walk each.-separated segment's own nestedproperties. - Fixed a redaction bug in
redactSecretLikeText: a secret value containing an escaped quote (\") could leak the remainder of the value in plaintext next to[REDACTED]. The matching pattern is now escape-aware. - Fixed
gaps's regression "growing" verdict: the threshold was scaled byMath.max(1, meanY), which collapsed toward zero whenever the mean gap was negative. Now scaled byMath.abs(meanY). - Corrected
gaps's self-time documentation and tests: the raw signed-gap sum is only approximately, not exactly, equal to the naive self-time formula (a child span nested with slack breaks the exact identity). - Refreshed the OpenTelemetry semantic-convention attribute list backing
--with-samples, which had gone stale and rendered an empty samples column against current data. - Fixed VCAP
instance_namematching,track_total_hitsnot being set on paginated queries, four commands silently discarding their own truncation flag, a--limit 0footgun onfind/samplethat returned zero rows without warning, adiff --sort pctranking bug when both sides were zero, and a missing fallback ingetRegionKeyForApifor valid SAP regions not yet in the static map. - Added regression tests covering every fix above.
- Fixed
Other 6
Notes
Initial release.
- Read-only query and analysis CLI over OpenTelemetry spans already ingested into SAP Cloud
Logging's OpenSearch backend (
otel-v1-apm-span-*), reached through the OpenSearch Dashboards console-proxy. - Twelve commands:
sample,mapping,find,top,count,spans,span,fields,selftime,gaps,detached,diff, plus aresultsubtree (show/list/prune/clear) for refs saved via--save. --region/--org/--spacetargeting with ambientcf targetfallback and a resolved-target stderr notice, matchingcf-hana's convention.- A credential-discovery decision tree for the Cloud Logging dashboards basic-auth credential:
existing service keys, then pre-SAML app bindings, then (only behind the explicit
--allow-mint-credentialopt-in) a temporary SAML disable/mint/restore cycle with redacted logging and a loud, distinct error if the restore step itself fails. --format table|json|json-compact|csvand--save/result showon every row-returning command, mirroringcf-hana.search_after-based pagination past OpenSearch's defaultmax_result_window(10000) forspans, verified against a 10000+ span synthetic trace in both the unit and e2e suites.
- Read-only query and analysis CLI over OpenTelemetry spans already ingested into SAP Cloud
Logging's OpenSearch backend (
Others providing
Compare all 6 on the Telemetry page →- SAP Cloud LoggingSAP Cloud LoggingAn instance-based observability service built on OpenSearch that stores and analyses logs, metrics and traces from Cloud Foundry, Kyma, Kubernetes and other runtime environments.sap official, btpOpenSearch-based store for logs, metrics and traces
- Agent Activation for Dynatrace Service for SAP BTPAgent Activation for Dynatrace Service for SAP BTPInjects the Dynatrace agent into Java applications on SAP BTP, so their metrics, events and end-to-end traces go to a Dynatrace SaaS monitoring environment the customer already runs.sap official, btpInjects the Dynatrace agent into Java applications on SAP BTP
- SAP Monitoring Service for SAP BTPSAP Monitoring Service for SAP BTPMetrics, availability and JMX checks, and email or webhook alerts for Java, HTML5 and SAP HANA XS applications and database systems, in Neo regions of SAP BTP.sap official, btpMetrics, availability checks and alerts for apps in Neo regions
- @sap/e2e-trace@sap/e2e-traceSAP Passport handling for a Node.js component: it reads the identifiers a request arrives with, updates them for the next system, and writes a statistics record per request to standard output.sap official, btpSAP Passport handling for Node.js components
- @saptools/cf-metrics@saptools/cf-metricsA read-only command line that queries the container and OpenTelemetry metrics SAP Cloud Logging has ingested for Cloud Foundry apps, as history over a time range, a ranking across apps or a live watch.independent, btpRead-only query tool for metrics in SAP Cloud Logging
Details
Sources
Sources are graph entities: the same page may back several entries, and an unreachable source flags every row that rests on it.
Compatibility and provenance
History in the atlas
- First seen.
For your agent
- Ref
artifact:saptools-cf-otel- JSON
- /artifacts/saptools-cf-otel.json
- MCP
- Connecting an agent