Skip to content

Observability

Use the Aspire dashboard for local telemetry and Dynatrace for deployed applications. Forge sends OpenTelemetry logs, traces, and metrics to these destinations; Dynatrace also provides Azure resource logs, synthetic monitoring, application maps, and infrastructure monitoring.

Prerequisites

Run the AppHost to inspect local telemetry. To query deployed telemetry, use the SAIF CLI with access to SAIF's Dynatrace tenant and its configured authentication. The CLI reference describes authentication and command options.

Run saif auth dynatrace login for the interactive platform-token setup. It opens the token page, lists the required scopes, validates access, and stores the token. Prefer the secret prompt to putting a token in command history.

Do not log secrets or sensitive payloads merely to make them searchable. Telemetry from deployed applications leaves the application process and enters the configured backend.

Forge collection defaults

ConfigureOpenTelemetry() in SAIF.Platform/Extensions.cs configures these defaults:

  • Application-side sampling: AlwaysOnSampler samples every span presented to the SDK. That increases visibility and ingest volume, but does not guarantee backend retention or successful export. Export failures, downstream processing, and backend retention remain separate concerns. See OpenTelemetry's sampling concepts. Override the sampler in the service's own OpenTelemetry configuration when its traffic and visibility requirements justify a different choice.
  • Instrumentation: AddSource("*") and AddMeter("*") collect every matching registered activity source and meter rather than requiring a per-source opt-in list. The platform also registers ASP.NET Core and HTTP client instrumentation, plus runtime metrics.
  • Logs: formatted messages and scopes are included.
  • Export: OTLP export is enabled only when OTEL_EXPORTER_OTLP_ENDPOINT has a value.
  • Platform diagnostics: Diagnostics.ActivitySource emits platform operations and platform.* tags. Use the SAIF.Platform package reference for the API surface rather than treating these tags as a replacement for your application's own instrumentation.

Querying telemetry from the CLI

The saif otel commands query Dynatrace directly. Quote relative time expressions so PowerShell passes them unchanged:

# Application and container logs for a service, narrowed to errors
saif otel logs smithy --environment test --level error --from 'now()-2h'

# Request counts, failures, and P50/P95/P99 latency for a service
saif otel metrics smithy --environment test --from 'now()-2h'

# Failed spans in one deployment environment
saif otel traces smithy --environment test --failed --from 'now()-2h'

Defaults and output

The curated queries default to the last two hours and a 100-record limit, capped at the configured maximum of 1,000 records. Use --from, --to, and --limit to narrow the result. These defaults come from DynatraceSettings.

Command Curated query behavior
logs [service] Newest logs first; optional case-insensitive substring match on service.name
traces [service] Newest spans first; service match and optional deployment, failure, or trace-ID filters
metrics <service> Request count, failures, and P50/P95/P99 latency, grouped by service entity and matched against its name
metrics without a service Average host CPU usage by host, not application request metrics

--source all is the logs default. Use --source app for OpenTelemetry application logs or --source console for container capture, such as startup failures that never reach the application's OTLP exporter. --level is an alias for --loglevel and matches one level case-insensitively.

Table output includes links into Dynatrace and per-trace links where available. Use --format json for machine-readable output without those human-facing notes.

Environment selection is not trace correlation

  • --environment test selects the mapped Dynatrace tenant and filters traces by deployment.environment.name.
  • For logs and metrics, --environment selects the tenant only. Their curated queries do not filter deployment environment; the CLI prints a note in human-readable output. Do not interpret their results as test-only traffic.
  • The shipped mapping sends platformdev, test, qa, and uat to the dev tenant, and prod to the prod tenant.
  • --dynatrace-environment dev selects a tenant directly. With both selectors present, it overrides tenant selection while --environment still scopes the curated trace query.
  • With neither selector, the CLI uses its configured default tenant (dev by default) without a deployment filter.

See ResolveTarget() and the shipped tenant mapping.

Follow one trace

When an alert or dashboard gives you a trace ID, use it to correlate spans and logs:

saif otel traces --environment test --trace-id e1f7f740cb2c79175ed238017dd80bc4 --from 'now()-7d'
saif otel logs --environment test --trace-id e1f7f740cb2c79175ed238017dd80bc4 --level error --from 'now()-7d'

--trace-id handles the schema difference: spans use trace.id as a UID, while logs use the string trace_id. It composes with the curated filters, but does not make logs without a trace ID correlatable. Metrics have no --trace-id option.

Raw DQL

--query replaces the curated query. Write the service, deployment, trace, time range, limit, and probe conditions inside your DQL; the corresponding convenience filters no longer apply. Tenant selection still applies. The CLI warns about ignored options in human-readable output. See OtelQueryCommandHandler for this distinction.

Health-probe routes are excluded by default

Curated service metrics and trace queries exclude endpoint.name values matching health, ready, or live as whole words or path segments. They use DQL matchesPhrase, not a substring check that would confuse /live with /deliveries. Records with no endpoint.name pass through; the CLI does not assume they are probes.

Metrics apply this exclusion before aggregation with the timeseries filter parameter so probes do not distort request counts or percentiles. Logs have no reliable structured route filter and remain unaffected. Use --include-probes to disable the exclusion:

saif otel metrics smithy --environment test --from 'now()-2h' --include-probes
saif otel traces smithy --environment test --include-probes

The CLI's Forge:Dynatrace:ProbeEndpointSubstrings setting extends the word-boundary phrase matches. ProbeEndpointExactMatches adds exact, case-sensitive route names and starts empty. These settings bind from the CLI's appsettings.json, not saif config set or ~/.saif/config.json; consumers cannot customize them through those user-config commands.

{
  "Forge": {
    "Dynatrace": {
      "ProbeEndpointSubstrings": ["healthz", "startup"],
      "ProbeEndpointExactMatches": ["/*"]
    }
  }
}

The .NET configuration binder appends to the built-in list, and query construction removes duplicates. Do not add /* or * unless you have confirmed that the tenant's catch-all route represents probes for the affected services; it can also hold real, unclassified traffic. See DynatraceDqlBuilder for the query construction.