---
title: Observability
description: Inspect Forge logs, metrics, and traces locally and query deployed telemetry with the SAIF CLI.
moved_from:
  - reference/tools/opentelemetry.md
  - reference/tools/dynatrace.md
---

# Observability

Use the Aspire dashboard for local telemetry and Dynatrace for deployed applications. Forge sends OpenTelemetry logs, traces, and metrics to these destinations; Dynatrace also provides Azure resource logs, synthetic monitoring, application maps, and infrastructure monitoring.

## Prerequisites

Run the AppHost to inspect local telemetry. To query deployed telemetry, use the [SAIF CLI](../../learn/install-saif-cli.md) with access to SAIF's Dynatrace tenant and its configured authentication. The [CLI reference](../../reference/dotnet/SAIF.Platform.CLI/commands.md) describes authentication and command options.

Run `saif auth dynatrace login` for the interactive platform-token setup. It opens the token page, lists the required scopes, validates access, and stores the token. Prefer the secret prompt to putting a token in command history.

Do not log secrets or sensitive payloads merely to make them searchable. Telemetry from deployed applications leaves the application process and enters the configured backend.

## Forge collection defaults

[`ConfigureOpenTelemetry()` in `SAIF.Platform/Extensions.cs`](https://github.com/saif-corp/forge/blob/main/src/dotnet/SAIF.Platform/Extensions.cs) configures these defaults:

- **Application-side sampling:** `AlwaysOnSampler` samples every span presented to the SDK. That increases visibility and ingest volume, but does **not** guarantee backend retention or successful export. Export failures, downstream processing, and backend retention remain separate concerns. See OpenTelemetry's [sampling concepts](https://opentelemetry.io/docs/concepts/sampling/). Override the sampler in the service's own OpenTelemetry configuration when its traffic and visibility requirements justify a different choice.
- **Instrumentation:** `AddSource("*")` and `AddMeter("*")` collect every matching registered activity source and meter rather than requiring a per-source opt-in list. The platform also registers ASP.NET Core and HTTP client instrumentation, plus runtime metrics.
- **Logs:** formatted messages and scopes are included.
- **Export:** OTLP export is enabled only when `OTEL_EXPORTER_OTLP_ENDPOINT` has a value.
- **Platform diagnostics:** `Diagnostics.ActivitySource` emits platform operations and `platform.*` tags. Use the [`SAIF.Platform` package reference](../../reference/dotnet/SAIF.Platform/index.md) for the API surface rather than treating these tags as a replacement for your application's own instrumentation.

## Querying telemetry from the CLI

The `saif otel` commands query Dynatrace directly. Quote relative time expressions so PowerShell passes them unchanged:

```powershell
# Application and container logs for a service, narrowed to errors
saif otel logs smithy --environment test --level error --from 'now()-2h'

# Request counts, failures, and P50/P95/P99 latency for a service
saif otel metrics smithy --environment test --from 'now()-2h'

# Failed spans in one deployment environment
saif otel traces smithy --environment test --failed --from 'now()-2h'
```

### Defaults and output

The curated queries default to the last two hours and a 100-record limit, capped at the configured maximum of 1,000 records. Use `--from`, `--to`, and `--limit` to narrow the result. These defaults come from [`DynatraceSettings`](https://github.com/saif-corp/forge/blob/main/src/dotnet/SAIF.Platform.Sdk/Dynatrace/DynatraceSettings.cs).

| Command | Curated query behavior |
| --- | --- |
| `logs [service]` | Newest logs first; optional case-insensitive substring match on `service.name` |
| `traces [service]` | Newest spans first; service match and optional deployment, failure, or trace-ID filters |
| `metrics <service>` | Request count, failures, and P50/P95/P99 latency, grouped by service entity and matched against its name |
| `metrics` without a service | Average host CPU usage by host, not application request metrics |

`--source all` is the logs default. Use `--source app` for OpenTelemetry application logs or `--source console` for container capture, such as startup failures that never reach the application's OTLP exporter. `--level` is an alias for `--loglevel` and matches one level case-insensitively.

Table output includes links into Dynatrace and per-trace links where available. Use `--format json` for machine-readable output without those human-facing notes.

### Environment selection is not trace correlation

- `--environment test` selects the mapped Dynatrace tenant and filters **traces** by `deployment.environment.name`.
- For **logs and metrics**, `--environment` selects the tenant only. Their curated queries do not filter deployment environment; the CLI prints a note in human-readable output. Do not interpret their results as test-only traffic.
- The shipped mapping sends `platformdev`, `test`, `qa`, and `uat` to the `dev` tenant, and `prod` to the `prod` tenant.
- `--dynatrace-environment dev` selects a tenant directly. With both selectors present, it overrides tenant selection while `--environment` still scopes the curated trace query.
- With neither selector, the CLI uses its configured default tenant (`dev` by default) without a deployment filter.

See [`ResolveTarget()`](https://github.com/saif-corp/forge/blob/main/src/dotnet/SAIF.Platform.Sdk/Dynatrace/DynatraceEnvironmentResolver.cs) and the [shipped tenant mapping](https://github.com/saif-corp/forge/blob/main/src/dotnet/SAIF.Platform.CLI/appsettings.json).

### Follow one trace

When an alert or dashboard gives you a trace ID, use it to correlate spans and logs:

```powershell
saif otel traces --environment test --trace-id e1f7f740cb2c79175ed238017dd80bc4 --from 'now()-7d'
saif otel logs --environment test --trace-id e1f7f740cb2c79175ed238017dd80bc4 --level error --from 'now()-7d'
```

`--trace-id` handles the schema difference: spans use `trace.id` as a UID, while logs use the string `trace_id`. It composes with the curated filters, but does not make logs without a trace ID correlatable. Metrics have no `--trace-id` option.

### Raw DQL

`--query` replaces the curated query. Write the service, deployment, trace, time range, limit, and probe conditions inside your DQL; the corresponding convenience filters no longer apply. Tenant selection still applies. The CLI warns about ignored options in human-readable output. See [`OtelQueryCommandHandler`](https://github.com/saif-corp/forge/blob/main/src/dotnet/SAIF.Platform.CLI/Commands/Otel/OtelQueryCommand.cs) for this distinction.

### Health-probe routes are excluded by default

Curated service metrics and trace queries exclude `endpoint.name` values matching `health`, `ready`, or `live` as whole words or path segments. They use DQL `matchesPhrase`, not a substring check that would confuse `/live` with `/deliveries`. Records with no `endpoint.name` pass through; the CLI does not assume they are probes.

Metrics apply this exclusion **before aggregation** with the `timeseries` filter parameter so probes do not distort request counts or percentiles. Logs have no reliable structured route filter and remain unaffected. Use `--include-probes` to disable the exclusion:

```powershell
saif otel metrics smithy --environment test --from 'now()-2h' --include-probes
saif otel traces smithy --environment test --include-probes
```

The CLI's `Forge:Dynatrace:ProbeEndpointSubstrings` setting extends the word-boundary phrase matches. `ProbeEndpointExactMatches` adds exact, case-sensitive route names and starts empty. These settings bind from the CLI's `appsettings.json`, not `saif config set` or `~/.saif/config.json`; consumers cannot customize them through those user-config commands.

```json
{
  "Forge": {
    "Dynatrace": {
      "ProbeEndpointSubstrings": ["healthz", "startup"],
      "ProbeEndpointExactMatches": ["/*"]
    }
  }
}
```

The .NET configuration binder appends to the built-in list, and query construction removes duplicates. Do not add `/*` or `*` unless you have confirmed that the tenant's catch-all route represents probes for the affected services; it can also hold real, unclassified traffic. See [`DynatraceDqlBuilder`](https://github.com/saif-corp/forge/blob/main/src/dotnet/SAIF.Platform.Sdk/Dynatrace/Dql/DynatraceDqlBuilder.cs) for the query construction.

## Related Documentation

- [Local Development](../local-dev/index.md)
- [Integration testing](../testing/index.md)
- [OpenTelemetry .NET](https://opentelemetry.io/docs/languages/dotnet/)
- [OpenTelemetry and Dynatrace](https://docs.dynatrace.com/docs/ingest-from/opentelemetry)
- [Dynatrace DQL string functions](https://docs.dynatrace.com/docs/platform/grail/dynatrace-query-language/functions/string-functions)