Build traces without secret leaks

Trace decisions, not unrestricted payloads

A useful trace answers which task version ran, what evidence classes were consulted, which tool and target were selected, what policy decision occurred, and whether an external effect was confirmed. None of those questions always requires storing the full prompt, retrieved source, shell output, HTTP body, or secret-bearing environment. Start with a minimal event schema: run and operation identifiers, component and version, timestamps, status, reason code, data classification, input and output hashes where appropriate, and links to separately controlled artifacts. OWASP's logging guidance says access tokens, passwords, connection strings, encryption keys, sensitive personal data, and higher-classification data should usually be removed, masked, hashed, or encrypted rather than recorded directly. Apply that rule before collection, not only in a dashboard.

Redact by structure and provenance

Pattern matching catches familiar token formats but misses novel credentials and transformed values. Build redaction into data producers. A tool wrapper knows which arguments are secrets, an HTTP client knows header names, and a retrieval layer knows a document's classification. Replace sensitive values with stable, scoped references such as credential-slot names or one-run hashes. Keep source snippets out of default spans; record path, revision, line range, and content digest, then grant controlled access to the underlying artifact when investigation requires it. OpenTelemetry's convention guidance explicitly calls for noting attributes that may contain personal or sensitive information. Semantic consistency helps, but the organization still decides which attributes are safe to emit.

Worked example: a hypothetical deployment failure

Suppose a hypothetical release agent calls a deployment API and receives an authentication error. An unsafe trace stores the complete request headers, environment snapshot, response body, and prompt that contained an internal incident link. A safer trace records operation deploy, target environment staging, credential reference deployer-staging, request-body hash, HTTP status, provider request ID, policy decision approved, and result failed-authentication. The wrapper suppresses the authorization header and classifies the response before preserving a short sanitized error. A security-only artifact store may retain encrypted raw evidence for a brief incident window under separate access control. Ordinary developers can diagnose the failed credential slot without receiving the credential, customer payload, or unrelated environment values.

Test telemetry as an attack surface

Seed non-production canary secrets in prompts, files, tool arguments, outputs, exception messages, and transformed forms, then search every destination: local logs, traces, metrics labels, error trackers, exports, backups, and support bundles. Verify that changing log level cannot silently enable raw model or tool payloads without an approved configuration change. Test log injection with line breaks and structured delimiters, and bound field sizes so hostile content cannot distort the record or create denial of service. Give trace readers less privilege than production operators where possible, record access to sensitive evidence, and enforce retention deletion across replicas. Observability fails safely only when telemetry loss or sanitization errors are visible without falling back to dumping the original payload.

OWASP Logging Cheat Sheet ↗

OpenTelemetry convention guidance ↗

Include every recipient

Inventory third-party collectors, customer-visible diagnostics, downloaded support bundles, and copied incident notes as trace recipients. A clean application dashboard does not prove those paths are clean. Apply classification and retention rules before export, and make any exceptional raw-data transfer a recorded, time-bounded security decision separately.