Engineering Notes21 October 20268 min read

Log an Event You Can Search Later

A sentence in a log file is hard to aggregate. A record with a stable event name, ids, and a result is something you can count during an incident.

NaveenInfrastructure Engineer
FIGURE 1.0
EDITORIAL REFERENCE
A monitoring display used during operations
CONTEXT:A monitoring display used during operations
REFERENCE:Carlos Muza
ON THIS PAGE (7 SECTIONS)
EXECUTIVE BRIEFING · KEY TAKEAWAYS
  • ✓OpenTelemetry treats a log record as structured data, not only as a text body.
  • ✓Include a correlation id, the operation, the outcome, and the duration. Omit secrets and full payloads.
  • ✓An error without the request id cannot be tied to the response the client saw.
  • ✓Model traces are a specialization of this. Ordinary services need the same discipline.

During an incident someone greps for "failed." The word appears in success messages, in a user name, and in the one error that matters. The line does not include the order id. The person who wrote it knew which order it was. The person reading it does not.

The OpenTelemetry logs data model defines a log record with attributes, a timestamp, a severity, and an optional body. The useful part for an application is the attributes. Give the event a stable name, such as order.submit, and fields for tenant, actor, subject id, outcome, and duration. The body can stay empty. A dashboard can count outcome=error. A search for a sentence cannot.

Fields worth storing

FieldWhy it is thereLeave it out when
eventStable name you can aggregateYou would have to edit code to rename a sentence
correlation_idTies the log to the response and to downstream callsYou have no id to put on the response either
outcomeok, rejected, or error, as a valueThe only signal is the English word failed
duration_msShows slowness without a profilerThe work is a batch with its own timer
subject idThe order, user, or jobThe id is itself a secret

Do not log passwords, session tokens, card numbers, or the full body of a request. Log that validation rejected a field, not the value of the field, when the value might be personal. LLM Observability applies the same idea to model calls: record the decision and the ids, not a copy of every token, unless you have a retention reason and a place to keep it.

Illustrative scenario

This scenario is illustrative. A checkout emits order.submit with outcome=rejected and reason=address_incomplete. Support pastes the correlation id from the error response into the log search and sees the rejection, the warehouse id, and the duration. They do not see the street address. A spike in outcome=error with reason=dependency_timeout is a graph, not a pile of stack traces. The stack trace is on the error records only, and it stays on the server.

Checklist

  • Pick event names and stick to them.
  • Put a correlation id on the response and on every log from that request.
  • Store outcome as a field.
  • Ban secrets and raw credentials in the logging helper, not only in a code review comment.
  • Keep stack traces on error records.
  • Alert on a rate of a specific event, not on the presence of the word error in free text.

Questions that usually come next

Are access logs enough?

They show that a request arrived and which status left. They do not show which order failed validation. Keep access logs. Add event logs for the decisions you will be asked about.

How long do we keep them?

Long enough to investigate the incidents you actually have, and no longer when the fields are personal. Write the period down. An infinite log of request bodies is a data store you did not mean to build.

Direct answers

Are access logs enough?

They show status and path. Add event logs for the business decision you will need to explain, with ids and an outcome.

How long should logs be kept?

Long enough for the incidents you investigate. A body that contains personal data needs a retention decision, not a default of forever.

ACADEMIC & INDUSTRY CITATIONS

References & Prior Art

REF 01Specification

OpenTelemetry Logs Data Model

OpenTelemetry Specification • OpenTelemetry (2026)

Source
VERIFIED CONTRIBUTOR

Naveen

Infrastructure Engineer

Naveen handles infrastructure, edge network resilience, chunked media pipelines, and deterministic CI/CD release verification at Vasiva.

CURATED ARCHIVE

Continue Reading

Contextually relevant engineering logs and production architecture notes.

SYSTEM ADVISORY

Have an AI workflow or complex system worth exploring?

Work directly with the senior engineers who designed and shipped these systems. Zero vendor lock-in, deterministic architectures, and first-party audit integrity.

Evaluate the opportunity