Field Notes on Log Level Escalation
Log verbosity raised during an incident was not lowered after, and has now been classified as the current standard.
Observation Log: Production Service billing-processor
Log level at time of first observation: DEBUG (raised from INFO, incident #2024-47)
Current log level: DEBUG
Days at current level: 34
Day 1 (Incident Night)
DEBUG logging is enabled at 2:14 AM. The incident concerns a payment processor timeout. The engineer needs to see request headers, retry counts, and response timing. The escalation is correct. The escalation takes four minutes. The incident resolves at 3:07 AM.
The engineer goes to sleep. The log level is DEBUG.
Day 4
A second engineer notices the verbose output in CloudWatch. She considers lowering it. She does not know if the incident is fully resolved. She decides to leave it in case the incident resurfaces. This decision is not recorded anywhere.
Day 9
A third engineer raises a PR. The PR includes a comment in the code: "FYI logs are still debug from the 47 incident." The reviewer responds: "Should be fine for now." The PR merges. The log level is DEBUG.
The governing rule is now in effect: once log verbosity has survived two PR reviews without being made the explicit subject of a remediation ticket, it is considered the current standard. Not the new standard. The current standard.
Day 18
The monthly AWS bill arrives. The CloudWatch costs are higher than forecast. Nobody connects this to log level escalation. Somebody opens a ticket to review storage costs. The ticket is assigned to the person who raised the original incident.
He is on vacation.
Day 34
A new engineer joins the team. She reads the README. The README does not mention the log level. She runs the service locally and assumes DEBUG is the intended configuration, because that is what the deployment currently specifies.
She is correct about what the configuration currently specifies. She is not correct about intent. These facts are compatible, and nobody has been assigned to resolve the difference.
The log level remains DEBUG. The incident that required DEBUG ended thirty-three days ago. The logs do not know this.