Preliminary Findings on Production Query Caution
On staging you type fast. On production your hands know something your brain forgot.
Preliminary Findings on Production Query Caution
Subject: The Behavioral Change That Occurs When an Engineer Connects to a Production Database Status: Preliminary. Consistent with all prior observations across multiple engineers.
Observation
On a staging or development environment, an engineer writes a query with the fluency of someone who writes queries regularly. They type without pause. They run the query. They iterate.
On a production environment, the same engineer types the same query more slowly.
They reach the end of the query and do not run it.
They reread it.
They scroll back to the top.
They reread it from the top.
They call over a colleague to read it.
The colleague reads it. The colleague nods. The engineer reads it one more time.
Then they run it.
What the Query Was
A SELECT statement.
The SELECT statement could not modify data. The SELECT statement was known to be safe before the ritual began. The ritual was performed anyway.
Why the Ritual Persists
The ritual was created after an incident. Not necessarily an incident this engineer experienced personally. Sometimes an incident they heard about. Sometimes an incident they heard about secondhand.
Ritual resolves uncertainty before the source of uncertainty is confirmed. The ritual spread to queries that warranted no caution because the ritual had proven its value in cases that did.
The team now trusts the ritual more than the explanation. When a new engineer runs a production query without the careful reread, senior engineers wince slightly. The wince is not because the query was dangerous. The wince is because the ritual was not performed.
Known Limitation
The caution applied at query time cannot prevent incidents caused by incorrect data already in production, upstream systems, or queries run last week that nobody reviewed carefully enough.
The ritual addresses what can be seen. It does not address what cannot be seen. This is not a criticism. The ritual is doing everything it was designed to do.
The engineer whose typing speed drops near production has absorbed the lesson. The lesson is now automatic. The lesson no longer requires the original incident to justify itself.