Preliminary Findings on Staging Confidence

PRELIMINARY FINDINGS — Staging Confidence Investigation

ISSUE: A feature that passed staging testing has failed in production. STAGE OF INVESTIGATION: Early. Several explanations remain open.

The Staging Account

Staging confirms that all tests were conducted and passed. The feature was deployed to staging on Tuesday at 2pm. Testing concluded at 4:30pm. At 4:30pm, staging reported: no errors, expected behavior across all test cases, performance within parameters.

Staging maintains this account. Staging's records are accurate for the environment that staging represents.

The Production Account

Production confirms that the feature was deployed Wednesday morning. At 2:14pm Wednesday, the first error report was filed. By 3pm, six reports had accumulated describing behavior that staging did not reproduce.

Production's records are also accurate.

The Discrepancy

The discrepancy arises from the difference between what staging is and what production is.

Staging contains: a subset of production data, a configuration that approximates production configuration, a load that is lower than production load, and a user base of one to three developers who know the expected behavior and interact with the feature accordingly.

Production contains: all production data, the actual configuration that was slightly different from staging on three keys that were missed during staging setup, full production load, and users who interact with the feature in ways the developers did not specifically anticipate.

The Findings

Staging was correct about staging. Production is correct about production. The preliminary finding is that staging and production were not sufficiently similar to make staging confidence transferable.

This is a known issue in the industry. The investigation continues.

The three configuration keys have been identified. A fix is in staging now.