Add test determinism and verification guidance to AGENTS.md

XMLWordPrintableJSON

    • Storage Engines
    • 282.519
    • None
    • None

      The Test Frameworks section of AGENTS.md lists where each test framework lives but gives no guidance on how to write a test. Agent-assisted test authoring has been a recurring source of flaky tests, and nearly all of it traces to a small, enumerable set of causes:

      • Asserting on state owned by a background thread (eviction, checkpoint, sweep, garbage collection, the disaggregated-storage pickup server) without waiting for it. Several recent flakes were the same mistake: assuming a state transition is synchronous with the call that triggered it.
      • Using a sleep as a synchronization primitive, where the duration is long enough on an idle machine but not on an ASan or loaded variant.
      • Exact-value assertions on statistics that move with eviction timing and page splits.
      • Unseeded randomness, and workloads sized to only barely trigger the condition under test.
      • Depending on state left behind by another test.

      A second gap is the definition of done. A single green local run is treated as evidence that a new test is sound, when the checks that actually catch nondeterminism are: confirming the test fails without the fix, running it repeatedly, and running an Evergreen patch build across the sanitizer and slower variants.

      Adding this to AGENTS.md puts it in context for every agent session in the repo and gives reviewers a written rubric to hold new tests against. The section follows the shape of the existing Comment Prose Style guidance.

            Assignee:
            [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            Chenhao Qu
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: