Uploaded image for project: 'Drivers'
  1. Drivers
  2. DRIVERS-2139

Test retryable writes against real shutdown scenarios

    • Type: Icon: Spec Change Spec Change
    • Resolution: Unresolved
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Component/s: Retryability
    • Labels:
      None
    • Needed

      In PYTHON-1650, a user discovered a bug in PyMongo's implementation of retryable writes where it was possible for both the initial retryable write attempt and the retry attempt to never increment the txnNumber. Worse, this could occur when a primary was shut down normally. Testing retryable writes against real shut down scenarios (shutdowns, crashes, stepdowns) would have caught this issue.

      The specific scenario where PyMongo's bug would be hit is when:

      1. Primary shuts down.
      2. Client initiates a retryable write, retryWrites=true, and the server is 3.6 or 4.0.
      3. Server selection succeeds on the stale primary yielding server P.
      4. Socket is checked out of P's pool.
      5. The pool notices the socket has been closed (because this socket has been idle for >1 second) and attempts to create a new connection to the old primary.
      6. The connection attempt fails with a (retryable) "connection refused" error because the old primary is not running.
      7. The initial write attempt fails with a retryable error before incrementing the txnNumber.
      8. The retry attempt waits until the new primary is elected, does not increment the txnNumber, and runs the write with a previous txnNumber.

      The fix implemented in PyMongo 3.7.2 is to always increment the txnNumber before server/socket selection (similar to how start_transaction works).

      Note that it would be much less likely for this to occur if PYTHON-1435 was implemented because server selection would wait for the new primary on the initial attempt.

            Assignee:
            Unassigned Unassigned
            Reporter:
            shane.harvey@mongodb.com Shane Harvey
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated: