[DRIVERS-2139] Test retryable writes against real shutdown scenarios Created: 12/Oct/18  Updated: 21/Dec/23

Status: Backlog
Project: Drivers
Component/s: Retryability
Fix Version/s: None

Type: Spec Change Priority: Major - P3
Reporter: Shane Harvey Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Related
related to PYTHON-1650 Replica failover with retryWrites not... Closed
is related to DRIVERS-2121 Test retryable reads against live fai... Backlog
Driver Changes: Needed

 Description   

In PYTHON-1650, a user discovered a bug in PyMongo's implementation of retryable writes where it was possible for both the initial retryable write attempt and the retry attempt to never increment the txnNumber. Worse, this could occur when a primary was shut down normally. Testing retryable writes against real shut down scenarios (shutdowns, crashes, stepdowns) would have caught this issue.

The specific scenario where PyMongo's bug would be hit is when:

  1. Primary shuts down.
  2. Client initiates a retryable write, retryWrites=true, and the server is 3.6 or 4.0.
  3. Server selection succeeds on the stale primary yielding server P.
  4. Socket is checked out of P's pool.
  5. The pool notices the socket has been closed (because this socket has been idle for >1 second) and attempts to create a new connection to the old primary.
  6. The connection attempt fails with a (retryable) "connection refused" error because the old primary is not running.
  7. The initial write attempt fails with a retryable error before incrementing the txnNumber.
  8. The retry attempt waits until the new primary is elected, does not increment the txnNumber, and runs the write with a previous txnNumber.

The fix implemented in PyMongo 3.7.2 is to always increment the txnNumber before server/socket selection (similar to how start_transaction works).

Note that it would be much less likely for this to occur if PYTHON-1435 was implemented because server selection would wait for the new primary on the initial attempt.



 Comments   
Comment by Esha Bhargava [ 10/Feb/20 ]

This work will be done post 4.4

Comment by Prashant Mital (Inactive) [ 04/Nov/19 ]

jeff.yemin Indeed it is. Thanks for pointing it out!

Comment by Jeffrey Yemin [ 01/Nov/19 ]

CC prashant.mital maybe relevant for planned maintenance testing scenarios.

Comment by Bernie Hackett [ 12/Oct/18 ]

That is precisely the bug, and what happened in the reproduction. Even worse, the initial write was an insert, the retry write was an update, and the server converted the reply to the retry (which was the reply from the successful insert) into an upsert...

Comment by A. Jesse Jiryu Davis [ 12/Oct/18 ]

So PyMongo could use a completely unrelated txnNumber for an operation?

collection.update_one(filter, update)  # Uses txnNumber 1.
# In between, the primary dies.
collection.insert_one(doc)  # Also uses txnNumber 1 after finding new primary.

Comment by Shane Harvey [ 12/Oct/18 ]

We do have a single prose test that tests with a real replica set stepdown (DRIVERS-579). It's possible this test would have caught this bug so I'll report back once we implement it in PYTHON-1657.

Generated at Thu Feb 08 08:24:49 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.