When storeImageInSideCollection=true, we reserve two oplog slots, T-1 and T, for every findAndModify operation with a pre or post image:
- T-1 is reserved for the potential forged noop entry for the pre or post image during tenant migration or resharding.
- T is reserved for the update oplog entry itself.
For a collection with preImageRecordingEnabledForCollection=false, we write the update oplog entry in the reserved slot T and write the pre or post image to a config.image_collection entry. However, for a collection with preImageRecordingEnabledForCollection=true, we currently write the pre-image to a noop oplog entry in some other oplog slot T' > T, and then write the update oplog entry in the reserved slot T. That latter is invalid so we hit this invariant in the ReplClientInfo. That is:
- For a findAndModify operation with preImage, we end up with the following oplog entries:
- T-1: reserved for potential forged pre-image noop oplog entry
- T: update oplog entry
- T': pre-image noop oplog entry
- For a findAndModify operation with postImage, we end up with the following oplog entries:
- T-1: reserved for potential forged post-image noop oplog entry
- T: update oplog entry
- T': pre-image noop oplog entry
This invariant failure can be reproduced by applying the attached diffs (containing code taken from reserveOplogSlotsForRetryableFindAndModify()) and running the following test cases in OpObserverTest/TestFundamentalOnUpdateOutputs.
2021-11-02T06:52:46.159Z I TEST [thread1] UpdateTestCase {"ImageType":"PreImage","PreImageRecording":true,"ChangeStreamPreAndPostImagesEnabled":false,"RetryableOptions":"Images in side collection","ExpectedOplogEntries":2}
...
2021-11-02T06:52:46.159Z I TEST [thread1] Reserved {"slots":[{"ts":{"$timestamp":{"t":1635835966,"i":27}},"t":0},{"ts":{"$timestamp":{"t":1635835966,"i":28}},"t":0}]}
2021-11-02T06:52:46.159Z I REPL [thread1] replLogUpdate {"preImage noop":{"ts":{"$timestamp":{"t":1635835966,"i":29}},"t":0}}
2021-11-02T06:52:46.159Z I REPL [thread1] replLogUpdate {"oplogSlot":{"ts":{"$timestamp":{"t":1635835966,"i":28}},"t":0}}
2021-11-02T06:52:46.159Z F ASSERT [thread1] Invariant failure {"expr":"ot >= _lastOp","file":"src/mongo/db/repl/repl_client_info.cpp","line":64}
2021-11-02T08:05:39.296Z I TEST [main] UpdateTestCase {"ImageType":"PostImage","PreImageRecording":true,"ChangeStreamPreAndPostImagesEnabled":false,"RetryableOptions":"Images in side collection","ExpectedOplogEntries":2}
...
2021-11-02T08:05:39.296Z I TEST [main] Reserved {"slots":[{"ts":{"$timestamp":{"t":1635840339,"i":42}},"t":0},{"ts":{"$timestamp":{"t":1635840339,"i":43}},"t":0}]}
2021-11-02T08:05:39.296Z I REPL [main] replLogUpdate {"preImage noop":{"ts":{"$timestamp":{"t":1635840339,"i":44}},"t":0}}
2021-11-02T08:05:39.296Z I REPL [main] replLogUpdate {"oplogSlot":{"ts":{"$timestamp":{"t":1635840339,"i":43}},"t":0}}
2021-11-02T08:05:39.296Z F ASSERT [main] Invariant failure {"expr":"ot >= _lastOp","file":"src/mongo/db/repl/repl_client_info.cpp","line":64}
- is related to
-
SERVER-61498 Add reserveOplogSlots table from SERVER-61188 into architecture guide
-
- Closed
-
- related to
-
SERVER-81080 rename OplogDeleteEntryArgs::oplogSlots to clarify retryable findAndModify usage
-
- Closed
-