[SERVER-71950] Fail and log the operation when out-of-order keys are detected in WiredTiger Created: 07/Dec/22  Updated: 29/Oct/23  Resolved: 31/Jan/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 4.2.24, 4.4.19, 5.0.15, 6.3.0-rc0, 6.0.5

Type: Bug Priority: Major - P3
Reporter: Will Korteland Assignee: Gregory Wlodarek
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Related
is related to SERVER-76232 Do not crash when index inconsistenci... Closed
is related to SERVER-72089 Check that keys are returned in decre... Closed
is related to SERVER-73481 Validate does not detect out-of-order... Closed
Assigned Teams:
Storage Execution
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v6.2, v6.0, v5.0, v4.4, v4.2
Sprint: Execution Team 2022-12-26, Execution Team 2023-02-06
Participants:

 Description   

During an index rebuild, IndexBuildsManager::startBuildingIndexForRecovery will go back to the beginning in the case of a write conflict exception in a partial batch. This can lead to "infinite looping" behaviour for some classes of cursor problems (see HELP-40283).

It should do something that makes failures a bit clearer, e.g. having a retry limit if the occasional write conflict is expected, or possibly aborting the build for that index.

This issue can occur when data corruption is present such that keys returned from the storage engine are not in ascending order.

Update
When an out-of-order key is detected, we will no longer throw a WriteConflictException when performing a collection scan. The DataCorruptionDetected error will be thrown instead. For v6.3 and up a new health log entry will be inserted into the local.system.healthlog collection that includes both keys, namespace, ident, scan direction and a stack trace.



 Comments   
Comment by Githook User [ 11/Feb/23 ]

Author:

{'name': 'Gregory Wlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'username': 'GWlodarek'}

Message: SERVER-71950 Fail and log the operation when out-of-order keys are detected in WiredTiger

(cherry picked from commit b2085e2d422278733bb6925614eb0c2f36348034)
Branch: v4.2
https://github.com/mongodb/mongo/commit/2c754d856173e399f21e2c2d8164b8cd319f5c19

Comment by Githook User [ 10/Feb/23 ]

Author:

{'name': 'Gregory Wlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'username': 'GWlodarek'}

Message: SERVER-71950 Fail and log the operation when out-of-order keys are detected in WiredTiger

(cherry picked from commit b2085e2d422278733bb6925614eb0c2f36348034)
Branch: v4.4
https://github.com/mongodb/mongo/commit/835768e45b76a71cc9e12f6684e51a6d1d8dcc87

Comment by Githook User [ 10/Feb/23 ]

Author:

{'name': 'Gregory Wlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'username': 'GWlodarek'}

Message: SERVER-71950 Fail and log the operation when out-of-order keys are detected in WiredTiger

(cherry picked from commit b2085e2d422278733bb6925614eb0c2f36348034)
Branch: v6.0
https://github.com/mongodb/mongo/commit/befb5b1bf6b71bd168f73467d1c3d7c66d737ed2

Comment by Githook User [ 10/Feb/23 ]

Author:

{'name': 'Gregory Wlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'username': 'GWlodarek'}

Message: SERVER-71950 Fail and log the operation when out-of-order keys are detected in WiredTiger

(cherry picked from commit b2085e2d422278733bb6925614eb0c2f36348034)
Branch: v5.0
https://github.com/mongodb/mongo/commit/94ed9950c2a163de5715820e3268f2b9e7d42277

Comment by Githook User [ 31/Jan/23 ]

Author:

{'name': 'Gregory Wlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'username': 'GWlodarek'}

Message: SERVER-71950 Fail and log the operation when out-of-order keys are detected in WiredTiger
Branch: master
https://github.com/mongodb/mongo/commit/b2085e2d422278733bb6925614eb0c2f36348034

Comment by Githook User [ 03/Jan/23 ]

Author:

{'name': 'Gregory Wlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'username': 'GWlodarek'}

Message: Revert "SERVER-71950 Throw a fatal assertion when out-of-order keys are detected in WiredTiger"

This reverts commit 6abf3a43cd8419d914627cbeb9fa54e8d2a46f11.
Branch: master
https://github.com/mongodb/mongo/commit/a191242084142413fd33214285ca7180f03ff074

Comment by Githook User [ 13/Dec/22 ]

Author:

{'name': 'Gregory Wlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'username': 'GWlodarek'}

Message: SERVER-71950 Throw a fatal assertion when out-of-order keys are detected in WiredTiger
Branch: master
https://github.com/mongodb/mongo/commit/6abf3a43cd8419d914627cbeb9fa54e8d2a46f11

Comment by Will Korteland [ 07/Dec/22 ]

(cc'ing gregory.wlodarek@mongodb.com just so you know I ended up making the ticket)

Generated at Thu Feb 08 06:20:25 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.