[SERVER-70879] Fix race where multiple threads are turning an index multikey concurrently Created: 26/Oct/22  Updated: 29/Oct/23  Resolved: 04/Nov/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 6.1.1, 5.0.14, 6.0.3, 6.2.0-rc0

Type: Bug Priority: Major - P3
Reporter: Fausto Leyva (Inactive) Assignee: Henrik Edin
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v6.1, v6.0, v5.0
Sprint: Execution Team 2022-11-14
Participants:
Linked BF Score: 115

 Description   

In BF-26686 we see an inconsistent multikey state in the in-memory catalog.
From the comments under BF-26686 we witness two (presumably) concurrent inserts occur:

{ “_id” : ObjectId(“633a9b678c7aa637fbd079a6"), “a” : [ { “b” : [ 2 ] } ] }
{ “_id” : ObjectId(“633a9b678c7aa637fbd079ac”), “a” : { “b” : [ -1, -2 ] } }

Notably, the first insert will set the multikey path for a and b while the second document will only set b. Ultimately, we only have b as a multikey path (on disk).

We potentially have a situation where writers could be obtaining copies of the collection metadata before it is marked as multikey and the second writer could overwrite the other writer’s collection metadata write. 

This ticket would include writing a detailed c++ test to verify the findings and potential bug.



 Comments   
Comment by Githook User [ 09/Nov/22 ]

Author:

{'name': 'Henrik Edin', 'email': 'henrik.edin@mongodb.com', 'username': 'henrikedin'}

Message: SERVER-70879 Fix race where multiple threads are turning an index multikey concurrently

Fix race where the writes to the durable catalog are serialized but the second writer is using a stale in-memory state due to the commit handler not being executed yet. This interleaving does not cause a write conflict as the writes are serialized but the second writer overwrote the multikey paths set by the first writer.

To handle it we always read the latest state from the durable catalog before performing multikey writes. This guarantees that we always are operating on the latest state. Some care was needed to ensure index offsets remain stable.

(cherry picked from commit 74bf9118e38ff4ee63db8bb17146dabe3ef07392)
Branch: v5.0
https://github.com/mongodb/mongo/commit/defa7826e96929cabbaa204f91a39d6bd6e30239

Comment by Githook User [ 08/Nov/22 ]

Author:

{'name': 'Henrik Edin', 'email': 'henrik.edin@mongodb.com', 'username': 'henrikedin'}

Message: SERVER-70879 Fix race where multiple threads are turning an index multikey concurrently

Fix race where the writes to the durable catalog are serialized but the second writer is using a stale in-memory state due to the commit handler not being executed yet. This interleaving does not cause a write conflict as the writes are serialized but the second writer overwrote the multikey paths set by the first writer.

To handle it we always read the latest state from the durable catalog before performing multikey writes. This guarantees that we always are operating on the latest state. Some care was needed to ensure index offsets remain stable.

(cherry picked from commit 74bf9118e38ff4ee63db8bb17146dabe3ef07392)
Branch: v6.1
https://github.com/mongodb/mongo/commit/7465f99eeb0a3c5ff05dc70a2e4e71de2f98ffd8

Comment by Githook User [ 08/Nov/22 ]

Author:

{'name': 'Henrik Edin', 'email': 'henrik.edin@mongodb.com', 'username': 'henrikedin'}

Message: SERVER-70879 Fix race where multiple threads are turning an index multikey concurrently

Fix race where the writes to the durable catalog are serialized but the second writer is using a stale in-memory state due to the commit handler not being executed yet. This interleaving does not cause a write conflict as the writes are serialized but the second writer overwrote the multikey paths set by the first writer.

To handle it we always read the latest state from the durable catalog before performing multikey writes. This guarantees that we always are operating on the latest state. Some care was needed to ensure index offsets remain stable.

(cherry picked from commit 74bf9118e38ff4ee63db8bb17146dabe3ef07392)
Branch: v6.0
https://github.com/mongodb/mongo/commit/ac5d0034debcc646ffe610f8d63302c25a3598b7

Comment by Githook User [ 04/Nov/22 ]

Author:

{'name': 'Henrik Edin', 'email': 'henrik.edin@mongodb.com', 'username': 'henrikedin'}

Message: SERVER-70879 Fix race where multiple threads are turning an index multikey concurrently

Fix race where the writes to the durable catalog are serialized but the second writer is using a stale in-memory state due to the commit handler not being executed yet. This interleaving does not cause a write conflict as the writes are serialized but the second writer overwrote the multikey paths set by the first writer.

To handle it we always read the latest state from the durable catalog before performing multikey writes. This guarantees that we always are operating on the latest state. Some care was needed to ensure index offsets remain stable.
Branch: master
https://github.com/mongodb/mongo/commit/74bf9118e38ff4ee63db8bb17146dabe3ef07392

Generated at Thu Feb 08 06:17:23 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.