[SERVER-43882] Building indexes for startup recovery uses unowned RecordData after yielding its cursor Created: 08/Oct/19  Updated: 29/Oct/23  Resolved: 08/Nov/19

Status: Closed
Project: Core Server
Component/s: Index Maintenance, Storage
Affects Version/s: 4.2.0
Fix Version/s: 4.3.1, 4.2.2

Type: Bug Priority: Major - P3
Reporter: Louis Williams Assignee: Eric Milkie
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Related
related to SERVER-44904 Startup recovery should not delete co... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v4.2
Sprint: Execution Team 2019-11-04, Execution Team 2019-11-18
Participants:

 Description   
Issue Status as of Jan 6, 2020

ISSUE DESCRIPTION AND IMPACT
The code for building indexes during the restart of a replica set secondary accesses an un-owned memory object. If the memory backing the object has been re-used by a new owner, the index build process accesses unexpected memory locations and is likely to fail when parsing BSON in those locations.

This causes documents to be treated as corrupt when they are not, and can trigger SERVER-44904 which removes the documents without removing keys from all indexes that reference removed documents. This creates index inconsistencies on the node.

In the extremely unlikely event that consistent BSON is parsed from inconsistent memory, index keys themselves can become corrupted and will not reflect values in the document. This would cause incorrect results for queries that use the new index.

DIAGNOSIS AND AFFECTED VERSIONS
Replica sets running versions 4.2.0 and 4.2.1 are affected by this bug.

The presence of "Invalid BSON detected" errors in the server process logs during startup initialization indicate that this bug has had an impact, and that documents have been lost on that node. This data inconsistency between nodes can lead to data loss.

The validate command detects index inconsistencies and key corruption.

REMEDIATION AND WORKAROUNDS
The fix is included in the 4.2.2 production release. All users running 4.2.0 or 4.2.1 should upgrade to this version.

If a Secondary node crashes while building an index, do not restart the node immediately. Instead, perform an upgrade to MongoDB 4.2.2 starting with the impacted node.

If you see "Invalid BSON detected" errors logged on a replica set secondary that was restarted during an index build, resync that node from an un-impacted node.

Original description

startBuildingIndexesForRecovery uses an unowned RecordData after saving, committing, and restoring its cursor.

This is problematic because the RecordData can point to freed or overwritten memory.



 Comments   
Comment by Githook User [ 25/Nov/19 ]

Author:

{'name': 'Eric Milkie', 'username': 'milkie', 'email': 'milkie@mongodb.com'}

Message: SERVER-43882 startBuildingIndexForRecovery needs to own record data prior to yielding

(cherry picked from commit 4b92498a94f42790d2f6a3f26a965acb1de4702f)
Branch: v4.2
https://github.com/mongodb/mongo/commit/253caebbb4531e4852d18446c061de6f4e21f074

Comment by Githook User [ 08/Nov/19 ]

Author:

{'username': 'milkie', 'email': 'milkie@mongodb.com', 'name': 'Eric Milkie'}

Message: SERVER-43882 startBuildingIndexForRecovery needs to own record data prior to yielding
Branch: master
https://github.com/mongodb/mongo/commit/4b92498a94f42790d2f6a3f26a965acb1de4702f

Generated at Thu Feb 08 05:04:22 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.