ISSUE DESCRIPTION AND IMPACT
The code for building indexes during the restart of a replica set secondary accesses an un-owned memory object. If the memory backing the object has been re-used by a new owner, the index build process accesses unexpected memory locations and is likely to fail when parsing BSON in those locations.
This causes documents to be treated as corrupt when they are not, and can trigger
SERVER-44904 which removes the documents without removing keys from all indexes that reference removed documents. This creates index inconsistencies on the node.
In the extremely unlikely event that consistent BSON is parsed from inconsistent memory, index keys themselves can become corrupted and will not reflect values in the document. This would cause incorrect results for queries that use the new index.
DIAGNOSIS AND AFFECTED VERSIONS
Replica sets running versions 4.2.0 and 4.2.1 are affected by this bug.
The presence of "Invalid BSON detected" errors in the server process logs during startup initialization indicate that this bug has had an impact, and that documents have been lost on that node. This data inconsistency between nodes can lead to data loss.
The validate command detects index inconsistencies and key corruption.
REMEDIATION AND WORKAROUNDS
The fix is included in the 4.2.2 production release. All users running 4.2.0 or 4.2.1 should upgrade to this version.
If a Secondary node crashes while building an index, do not restart the node immediately. Instead, perform an upgrade to MongoDB 4.2.2 starting with the impacted node.
If you see "Invalid BSON detected" errors logged on a replica set secondary that was restarted during an index build, resync that node from an un-impacted node.