[SERVER-43883] Enable WiredTiger cursor copy debug mode in asan builds Created: 08/Oct/19  Updated: 29/Oct/23  Resolved: 06/Apr/20

Status: Closed
Project: Core Server
Component/s: Internal Code
Affects Version/s: None
Fix Version/s: 4.4.0-rc1, 4.7.0

Type: Improvement Priority: Major - P3
Reporter: Louis Williams Assignee: Daniel Gottlieb (Inactive)
Resolution: Fixed Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
depends on WT-5574 Rolling back a prepared transaction w... Closed
depends on WT-5588 Only release cursor memory allocated ... Closed
Related
related to WT-5165 Add cursor copy debug mode Closed
Backwards Compatibility: Fully Compatible
Backport Requested:
v4.4
Sprint: Execution Team 2020-02-24, Execution Team 2020-04-20
Participants:

 Description   

There is a class of bug that results in using unowned BSON or RecordData objects, which reference memory owned by something else, and accessing that memory after it has been freed or overwritten. This has the potential to lead to undefined behavior and in-memory data corruption.

See SERVER-42744, SERVER-43879, SERVER-43880, and SERVER-43882 for recent examples.

We should enable WT cursor copy debug mode with the configuration "debug_mode=(cursor_copy=true)" on the call to wiredtiger_open.

 

Old Description:
We should create a build variant or suite that does the following:

  • Before freeing owned BSONObjs, overwrite the memory with garbage. This will blow up when an unowned BSONObjs is used after the owned object has been freed.
  • When returning data from a cursor, copy memory from WiredTiger into a managed buffer, and return that unowned buffer to the caller. When that cursor is invalidated in any way, from an advance, close, or reset, overwrite the buffer with garbage and free the memory immediately. This will blow up in cases where callers use data from cursors after repositioning or yielding. See an example implementation here.


 Comments   
Comment by Githook User [ 10/Apr/20 ]

Author:

{'name': 'Daniel Gottlieb', 'email': 'daniel.gottlieb@mongodb.com', 'username': 'dgottlieb'}

Message: SERVER-43883: Enable WTs cursor copy mode for ASAN builds.

When a cursor changes its position, accessing the address of the
previous key and value is illegal, but often works by accident in production.

Cursor copy mode will allocate memory for a cursor's keys and values when
positioned. When the cursor changes its position, that memory is freed. This
allows address sanitizer builds to catch bugs in MongoDBs use of WiredTiger.

(cherry picked from commit a5f2ec58c2360effcfeb338804bfa31dc4b7cff6)
Branch: v4.4
https://github.com/mongodb/mongo/commit/e31417bd9aab2cc9dd06531ad37944433f9a73d6

Comment by Githook User [ 06/Apr/20 ]

Author:

{'name': 'Daniel Gottlieb', 'email': 'daniel.gottlieb@mongodb.com', 'username': 'dgottlieb'}

Message: SERVER-43883: Enable WTs cursor copy mode for ASAN builds.

When a cursor changes its position, accessing the address of the
previous key and value is illegal, but often works by accident in production.

Cursor copy mode will allocate memory for a cursor's keys and values when
positioned. When the cursor changes its position, that memory is freed. This
allows address sanitizer builds to catch bugs in MongoDBs use of WiredTiger.
Branch: master
https://github.com/mongodb/mongo/commit/a5f2ec58c2360effcfeb338804bfa31dc4b7cff6

Comment by Eric Milkie [ 03/Apr/20 ]

The dependent tickets are now resolved so I'm moving this back to Open status.

Comment by Daniel Gottlieb (Inactive) [ 13/Feb/20 ]

Attempting to turn this on for ASAN builds found some bugs with the cursor copy code itself in some trickier scenarios. I'm moving this ticket to blocked.

Comment by Louis Williams [ 14/Jan/20 ]

Since WT-5165 is now complete, I am repurposing this ticket to enable cursor copy debug mode in WiredTiger, which will accomplish the goals I described.

Generated at Thu Feb 08 05:04:22 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.