[SERVER-23925] Potential use-after-free when WSM is refetched after snapshot id changes in UpdateStage and DeleteStage Created: 26/Apr/16 Updated: 02/Dec/16 Resolved: 02/May/16 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Querying, Write Ops |
| Affects Version/s: | 3.3.4 |
| Fix Version/s: | 3.3.6 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Igor Canadi | Assignee: | Max Hirschhorn |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||
| Operating System: | ALL | ||||||||||||||||
| Sprint: | Query 14 (05/13/16) | ||||||||||||||||
| Participants: | |||||||||||||||||
| Linked BF Score: | 0 | ||||||||||||||||
| Description |
|
MongoRocks tests have been failing for a while with weird errors. I just ran an ASAN test and I think I discovered the root cause. I don't think it's specific to RocksDB storage engine, so I'm surprised that it's not failing in the WiredTiger case. Here's the stack trace of use-after-free: https://gist.github.com/igorcanadi/76235761eb90f1e1a23b2f35b1627b90 Bear with me:
I believe that the offending commit may be https://github.com/mongodb/mongo/commit/178e241b81882f85a58deda960d80607a77e1c3a (based on 'git blame', I haven't actually bisected it) Does this make sense? |
| Comments |
| Comment by Githook User [ 02/May/16 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Author: {u'username': u'visemet', u'name': u'Max Hirschhorn', u'email': u'max.hirschhorn@mongodb.com'}Message: | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Max Hirschhorn [ 26/Apr/16 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Thanks for tracking this one down igor! Using a patch similar to the one I used while doing
The changes from 178e241 inadvertently made it so that WorkingSetMember::makeObjOwnedIfNeeded() is called after cursor (i.e. the SeekableRecordCursor) was destructed. This leads to the heap-use-after-free that Igor described. Note: A similar issue affects the DeleteStage in addition to the UpdateStage. After applying the following patch, I can no longer reproduce the issue.
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Andrew Morrow (Inactive) [ 26/Apr/16 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Thanks, yes that is what I was after. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Igor Canadi [ 26/Apr/16 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
That's awesome, thanks! I do have an error report, I have a link in the original comment: https://gist.github.com/igorcanadi/76235761eb90f1e1a23b2f35b1627b90. Is this what you're asking for? | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Andrew Morrow (Inactive) [ 26/Apr/16 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
igor - Per our email discussion, I had actually just launched an ASAN/RocksDB test run when your email came in. I'll let you know if it shows results consistent with what you are seeing. Do you have an ASAN error report you can paste in? | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Igor Canadi [ 26/Apr/16 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
MongoRocks tests started failing between these two commits (according to evergreen):
The commit that I blame in the original ticket (https://github.com/mongodb/mongo/commit/178e241b81882f85a58deda960d80607a77e1c3a) is actually between the two commits! |