[SERVER-21568]  use after free in bt_split.c Created: 19/Nov/15  Updated: 07/Dec/15  Resolved: 20/Nov/15

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: 3.2.0-rc3
Fix Version/s: 3.2.0-rc4

Type: Bug Priority: Major - P3
Reporter: Michael Grundy Assignee: Michael Cahill (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File d20000-afterfix-3.log     File ett_load.diff    
Issue Links:
Depends
depends on WT-2224 backup_restore.js fails with tcmalloc... Closed
is depended on by SERVER-21442 WiredTiger changes for MongoDB 3.0.8 Closed
is depended on by WT-1973 MongoDB changes for WiredTiger 2.7.0 Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Steps To Reproduce:

Apply the attached patch that contains the election_timing_workload test and associated framework changes.

run:

./mongo --nodb jstests/replsets/election_timing_workload.js

Participants:

 Description   

Running election timing tests with a workload, the Primary will crash, frequently, after oplog truncation. Output from an ASAN build:

2187817-d20000| 2015-11-19T12:01:29.785-0500 D STORAGE  [WT RecordStoreThread: local.oplog.rs] Truncating the oplog between RecordId(6218908474747668007) and RecordId(6218908479042628027) to remove approximately 20263 records totaling to 4194441 bytes
2188061-d20000| 2015-11-19T12:01:29.786-0500 D STORAGE  [WT RecordStoreThread: local.oplog.rs] Finished truncating the oplog, it now contains approximately 206347 records totaling to 42713539 bytes
2188251-d20000| 2015-11-19T12:01:30.007-0500 D STORAGE  [ftdc] WT queued drop of  table:collection-11-570436650327984530 res 16
2188371-d20000| 2015-11-19T12:01:30.007-0500 D STORAGE  [ftdc] WT queued drop of  table:index-12-570436650327984530 res 16
2188486-d20000| 2015-11-19T12:01:30.007-0500 D STORAGE  [ftdc] WT queued drop of  table:index-13-570436650327984530 res 16
2188601-d20000| 2015-11-19T12:01:30.008-0500 D STORAGE  [ftdc] WT queued drop of  table:index-14-570436650327984530 res 16
2188716-d20000| 2015-11-19T12:01:30.008-0500 D STORAGE  [ftdc] WT queued drop of  table:index-15-570436650327984530 res 16
2188831-d20000| =================================================================
2188905:d20000| ==27065== ERROR: AddressSanitizer: heap-use-after-free on address 0x6004000a4138 at pc 0x3f0f620 bp 0x7f05d763e290 sp 0x7f05d763e288
2189046-d20000| READ of size 8 at 0x6004000a4138 thread T79 
2189098-d20000|     #0 0x3f0f61f in __split_parent /home/grund/MongoDB/replcrsh/src/third_party/wiredtiger/src/btree/bt_split.c:887
2189222-d20000|     #1 0x3f1514a in __split_multi /home/grund/MongoDB/replcrsh/src/third_party/wiredtiger/src/btree/bt_split.c:1935
2189346-d20000|     #2 0x3f1514a in __wt_split_multi /home/grund/MongoDB/replcrsh/src/third_party/wiredtiger/src/btree/bt_split.c:1978
2189473-d20000|     #3 0x3fec912 in __evict_page_dirty_update /home/grund/MongoDB/replcrsh/src/third_party/wiredtiger/src/evict/evict_page.c:293
2189610-d20000|     #4 0x3fec912 in __wt_evict /home/grund/MongoDB/replcrsh/src/third_party/wiredtiger/src/evict/evict_page.c:123
2189732-d20000|     #5 0x3fe2b29 in __evict_page /home/grund/MongoDB/replcrsh/src/third_party/wiredtiger/src/evict/evict_lru.c:1467
2189856-d20000|     #6 0x3fe3719 in __evict_lru_pages /home/grund/MongoDB/replcrsh/src/third_party/wiredtiger/src/evict/evict_lru.c:818
2189984-d20000|     #7 0x3fe3719 in __evict_worker /home/grund/MongoDB/replcrsh/src/third_party/wiredtiger/src/evict/evict_lru.c:434
2190109-d20000|     #8 0x7f0608a96b97 (/usr/lib/x86_64-linux-gnu/libasan.so.0+0x18b97)
2190188-d20000|     #9 0x7f0607c3c181 in start_thread /build/buildd/eglibc-2.19/nptl/pthread_create.c:312
2190286-d20000|     #10 0x7f060796947c in clone /build/buildd/eglibc-2.19/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:111
2190403-d20000| 0x6004000a4138 is located 8 bytes inside of 16-byte region [0x6004000a4130,0x6004000a4140)

db version v3.2.0-rc3-49-ga0771ea
git version: a0771ea5ec1b44537d3c409e3d712db24fd8e6bb

mongod has the fix for WT-2224



 Comments   
Comment by Githook User [ 02/Dec/15 ]

Author:

{u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

Message: Import wiredtiger-wiredtiger-mongodb-3.0.7-20-g27d0cbd.tar.gz from wiredtiger branch mongodb-3.0

ref: deb2d81..27d0cbd

6feaa28 WT-2251 Fix leaf of deleted page addresses.
1da2d3a SERVER-21568 Fix a use-after-free.
4228295 SERVER-21553 Free blocks during reverse splits.
0398515 WT-2241 Use a lock to protect transaction ID allocation.
06a5c7b WT-2237 Avoid yields if we race allocating transaction IDs.
16a418b WT-2237 Have threads publish unique transaction IDs so that updates always become visible immediately on commit.
Branch: v3.0
https://github.com/mongodb/mongo/commit/eb8bc24a0d36b673022faa83976a4b33bc8f676b

Comment by Githook User [ 02/Dec/15 ]

Author:

{u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'}

Message: SERVER-21568 Fix a use after-free

(cherry picked from commit 2d01a566)

Conflicts:
src/btree/bt_split.c
Branch: mongodb-3.0
https://github.com/wiredtiger/wiredtiger/commit/1da2d3a517392d71e13b7ecdeaa97b01a2627d2d

Comment by Michael Cahill (Inactive) [ 20/Nov/15 ]

Should be fixed by https://github.com/mongodb/mongo/commit/fb959fb6020ade800db8674b598a04a1ab4c7e67

Comment by Githook User [ 20/Nov/15 ]

Author:

{u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'}

Message: Merge pull request #2323 from wiredtiger/SERVER-21568

SERVER-21568 Fix a use after-free
Branch: develop
https://github.com/wiredtiger/wiredtiger/commit/2d01a566ba7b3576cb5b45af55479b3c06995589

Comment by Githook User [ 20/Nov/15 ]

Author:

{u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'}

Message: Merge pull request #2323 from wiredtiger/SERVER-21568

SERVER-21568 Fix a use after-free
Branch: develop
https://github.com/wiredtiger/wiredtiger/commit/2d01a566ba7b3576cb5b45af55479b3c06995589

Comment by Githook User [ 20/Nov/15 ]

Author:

{u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

Message: SERVER-21568 Free page_del information in refs causing reverse splits.
Branch: develop
https://github.com/wiredtiger/wiredtiger/commit/d7ac553f8e4ac466b7f48f086345b034a8b33a23

Comment by Githook User [ 20/Nov/15 ]

Author:

{u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

Message: SERVER-21568 Free deleted page structures as soon as possible.

The ref->page_del structure can be freed as soon as it becomes globally visible. Do that when we have the ref locked to avoid unnecessarily spinning on the page state.
Branch: develop
https://github.com/wiredtiger/wiredtiger/commit/ead5ddc8e2c32af7e3e7eeaeacc6ffdf7986833b

Comment by Githook User [ 20/Nov/15 ]

Author:

{u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

Message: SERVER-21568 Fix a use after-free.

If a page was truncated, then subsequently there were more inserts onto that page so it was reinstantiated and the page ended up splitting, WiredTiger would attempt to free a data structure twice.
Branch: develop
https://github.com/wiredtiger/wiredtiger/commit/244aca5ca73e694b809e551ddf2e691a9d040aac

Comment by Michael Cahill (Inactive) [ 20/Nov/15 ]

CR: https://github.com/wiredtiger/wiredtiger/pull/2323

Comment by Michael Grundy [ 19/Nov/15 ]

Rolling back the majority write that was in flight when the primary was stopped might be related. I wasn't able to reproduce this in a simpler test case that didn't have the primary being stopped.

Comment by Daniel Pasette (Inactive) [ 19/Nov/15 ]

WT-2224?

Generated at Thu Feb 08 03:57:44 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.