[SERVER-48606] Unhandled exception in WTCheckpointThread Created: 05/Jun/20  Updated: 05/Jun/20  Resolved: 05/Jun/20

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 4.2.7
Fix Version/s: None

Type: Bug Priority: Critical - P2
Reporter: Diana Zhivkova Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Windows Server 2019 Standard

Dell Server PER540

Processor: Intel(R) Xeon(R) Silver 4212 CPU @ 2.20GHz (2 processors)

RAM: 128 GB

Hard Disk: Dell PERC H740P Apd SCSI


Issue Links:
Duplicate
duplicates WT-6366 Off-by-one overflow in block-modifica... Closed
Operating System: ALL
Participants:

 Description   

After upgrading our nodes to 4.2.7 we've been getting unhandled exceptions sporadically. The mongod.exe ends up terminating and this is affecting terribly our live environment. 

The exception from the mongodb.log is as follows:

2020-06-04T06:29:59.276-0500 F CONTROL [WTCheckpointThread] *** unhandled exception (access violation) at 0x00007FF649BBB29A, terminating
2020-06-04T06:29:59.276-0500 F CONTROL [WTCheckpointThread] *** access violation was a read from 0x2667ffeee
2020-06-04T06:29:59.276-0500 F CONTROL [WTCheckpointThread] *** stack trace for unhandled exception:
2020-06-04T06:30:00.611-0500 I - [WTCheckpointThread] mongod.exe ...\src\third_party\gperftools-2.7\dist\src\tcmalloc.cc(1972) tc_calloc+0x9a
mongod.exe ...\src\third_party\wiredtiger\src\os_common\os_alloc.c(50) __wt_calloc+0x5a
mongod.exe ...\src\third_party\wiredtiger\src\block\block_session.c(34) __block_ext_alloc+0x57
mongod.exe ...\src\third_party\wiredtiger\src\block\block_ext.c(233) __block_off_insert+0x2a
mongod.exe ...\src\third_party\wiredtiger\src\block\block_ext.c(1005) __block_merge+0x1b0
mongod.exe ...\src\third_party\wiredtiger\src\block\block_ext.c(894) __wt_block_extlist_merge+0x30a
mongod.exe ...\src\third_party\wiredtiger\src\block\block_ckpt.c(654) __ckpt_process+0x3b0
mongod.exe ...\src\third_party\wiredtiger\src\block\block_ckpt.c(255) __wt_block_checkpoint+0xa4
mongod.exe ...\src\third_party\wiredtiger\src\btree\bt_io.c(346) __wt_bt_write+0x3cc
mongod.exe ...\src\third_party\wiredtiger\src\reconcile\rec_write.c(2303) __rec_write_wrapup+0x5f1
mongod.exe ...\src\third_party\wiredtiger\src\reconcile\rec_write.c(209) __reconcile+0x2e2
mongod.exe ...\src\third_party\wiredtiger\src\reconcile\rec_write.c(102) __wt_reconcile+0x176
mongod.exe ...\src\third_party\wiredtiger\src\reconcile\rec_write.c(516) __rec_root_write+0x20b
mongod.exe ...\src\third_party\wiredtiger\src\reconcile\rec_write.c(278) __reconcile+0x5ad
mongod.exe ...\src\third_party\wiredtiger\src\reconcile\rec_write.c(102) __wt_reconcile+0x176
mongod.exe ...\src\third_party\wiredtiger\src\btree\bt_sync.c(311) __wt_sync_file+0x620
mongod.exe ...\src\third_party\wiredtiger\src\txn\txn_ckpt.c(1578) __checkpoint_tree+0x169
mongod.exe ...\src\third_party\wiredtiger\src\txn\txn_ckpt.c(1686) __checkpoint_tree_helper+0x55
mongod.exe ...\src\third_party\wiredtiger\src\txn\txn_ckpt.c(199) __checkpoint_apply+0x56
mongod.exe ...\src\third_party\wiredtiger\src\txn\txn_ckpt.c(848) __txn_checkpoint+0x50d
mongod.exe ...\src\third_party\wiredtiger\src\txn\txn_ckpt.c(1043) __txn_checkpoint_wrapper+0x189
mongod.exe ...\src\third_party\wiredtiger\src\txn\txn_ckpt.c(1101) __wt_txn_checkpoint+0x112
mongod.exe ...\src\third_party\wiredtiger\src\session\session_api.c(1962) __session_checkpoint+0x3bc
mongod.exe ...\src\mongo\db\storage\wiredtiger\wiredtiger_kv_engine.cpp(375) mongo::WiredTigerKVEngine::WiredTigerCheckpointThread::run+0x78a
mongod.exe ...\src\mongo\util\background.cpp(154) mongo::BackgroundJob::jobBody+0x145
mongod.exe c:\program files (x86)\microsoft visual studio\2017\professional\vc\tools\msvc\14.16.27023\include\thr\xthread(230) std::_LaunchPad<std::unique_ptr<std::tuple<<lambda_e94772a372767cbd7bebbcdd24044cf3> >,std::default_delete<std::tuple<<lambda_e94772a372767cbd7bebbcdd24044cf3> > > > >::_Go+0x6d
mongod.exe c:\program files (x86)\microsoft visual studio\2017\professional\vc\tools\msvc\14.16.27023\include\thr\xthread(209) std::_Pad::_Call_func+0x9
ucrtbase.dll o_exp+0x5a
KERNEL32.DLL BaseThreadInitThunk+0x14
2020-06-04T06:30:00.612-0500 I CONTROL [WTCheckpointThread] writing minidump diagnostic file C:\MMSAutomation\versions\mongodb-win32-x86_64-4.2.7-ent\bin\mongod.2020-06-04T11-30-00.mdmp
2020-06-04T06:30:01.395-0500 I NETWORK [listener] connection accepted from 169.254.1.2:64568 #6240 (97 connections now open)
{{2020-06-04T06:30:01.395-0500 I NETWORK [conn6240] received client metadata from 169.254.1.2:64568 conn6240: { driver:

{ name: "mongo-go-driver", version: "v1.1.4" }

, os: { type: "windows", architecture: "amd64" }, platform: "go1.14.2" }}}
2020-06-04T06:30:01.401-0500 F CONTROL [WTCheckpointThread] *** immediate exit due to unhandled exception



 Comments   
Comment by Keith Smith [ 05/Jun/20 ]

pasette. That's correct, WT-6366 should only happen when incremental backup is used; It doesn't matter whether it's invoked via ops mgr.  I can't tell from the error whether the customer has incrementals enabled, but the stack trace is exactly what we've seen in other WT-6366 failures.

Comment by Diana Zhivkova [ 05/Jun/20 ]

Confirmed. Downgrading to 4.6.2 removes the issues.

Thanks

Comment by Daniel Pasette (Inactive) [ 05/Jun/20 ]

To workaround until 4.2.8 is released, please downgrade to 4.2.6

Comment by Alexander Gorrod [ 05/Jun/20 ]

Hi diana@orionswave.com - thanks for reporting this issue, and we are sorry that you encountered it. I believe it is a duplicate of WT-6366 - which we have a fix in process for, and will be available in an upcoming MongoDB 4.2.8 release. Please follow that ticket to track the progress of the fix.

Generated at Thu Feb 08 05:17:35 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.