[SERVER-71008] Fatal assertion 39001 BadValue: timestamp_transaction 22 Created: 02/Nov/22  Updated: 05/Dec/22  Resolved: 05/Dec/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 4.2.20
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Srikanta Patra Assignee: Yuan Fang
Resolution: Done Votes: 0
Labels: Bug
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-55894 WiredTiger error (22) and Fatal asser... Closed
related to SERVER-67790 [4.2] Running with enableMajorityRead... Closed
Operating System: ALL
Participants:

 Description   

Hi Team,

 

There was a issue with our mongodb cluster. We have 4 Node cluster (verison 4.2.20)

of which the primary got failover to one of the existing secondary server. 

 

Upon analysing the logs, we found the same issue that we had raised couple of months back as part of https://jira.mongodb.org/browse/SERVER-55894 .  

 

Below is the error log 

2022-11-02T08:02:34.615+0000 E  STORAGE  [conn30867] WiredTiger error (22) [1667376154:615441][23003:0x7fc88b557700], WT_SESSION.timestamp_transaction: __wt_txn_set_commit_timestamp, 676: commit timestamp (1667376154, 11928) is less than the oldest timestamp (1667376154, 11934): Invalid argument Raw: [1667376154:615441][23003:0x7fc88b557700], WT_SESSION.timestamp_transaction: __wt_txn_set_commit_timestamp, 676: commit timestamp (1667376154, 11928) is less than the oldest timestamp (1667376154, 11934): Invalid argument
2022-11-02T08:02:34.616+0000 F  -        [conn30867] Fatal assertion 39001 BadValue: timestamp_transaction 22: Invalid argument at src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp 1327
2022-11-02T08:02:34.616+0000 F  -        [conn30867] \n\n***aborting after fassert() failure\n\n
2022-11-02T08:02:34.690+0000 F  -        [conn30867] Got signal: 6 (Aborted).
 0x56236a09af21 0x56236a09a45e 0x56236a09a4f6 0x7fc8b5c5d8e0 0x7fc8b58d4ca0 0x7fc8b58d6148 0x562368448c1d 0x56236817470b 0x562368557c16 0x562368d79951 0x562368e12afc 0x562368e14765 0x562368bed0ce 0x5623684d3a3f 0x562368d7c7d8 0x562368c5f92c 0x562368c5fd10 0x562368c603c8 0x562368c529ff 0x562368c5059d 0x5623688ee37c 0x5623688ef325 0x5623688f136c 0x5623688f2132 0x5623688df48c 0x5623688ebf55 0x5623688e9737 0x5623688eafc5 0x56236979e102 0x5623688e6ccd 0x5623688e821d 0x5623688e8b66 0x5623688e96fb 0x5623688eafc5 0x56236979e587 0x562369dfee15 0x562369dfee74 0x7fc8b5c5344b 0x7fc8b598e40f
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"562367708000","o":"2992F21","s":"_ZN5mongo15printStackTraceERSo"},{"b":"562367708000","o":"299245E"},{"b":"562367708000","o":"29924F6"},{"b":"7FC8B5C4C000","o":"118E0"},{"b":"7FC8B58A1000","o":"33CA0","s":"gsignal"},{"b":"7FC8B58A1000","o":"35148","s":"abort"},{"b":"562367708000","o":"D40C1D","s":"_ZN5mongo42fassertFailedWithStatusNoTraceWithLocationEiRKNS_6StatusEPKcj"},{"b":"562367708000","o":"A6C70B"},{"b":"562367708000","o":"E4FC16","s":"_ZN5mongo21WiredTigerRecordStore26insertRecordsWithDocWriterEPNS_16OperationContextEPKPKNS_9DocWriterEPKNS_9TimestampEmPNS_8RecordIdE"},{"b":"562367708000","o":"1671951","s":"_ZN5mongo14CollectionImpl23insertDocumentsForOplogEPNS_16OperationContextEPKPKNS_9DocWriterEPNS_9TimestampEm"},{"b":"562367708000","o":"170AAFC","s":"_ZN5mongo4repl12_logOpsInnerEPNS_16OperationContextERKNS_15NamespaceStringEPKPKNS_9DocWriterEPNS_9TimestampEmPNS_10CollectionENS0_6OpTimeENS_6Date_tE"},{"b":"562367708000","o":"170C765","s":"_ZN5mongo4repl12logInsertOpsEPNS_16OperationContextERKNS_15NamespaceStringEN5boost8optionalINS_4UUIDEEEN9__gnu_cxx17__normal_iteratorIPKNS_15InsertStatementESt6vectorISC_SaISC_EEEESI_bNS_6Date_tE"},{"b":"562367708000","o":"14E50CE","s":"_ZN5mongo14OpObserverImpl9onInsertsEPNS_16OperationContextERKNS_15NamespaceStringEN5boost8optionalINS_4UUIDEEEN9__gnu_cxx17__normal_iteratorIPKNS_15InsertStatementESt6vectorISC_SaISC_EEEESI_b"},{"b":"562367708000","o":"DCBA3F","s":"_ZN5mongo18OpObserverRegistry9onInsertsEPNS_16OperationContextERKNS_15NamespaceStringEN5boost8optionalINS_4UUIDEEEN9__gnu_cxx17__normal_iteratorIPKNS_15InsertStatementESt6vectorISC_SaISC_EEEESI_b"},{"b":"562367708000","o":"16747D8","s":"_ZN5mongo14CollectionImpl15insertDocumentsEPNS_16OperationContextEN9__gnu_cxx17__normal_iteratorIPKNS_15InsertStatementESt6vectorIS5_SaIS5_EEEESB_PNS_7OpDebugEb"},{"b":"562367708000","o":"155792C"},{"b":"562367708000","o":"1557D10"},{"b":"562367708000","o":"15583C8","s":"_ZN5mongo14performInsertsEPNS_16OperationContextERKNS_9write_ops6InsertEb"},{"b":"562367708000","o":"154A9FF"},{"b":"562367708000","o":"154859D"},{"b":"562367708000","o":"11E637C"},{"b":"562367708000","o":"11E7325"},{"b":"562367708000","o":"11E936C"},{"b":"562367708000","o":"11EA132","s":"_ZN5mongo23ServiceEntryPointCommon13handleRequestEPNS_16OperationContextERKNS_7MessageERKNS0_5HooksE"},{"b":"562367708000","o":"11D748C","s":"_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE"},{"b":"562367708000","o":"11E3F55","s":"_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE"},{"b":"562367708000","o":"11E1737","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"562367708000","o":"11E2FC5"},{"b":"562367708000","o":"2096102","s":"_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE"},{"b":"562367708000","o":"11DECCD","s":"_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE"},{"b":"562367708000","o":"11E021D","s":"_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE"},{"b":"562367708000","o":"11E0B66","s":"_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE"},{"b":"562367708000","o":"11E16FB","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"562367708000","o":"11E2FC5"},{"b":"562367708000","o":"2096587"},{"b":"562367708000","o":"26F6E15"},{"b":"562367708000","o":"26F6E74"},{"b":"7FC8B5C4C000","o":"744B"},{"b":"7FC8B58A1000","o":"ED40F","s":"clone"}],"processInfo":{ "mongodbVersion" : "4.2.20", "gitVersion" : "15c0712952c356cb711c13a42cb3bca8617d4ebc", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.14.275-207.503.amzn2.x86_64", "version" : "#1 SMP Wed Apr 13 20:24:50 UTC 2022", "machine" : "x86_64" }, "somap" : [ { "b" : "562367708000", "elfType" : 3, "buildId" : "158183DD7D05EC87B0DE68A9D3A0278785590553" }, { "b" : "7FFEB0352000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "C43776337825FC5B24C9DA856E02319839982F93" }, { "b" : "7FC8B70A7000", "path" : "/lib64/libcurl.so.4", "elfType" : 3, "buildId" : "76E323C50375D93039B96955125C9F165AD3F11A" }, { "b" : "7FC8B6E91000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "E9DEEED0BDCA0711B57858014896D968FE2B5D27" }, { "b" : "7FC8B6A3B000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "FDF6260976321A07489C660F761CEBDE4CFD47D7" }, { "b" : "7FC8B67CC000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "7BE7054050B1BEEB0669CC3A5658FC79748DAD2B" }, { "b" : "7FC8B65C8000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "F15DE2F853AB538930719067AE37E75D4C3227DF" }, { "b" : "7FC8B63C0000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "AB9E6546305ABA17E6910218A3D959B2C7034938" }, { "b" : "7FC8B6080000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "DF9268F8221BCBF25207D9EB17581207C1F60062" }, { "b" : "7FC8B5E6A000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "A2082B3D4C249A26E0B21AB31DE4D251A07312C6" }, { "b" : "7FC8B5C4C000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "EB2DDD809A3C003790EBCBCB2B6F48DAD8047E54" }, { "b" : "7FC8B58A1000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "77254C111C51B5982752D1614611914631524291" }, { "b" : "7FC8B733E000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "32A2AF83D52A1E8C4444E52D5281A31294952343" }, { "b" : "7FC8B567A000", "path" : "/lib64/libnghttp2.so.14", "elfType" : 3, "buildId" : "ABE2627C22FDC7706352089D224CA49759CB74F9" }, { "b" : "7FC8B542B000", "path" : "/lib64/libidn2.so.0", "elfType" : 3, "buildId" : "979481B52FE3E6D03440A27BEEB8B42C9F358E03" }, { "b" : "7FC8B5203000", "path" : "/lib64/libssh2.so.1", "elfType" : 3, "buildId" : "DB7387A967A1DB497B643672C4E4DA5A9931D347" }, { "b" : "7FC8B4FB7000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "DEE4C16B9C3BBBBBF3D3E3A6947EA2918571BDEF" }, { "b" : "7FC8B4CD3000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "86B0F5A67C11C72D678DB9AF142B17DCF08D9FD0" }, { "b" : "7FC8B4AA2000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "A3D5952CD58619354C060E2B651490639A6C198C" }, { "b" : "7FC8B489E000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "A404F739EA284ED5BE28E9778857D8B016BBD657" }, { "b" : "7FC8B464C000", "path" : "/lib64/libldap-2.4.so.2", "elfType" : 3, "buildId" : "E0B0F4EA58494B2D558C56249972502BF76FAE07" }, { "b" : "7FC8B443D000", "path" : "/lib64/liblber-2.4.so.2", "elfType" : 3, "buildId" : "005BA2226678E21EAB03B457F1EFE2003791FCE9" }, { "b" : "7FC8B4228000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "89D3CC074FA2CAFE035EC68762FACC3EAC173E19" }, { "b" : "7FC8B3F10000", "path" : "/lib64/libunistring.so.0", "elfType" : 3, "buildId" : "DB84428D5AF12394B1E9DCE0DB7C36118C158C3A" }, { "b" : "7FC8B3D01000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "FC001245DC41D133866F7F7F6BBCFA3E2A375FDC" }, { "b" : "7FC8B3AFD000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "C966A5338211832BA7AF1F8EE75170F2FE2C378E" }, { "b" : "7FC8B38E1000", "path" : "/lib64/libsasl2.so.3", "elfType" : 3, "buildId" : "E0C9F3539BBD29B55B91EC5EF78C446E1F24AE7F" }, { "b" : "7FC8B3682000", "path" : "/lib64/libssl3.so", "elfType" : 3, "buildId" : "9F9FFDFA6BF1398B70FAAA1103599CC736B1C402" }, { "b" : "7FC8B345C000", "path" : "/lib64/libsmime3.so", "elfType" : 3, "buildId" : "2E3428F7750015CCC22F2460442A977CFF9E2413" }, { "b" : "7FC8B312F000", "path" : "/lib64/libnss3.so", "elfType" : 3, "buildId" : "E924C26079425FD432BDEFA4E6F1D19AC7A9A3ED" }, { "b" : "7FC8B2F00000", "path" : "/lib64/libnssutil3.so", "elfType" : 3, "buildId" : "15C43A56B199288739D2BA1D694173AEA7C6B4A5" }, { "b" : "7FC8B2CFC000", "path" : "/lib64/libplds4.so", "elfType" : 3, "buildId" : "0D96CF7ED26BAA5BA06AD6234575FA8BAB1B01FC" }, { "b" : "7FC8B2AF7000", "path" : "/lib64/libplc4.so", "elfType" : 3, "buildId" : "467E73CBB0223C865DE55CE2AD8D3DABD8247F7A" }, { "b" : "7FC8B28BB000", "path" : "/lib64/libnspr4.so", "elfType" : 3, "buildId" : "83866E18EABBF34398185BBAC565A15148259CF1" }, { "b" : "7FC8B2694000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "C17759D57199C1AD0E59C2567D9473E2D0878EFF" }, { "b" : "7FC8B245D000", "path" : "/lib64/libcrypt.so.1", "elfType" : 3, "buildId" : "AB959E1CA807CAE33B552942BE48AB46E57BC124" }, { "b" : "7FC8B21F9000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "803CA9FD9A024206DB65BA5018968E6A74285C68" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x56236a09af21]
 mongod(+0x299245E) [0x56236a09a45e]
 mongod(+0x29924F6) [0x56236a09a4f6]
 libpthread.so.0(+0x118E0) [0x7fc8b5c5d8e0]
 libc.so.6(gsignal+0x110) [0x7fc8b58d4ca0]
 libc.so.6(abort+0x148) [0x7fc8b58d6148]
 mongod(_ZN5mongo42fassertFailedWithStatusNoTraceWithLocationEiRKNS_6StatusEPKcj+0x0) [0x562368448c1d]
 mongod(+0xA6C70B) [0x56236817470b]
 mongod(_ZN5mongo21WiredTigerRecordStore26insertRecordsWithDocWriterEPNS_16OperationContextEPKPKNS_9DocWriterEPKNS_9TimestampEmPNS_8RecordIdE+0x1A6) [0x562368557c16]
 mongod(_ZN5mongo14CollectionImpl23insertDocumentsForOplogEPNS_16OperationContextEPKPKNS_9DocWriterEPNS_9TimestampEm+0x81) [0x562368d79951]
 mongod(_ZN5mongo4repl12_logOpsInnerEPNS_16OperationContextERKNS_15NamespaceStringEPKPKNS_9DocWriterEPNS_9TimestampEmPNS_10CollectionENS0_6OpTimeENS_6Date_tE+0x9C) [0x562368e12afc]
 mongod(_ZN5mongo4repl12logInsertOpsEPNS_16OperationContextERKNS_15NamespaceStringEN5boost8optionalINS_4UUIDEEEN9__gnu_cxx17__normal_iteratorIPKNS_15InsertStatementESt6vectorISC_SaISC_EEEESI_bNS_6Date_tE+0x8B5) [0x562368e14765]
 mongod(_ZN5mongo14OpObserverImpl9onInsertsEPNS_16OperationContextERKNS_15NamespaceStringEN5boost8optionalINS_4UUIDEEEN9__gnu_cxx17__normal_iteratorIPKNS_15InsertStatementESt6vectorISC_SaISC_EEEESI_b+0x10E) [0x562368bed0ce]
 mongod(_ZN5mongo18OpObserverRegistry9onInsertsEPNS_16OperationContextERKNS_15NamespaceStringEN5boost8optionalINS_4UUIDEEEN9__gnu_cxx17__normal_iteratorIPKNS_15InsertStatementESt6vectorISC_SaISC_EEEESI_b+0xAF) [0x5623684d3a3f]
 mongod(_ZN5mongo14CollectionImpl15insertDocumentsEPNS_16OperationContextEN9__gnu_cxx17__normal_iteratorIPKNS_15InsertStatementESt6vectorIS5_SaIS5_EEEESB_PNS_7OpDebugEb+0x298) [0x562368d7c7d8]
 mongod(+0x155792C) [0x562368c5f92c]
 mongod(+0x1557D10) [0x562368c5fd10]
 mongod(_ZN5mongo14performInsertsEPNS_16OperationContextERKNS_9write_ops6InsertEb+0x5F8) [0x562368c603c8]
 mongod(+0x154A9FF) [0x562368c529ff]
 mongod(+0x154859D) [0x562368c5059d]
 mongod(+0x11E637C) [0x5623688ee37c]
 mongod(+0x11E7325) [0x5623688ef325]
 mongod(+0x11E936C) [0x5623688f136c]
 mongod(_ZN5mongo23ServiceEntryPointCommon13handleRequestEPNS_16OperationContextERKNS_7MessageERKNS0_5HooksE+0x5C2) [0x5623688f2132]
 mongod(_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE+0x3C) [0x5623688df48c]
 mongod(_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE+0xE5) [0x5623688ebf55]
 mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x117) [0x5623688e9737]
 mongod(+0x11E2FC5) [0x5623688eafc5]
 mongod(_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE+0x182) [0x56236979e102]
 mongod(_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE+0x10D) [0x5623688e6ccd]
 mongod(_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE+0x5BD) [0x5623688e821d]
 mongod(_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE+0x336) [0x5623688e8b66]
 mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0xDB) [0x5623688e96fb]
 mongod(+0x11E2FC5) [0x5623688eafc5]
 mongod(+0x2096587) [0x56236979e587]
 mongod(+0x26F6E15) [0x562369dfee15]
 mongod(+0x26F6E74) [0x562369dfee74]
 libpthread.so.0(+0x744B) [0x7fc8b5c5344b]
 libc.so.6(clone+0x3F) [0x7fc8b598e40f]
-----  END BACKTRACE  ----- 



 Comments   
Comment by Yuan Fang [ 05/Dec/22 ]

We haven’t heard back from you for some time, so I’m going to close this ticket. If this is still an issue for you, please provide additional information and we will reopen the ticket.

Comment by Yuan Fang [ 21/Nov/22 ]

Hi srikanta@jungleegames.com,

We still need additional information to diagnose the problem. If this is still an issue for you, would you please check if any node running with enableMajorityReadConcern=false? if so, could you turn enableMajorityReadConcern: true (the default) or upgrade to 4.4?

If not, or if the issue still persists after trying the steps advised above,  could you archive (tar or zip) the mongod.log files covering the incident and the $dbpath/diagnostic.data directory (the contents are described here) and upload them to this support uploader location?

Files uploaded to this portal are visible only to MongoDB employees and are routinely deleted after some time.

Regards,
Yuan

Comment by Yuan Fang [ 02/Nov/22 ]

Hi srikanta@jungleegames.com,

Thank you for your report. We've seen issues that get similar WT error messages with enableMajorityReadConcern=false on 4.2, would you check if any node running with enableMajorityReadConcern=false? if so, could you turn enableMajorityReadConcern: true (the default) or upgrade to 4.4?

If not, or if the issue still persists after trying the steps advised above,  could you archive (tar or zip) the mongod.log files covering the incident and the $dbpath/diagnostic.data directory (the contents are described here) and upload them to this support uploader location?

Files uploaded to this portal are visible only to MongoDB employees and are routinely deleted after some time.

Regards,
Yuan

Generated at Thu Feb 08 06:17:47 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.