[SERVER-48637] error:WiredTiger error (22) and Fatal assertion 39001 Created: 06/Jun/20  Updated: 29/Sep/21  Resolved: 12/Apr/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 4.2.7
Fix Version/s: None

Type: Question Priority: Major - P3
Reporter: kkkkkk kopone Assignee: Dmitry Agranat
Resolution: Duplicate Votes: 2
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File mongod.log    
Issue Links:
Duplicate
is duplicated by SERVER-55746 Validate inserts into the oplog of a ... Backlog
Related
related to SERVER-55735 MongoDB version 4.2.5 WiredTiger erro... Closed
related to SERVER-55894 WiredTiger error (22) and Fatal asser... Closed
related to SERVER-55746 Validate inserts into the oplog of a ... Backlog
Participants:

 Description   

2020-06-06T11:02:36.661+0800 I NETWORK [listener] connection accepted from 10.10.0.63:46490 #25 (11 connections now open)
2020-06-06T11:02:36.662+0800 I NETWORK [conn25] received client metadata from 10.10.0.63:46490 conn25: \{ driver: { name: "mongo-go-driver", version: "v1.2.1" }, os: \{ type: "linux", architecture: "amd64" }, platform: "go1.12.17" }
2020-06-06T11:02:36.662+0800 I NETWORK [listener] connection accepted from 10.10.0.63:46492 #26 (12 connections now open)
2020-06-06T11:02:36.662+0800 I NETWORK [conn26] received client metadata from 10.10.0.63:46492 conn26: \{ driver: { name: "mongo-go-driver", version: "v1.2.1" }, os: \{ type: "linux", architecture: "amd64" }, platform: "go1.12.17", application: \{ name: "mongorestore" } }
2020-06-06T11:02:36.677+0800 I ACCESS [conn26] Successfully authenticated as principal root on admin from client ******:46492
2020-06-06T11:02:36.770+0800 E STORAGE [conn26] WiredTiger error (22) [1591412556:770705][14500:0x7f5c8ad3c700], WT_SESSION.timestamp_transaction: __wt_txn_set_commit_timestamp, 683: commit timestamp (1590732316, 2) is less than the oldest timestamp (1591412554, 1): Invalid argument Raw: [1591412556:770705][14500:0x7f5c8ad3c700], WT_SESSION.timestamp_transaction: __wt_txn_set_commit_timestamp, 683: commit timestamp (1590732316, 2) is less than the oldest timestamp (1591412554, 1): Invalid argument
2020-06-06T11:02:36.770+0800 F - [conn26] Fatal assertion 39001 BadValue: timestamp_transaction 22: Invalid argument at src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp 1323
2020-06-06T11:02:36.770+0800 F - [conn26]
 
***aborting after fassert() failure
 
 
2020-06-06T11:02:36.793+0800 F - [conn26] Got signal: 6 (Aborted).
 0x5561aa1779f1 0x5561aa17700c 0x5561aa177096 0x7f5ca5979890 0x7f5ca55b4e97 0x7f5ca55b6801 0x5561a85cf3a7 0x5561a83088fd 0x5561a86cd324 0x5561a8e9acb6 0x5561a8e9ccbc 0x5561a8d9240c 0x5561a8d927f0 0x5561a8d92ea2 0x5561a8d84c7f 0x5561a8d82ded 0x5561a8a468e8 0x5561a8a48b54 0x5561a8a498ea 0x5561a8a3746c 0x5561a8a431fc 0x5561a8a404df 0x5561a8a4216c 0x5561a989d432 0x5561a8a3d3ad 0x5561a8a3ec23 0x5561a8a3f7d6 0x5561a8a4043b 0x5561a8a4216c 0x5561a989d89b 0x5561a9ef31b5 0x5561a9ef3214 0x7f5ca596e6db 0x7f5ca569788f
----- BEGIN BACKTRACE -----



 Comments   
Comment by Dmitry Agranat [ 12/Apr/21 ]

Thanks srikanta@jungleegames.com, we have moved WT-7398 to SERVER-55894 and will follow up there. Closing this out as a duplicate of SERVER-55746.

Comment by Srikanta Patra [ 08/Apr/21 ]

@Dmitry Agranat, I create separete ticket. Please find below link ticket.

https://jira.mongodb.org/browse/WT-7398

 

Comment by Srikanta Patra [ 08/Apr/21 ]

@Dmitry Agranat, Sure I will create ticket and sharing to you.

Comment by Dmitry Agranat [ 06/Apr/21 ]

2857516870@qq.com, We believe the issue you've originally reported is related to SERVER-55746, please follow SERVER-55746 for further updates.

zubair.iub.pk@gmail.com, We are currently investigating the issue you've reported in SERVER-55735, which seems unrelated to SERVER-55746.

srikanta@jungleegames.com, divomen@gmail.com If this is still an issue for you, please open a separate ticket where we'll need to collect some diagnostics data in order to investigate it.

Comment by Srikanta Patra [ 05/Apr/21 ]

Hi All,

 

We are also getting same error on production server. Please help to get it resolve.

2021-04-02T22:52:49.692+0530 E STORAGE [conn4020] WiredTiger error (22) [1617384169:692561][5626:0x7f620f4f5700], WT_SESSION.timestamp_transaction: __wt_txn_set_commit_timestamp, 683: commit timestamp (1617384169, 810) is less than the oldest timestamp (1617384169, 811): Invalid argument Raw: [1617384169:692561][5626:0x7f620f4f5700], WT_SESSION.timestamp_transaction: __wt_txn_set_commit_timestamp, 683: commit timestamp (1617384169, 810) is less than the oldest timestamp (1617384169, 811): Invalid argument
2021-04-02T22:52:49.692+0530 F - [conn4020] Fatal assertion 39001 BadValue: timestamp_transaction 22: Invalid argument at src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp 1323
2021-04-02T22:52:49.692+0530 F - [conn4020]
2021-04-02T22:52:49.764+0530 F - [conn4020] Got signal: 6 (Aborted)

Comment by Zubair Ahmed [ 03/Apr/21 ]

Neither manually nor inserted documents directly into oplog of replica set.

Comment by Daniel Gottlieb (Inactive) [ 02/Apr/21 ]

zubair.iub.pk@gmail.com are you or an application being run inserting documents directly into the oplog of an active replica set primary?

Comment by Zubair Ahmed [ 02/Apr/21 ]

Mongodb Version : 4.2.5 service crashed on Centos 7.7.1908  with following backtrace

 

2021-03-31T13:32:03.820+0000 F - [conn527851316] Got signal: 6 (Aborted).
0x56454fe7df41 0x56454fe7d73e 0x56454fe7d7d6 0x7faa798e85f0 0x7faa79541337 0x7faa79542a28 0x56454e2db69d 0x56454e01a395 0x56454e3d80fc 0x56454ebac071 0x56454ec3f4e3 0x56454ec43c7c 0x56454ea2a3cb 0x56454ea303cc 0x56454e365dc7 0x56454ebb1e4b 0x56454ed6ef6d 0x56454ed6fe12 0x56454ed4e518 0x56454ed4e897 0x56454ed4e518 0x56454ed95850 0x56454ed95fad 0x56454ea5b617 0x56454ea5d410 0x56454ea5e66a 0x56454f87067f 0x56454e77ceaa 0x56454e77e1b2 0x56454e780204 0x56454e780f6a 0x56454e76ebbc 0x56454e77a94c 0x56454e777c2f 0x56454e7798bc 0x56454f5ab572 0x56454e774afd 0x56454e776373 0x56454e776f26 0x56454e777b8b 0x56454e7798bc 0x56454f5ab9db 0x56454fbfaf64 0x7faa798e0e65 0x7faa7960988d
----- BEGIN BACKTRACE -----

{"backtrace":[\{"b":"56454D5D4000","o":"28A9F41","s":"_ZN5mongo15printStackTraceERSo"}

,{"b":"56454D5D4000","o":"28A973E"},{"b":"56454D5D4000","o":"28A97D6"},{"b":"7FAA798D9000","o":"F5F0"},{"b":"7FAA7950B000","o":"36337","s":"gsignal"},{"b":"7FAA7950B000","o":"37A28","s":"abort"},{"b":"56454D5D4000","o":"D0769D","s":"_ZN5mongo42fassertFailedWithStatusNoTraceWithLocationEiRKNS_6StatusEPKcj"},{"b":"56454D5D4000","o":"A46395"},{"b":"56454D5D4000","o":"E040FC","s":"_ZN5mongo21WiredTigerRecordStore26insertRecordsWithDocWriterEPNS_16OperationContextEPKPKNS_9DocWriterEPKNS_9TimestampEmPNS_8RecordIdE"},{"b":"56454D5D4000","o":"15D8071","s":"_ZN5mongo14CollectionImpl23insertDocumentsForOplogEPNS_16OperationContextEPKPKNS_9DocWriterEPNS_9TimestampEm"},{"b":"56454D5D4000","o":"166B4E3","s":"_ZN5mongo4repl12_logOpsInnerEPNS_16OperationContextERKNS_15NamespaceStringEPKPKNS_9DocWriterEPNS_9TimestampEmPNS_10CollectionENS0_6OpTimeENS_6Date_tE"},{"b":"56454D5D4000","o":"166FC7C","s":"_ZN5mongo4repl5logOpEPNS_16OperationContextEPKcRKNS_15NamespaceStringEN5boost8optionalINS_4UUIDEEERKNS_7BSONObjEPSD_bNS_6Date_tERKNS_20OperationSessionInfoENS9_IiEERKNS0_9OplogLinkERKNS0_6OpTimeE"},{"b":"56454D5D4000","o":"14563CB"},{"b":"56454D5D4000","o":"145C3CC","s":"_ZN5mongo14OpObserverImpl8onUpdateEPNS_16OperationContextERKNS_20OplogUpdateEntryArgsE"},{"b":"56454D5D4000","o":"D91DC7","s":"_ZN5mongo18OpObserverRegistry8onUpdateEPNS_16OperationContextERKNS_20OplogUpdateEntryArgsE"},{"b":"56454D5D4000","o":"15DDE4B","s":"_ZN5mongo14CollectionImpl14updateDocumentEPNS_16OperationContextENS_8RecordIdERKNS_11SnapshottedINS_7BSONObjEEERKS5_bPNS_7OpDebugEPNS_20CollectionUpdateArgsE"},{"b":"56454D5D4000","o":"179AF6D","s":"_ZN5mongo11UpdateStage18transformAndUpdateERKNS_11SnapshottedINS_7BSONObjEEERNS_8RecordIdE"},{"b":"56454D5D4000","o":"179BE12","s":"_ZN5mongo11UpdateStage6doWorkEPm"},{"b":"56454D5D4000","o":"177A518","s":"_ZN5mongo9PlanStage4workEPm"},{"b":"56454D5D4000","o":"177A897","s":"_ZN5mongo15ProjectionStage6doWorkEPm"},{"b":"56454D5D4000","o":"177A518","s":"_ZN5mongo9PlanStage4workEPm"},{"b":"56454D5D4000","o":"17C1850","s":"_ZN5mongo16PlanExecutorImpl12_getNextImplEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE"},{"b":"56454D5D4000","o":"17C1FAD","s":"_ZN5mongo16PlanExecutorImpl7getNextEPNS_7BSONObjEPNS_8RecordIdE"},{"b":"56454D5D4000","o":"1487617"},{"b":"56454D5D4000","o":"1489410"},{"b":"56454D5D4000","o":"148A66A"},{"b":"56454D5D4000","o":"229C67F","s":"_ZN5mongo12BasicCommand10Invocation3runEPNS_16OperationContextEPNS_3rpc21ReplyBuilderInterfaceE"},{"b":"56454D5D4000","o":"11A8EAA"},{"b":"56454D5D4000","o":"11AA1B2"},{"b":"56454D5D4000","o":"11AC204"},{"b":"56454D5D4000","o":"11ACF6A","s":"_ZN5mongo23ServiceEntryPointCommon13handleRequestEPNS_16OperationContextERKNS_7MessageERKNS0_5HooksE"},{"b":"56454D5D4000","o":"119ABBC","s":"_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE"},{"b":"56454D5D4000","o":"11A694C","s":"_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE"},{"b":"56454D5D4000","o":"11A3C2F","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"56454D5D4000","o":"11A58BC"},{"b":"56454D5D4000","o":"1FD7572","s":"_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE"},{"b":"56454D5D4000","o":"11A0AFD","s":"_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE"},{"b":"56454D5D4000","o":"11A2373","s":"_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE"},{"b":"56454D5D4000","o":"11A2F26","s":"_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE"},{"b":"56454D5D4000","o":"11A3B8B","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"56454D5D4000","o":"11A58BC"},{"b":"56454D5D4000","o":"1FD79DB"},{"b":"56454D5D4000","o":"2626F64"},{"b":"7FAA798D9000","o":"7E65"},{"b":"7FAA7950B000","o":"FE88D","s":"clone"}],"processInfo":{ "mongodbVersion" : "4.2.5", "gitVersion" : "2261279b51ea13df08ae708ff278f0679c59dc32", "compiledModules" : [], "uname" :

{ "sysname" : "Linux", "release" : "3.10.0-1062.9.1.el7.x86_64", "version" : "#1 SMP Fri Dec 6 15:49:49 UTC 2019", "machine" : "x86_64" }

, "somap" : [ { "b" : "56454D5D4000", "elfType" : 3, "buildId" : "56B1321DBA5855D67E2937EDEADAE0C9D69BBBA4" }, { "b" : "7FFCE9CA6000", "elfType" : 3, "buildId" : "B5A5458535A1397FA6BAAF5E8C13A6395426A1B2" }, { "b" : "7FAA7AD07000", "path" : "/lib64/libcurl.so.4", "elfType" : 3, "buildId" : "89C83CEB5DE5FDEC4F3DFBA4FCAACB53D747A998" }, { "b" : "7FAA7AAEE000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "3009B26B33156EAAF99787AA3DA0C6AE99649755" }, { "b" : "7FAA7A68B000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "4CF1939F660008CFA869D8364651F31AACD2C1C4" }, { "b" : "7FAA7A419000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "3B305C3BA17FE394862E749763F2956C9C890C2E" }, { "b" : "7FAA7A215000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "18113E6E83D8E981B8E8D808F7F3DBB23F950A1D" }, { "b" : "7FAA7A00D000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "4749697BF078337576C4629F0D30B296A0939779" }, { "b" : "7FAA79D0B000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "5681C054FDABCF789F4DDA66E94F1F6ED1747327" }, { "b" : "7FAA79AF5000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "DAC0179F4555AEFEC9E97476201802FD20C03EC5" }, { "b" : "7FAA798D9000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "8B33F7F8C86F8D544C63C5541A8E42B3DDFEF8B1" }, { "b" : "7FAA7950B000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "398944D32CF16A67AF51067A326E6C0CC14F90ED" }, { "b" : "7FAA7AF71000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "5CC1A53B747A7E4D21198723C2B633E54F3C06D9" }, { "b" : "7FAA792D8000", "path" : "/lib64/libidn.so.11", "elfType" : 3, "buildId" : "2B77BBEFFF65E94F3E0B71A4E89BEB68C4B476C5" }, { "b" : "7FAA790AB000", "path" : "/lib64/libssh2.so.1", "elfType" : 3, "buildId" : "1AF123CADB2F2910E89CBD540A06D3B33692F95E" }, { "b" : "7FAA78E52000", "path" : "/lib64/libssl3.so", "elfType" : 3, "buildId" : "B6321C434B5C7386B144B925CEE2798D269FDDF5" }, { "b" : "7FAA78C2A000", "path" : "/lib64/libsmime3.so", "elfType" : 3, "buildId" : "BDA454441F59F41D2DA36E13CEA1FC4CE95B2BBB" }, { "b" : "7FAA788FB000", "path" : "/lib64/libnss3.so", "elfType" : 3, "buildId" : "D61EB90C9F32CA6E81E7FAC437F2C496438C8D9E" }, { "b" : "7FAA786CB000", "path" : "/lib64/libnssutil3.so", "elfType" : 3, "buildId" : "1E366A2153AD7488EE72E989D9AD6BD458BE8EDE" }, { "b" : "7FAA784C7000", "path" : "/lib64/libplds4.so", "elfType" : 3, "buildId" : "325B8CE57A776DE0B24B362A7E0C90E903B1A4B8" }, { "b" : "7FAA782C2000", "path" : "/lib64/libplc4.so", "elfType" : 3, "buildId" : "0460FF10A3C63749113D380C40E10DFCF066C76E" }, { "b" : "7FAA78084000", "path" : "/lib64/libnspr4.so", "elfType" : 3, "buildId" : "8840B019EDB66B0CFBD2F77EF196440F7928106E" }, { "b" : "7FAA77E37000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "E2AA8CA3D3164E7DBEC293BFA0B55D2B10DAC05D" }, { "b" : "7FAA77B4E000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "3EE7267AF7BFD3B132E6A222D997DA09C96C90DD" }, { "b" : "7FAA7791B000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "82E28CACB60C27CD6F14A6D2268F0CFF621664D0" }, { "b" : "7FAA77717000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "67E935BFABA2C914C01156B88947DD515EA51170" }, { "b" : "7FAA77508000", "path" : "/lib64/liblber-2.4.so.2", "elfType" : 3, "buildId" : "3192C56CD451E18EB9F29CB045432BA9C738DD29" }, { "b" : "7FAA772B3000", "path" : "/lib64/libldap-2.4.so.2", "elfType" : 3, "buildId" : "F1FADDDE0D21D5F4E2DCADEDD3B85B6E7AAC9883" }, { "b" : "7FAA7709D000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "B9D5F73428BD6AD68C96986B57BEA3B7CEDB9745" }, { "b" : "7FAA76E8D000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "4F5FBB2087BE132892467C4E7A46A3D07E5DA40B" }, { "b" : "7FAA76C89000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "2E01D5AC08C1280D013AAB96B292AC58BC30A263" }, { "b" : "7FAA76A6C000", "path" : "/lib64/libsasl2.so.3", "elfType" : 3, "buildId" : "E2F2017F821DD1B9D307DA1A9B8014F2941AEB7B" }, { "b" : "7FAA76845000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "D2DD4DA3FDE1477D25BFFF80F3A25FDB541A8179" }, { "b" : "7FAA7660E000", "path" : "/lib64/libcrypt.so.1", "elfType" : 3, "buildId" : "84467C988F41D853C58353BEB247670E15DA8BAD" }, { "b" : "7FAA763AC000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "9CA3D11F018BEEB719CDB34BE800BF1641350D0A" }, { "b" : "7FAA761A9000", "path" : "/lib64/libfreebl3.so", "elfType" : 3, "buildId" : "197680DAE6538245CB99723E57447C4EF2E98362" } ] }}
mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x56454fe7df41]
mongod(+0x28A973E) [0x56454fe7d73e]
mongod(+0x28A97D6) [0x56454fe7d7d6]
libpthread.so.0(+0xF5F0) [0x7faa798e85f0]
libc.so.6(gsignal+0x37) [0x7faa79541337]
libc.so.6(abort+0x148) [0x7faa79542a28]
mongod(_ZN5mongo42fassertFailedWithStatusNoTraceWithLocationEiRKNS_6StatusEPKcj+0x0) [0x56454e2db69d]
mongod(+0xA46395) [0x56454e01a395]
mongod(_ZN5mongo21WiredTigerRecordStore26insertRecordsWithDocWriterEPNS_16OperationContextEPKPKNS_9DocWriterEPKNS_9TimestampEmPNS_8RecordIdE+0x1AC) [0x56454e3d80fc]
mongod(_ZN5mongo14CollectionImpl23insertDocumentsForOplogEPNS_16OperationContextEPKPKNS_9DocWriterEPNS_9TimestampEm+0x81) [0x56454ebac071]
mongod(_ZN5mongo4repl12_logOpsInnerEPNS_16OperationContextERKNS_15NamespaceStringEPKPKNS_9DocWriterEPNS_9TimestampEmPNS_10CollectionENS0_6OpTimeENS_6Date_tE+0xA3) [0x56454ec3f4e3]
mongod(_ZN5mongo4repl5logOpEPNS_16OperationContextEPKcRKNS_15NamespaceStringEN5boost8optionalINS_4UUIDEEERKNS_7BSONObjEPSD_bNS_6Date_tERKNS_20OperationSessionInfoENS9_IiEERKNS0_9OplogLinkERKNS0_6OpTimeE+0x43C) [0x56454ec43c7c]
mongod(+0x14563CB) [0x56454ea2a3cb]
mongod(_ZN5mongo14OpObserverImpl8onUpdateEPNS_16OperationContextERKNS_20OplogUpdateEntryArgsE+0x7CC) [0x56454ea303cc]
mongod(_ZN5mongo18OpObserverRegistry8onUpdateEPNS_16OperationContextERKNS_20OplogUpdateEntryArgsE+0x57) [0x56454e365dc7]
mongod(_ZN5mongo14CollectionImpl14updateDocumentEPNS_16OperationContextENS_8RecordIdERKNS_11SnapshottedINS_7BSONObjEEERKS5_bPNS_7OpDebugEPNS_20CollectionUpdateArgsE+0x5CB) [0x56454ebb1e4b]
mongod(_ZN5mongo11UpdateStage18transformAndUpdateERKNS_11SnapshottedINS_7BSONObjEEERNS_8RecordIdE+0xD2D) [0x56454ed6ef6d]
mongod(_ZN5mongo11UpdateStage6doWorkEPm+0x2F2) [0x56454ed6fe12]
mongod(_ZN5mongo9PlanStage4workEPm+0x68) [0x56454ed4e518]
mongod(_ZN5mongo15ProjectionStage6doWorkEPm+0x47) [0x56454ed4e897]
mongod(_ZN5mongo9PlanStage4workEPm+0x68) [0x56454ed4e518]
mongod(_ZN5mongo16PlanExecutorImpl12_getNextImplEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE+0x230) [0x56454ed95850]
mongod(_ZN5mongo16PlanExecutorImpl7getNextEPNS_7BSONObjEPNS_8RecordIdE+0x4D) [0x56454ed95fad]
mongod(+0x1487617) [0x56454ea5b617]
mongod(+0x1489410) [0x56454ea5d410]
mongod(+0x148A66A) [0x56454ea5e66a]
mongod(_ZN5mongo12BasicCommand10Invocation3runEPNS_16OperationContextEPNS_3rpc21ReplyBuilderInterfaceE+0xAF) [0x56454f87067f]
mongod(+0x11A8EAA) [0x56454e77ceaa]
mongod(+0x11AA1B2) [0x56454e77e1b2]
mongod(+0x11AC204) [0x56454e780204]
mongod(_ZN5mongo23ServiceEntryPointCommon13handleRequestEPNS_16OperationContextERKNS_7MessageERKNS0_5HooksE+0x41A) [0x56454e780f6a]
mongod(_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE+0x3C) [0x56454e76ebbc]
mongod(_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE+0xEC) [0x56454e77a94c]
mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x17F) [0x56454e777c2f]
mongod(+0x11A58BC) [0x56454e7798bc]
mongod(_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE+0x182) [0x56454f5ab572]
mongod(_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE+0x10D) [0x56454e774afd]
mongod(_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE+0x753) [0x56454e776373]
mongod(_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE+0x316) [0x56454e776f26]
mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0xDB) [0x56454e777b8b]
mongod(+0x11A58BC) [0x56454e7798bc]
mongod(+0x1FD79DB) [0x56454f5ab9db]
mongod(+0x2626F64) [0x56454fbfaf64]
libpthread.so.0(+0x7E65) [0x7faa798e0e65]
libc.so.6(clone+0x6D) [0x7faa7960988d]
----- END BACKTRACE -----

Comment by Dmitry Voronenkov [ 15/Jun/20 ]

I have the same issue:

2020-06-15T00:57:16.354+0000 E STORAGE [conn23415] WiredTiger error (22) [1592182636:354084][1:0x7fbc81a77700], WT_SESSION.timestamp_transaction: __wt_txn_set_commit_timestamp, 683: commit timestamp (1592182636, 108) is less than the oldest timestamp (1592182636, 110): Invalid argument Raw: [1592182636:354084][1:0x7fbc81a77700], WT_SESSION.timestamp_transaction: __wt_txn_set_commit_timestamp, 683: commit timestamp (1592182636, 108) is less than the oldest timestamp (1592182636, 110): Invalid argument
2020-06-15T00:57:16.376+0000 F - [conn23415] Fatal assertion 39001 BadValue: timestamp_transaction 22: Invalid argument at src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp 1323
2020-06-15T00:57:16.376+0000 F - [conn23415]

***aborting after fassert() failure

2020-06-15T00:57:16.754+0000 F - [conn23415] Got signal: 6 (Aborted).
0x561c49b139f1 0x561c49b1300c 0x561c49b13096 0x7fbca9bff890 0x7fbca983ae97 0x7fbca983c801 0x561c47f6b3a7 0x561c47ca48fd 0x561c480694fc 0x561c48835eb1 0x561c488cad03 0x561c488cf49c 0x561c486b7cab 0x561c486bdcac 0x561c47ff66d7 0x561c4883c35c 0x561c489fafad 0x561c489fc692 0x561c489fe70b 0x561c489dad88 0x561c48a222b0 0x561c48a22a0d 0x561c486e87d7 0x561c486ea5eb 0x561c486eb85a 0x561c49502f4f 0x561c483e17ea 0x561c483e2ad5 0x561c483e4b54 0x561c483e58ea 0x561c483d346c 0x561c483df1fc 0x561c483dc4df 0x561c483de16c 0x561c49239432 0x561c483d93ad 0x561c483dac23 0x561c483db7d6 0x561c483dc43b 0x561c483de16c 0x561c4923989b 0x561c4988f1b5 0x561c4988f214 0x7fbca9bf46db 0x7fbca991d88f
----- BEGIN BACKTRACE -----

Comment by Carl Champain (Inactive) [ 08/Jun/20 ]

Hi 2857516870@qq.com,

Thanks for the report.
Can you please provide the full mongod.log file for this node?

Kind regards,
Carl
 

Generated at Thu Feb 08 05:17:41 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.