[SERVER-55735] MongoDB version 4.2.5 WiredTiger error (22) Created: 02/Apr/21  Updated: 05/Jul/22  Resolved: 22/Apr/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 4.2.5
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Zubair Ahmed Assignee: Dmitry Agranat
Resolution: Incomplete Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
is related to SERVER-67790 [4.2] Running with enableMajorityRead... Closed
is related to SERVER-48637 error:WiredTiger error (22) and Fatal... Closed
Operating System: ALL
Participants:

 Description   

2021-03-31T13:32:03.651+0000 E STORAGE [conn527851316] WiredTiger error (22) [1617197523:650992][27216:0x7faa1b3f6700], WT_SESSION.timestamp_transaction: __wt_txn_set_commit_timestamp, 683: commit timestamp (1617197523, 1759) is less than the oldest timestamp (1617197523, 1762): Invalid argument Raw: [1617197523:650992][27216:0x7faa1b3f6700], WT_SESSION.timestamp_transaction: __wt_txn_set_commit_timestamp, 683: commit timestamp (1617197523, 1759) is less than the oldest timestamp (1617197523, 1762): Invalid argument
2021-03-31T13:32:03.665+0000 F - [conn527851316] Fatal assertion 39001 BadValue: timestamp_transaction 22: Invalid argument at src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp 1321
2021-03-31T13:32:03.667+0000 F - [conn527851316] 
2021-03-31T13:32:03.820+0000 F - [conn527851316] Got signal: 6 (Aborted).
 0x56454fe7df41 0x56454fe7d73e 0x56454fe7d7d6 0x7faa798e85f0 0x7faa79541337 0x7faa79542a28 0x56454e2db69d 0x56454e01a395 0x56454e3d80fc 0x56454ebac071 0x56454ec3f4e3 0x56454ec43c7c 0x56454ea2a3cb 0x56454ea303cc 0x56454e365dc7 0x56454ebb1e4b 0x56454ed6ef6d 0x56454ed6fe12 0x56454ed4e518 0x56454ed4e897 0x56454ed4e518 0x56454ed95850 0x56454ed95fad 0x56454ea5b617 0x56454ea5d410 0x56454ea5e66a 0x56454f87067f 0x56454e77ceaa 0x56454e77e1b2 0x56454e780204 0x56454e780f6a 0x56454e76ebbc 0x56454e77a94c 0x56454e777c2f 0x56454e7798bc 0x56454f5ab572 0x56454e774afd 0x56454e776373 0x56454e776f26 0x56454e777b8b 0x56454e7798bc 0x56454f5ab9db 0x56454fbfaf64 0x7faa798e0e65 0x7faa7960988d
----- BEGIN BACKTRACE -----
{"backtrace":[\{"b":"56454D5D4000","o":"28A9F41","s":"_ZN5mongo15printStackTraceERSo"},\{"b":"56454D5D4000","o":"28A973E"},\{"b":"56454D5D4000","o":"28A97D6"},\{"b":"7FAA798D9000","o":"F5F0"},\{"b":"7FAA7950B000","o":"36337","s":"gsignal"},\{"b":"7FAA7950B000","o":"37A28","s":"abort"},\{"b":"56454D5D4000","o":"D0769D","s":"_ZN5mongo42fassertFailedWithStatusNoTraceWithLocationEiRKNS_6StatusEPKcj"},\{"b":"56454D5D4000","o":"A46395"},\{"b":"56454D5D4000","o":"E040FC","s":"_ZN5mongo21WiredTigerRecordStore26insertRecordsWithDocWriterEPNS_16OperationContextEPKPKNS_9DocWriterEPKNS_9TimestampEmPNS_8RecordIdE"},\{"b":"56454D5D4000","o":"15D8071","s":"_ZN5mongo14CollectionImpl23insertDocumentsForOplogEPNS_16OperationContextEPKPKNS_9DocWriterEPNS_9TimestampEm"},\{"b":"56454D5D4000","o":"166B4E3","s":"_ZN5mongo4repl12_logOpsInnerEPNS_16OperationContextERKNS_15NamespaceStringEPKPKNS_9DocWriterEPNS_9TimestampEmPNS_10CollectionENS0_6OpTimeENS_6Date_tE"},\{"b":"56454D5D4000","o":"166FC7C","s":"_ZN5mongo4repl5logOpEPNS_16OperationContextEPKcRKNS_15NamespaceStringEN5boost8optionalINS_4UUIDEEERKNS_7BSONObjEPSD_bNS_6Date_tERKNS_20OperationSessionInfoENS9_IiEERKNS0_9OplogLinkERKNS0_6OpTimeE"},\{"b":"56454D5D4000","o":"14563CB"},\{"b":"56454D5D4000","o":"145C3CC","s":"_ZN5mongo14OpObserverImpl8onUpdateEPNS_16OperationContextERKNS_20OplogUpdateEntryArgsE"},\{"b":"56454D5D4000","o":"D91DC7","s":"_ZN5mongo18OpObserverRegistry8onUpdateEPNS_16OperationContextERKNS_20OplogUpdateEntryArgsE"},\{"b":"56454D5D4000","o":"15DDE4B","s":"_ZN5mongo14CollectionImpl14updateDocumentEPNS_16OperationContextENS_8RecordIdERKNS_11SnapshottedINS_7BSONObjEEERKS5_bPNS_7OpDebugEPNS_20CollectionUpdateArgsE"},\{"b":"56454D5D4000","o":"179AF6D","s":"_ZN5mongo11UpdateStage18transformAndUpdateERKNS_11SnapshottedINS_7BSONObjEEERNS_8RecordIdE"},\{"b":"56454D5D4000","o":"179BE12","s":"_ZN5mongo11UpdateStage6doWorkEPm"},\{"b":"56454D5D4000","o":"177A518","s":"_ZN5mongo9PlanStage4workEPm"},\{"b":"56454D5D4000","o":"177A897","s":"_ZN5mongo15ProjectionStage6doWorkEPm"},\{"b":"56454D5D4000","o":"177A518","s":"_ZN5mongo9PlanStage4workEPm"},\{"b":"56454D5D4000","o":"17C1850","s":"_ZN5mongo16PlanExecutorImpl12_getNextImplEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE"},\{"b":"56454D5D4000","o":"17C1FAD","s":"_ZN5mongo16PlanExecutorImpl7getNextEPNS_7BSONObjEPNS_8RecordIdE"},\{"b":"56454D5D4000","o":"1487617"},\{"b":"56454D5D4000","o":"1489410"},\{"b":"56454D5D4000","o":"148A66A"},\{"b":"56454D5D4000","o":"229C67F","s":"_ZN5mongo12BasicCommand10Invocation3runEPNS_16OperationContextEPNS_3rpc21ReplyBuilderInterfaceE"},\{"b":"56454D5D4000","o":"11A8EAA"},\{"b":"56454D5D4000","o":"11AA1B2"},\{"b":"56454D5D4000","o":"11AC204"},\{"b":"56454D5D4000","o":"11ACF6A","s":"_ZN5mongo23ServiceEntryPointCommon13handleRequestEPNS_16OperationContextERKNS_7MessageERKNS0_5HooksE"},\{"b":"56454D5D4000","o":"119ABBC","s":"_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE"},\{"b":"56454D5D4000","o":"11A694C","s":"_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE"},\{"b":"56454D5D4000","o":"11A3C2F","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},\{"b":"56454D5D4000","o":"11A58BC"},\{"b":"56454D5D4000","o":"1FD7572","s":"_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE"},\{"b":"56454D5D4000","o":"11A0AFD","s":"_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE"},\{"b":"56454D5D4000","o":"11A2373","s":"_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE"},\{"b":"56454D5D4000","o":"11A2F26","s":"_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE"},\{"b":"56454D5D4000","o":"11A3B8B","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},\{"b":"56454D5D4000","o":"11A58BC"},\{"b":"56454D5D4000","o":"1FD79DB"},\{"b":"56454D5D4000","o":"2626F64"},\{"b":"7FAA798D9000","o":"7E65"},\{"b":"7FAA7950B000","o":"FE88D","s":"clone"}],"processInfo":\{ "mongodbVersion" : "4.2.5", "gitVersion" : "2261279b51ea13df08ae708ff278f0679c59dc32", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.0-1062.9.1.el7.x86_64", "version" : "#1 SMP Fri Dec 6 15:49:49 UTC 2019", "machine" : "x86_64" }, "somap" : [ \{ "b" : "56454D5D4000", "elfType" : 3, "buildId" : "56B1321DBA5855D67E2937EDEADAE0C9D69BBBA4" }, \{ "b" : "7FFCE9CA6000", "elfType" : 3, "buildId" : "B5A5458535A1397FA6BAAF5E8C13A6395426A1B2" }, \{ "b" : "7FAA7AD07000", "path" : "/lib64/libcurl.so.4", "elfType" : 3, "buildId" : "89C83CEB5DE5FDEC4F3DFBA4FCAACB53D747A998" }, \{ "b" : "7FAA7AAEE000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "3009B26B33156EAAF99787AA3DA0C6AE99649755" }, \{ "b" : "7FAA7A68B000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "4CF1939F660008CFA869D8364651F31AACD2C1C4" }, \{ "b" : "7FAA7A419000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "3B305C3BA17FE394862E749763F2956C9C890C2E" }, \{ "b" : "7FAA7A215000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "18113E6E83D8E981B8E8D808F7F3DBB23F950A1D" }, \{ "b" : "7FAA7A00D000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "4749697BF078337576C4629F0D30B296A0939779" }, \{ "b" : "7FAA79D0B000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "5681C054FDABCF789F4DDA66E94F1F6ED1747327" }, \{ "b" : "7FAA79AF5000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "DAC0179F4555AEFEC9E97476201802FD20C03EC5" }, \{ "b" : "7FAA798D9000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "8B33F7F8C86F8D544C63C5541A8E42B3DDFEF8B1" }, \{ "b" : "7FAA7950B000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "398944D32CF16A67AF51067A326E6C0CC14F90ED" }, \{ "b" : "7FAA7AF71000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "5CC1A53B747A7E4D21198723C2B633E54F3C06D9" }, \{ "b" : "7FAA792D8000", "path" : "/lib64/libidn.so.11", "elfType" : 3, "buildId" : "2B77BBEFFF65E94F3E0B71A4E89BEB68C4B476C5" }, \{ "b" : "7FAA790AB000", "path" : "/lib64/libssh2.so.1", "elfType" : 3, "buildId" : "1AF123CADB2F2910E89CBD540A06D3B33692F95E" }, \{ "b" : "7FAA78E52000", "path" : "/lib64/libssl3.so", "elfType" : 3, "buildId" : "B6321C434B5C7386B144B925CEE2798D269FDDF5" }, \{ "b" : "7FAA78C2A000", "path" : "/lib64/libsmime3.so", "elfType" : 3, "buildId" : "BDA454441F59F41D2DA36E13CEA1FC4CE95B2BBB" }, \{ "b" : "7FAA788FB000", "path" : "/lib64/libnss3.so", "elfType" : 3, "buildId" : "D61EB90C9F32CA6E81E7FAC437F2C496438C8D9E" }, \{ "b" : "7FAA786CB000", "path" : "/lib64/libnssutil3.so", "elfType" : 3, "buildId" : "1E366A2153AD7488EE72E989D9AD6BD458BE8EDE" }, \{ "b" : "7FAA784C7000", "path" : "/lib64/libplds4.so", "elfType" : 3, "buildId" : "325B8CE57A776DE0B24B362A7E0C90E903B1A4B8" }, \{ "b" : "7FAA782C2000", "path" : "/lib64/libplc4.so", "elfType" : 3, "buildId" : "0460FF10A3C63749113D380C40E10DFCF066C76E" }, \{ "b" : "7FAA78084000", "path" : "/lib64/libnspr4.so", "elfType" : 3, "buildId" : "8840B019EDB66B0CFBD2F77EF196440F7928106E" }, \{ "b" : "7FAA77E37000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "E2AA8CA3D3164E7DBEC293BFA0B55D2B10DAC05D" }, \{ "b" : "7FAA77B4E000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "3EE7267AF7BFD3B132E6A222D997DA09C96C90DD" }, \{ "b" : "7FAA7791B000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "82E28CACB60C27CD6F14A6D2268F0CFF621664D0" }, \{ "b" : "7FAA77717000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "67E935BFABA2C914C01156B88947DD515EA51170" }, \{ "b" : "7FAA77508000", "path" : "/lib64/liblber-2.4.so.2", "elfType" : 3, "buildId" : "3192C56CD451E18EB9F29CB045432BA9C738DD29" }, \{ "b" : "7FAA772B3000", "path" : "/lib64/libldap-2.4.so.2", "elfType" : 3, "buildId" : "F1FADDDE0D21D5F4E2DCADEDD3B85B6E7AAC9883" }, \{ "b" : "7FAA7709D000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "B9D5F73428BD6AD68C96986B57BEA3B7CEDB9745" }, \{ "b" : "7FAA76E8D000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "4F5FBB2087BE132892467C4E7A46A3D07E5DA40B" }, \{ "b" : "7FAA76C89000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "2E01D5AC08C1280D013AAB96B292AC58BC30A263" }, \{ "b" : "7FAA76A6C000", "path" : "/lib64/libsasl2.so.3", "elfType" : 3, "buildId" : "E2F2017F821DD1B9D307DA1A9B8014F2941AEB7B" }, \{ "b" : "7FAA76845000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "D2DD4DA3FDE1477D25BFFF80F3A25FDB541A8179" }, \{ "b" : "7FAA7660E000", "path" : "/lib64/libcrypt.so.1", "elfType" : 3, "buildId" : "84467C988F41D853C58353BEB247670E15DA8BAD" }, \{ "b" : "7FAA763AC000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "9CA3D11F018BEEB719CDB34BE800BF1641350D0A" }, \{ "b" : "7FAA761A9000", "path" : "/lib64/libfreebl3.so", "elfType" : 3, "buildId" : "197680DAE6538245CB99723E57447C4EF2E98362" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x56454fe7df41]
 mongod(+0x28A973E) [0x56454fe7d73e]
 mongod(+0x28A97D6) [0x56454fe7d7d6]
 libpthread.so.0(+0xF5F0) [0x7faa798e85f0]
 libc.so.6(gsignal+0x37) [0x7faa79541337]
 libc.so.6(abort+0x148) [0x7faa79542a28]
 mongod(_ZN5mongo42fassertFailedWithStatusNoTraceWithLocationEiRKNS_6StatusEPKcj+0x0) [0x56454e2db69d]
 mongod(+0xA46395) [0x56454e01a395]
 mongod(_ZN5mongo21WiredTigerRecordStore26insertRecordsWithDocWriterEPNS_16OperationContextEPKPKNS_9DocWriterEPKNS_9TimestampEmPNS_8RecordIdE+0x1AC) [0x56454e3d80fc]
 mongod(_ZN5mongo14CollectionImpl23insertDocumentsForOplogEPNS_16OperationContextEPKPKNS_9DocWriterEPNS_9TimestampEm+0x81) [0x56454ebac071]
 mongod(_ZN5mongo4repl12_logOpsInnerEPNS_16OperationContextERKNS_15NamespaceStringEPKPKNS_9DocWriterEPNS_9TimestampEmPNS_10CollectionENS0_6OpTimeENS_6Date_tE+0xA3) [0x56454ec3f4e3]
 mongod(_ZN5mongo4repl5logOpEPNS_16OperationContextEPKcRKNS_15NamespaceStringEN5boost8optionalINS_4UUIDEEERKNS_7BSONObjEPSD_bNS_6Date_tERKNS_20OperationSessionInfoENS9_IiEERKNS0_9OplogLinkERKNS0_6OpTimeE+0x43C) [0x56454ec43c7c]
 mongod(+0x14563CB) [0x56454ea2a3cb]
 mongod(_ZN5mongo14OpObserverImpl8onUpdateEPNS_16OperationContextERKNS_20OplogUpdateEntryArgsE+0x7CC) [0x56454ea303cc]
 mongod(_ZN5mongo18OpObserverRegistry8onUpdateEPNS_16OperationContextERKNS_20OplogUpdateEntryArgsE+0x57) [0x56454e365dc7]
 mongod(_ZN5mongo14CollectionImpl14updateDocumentEPNS_16OperationContextENS_8RecordIdERKNS_11SnapshottedINS_7BSONObjEEERKS5_bPNS_7OpDebugEPNS_20CollectionUpdateArgsE+0x5CB) [0x56454ebb1e4b]
 mongod(_ZN5mongo11UpdateStage18transformAndUpdateERKNS_11SnapshottedINS_7BSONObjEEERNS_8RecordIdE+0xD2D) [0x56454ed6ef6d]
 mongod(_ZN5mongo11UpdateStage6doWorkEPm+0x2F2) [0x56454ed6fe12]
 mongod(_ZN5mongo9PlanStage4workEPm+0x68) [0x56454ed4e518]
 mongod(_ZN5mongo15ProjectionStage6doWorkEPm+0x47) [0x56454ed4e897]
 mongod(_ZN5mongo9PlanStage4workEPm+0x68) [0x56454ed4e518]
 mongod(_ZN5mongo16PlanExecutorImpl12_getNextImplEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE+0x230) [0x56454ed95850]
 mongod(_ZN5mongo16PlanExecutorImpl7getNextEPNS_7BSONObjEPNS_8RecordIdE+0x4D) [0x56454ed95fad]
 mongod(+0x1487617) [0x56454ea5b617]
 mongod(+0x1489410) [0x56454ea5d410]
 mongod(+0x148A66A) [0x56454ea5e66a]
 mongod(_ZN5mongo12BasicCommand10Invocation3runEPNS_16OperationContextEPNS_3rpc21ReplyBuilderInterfaceE+0xAF) [0x56454f87067f]
 mongod(+0x11A8EAA) [0x56454e77ceaa]
 mongod(+0x11AA1B2) [0x56454e77e1b2]
 mongod(+0x11AC204) [0x56454e780204]
 mongod(_ZN5mongo23ServiceEntryPointCommon13handleRequestEPNS_16OperationContextERKNS_7MessageERKNS0_5HooksE+0x41A) [0x56454e780f6a]
 mongod(_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE+0x3C) [0x56454e76ebbc]
 mongod(_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE+0xEC) [0x56454e77a94c]
 mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x17F) [0x56454e777c2f]
 mongod(+0x11A58BC) [0x56454e7798bc]
 mongod(_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE+0x182) [0x56454f5ab572]
 mongod(_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE+0x10D) [0x56454e774afd]
 mongod(_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE+0x753) [0x56454e776373]
 mongod(_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE+0x316) [0x56454e776f26]
 mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0xDB) [0x56454e777b8b]
 mongod(+0x11A58BC) [0x56454e7798bc]
 mongod(+0x1FD79DB) [0x56454f5ab9db]
 mongod(+0x2626F64) [0x56454fbfaf64]
 libpthread.so.0(+0x7E65) [0x7faa798e0e65]
 libc.so.6(clone+0x6D) [0x7faa7960988d]
----- END BACKTRACE -----



 Comments   
Comment by Dmitry Agranat [ 22/Apr/21 ]

Hi zubair.iub.pk@gmail.com,

We haven’t heard back from you for some time, so I’m going to close this ticket. If this is still an issue for you, please provide additional information and we will reopen the ticket.

Regards,
Dima

Comment by Dmitry Agranat [ 08/Apr/21 ]

Hi zubair.iub.pk@gmail.com, the provided diagnostic.data does not cover the time of the event, and the provided mongod log only contains the same error you've already posted in your original comment.

Do you still have diagnostic.data covering the time of the event and full mongod log covering at least several hours prior to the reported incident?

Also, was it a one-time event or did this happen again since 2021-03-31T13:32:03.651+0000?

Regards,
Dima

Comment by Zubair Ahmed [ 07/Apr/21 ]

dmitry.agranat   Thanks . incident logs and diagnostic data has been up loaded at support location.

Comment by Dmitry Agranat [ 06/Apr/21 ]

Hi zubair.iub.pk@gmail.com,

Would you please archive (tar or zip) the mongod.log files covering the incident and the $dbpath/diagnostic.data directory (the contents are described here) and upload them to this support uploader location?

Files uploaded to this portal are visible only to MongoDB employees and are routinely deleted after some time.

Dima

Generated at Thu Feb 08 05:37:18 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.