[SERVER-55894] WiredTiger error (22) and Fatal assertion 39001 Created: 08/Apr/21 Updated: 02/Nov/22 Resolved: 22/Apr/21 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | 4.2.8 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Srikanta Patra | Assignee: | Dmitry Agranat |
| Resolution: | Incomplete | Votes: | 0 |
| Labels: | WiredTiger | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Please find below OS details and hardware specification [srikanta@ip-172-31-4-86 etc]$ lscpu |
||
| Attachments: |
|
||||||||||||||||||||
| Issue Links: |
|
||||||||||||||||||||
| Participants: | |||||||||||||||||||||
| Description |
|
We are also getting same error on production server. Please help to get it resolve. 2021-04-02T22:52:49.692+0530 E STORAGE [conn4020] WiredTiger error (22) [1617384169:692561][5626:0x7f620f4f5700], WT_SESSION.timestamp_transaction: __wt_txn_set_commit_timestamp, 683: commit timestamp (1617384169, 810) is less than the oldest timestamp (1617384169, 811): Invalid argument Raw: [1617384169:692561][5626:0x7f620f4f5700], WT_SESSION.timestamp_transaction: __wt_txn_set_commit_timestamp, 683: commit timestamp (1617384169, 810) is less than the oldest timestamp (1617384169, 811): Invalid argument |
| Comments |
| Comment by Eduardo Legatti [ 26/Jan/22 ] |
|
Hi, Is there a way to reopen this thread? I'm using MongoDB version 4.2.17 and I got the same error today. |
| Comment by Eduardo Legatti [ 19/Nov/21 ] |
|
@Dmitry Agranat Same error here in 4.2.8 version Linux 86/64 in production environment. This problem ocurred two times in the last 30 days. 2021-11-19T06:19:03.936-0300 E STORAGE [conn534610] WiredTiger error (22) [1637313543:936727][16877:0x7f6d29bcf700], WT_SESSION.timestamp_transaction: __wt_txn_set_commit_timestamp, 683: commit timestamp (1637313543, 2321) is less than the oldest timestamp (1637313543, 2322): Invalid argument Raw: [1637313543:936727][16877:0x7f6d29bcf700], WT_SESSION.timestamp_transaction: __wt_txn_set_commit_timestamp, 683: commit timestamp (1637313543, 2321) is less than the oldest timestamp (1637313543, 2322): Invalid argument |
| Comment by Srikanta Patra [ 29/Sep/21 ] |
|
Hi @Dmitry Agranat,
I attached mongodb log as well. Please let us know to fix this issue.
Thanks, Srikanta Patra |
| Comment by Srikanta Patra [ 28/Sep/21 ] |
|
Hi @Dmitry Agranat, Please find attached diagnostic.data dir logs.[^diagnostic.data.rar] I will upload the mongodb log as well. Thanks, Srikanta Patra |
| Comment by Srikanta Patra [ 28/Sep/21 ] |
|
Hi @Dmitry Agranat, Again we got the same error. Please find below logs. 2021-09-21T22:37:13.898+0530 E STORAGE [conn56586] WiredTiger error (22) [1632244033:898964][1411:0x7f18a5c77700], WT_SESSION.timestamp_transaction: __wt_txn_set_commit_timestamp, 683: commit timestamp (1632244033, 1048) is less than the oldest timestamp (1632244033, 1051): Invalid argument Raw: [1632244033:898964][1411:0x7f18a5c77700], WT_SESSION.timestamp_transaction: __wt_txn_set_commit_timestamp, 683: commit timestamp (1632244033, 1048) is less than the oldest timestamp (1632244033, 1051): Invalid argument ***aborting after fassert() failure 2021-09-21T22:37:13.966+0530 F - [conn56586] Got signal: 6 (Aborted). ,{"b":"556EDF606000","o":"28BE20C"},{"b":"556EDF606000","o":"28BE296"},{"b":"7F18CC036000","o":"117E0"},{"b":"7F18CBC8B000","o":"33C20","s":"gsignal"},{"b":"7F18CBC8B000","o":"350C8","s":"abort"},{"b":"556EDF606000","o":"D14BA1","s":"_ZN5mongo42fassertFailedWithStatusNoTraceWithLocationEiRKNS_6StatusEPKcj"},{"b":"556EDF606000","o":"A4DD4D"},{"b":"556EDF606000","o":"E12F2C","s":"_ZN5mongo21WiredTigerRecordStore26insertRecordsWithDocWriterEPNS_16OperationContextEPKPKNS_9DocWriterEPKNS_9TimestampEmPNS_8RecordIdE"},{"b":"556EDF606000","o":"15E07E1","s":"_ZN5mongo14CollectionImpl23insertDocumentsForOplogEPNS_16OperationContextEPKPKNS_9DocWriterEPNS_9TimestampEm"},{"b":"556EDF606000","o":"1675613","s":"_ZN5mongo4repl12_logOpsInnerEPNS_16OperationContextERKNS_15NamespaceStringEPKPKNS_9DocWriterEPNS_9TimestampEmPNS_10CollectionENS0_6OpTimeENS_6Date_tE"},{"b":"556EDF606000","o":"1679DAC","s":"_ZN5mongo4repl5logOpEPNS_16OperationContextEPKcRKNS_15NamespaceStringEN5boost8optionalINS_4UUIDEEERKNS_7BSONObjEPSD_bNS_6Date_tERKNS_20OperationSessionInfoENS9_IiEERKNS0_9OplogLinkERKNS0_6OpTimeE"},{"b":"556EDF606000","o":"1461DBB"},{"b":"556EDF606000","o":"1467DBC","s":"_ZN5mongo14OpObserverImpl8onUpdateEPNS_16OperationContextERKNS_20OplogUpdateEntryArgsE"},{"b":"556EDF606000","o":"DA0107","s":"_ZN5mongo18OpObserverRegistry8onUpdateEPNS_16OperationContextERKNS_20OplogUpdateEntryArgsE"},{"b":"556EDF606000","o":"15E6C8C","s":"_ZN5mongo14CollectionImpl25updateDocumentWithDamagesEPNS_16OperationContextENS_8RecordIdERKNS_11SnapshottedINS_10RecordDataEEEPKcRKSt6vectorINS_11mutablebson11DamageEventESaISD_EEPNS_20CollectionUpdateArgsE"},{"b":"556EDF606000","o":"17A5B2D","s":"_ZN5mongo11UpdateStage18transformAndUpdateERKNS_11SnapshottedINS_7BSONObjEEERNS_8RecordIdE"},{"b":"556EDF606000","o":"17A7212","s":"_ZN5mongo11UpdateStage6doWorkEPm"},{"b":"556EDF606000","o":"1785908","s":"_ZN5mongo9PlanStage4workEPm"},{"b":"556EDF606000","o":"17CCE30","s":"_ZN5mongo16PlanExecutorImpl12_getNextImplEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE"},{"b":"556EDF606000","o":"17CD58D","s":"_ZN5mongo16PlanExecutorImpl7getNextEPNS_7BSONObjEPNS_8RecordIdE"},{"b":"556EDF606000","o":"14928E7"},{"b":"556EDF606000","o":"14946FB"},{"b":"556EDF606000","o":"149596A"},{"b":"556EDF606000","o":"22ADCEF","s":"_ZN5mongo12BasicCommand10Invocation3runEPNS_16OperationContextEPNS_3rpc21ReplyBuilderInterfaceE"},{"b":"556EDF606000","o":"118B58A"},{"b":"556EDF606000","o":"118C875"},{"b":"556EDF606000","o":"118E8F4"},{"b":"556EDF606000","o":"118F68A","s":"_ZN5mongo23ServiceEntryPointCommon13handleRequestEPNS_16OperationContextERKNS_7MessageERKNS0_5HooksE"},{"b":"556EDF606000","o":"117D20C","s":"_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE"},{"b":"556EDF606000","o":"1188F9C","s":"_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE"},{"b":"556EDF606000","o":"118627F","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"556EDF606000","o":"1187F0C"},{"b":"556EDF606000","o":"1FE3F82","s":"_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE"},{"b":"556EDF606000","o":"118314D","s":"_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE"},{"b":"556EDF606000","o":"11849C3","s":"_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE"},{"b":"556EDF606000","o":"1185576","s":"_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE"},{"b":"556EDF606000","o":"11861DB","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"556EDF606000","o":"1187F0C"},{"b":"556EDF606000","o":"1FE43EB"},{"b":"556EDF606000","o":"2639F35"},{"b":"556EDF606000","o":"2639F94"},{"b":"7F18CC036000","o":"740B"},{"b":"7F18CBC8B000","o":"ED09F","s":"clone"}],"processInfo":{ "mongodbVersion" : "4.2.8", "gitVersion" : "43d25964249164d76d5e04dd6cf38f6111e21f5f", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.14.231-173.361.amzn2.x86_64", "version" : "#1 SMP Mon Apr 26 20:57:08 UTC 2021", "machine" : "x86_64" }, "somap" : [ { "b" : "556EDF606000", "elfType" : 3, "buildId" : "1118CB24C802841614B9E8C04394950FD56BD9C3" }, { "b" : "7FFD59FA8000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "FD7ED28E435B892351687132649AD89F7DDB810E" }, { "b" : "7F18CD491000", "path" : "/lib64/libcurl.so.4", "elfType" : 3, "buildId" : "B737D249E957BD15A72E1E0819C56A24EBF6E2F3" }, { "b" : "7F18CD27B000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "D54CF97BF07398E614DFCC7C91BCFC5E9071A947" }, { "b" : "7F18CCE26000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "6A0BDB33426B1355207BF1450AAE789918AE14CC" }, { "b" : "7F18CCBB7000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "C589FBDB14C2DB872D9A9BE1B7692A40E25B35DE" }, { "b" : "7F18CC9B3000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "765CB7C0C3F23012206E28836DA842B4E26CEA32" }, { "b" : "7F18CC7AB000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "73AAF1068EFB9CDB2C5D7B30A3737A3632BD8BD6" }, { "b" : "7F18CC46B000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "2EDF81157EE48F2E86E342A8129249A69FFB6358" }, { "b" : "7F18CC254000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "025CC3324B759F1D51215F04B7976A87BA655534" }, { "b" : "7F18CC036000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "FBB1EBCD3AA68F9FF71EAFF037EC3F0EB1FE45B9" }, { "b" : "7F18CBC8B000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "8779C8D644555E9918F28FB0B801AEA29A6692B8" }, { "b" : "7F18CD716000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "BB7472F0482DB35A29453A96BEE279DAF83C0ADB" }, { "b" : "7F18CBA64000", "path" : "/lib64/libnghttp2.so.14", "elfType" : 3, "buildId" : "ABE2627C22FDC7706352089D224CA49759CB74F9" }, { "b" : "7F18CB817000", "path" : "/lib64/libidn2.so.0", "elfType" : 3, "buildId" : "F570AD4111FD2ADDA7A7052E4A8ADB2427D62703" }, { "b" : "7F18CB5EF000", "path" : "/lib64/libssh2.so.1", "elfType" : 3, "buildId" : "DB7387A967A1DB497B643672C4E4DA5A9931D347" }, { "b" : "7F18CB3A2000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "B66D7C7DE3D59F794AD60D2B71482E17484CF860" }, { "b" : "7F18CB0BA000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "9677E50188E2ADDBED7302D36BECA8E1F616E77E" }, { "b" : "7F18CAE88000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "4D65FAB419F439A47361C55866A78A9768774517" }, { "b" : "7F18CAC84000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "617A0D6614294A0460D068C62AC3EEB5D6E93A25" }, { "b" : "7F18CAA32000", "path" : "/lib64/libldap-2.4.so.2", "elfType" : 3, "buildId" : "203805A61C19203C8042A8360F3908C34DA43E4B" }, { "b" : "7F18CA823000", "path" : "/lib64/liblber-2.4.so.2", "elfType" : 3, "buildId" : "2FFCC7CCB69677D18DD4EEAEFBACAB966303539D" }, { "b" : "7F18CA60D000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "B3D7471B1072E67716DF857D800318A7F47A09FB" }, { "b" : "7F18CA2F5000", "path" : "/lib64/libunistring.so.0", "elfType" : 3, "buildId" : "058ACB5009C38557BD075015D6D55DFCB0118EB4" }, { "b" : "7F18CA0E6000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "E35E6A8E73183AD3B40E8A21574E1D2780ADB599" }, { "b" : "7F18C9EE2000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "895D9F7FAA8E8ADCF110AA559D251258EFD4C924" }, { "b" : "7F18C9CC5000", "path" : "/lib64/libsasl2.so.3", "elfType" : 3, "buildId" : "5E6E0F28762D9135F24BAD9D1B2723833DFE4C29" }, { "b" : "7F18C9A6B000", "path" : "/lib64/libssl3.so", "elfType" : 3, "buildId" : "366998CC2B48C772F744CA5CA8E5253ED665EE3F" }, { "b" : "7F18C9845000", "path" : "/lib64/libsmime3.so", "elfType" : 3, "buildId" : "D818A3C2CE10C9180464EE505794530C1A90AEF7" }, { "b" : "7F18C951D000", "path" : "/lib64/libnss3.so", "elfType" : 3, "buildId" : "54B32A2179988CAE4A11FA8F86AF1E9E2760073E" }, { "b" : "7F18C92EE000", "path" : "/lib64/libnssutil3.so", "elfType" : 3, "buildId" : "450373B22AF5668A9C0325FEFC918B396A66273C" }, { "b" : "7F18C90EA000", "path" : "/lib64/libplds4.so", "elfType" : 3, "buildId" : "A9EF38B34F69655160214C303487A6881992B229" }, { "b" : "7F18C8EE5000", "path" : "/lib64/libplc4.so", "elfType" : 3, "buildId" : "CD8713077A5983BFA6C267691EC82A077A46842C" }, { "b" : "7F18C8CA9000", "path" : "/lib64/libnspr4.so", "elfType" : 3, "buildId" : "05DADE6B01040412C2F7C18DE47056558821D028" }, { "b" : "7F18C8A82000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "8271DC15F869AE83F5A370302B4C3F0D9D6338E4" }, { "b" : "7F18C884B000", "path" : "/lib64/libcrypt.so.1", "elfType" : 3, "buildId" : "89E07B214D9DE630668A0E514B097098A90EA589" }, { "b" : "7F18C85E7000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "1E027913526FE9BC71D008691D3B7DBE88C880DE" } ] }} |
| Comment by Dmitry Agranat [ 22/Apr/21 ] |
|
We haven’t heard back from you for some time, so I’m going to close this ticket. If this is still an issue for you, please provide additional information and we will reopen the ticket. Regards, |
| Comment by Dmitry Agranat [ 08/Apr/21 ] |
|
Would you please archive (tar or zip) the mongod.log files covering the incident and the $dbpath/diagnostic.data directory (the contents are described here) and upload them to this support uploader location? Files uploaded to this portal are visible only to MongoDB employees and are routinely deleted after some time. Regards, |