[SERVER-58318] Mongo 4.4.6 : rollback_to_stable illegal with active transactions Created: 07/Jul/21  Updated: 20/Jul/21  Resolved: 20/Jul/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Zubair Ahmed Assignee: Dmitry Agranat
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
is duplicated by WT-6729 Quiesce eviction prior running rollba... Closed
Related
related to WT-6729 Quiesce eviction prior running rollba... Closed
Operating System: ALL
Participants:

 Description   

{"t":\{"$date":"2021-07-06T15:09:30.157+00:00"},"s":"I", "c":"STORAGE", "id":22372, "ctx":"OplogVisibilityThread","msg":"Oplog visibility thread shutting down."}
{"t":\{"$date":"2021-07-06T15:09:30.160+00:00"},"s":"I", "c":"ROLLBACK", "id":23991, "ctx":"BackgroundSync","msg":"Rolling back to the stable timestamp","attr":\{"stableTimestamp":{"$timestamp":{"t":1625584094,"i":593}},"initialDataTimestamp":\{"$timestamp":{"t":1621325724,"i":5}}}}
{"t":\{"$date":"2021-07-06T15:09:30.198+00:00"},"s":"E", "c":"STORAGE", "id":22435, "ctx":"BackgroundSync","msg":"WiredTiger error","attr":\{"error":22,"message":"[1625584170:198447][2037356:0x7f244849d700], txn rollback_to_stable: __rollback_to_stable_check, 1254: rollback_to_stable illegal with active transactions: Invalid argument"}}
{"t":\{"$date":"2021-07-06T15:09:30.200+00:00"},"s":"F", "c":"-", "id":23093, "ctx":"BackgroundSync","msg":"Fatal assertion","attr":\{"msgid":31049,"error":"UnrecoverableRollbackError: Error rolling back to stable. Err: Invalid argument","file":"src/mongo/db/repl/storage_interface_impl.cpp","line":1219}}
{"t":\{"$date":"2021-07-06T15:09:30.200+00:00"},"s":"F", "c":"-", "id":23094, "ctx":"BackgroundSync","msg":"\n\n***aborting after fassert() failure\n\n"}
{"t":\{"$date":"2021-07-06T15:09:30.201+00:00"},"s":"F", "c":"CONTROL", "id":4757800, "ctx":"BackgroundSync","msg":"Writing fatal message","attr":\{"message":"Got signal: 6 (Aborted).\n"}}

{"t":\{"$date":"2021-07-06T15:09:32.359+00:00"},"s":"I", "c":"CONTROL", "id":31431, "ctx":"BackgroundSync","msg":"BACKTRACE: \{bt}","attr":\{"bt":{"backtrace":[{"a":"55873F4A8CCA","b":"55873C718000","o":"2D90CCA","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.606","s+":"1EA"},\{"a":"55873F4AA749","b":"55873C718000","o":"2D92749","s":"_ZN5mongo15printStackTraceEv","s+":"29"},\{"a":"55873F4A7AF6","b":"55873C718000","o":"2D8FAF6","s":"_ZN5mongo12_GLOBAL__N_116abruptQuitActionEiP9siginfo_tPv","s+":"66"},\{"a":"7F2470D10B20","b":"7F2470CFE000","o":"12B20","s":"funlockfile","s+":"50"},\{"a":"7F24709727FF","b":"7F247093B000","o":"377FF","s":"gsignal","s+":"10F"},\{"a":"7F247095CC35","b":"7F247093B000","o":"21C35","s":"abort","s+":"127"},\{"a":"55873D64754F","b":"55873C718000","o":"F2F54F","s":"_ZN5mongo35fassertFailedWithStatusWithLocationEiRKNS_6StatusEPKcj","s+":"178"},\{"a":"55873D35C3C9","b":"55873C718000","o":"C443C9","s":"_ZN5mongo4repl20StorageInterfaceImpl24recoverToStableTimestampEPNS_16OperationContextE","s+":"AF"},\{"a":"55873DA7B7A0","b":"55873C718000","o":"13637A0","s":"_ZN5mongo4repl12RollbackImpl25_recoverToStableTimestampEPNS_16OperationContextE","s+":"50"},\{"a":"55873DA84693","b":"55873C718000","o":"136C693","s":"_ZN5mongo4repl12RollbackImpl43_runPhaseFromAbortToReconstructPreparedTxnsEPNS_16OperationContextENS0_23RollBackLocalOperations19RollbackCommonPointE","s+":"143"},\{"a":"55873DA863D3","b":"55873C718000","o":"136E3D3","s":"_ZN5mongo4repl12RollbackImpl11runRollbackEPNS_16OperationContextE","s+":"4B3"},\{"a":"55873DA52653","b":"55873C718000","o":"133A653","s":"_ZN5mongo4repl14BackgroundSync34_runRollbackViaRecoverToCheckpointEPNS_16OperationContextERKNS_11HostAndPortEPNS0_14OplogInterfaceEPNS0_16StorageInterfaceESt8functionIFPNS_12DBClientBaseEvEE","s+":"233"},\{"a":"55873DA53321","b":"55873C718000","o":"133B321","s":"_ZN5mongo4repl14BackgroundSync12_runRollbackEPNS_16OperationContextERKNS_6StatusERKNS_11HostAndPortEiPNS0_16StorageInterfaceE","s+":"A71"},\{"a":"55873D374A0B","b":"55873C718000","o":"C5CA0B","s":"_ZN5mongo4repl14BackgroundSync8_produceEv.cold.1201","s+":"254F"},\{"a":"55873DA53816","b":"55873C718000","o":"133B816","s":"_ZN5mongo4repl14BackgroundSync12_runProducerEv","s+":"E6"},\{"a":"55873DA53AF0","b":"55873C718000","o":"133BAF0","s":"_ZN5mongo4repl14BackgroundSync4_runEv","s+":"B0"},\{"a":"55873DA53BAC","b":"55873C718000","o":"133BBAC","s":"_ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN5mongo4stdx6threadC4IZNS3_4repl14BackgroundSync7startupEPNS3_16OperationContextEEUlvE_JELi0EEET_DpOT0_EUlvE_EEEEE6_M_runEv","s+":"5C"},\{"a":"55873F651E0F","b":"55873C718000","o":"2F39E0F","s":"execute_native_thread_routine","s+":"F"},\{"a":"7F2470D0614A","b":"7F2470CFE000","o":"814A","s":"start_thread","s+":"EA"},\{"a":"7F2470A37F23","b":"7F247093B000","o":"FCF23","s":"clone","s+":"43"}],"processInfo":\{"mongodbVersion":"4.4.6","gitVersion":"72e66213c2c3eab37d9358d5e78ad7f5c1d0d0d7","compiledModules":[],"uname":{"sysname":"Linux","release":"4.18.0-193.6.3.el8_2.x86_64","version":"#1 SMP Wed Jun 10 11:09:32 UTC 2020","machine":"x86_64"},"somap":[\{"b":"55873C718000","elfType":3,"buildId":"264EA6EE10C77300B36F30AE15B879E4CE140A3C"},\{"b":"7F2470CFE000","path":"/lib64/libpthread.so.0","elfType":3,"buildId":"634E9EA2594F8C82DDDE20AB179BDA7DAEE7E18F"},\{"b":"7F247093B000","path":"/lib64/libc.so.6","elfType":3,"buildId":"ADE58D86662ACEEE2210A9EF12018705E978965D"}]}}}}
{"t":\{"$date":"2021-07-06T15:09:32.360+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"BackgroundSync","msg":" Frame: \{frame}","attr":\{"frame":{"a":"55873F4A8CCA","b":"55873C718000","o":"2D90CCA","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.606","s+":"1EA"}}}
{"t":\{"$date":"2021-07-06T15:09:32.360+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"BackgroundSync","msg":" Frame: \{frame}","attr":\{"frame":{"a":"55873F4AA749","b":"55873C718000","o":"2D92749","s":"_ZN5mongo15printStackTraceEv","s+":"29"}}}
{"t":\{"$date":"2021-07-06T15:09:32.360+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"BackgroundSync","msg":" Frame: \{frame}","attr":\{"frame":{"a":"55873F4A7AF6","b":"55873C718000","o":"2D8FAF6","s":"_ZN5mongo12_GLOBAL__N_116abruptQuitActionEiP9siginfo_tPv","s+":"66"}}}
{"t":\{"$date":"2021-07-06T15:09:32.360+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"BackgroundSync","msg":" Frame: \{frame}","attr":\{"frame":{"a":"7F2470D10B20","b":"7F2470CFE000","o":"12B20","s":"funlockfile","s+":"50"}}}
{"t":\{"$date":"2021-07-06T15:09:32.360+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"BackgroundSync","msg":" Frame: \{frame}","attr":\{"frame":{"a":"7F24709727FF","b":"7F247093B000","o":"377FF","s":"gsignal","s+":"10F"}}}
{"t":\{"$date":"2021-07-06T15:09:32.360+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"BackgroundSync","msg":" Frame: \{frame}","attr":\{"frame":{"a":"7F247095CC35","b":"7F247093B000","o":"21C35","s":"abort","s+":"127"}}}
{"t":\{"$date":"2021-07-06T15:09:32.360+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"BackgroundSync","msg":" Frame: \{frame}","attr":\{"frame":{"a":"55873D64754F","b":"55873C718000","o":"F2F54F","s":"_ZN5mongo35fassertFailedWithStatusWithLocationEiRKNS_6StatusEPKcj","s+":"178"}}}
{"t":\{"$date":"2021-07-06T15:09:32.360+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"BackgroundSync","msg":" Frame: \{frame}","attr":\{"frame":{"a":"55873D35C3C9","b":"55873C718000","o":"C443C9","s":"_ZN5mongo4repl20StorageInterfaceImpl24recoverToStableTimestampEPNS_16OperationContextE","s+":"AF"}}}
{"t":\{"$date":"2021-07-06T15:09:32.360+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"BackgroundSync","msg":" Frame: \{frame}","attr":\{"frame":{"a":"55873DA7B7A0","b":"55873C718000","o":"13637A0","s":"_ZN5mongo4repl12RollbackImpl25_recoverToStableTimestampEPNS_16OperationContextE","s+":"50"}}}
{"t":\{"$date":"2021-07-06T15:09:32.360+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"BackgroundSync","msg":" Frame: \{frame}","attr":\{"frame":{"a":"55873DA84693","b":"55873C718000","o":"136C693","s":"_ZN5mongo4repl12RollbackImpl43_runPhaseFromAbortToReconstructPreparedTxnsEPNS_16OperationContextENS0_23RollBackLocalOperations19RollbackCommonPointE","s+":"143"}}}
{"t":\{"$date":"2021-07-06T15:09:32.360+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"BackgroundSync","msg":" Frame: \{frame}","attr":\{"frame":{"a":"55873DA863D3","b":"55873C718000","o":"136E3D3","s":"_ZN5mongo4repl12RollbackImpl11runRollbackEPNS_16OperationContextE","s+":"4B3"}}}
{"t":\{"$date":"2021-07-06T15:09:32.360+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"BackgroundSync","msg":" Frame: \{frame}","attr":\{"frame":{"a":"55873DA52653","b":"55873C718000","o":"133A653","s":"_ZN5mongo4repl14BackgroundSync34_runRollbackViaRecoverToCheckpointEPNS_16OperationContextERKNS_11HostAndPortEPNS0_14OplogInterfaceEPNS0_16StorageInterfaceESt8functionIFPNS_12DBClientBaseEvEE","s+":"233"}}}
{"t":\{"$date":"2021-07-06T15:09:32.360+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"BackgroundSync","msg":" Frame: \{frame}","attr":\{"frame":{"a":"55873DA53321","b":"55873C718000","o":"133B321","s":"_ZN5mongo4repl14BackgroundSync12_runRollbackEPNS_16OperationContextERKNS_6StatusERKNS_11HostAndPortEiPNS0_16StorageInterfaceE","s+":"A71"}}}
{"t":\{"$date":"2021-07-06T15:09:32.360+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"BackgroundSync","msg":" Frame: \{frame}","attr":\{"frame":{"a":"55873D374A0B","b":"55873C718000","o":"C5CA0B","s":"_ZN5mongo4repl14BackgroundSync8_produceEv.cold.1201","s+":"254F"}}}
{"t":\{"$date":"2021-07-06T15:09:32.360+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"BackgroundSync","msg":" Frame: \{frame}","attr":\{"frame":{"a":"55873DA53816","b":"55873C718000","o":"133B816","s":"_ZN5mongo4repl14BackgroundSync12_runProducerEv","s+":"E6"}}}
{"t":\{"$date":"2021-07-06T15:09:32.360+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"BackgroundSync","msg":" Frame: \{frame}","attr":\{"frame":{"a":"55873DA53AF0","b":"55873C718000","o":"133BAF0","s":"_ZN5mongo4repl14BackgroundSync4_runEv","s+":"B0"}}}
{"t":\{"$date":"2021-07-06T15:09:32.360+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"BackgroundSync","msg":" Frame: \{frame}","attr":\{"frame":{"a":"55873DA53BAC","b":"55873C718000","o":"133BBAC","s":"_ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN5mongo4stdx6threadC4IZNS3_4repl14BackgroundSync7startupEPNS3_16OperationContextEEUlvE_JELi0EEET_DpOT0_EUlvE_EEEEE6_M_runEv","s+":"5C"}}}
{"t":\{"$date":"2021-07-06T15:09:32.360+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"BackgroundSync","msg":" Frame: \{frame}","attr":\{"frame":{"a":"55873F651E0F","b":"55873C718000","o":"2F39E0F","s":"execute_native_thread_routine","s+":"F"}}}
{"t":\{"$date":"2021-07-06T15:09:32.360+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"BackgroundSync","msg":" Frame: \{frame}","attr":\{"frame":{"a":"7F2470D0614A","b":"7F2470CFE000","o":"814A","s":"start_thread","s+":"EA"}}}
{"t":\{"$date":"2021-07-06T15:09:32.360+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"BackgroundSync","msg":" Frame: \{frame}","attr":\{"frame":{"a":"7F2470A37F23","b":"7F247093B000","o":"FCF23","s":"clone","s+":"43"}}}



 Comments   
Comment by Dmitry Agranat [ 11/Jul/21 ]

Thanks zubair.iub.pk@gmail.com for providing the requested data. We believe this issue is related to WT-6729, please watch WT-6729 for updates and fixed versions.

Comment by Zubair Ahmed [ 08/Jul/21 ]

Hi dmitry.agranat,

Required incident logs and diagnostic data has been uploaded at support uploader location. Please have a look on it. Thanks.

 

Comment by Dmitry Agranat [ 08/Jul/21 ]

Hi zubair.iub.pk@gmail.com,

Would you please archive (tar or zip) the full mongod.log file covering the incident and the $dbpath/diagnostic.data directory (the contents are described here) and upload them to this support uploader location?

Files uploaded to this portal are visible only to MongoDB employees and are routinely deleted after some time.

Thanks,
Dima

Generated at Thu Feb 08 05:44:11 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.