[SERVER-67799] [WTCheckpointThread] Got signal: 6 (Aborted) Created: 06/Jul/22 Updated: 04/Oct/22 Resolved: 04/Oct/22 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | 4.2 Required |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Daniel Raj | Assignee: | Edwin Zhou |
| Resolution: | Done | Votes: | 0 |
| Labels: | Bug | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Operating System: | ALL |
| Steps To Reproduce: | Load testing |
| Participants: |
| Description |
|
Hi, Mongodb version 4.2.20 is being used.
Any leads about why it was aborted would help and how to analyze such issues?
2022-06-21T11:58:48.644+0000 I COMMAND [conn10287] command DB_PROFILE.$cmd command: update { update: "micro_service", ordered: false, writeConcern: { w: "majority" }, $db: "DB_PROFILE" } numYields:0 reslen:260 locks:{ ParallelBatchWriterMode: { acquireCount: { r: 1 }}, ReplicationStateTransition: { acquireCount: { w: 1 }}, Global: { acquireCount: { w: 1 }}, Database: { acquireCount: { w: 1 }}, Collection: { acquireCount: { w: 1 }}, Mutex: { acquireCount: { r: 2 } } } flowControl:{ acquireCount: 1, timeAcquiringMicros: 1 } storage:{} protocol:op_query 463ms , txnNumber: 208326, $db: "DB_PROFILE", $clusterTime: { clusterTime: Timestamp(1655812725, 1), signature: { hash: BinData(0, DAB13C0AD573C8D0D1B3C5986F4DDFAF096811E1), keyId: 7110219703129210885 }}, lsid: { id: UUID("798984a2-96eb-42ac-93d3-3747b08bfe80") } } numYields:0 reslen:245 locks:{ ParallelBatchWriterMode: { acquireCount: { r: 2 }}, ReplicationStateTransition: { acquireCount: { w: 3 }}, Global: { acquireCount: { w: 2 }}, Database: { acquireCount: { w: 2 }}, Collection: { acquireCount: { w: 2 }}, Mutex: { acquireCount: { r: 3 } } } flowControl:{ acquireCount: 1, timeAcquiringMicros: 1 } storage:{} protocol:op_msg 464ms , $db: "DB_SUBSCRIBER" } numYields:0 reslen:260 locks:{ ParallelBatchWriterMode: { acquireCount: { r: 1 }}, ReplicationStateTransition: { acquireCount: { w: 1 }}, Global: { acquireCount: { w: 1 }}, Database: { acquireCount: { w: 1 }}, Collection: { acquireCount: { w: 1 }}, Mutex: { acquireCount: { r: 2 } } } flowControl:{ acquireCount: 1 } storage:{} protocol:op_query 808ms , $db: "DB_PROFILE" } numYields:0 reslen:260 locks:{ ParallelBatchWriterMode: { acquireCount: { r: 1 }}, ReplicationStateTransition: { acquireCount: { w: 1 }}, Global: { acquireCount: { w: 1 }}, Database: { acquireCount: { w: 1 }}, Collection: { acquireCount: { w: 1 }}, Mutex: { acquireCount: { r: 1 } } } flowControl:{ acquireCount: 1, timeAcquiringMicros: 1 } storage:{} protocol:op_query 769ms , $db: "DB_SUBSCRIBER" } numYields:0 reslen:245 locks:{ ParallelBatchWriterMode: { acquireCount: { r: 1 }}, ReplicationStateTransition: { acquireCount: { w: 1 }}, Global: { acquireCount: { w: 1 }}, Database: { acquireCount: { w: 1 }}, Collection: { acquireCount: { w: 1 }}, Mutex: { acquireCount: { r: 1 } } } flowControl:{ acquireCount: 1, timeAcquiringMicros: 1 } storage:{} protocol:op_query 1024ms , $db: "DB_SUBSCRIBER" } numYields:0 reslen:245 locks:{ ParallelBatchWriterMode: { acquireCount: { r: 1 }}, ReplicationStateTransition: { acquireCount: { w: 1 }}, Global: { acquireCount: { w: 1 }}, Database: { acquireCount: { w: 1 }}, Collection: { acquireCount: { w: 1 }}, Mutex: { acquireCount: { r: 1 } } } flowControl:{ acquireCount: 1, timeAcquiringMicros: 1 } storage:{} protocol:op_query 958ms , $db: "DB_SUBSCRIBER" } numYields:0 reslen:260 locks:{ ParallelBatchWriterMode: { acquireCount: { r: 1 }}, ReplicationStateTransition: { acquireCount: { w: 1 }}, Global: { acquireCount: { w: 1 }}, Database: { acquireCount: { w: 1 }}, Collection: { acquireCount: { w: 1 }}, Mutex: { acquireCount: { r: 2 } } } flowControl:{ acquireCount: 1, timeAcquiringMicros: 1 } storage:{} protocol:op_query 1088ms , $db: "DB_PROFILE" } numYields:0 reslen:260 locks:{ ParallelBatchWriterMode: { acquireCount: { r: 1 }}, ReplicationStateTransition: { acquireCount: { w: 1 }}, Global: { acquireCount: { w: 1 }}, Database: { acquireCount: { w: 1 }}, Collection: { acquireCount: { w: 1 }}, Mutex: { acquireCount: { r: 2 }} } flowControl:{ acquireCount: 1, timeAcquiringMicros: 1 } storage:{} protocol:op_query 786ms
2022-06-21T11:58:49.182+0000 I COMMAND [conn10159] command DB_PROFILE.$cmd command: update { update: "micro_service", ordered: false, writeConcern: { w: "majority" }, $db: "DB_PROFILE" } numYields:0 reslen:260 locks:{ ParallelBatchWriterMode: { acquireCount: { r: 1 }}, ReplicationStateTransition: { acquireCount: { w: 1 }}, Global: { acquireCount: { w: 1 }}, Database: { acquireCount: { w: 1 }}, Collection: { acquireCount: { w: 1 }}, Mutex: { acquireCount: { r: 1 } } } flowControl:{ acquireCount: 1, timeAcquiringMicros: 8 } storage:{} protocol:op_query 193ms , $db: "DB_SUBSCRIBER" } numYields:0 reslen:260 locks:{ ParallelBatchWriterMode: { acquireCount: { r: 1 }}, ReplicationStateTransition: { acquireCount: { w: 1 }}, Global: { acquireCount: { w: 1 }}, Database: { acquireCount: { w: 1 }}, Collection: { acquireCount: { w: 1 }}, Mutex: { acquireCount: { r: 1 }} } flowControl:{ acquireCount: 1, timeAcquiringMicros: 2 } storage:{ data: { bytesRead: 128080, timeReadingMicros: 330 } } protocol:op_query 194ms , $db: "DB_SUBSCRIBER" } numYields:0 reslen:260 locks:{ ParallelBatchWriterMode: { acquireCount: { r: 1 }}, ReplicationStateTransition: { acquireCount: { w: 1 }}, Global: { acquireCount: { w: 1 }}, Database: { acquireCount: { w: 1 }}, Collection: { acquireCount: { w: 1 }}, Mutex: { acquireCount: { r: 2 } } } flowControl:{ acquireCount: 1, timeAcquiringMicros: 2 } storage:{} protocol:op_query 194ms ,{"b":"55884CF53000","o":"2991DBE"},{"b":"55884CF53000","o":"2991E56"},{"b":"7FEF124DD000","o":"12980"},{"b":"7FEF120EC000","o":"3EE87","s":"gsignal"},{"b":"7FEF120EC000","o":"407F1","s":"abort"},{"b":"55884CF53000","o":"D40600","s":"ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"55884CF53000","o":"A6EF50"},{"b":"55884CF53000","o":"EF66FB"},{"b":"55884CF53000","o":"A80122","s":"wt_err_func"},{"b":"55884CF53000","o":"A80589","s":"wt_panic"},{"b":"55884CF53000","o":"FDDCF9","s":"wt_block_checkpoint"},{"b":"55884CF53000","o":"ED821C"},{"b":"55884CF53000","o":"ED94EE","s":"wt_reconcile"},{"b":"55884CF53000","o":"F5C08D","s":"wt_sync_file"},{"b":"55884CF53000","o":"F092A4"},{"b":"55884CF53000","o":"F0BBCD"},{"b":"55884CF53000","o":"F0D156","s":"_wt_txn_checkpoint"},{"b":"55884CF53000","o":"A7F728"},{"b":"55884CF53000","o":"E39AA4","s":"_ZN5mongo18WiredTigerKVEngine26WiredTigerCheckpointThread3runEv"},{"b":"55884CF53000","o":"2859D90","s":"_ZN5mongo13BackgroundJob7jobBodyEv"},{"b":"55884CF53000","o":"285A3CE"},{"b":"55884CF53000","o":"2ABB09F"},{"b":"7FEF124DD000","o":"76DB"},{"b":"7FEF120EC000","o":"12161F","s":"clone"}],"processInfo":{ "mongodbVersion" : "4.2.20", "gitVersion" : "15c0712952c356cb711c13a42cb3bca8617d4ebc", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.15.0-176-generic", "version" : "#185-Ubuntu SMP Tue Mar 29 17:40:04 UTC 2022", "machine" : "x86_64" }, "somap" : [ { "b" : "55884CF53000", "elfType" : 3, "buildId" : "EC22436D24BD41324558A58015C26F8711C26419" }, { "b" : "7FFD0514E000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "4BA8BE578511382D0AEC3A10C107C8BDD0BFB52A" }, { "b" : "7FEF13A30000", "path" : "/usr/lib/x86_64-linux-gnu/libcurl.so.4", "elfType" : 3, "buildId" : "7C9EF9CA1C1C9F9925ECF63E033DA8859BC1E108" }, { "b" : "7FEF13816000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "052338C7AE2BAC6E1E5C0EE5924F7560FD9E025B" }, { "b" : "7FEF1334B000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1", "elfType" : 3, "buildId" : "C518099E5DE8D6AD61A166C63F4254D6E2DA3A93" }, { "b" : "7FEF130BE000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.1", "elfType" : 3, "buildId" : "CA742A427E7AEF089B39C4D773C20EA9E074CE8E" }, { "b" : "7FEF12EBA000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "52E7F9CD225B05DCEE4DE1BA87004D670DF130D2" }, { "b" : "7FEF12CB2000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "35E6D93BDF9DA528ABB754DCF2FA8D09DFCB8BF8" }, { "b" : "7FEF12914000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "D3522C3C754AC7C2E933A71E636F0C0520449263" }, { "b" : "7FEF126FC000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "679F3AE11120EC7C483BC9295345D836F5C104F7" }, { "b" : "7FEF124DD000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "8B60B9F8D9A27F2264A0D8E6C0768B3B02C29926" }, { "b" : "7FEF120EC000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "71F0F3074A929E519E85F6A5C03A7D1FD976BFE4" }, { "b" : "7FEF13CB2000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "02AA5DB099824E560F98F6E2E3F6848B053BFEED" }, { "b" : "7FEF11EC7000", "path" : "/usr/lib/x86_64-linux-gnu/libnghttp2.so.14", "elfType" : 3, "buildId" : "4F00E5207693FDC249DA42EC6472ACA6A7B929AE" }, { "b" : "7FEF11CAA000", "path" : "/usr/lib/x86_64-linux-gnu/libidn2.so.0", "elfType" : 3, "buildId" : "EE6E9462BA2491F4EE8C4E52C3323274A9366614" }, { "b" : "7FEF11A8E000", "path" : "/usr/lib/x86_64-linux-gnu/librtmp.so.1", "elfType" : 3, "buildId" : "69465D8AA6B19086ABF2455A703F9168BF82A69F" }, { "b" : "7FEF11880000", "path" : "/usr/lib/x86_64-linux-gnu/libpsl.so.5", "elfType" : 3, "buildId" : "CDAF1F1946846941F9D06414EC8C812D131A168E" }, { "b" : "7FEF11635000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "9FD187FE6579DB3B458E021CCAC0ECCAFC66B1B7" }, { "b" : "7FEF113E3000", "path" : "/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2", "elfType" : 3, "buildId" : "3455E5726038F673FC9C90385079A2113FE4F55E" }, { "b" : "7FEF111D5000", "path" : "/usr/lib/x86_64-linux-gnu/liblber-2.4.so.2", "elfType" : 3, "buildId" : "5F842A7CA8F5C84B338CCCB703F88F020057C77E" }, { "b" : "7FEF10FB8000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "6517F816C946C49AD246642F79493CD0CA2629DB" }, { "b" : "7FEF10C3A000", "path" : "/usr/lib/x86_64-linux-gnu/libunistring.so.2", "elfType" : 3, "buildId" : "0E2784298E7D3F4D894FE130ACEFA77C3E624F72" }, { "b" : "7FEF108D4000", "path" : "/usr/lib/x86_64-linux-gnu/libgnutls.so.30", "elfType" : 3, "buildId" : "DA3C78836D1C97B4158870FB2A84817C0653D8E9" }, { "b" : "7FEF1069E000", "path" : "/usr/lib/x86_64-linux-gnu/libhogweed.so.4", "elfType" : 3, "buildId" : "372EE4A2E4BB08F1A2B01AE486F88E7466ADD389" }, { "b" : "7FEF10468000", "path" : "/usr/lib/x86_64-linux-gnu/libnettle.so.6", "elfType" : 3, "buildId" : "433053B348A412F5CF22D07F699D412AD753FBE5" }, { "b" : "7FEF101E7000", "path" : "/usr/lib/x86_64-linux-gnu/libgmp.so.10", "elfType" : 3, "buildId" : "D40EA9B5EC5BC46799E4A412319617BD38BE9341" }, { "b" : "7FEF0FF11000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.3", "elfType" : 3, "buildId" : "320112EE17C7F1886DBEB108429063B22D996E9E" }, { "b" : "7FEF0FCDF000", "path" : "/usr/lib/x86_64-linux-gnu/libk5crypto.so.3", "elfType" : 3, "buildId" : "729F89DD92E3441FA99F8682533F7B92EE48880D" }, { "b" : "7FEF0FADB000", "path" : "/lib/x86_64-linux-gnu/libcom_err.so.2", "elfType" : 3, "buildId" : "17107881DF65C66B4C6D38CAB37C285FA44663BD" }, { "b" : "7FEF0F8D0000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5support.so.0", "elfType" : 3, "buildId" : "EAE217D2281E12BCD43675C73C59DF29D7167C66" }, { "b" : "7FEF0F6B5000", "path" : "/usr/lib/x86_64-linux-gnu/libsasl2.so.2", "elfType" : 3, "buildId" : "AA489C2B43FF4886958E90DFBD9DB3825D8F6A9A" }, { "b" : "7FEF0F474000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi.so.3", "elfType" : 3, "buildId" : "A1A98DB481968073636BBAECB561A3EA8ED198AE" }, { "b" : "7FEF0F145000", "path" : "/usr/lib/x86_64-linux-gnu/libp11-kit.so.0", "elfType" : 3, "buildId" : "8CFF559D91E3376842A1BA41CCF74EB9DECD159B" }, { "b" : "7FEF0EF32000", "path" : "/usr/lib/x86_64-linux-gnu/libtasn1.so.6", "elfType" : 3, "buildId" : "6036B89A3BB671B32E01464C0C82BFA016186352" }, { "b" : "7FEF0ED2E000", "path" : "/lib/x86_64-linux-gnu/libkeyutils.so.1", "elfType" : 3, "buildId" : "CCF39E49333249E43F465986B782D687686D9BBB" }, { "b" : "7FEF0EB25000", "path" : "/usr/lib/x86_64-linux-gnu/libheimntlm.so.0", "elfType" : 3, "buildId" : "C2376C5B831991591F1A67B976758185F86896D8" }, { "b" : "7FEF0E898000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.26", "elfType" : 3, "buildId" : "69BDEE5FA0FEEDF317308BE850F78761861D520A" }, { "b" : "7FEF0E5F6000", "path" : "/usr/lib/x86_64-linux-gnu/libasn1.so.8", "elfType" : 3, "buildId" : "315D74995AAA32DE4D15BA25F335066988B1B230" }, { "b" : "7FEF0E3C0000", "path" : "/usr/lib/x86_64-linux-gnu/libhcrypto.so.4", "elfType" : 3, "buildId" : "6673972A1C24A89EBAFBAE696188A4CB26C6DDEB" }, { "b" : "7FEF0E1AA000", "path" : "/usr/lib/x86_64-linux-gnu/libroken.so.18", "elfType" : 3, "buildId" : "430827C33259C12248CF44B91A9A9821114376F5" }, { "b" : "7FEF0DFA2000", "path" : "/usr/lib/x86_64-linux-gnu/libffi.so.6", "elfType" : 3, "buildId" : "3555B5F599C9787DFDDBF9E8DF6F706B9044D985" }, { "b" : "7FEF0DD79000", "path" : "/usr/lib/x86_64-linux-gnu/libwind.so.0", "elfType" : 3, "buildId" : "93A0931B1C2818F0EA224CE6FE5E31E84A9B55BB" }, { "b" : "7FEF0DB6A000", "path" : "/usr/lib/x86_64-linux-gnu/libheimbase.so.1", "elfType" : 3, "buildId" : "669D4CCE42FA4382796EFFCF0C16F459F4382C4C" }, { "b" : "7FEF0D920000", "path" : "/usr/lib/x86_64-linux-gnu/libhx509.so.5", "elfType" : 3, "buildId" : "4B80C543356EE0AF9039EFE7C9EA1CC1F74C426A" }, { "b" : "7FEF0D617000", "path" : "/usr/lib/x86_64-linux-gnu/libsqlite3.so.0", "elfType" : 3, "buildId" : "1E52E04B0722C6CFB8E534FB8DA45DDF95B46C19" }, { "b" : "7FEF0D3DF000", "path" : "/lib/x86_64-linux-gnu/libcrypt.so.1", "elfType" : 3, "buildId" : "A8BAF433206E7FA4E8C524EC445162ECBA55A2F4" } ] }} |
| Comments |
| Comment by Edwin Zhou [ 04/Oct/22 ] | |
|
We haven’t heard back from you for some time, so I’m going to close this ticket. If this is still an issue for you, please provide additional information and we will reopen the ticket. | |
| Comment by Edwin Zhou [ 20/Sep/22 ] | |
|
We still need additional information to diagnose the problem. If this is still an issue for you, would you please archive (tar or zip) and upload to the secure upload portal? For each node in the replica set spanning a time period that includes the incident, would you please archive (tar or zip) and upload to that link:
Best, | |
| Comment by Edwin Zhou [ 30/Aug/22 ] | |
|
Thank you for the follow up and I apologize for the delay in my response. Disk utilization usually refers to the activity on the disk. Do you mean that the space used on the disk is at 40%? If behavior is still a problem for you, I've created a secure upload portal for you to upload additional diagnostic information. Files uploaded to this portal are hosted on Box, are visible only to MongoDB employees, and are routinely deleted after some time. For each node in the replica set spanning a time period that includes the incident, would you please archive (tar or zip) and upload to that link:
Best, | |
| Comment by Daniel Raj [ 07/Jul/22 ] | |
|
Hi Edwin
Thanks for your quick revert. Initially I was also under the "No Space left" impression. But however as mentioned a) disk utilization was just 40% and b) after the "No Space left" warning there were lot more successful events.
So im not sure if it crashed because of disk issue.
Thank You Daniel Raj. | |
| Comment by Edwin Zhou [ 06/Jul/22 ] | |
|
Thank you for your report and information surrounding this issue. Looking at the log line you submitted, it seems that the device running mongodb has run out of storage space and fasserted with
You may resolve this by ensuring that MongoDB is using a data directory that has enough disk space to operate. Best, |