[SERVER-67799] [WTCheckpointThread] Got signal: 6 (Aborted) Created: 06/Jul/22  Updated: 04/Oct/22  Resolved: 04/Oct/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 4.2 Required
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Daniel Raj Assignee: Edwin Zhou
Resolution: Done Votes: 0
Labels: Bug
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Steps To Reproduce:

Load testing

Participants:

 Description   

Hi,

Mongodb version 4.2.20 is being used.
replicaset with 3 replicas, with readPreference=primary
During load testing, the primary node mongodb had crashed with following error and it never came back.
there was no core file generated
mongod service had to be manually restarted
disk utilization was at 40%

 

Any leads about why it was aborted would help and how to analyze such issues?

 

2022-06-21T11:58:48.644+0000 I  COMMAND  [conn10287] command DB_PROFILE.$cmd command: update { update: "micro_service", ordered: false, writeConcern:

{ w: "majority" }

, $db: "DB_PROFILE" } numYields:0 reslen:260 locks:{ ParallelBatchWriterMode: { acquireCount:

{ r: 1 }

}, ReplicationStateTransition: { acquireCount:

{ w: 1 }

}, Global: { acquireCount:

{ w: 1 }

}, Database: { acquireCount:

{ w: 1 }

}, Collection: { acquireCount:

{ w: 1 }

}, Mutex: { acquireCount:

{ r: 2 }

} } flowControl:{ acquireCount: 1, timeAcquiringMicros: 1 } storage:{} protocol:op_query 463ms
2022-06-21T11:58:48.643+0000 I  COMMAND  [conn10028] command DB_PROFILE.$cmd command: update { update: "micro_service", ordered: true, writeConcern:

{ w: "majority" }

, txnNumber: 208326, $db: "DB_PROFILE", $clusterTime: { clusterTime: Timestamp(1655812725, 1), signature:

{ hash: BinData(0, DAB13C0AD573C8D0D1B3C5986F4DDFAF096811E1), keyId: 7110219703129210885 }

}, lsid: { id: UUID("798984a2-96eb-42ac-93d3-3747b08bfe80") } } numYields:0 reslen:245 locks:{ ParallelBatchWriterMode: { acquireCount:

{ r: 2 }

}, ReplicationStateTransition: { acquireCount:

{ w: 3 }

}, Global: { acquireCount:

{ w: 2 }

}, Database: { acquireCount:

{ w: 2 }

}, Collection: { acquireCount:

{ w: 2 }

}, Mutex: { acquireCount:

{ r: 3 }

} } flowControl:{ acquireCount: 1, timeAcquiringMicros: 1 } storage:{} protocol:op_msg 464ms
2022-06-21T11:58:48.704+0000 E  STORAGE  [WTCheckpointThread] WiredTiger error (28) [1655812728:700987][1828:0x7fef069bf700], file:index-21286-3590310073709246823.wt, WT_SESSION.checkpoint: __posix_file_write, 539: /var/lib/mongodb/index-21286-3590310073709246823.wt: handle-write: pwrite: failed to write 4096 bytes at offset 8192: No space left on device Raw: [1655812728:700987][1828:0x7fef069bf700], file:index-21286-3590310073709246823.wt, WT_SESSION.checkpoint: __posix_file_write, 539: /var/lib/mongodb/index-21286-3590310073709246823.wt: handle-write: pwrite: failed to write 4096 bytes at offset 8192: No space left on device
2022-06-21T11:58:48.704+0000 E  STORAGE  [WTCheckpointThread] WiredTiger error (28) [1655812728:704947][1828:0x7fef069bf700], file:index-21286-3590310073709246823.wt, WT_SESSION.checkpoint: __ckpt_process, 764: index-21286-3590310073709246823.wt: fatal checkpoint failure: No space left on device Raw: [1655812728:704947][1828:0x7fef069bf700], file:index-21286-3590310073709246823.wt, WT_SESSION.checkpoint: __ckpt_process, 764: index-21286-3590310073709246823.wt: fatal checkpoint failure: No space left on device
2022-06-21T11:58:48.707+0000 E  STORAGE  [WTCheckpointThread] WiredTiger error (-31804) [1655812728:707344][1828:0x7fef069bf700], file:index-21286-3590310073709246823.wt, WT_SESSION.checkpoint: __wt_panic, 489: the process must exit and restart: WT_PANIC: WiredTiger library panic Raw: [1655812728:707344][1828:0x7fef069bf700], file:index-21286-3590310073709246823.wt, WT_SESSION.checkpoint: __wt_panic, 489: the process must exit and restart: WT_PANIC: WiredTiger library panic
2022-06-21T11:58:48.707+0000 I  COMMAND  [conn15414] command DB_SUBSCRIBER.$cmd command: update { update: "subscriber_service_kpi_hourly_history", ordered: false, writeConcern:

{ w: "majority" }

, $db: "DB_SUBSCRIBER" } numYields:0 reslen:260 locks:{ ParallelBatchWriterMode: { acquireCount:

{ r: 1 }

}, ReplicationStateTransition: { acquireCount:

{ w: 1 }

}, Global: { acquireCount:

{ w: 1 }

}, Database: { acquireCount:

{ w: 1 }

}, Collection: { acquireCount:

{ w: 1 }

}, Mutex: { acquireCount:

{ r: 2 }

} } flowControl:{ acquireCount: 1 } storage:{} protocol:op_query 808ms
2022-06-21T11:58:48.708+0000 I  COMMAND  [conn10297] command DB_PROFILE.$cmd command: update { update: "micro_service", ordered: false, writeConcern:

{ w: "majority" }

, $db: "DB_PROFILE" } numYields:0 reslen:260 locks:{ ParallelBatchWriterMode: { acquireCount:

{ r: 1 }

}, ReplicationStateTransition: { acquireCount:

{ w: 1 }

}, Global: { acquireCount:

{ w: 1 }

}, Database: { acquireCount:

{ w: 1 }

}, Collection: { acquireCount:

{ w: 1 }

}, Mutex: { acquireCount:

{ r: 1 }

} } flowControl:{ acquireCount: 1, timeAcquiringMicros: 1 } storage:{} protocol:op_query 769ms
2022-06-21T11:58:48.711+0000 F  -        [WTCheckpointThread] Fatal Assertion 50853 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 487
2022-06-21T11:58:48.713+0000 F  -        [WTCheckpointThread] \n\n***aborting after fassert() failure\n\n
2022-06-21T11:58:48.986+0000 I  COMMAND  [conn21986] command DB_SUBSCRIBER.$cmd command: delete { delete: "channel_details", ordered: false, writeConcern:

{ w: "majority" }

, $db: "DB_SUBSCRIBER" } numYields:0 reslen:245 locks:{ ParallelBatchWriterMode: { acquireCount:

{ r: 1 }

}, ReplicationStateTransition: { acquireCount:

{ w: 1 }

}, Global: { acquireCount:

{ w: 1 }

}, Database: { acquireCount:

{ w: 1 }

}, Collection: { acquireCount:

{ w: 1 }

}, Mutex: { acquireCount:

{ r: 1 }

} } flowControl:{ acquireCount: 1, timeAcquiringMicros: 1 } storage:{} protocol:op_query 1024ms
2022-06-21T11:58:48.986+0000 I  COMMAND  [conn15416] command DB_SUBSCRIBER.$cmd command: delete { delete: "channel_details", ordered: false, writeConcern:

{ w: "majority" }

, $db: "DB_SUBSCRIBER" } numYields:0 reslen:245 locks:{ ParallelBatchWriterMode: { acquireCount:

{ r: 1 }

}, ReplicationStateTransition: { acquireCount:

{ w: 1 }

}, Global: { acquireCount:

{ w: 1 }

}, Database: { acquireCount:

{ w: 1 }

}, Collection: { acquireCount:

{ w: 1 }

}, Mutex: { acquireCount:

{ r: 1 }

} } flowControl:{ acquireCount: 1, timeAcquiringMicros: 1 } storage:{} protocol:op_query 958ms
2022-06-21T11:58:48.987+0000 I  COMMAND  [conn15402] command DB_SUBSCRIBER.$cmd command: update { update: "subscriber_service_kpi_15min_history", ordered: false, writeConcern:

{ w: "majority" }

, $db: "DB_SUBSCRIBER" } numYields:0 reslen:260 locks:{ ParallelBatchWriterMode: { acquireCount:

{ r: 1 }

}, ReplicationStateTransition: { acquireCount:

{ w: 1 }

}, Global: { acquireCount:

{ w: 1 }

}, Database: { acquireCount:

{ w: 1 }

}, Collection: { acquireCount:

{ w: 1 }

}, Mutex: { acquireCount:

{ r: 2 }

} } flowControl:{ acquireCount: 1, timeAcquiringMicros: 1 } storage:{} protocol:op_query 1088ms
2022-06-21T11:58:48.987+0000 I  COMMAND  [conn10159] command DB_PROFILE.$cmd command: update { update: "micro_service", ordered: false, writeConcern:

{ w: "majority" }

, $db: "DB_PROFILE" } numYields:0 reslen:260 locks:{ ParallelBatchWriterMode: { acquireCount:

{ r: 1 }

}, ReplicationStateTransition: { acquireCount:

{ w: 1 }

}, Global: { acquireCount:

{ w: 1 }

}, Database: { acquireCount:

{ w: 1 }

}, Collection: { acquireCount:

{ w: 1 }

}, Mutex: { acquireCount:

{ r: 2 }

} } flowControl:{ acquireCount: 1, timeAcquiringMicros: 1 } storage:{} protocol:op_query 786ms

 

 

2022-06-21T11:58:49.182+0000 I  COMMAND  [conn10159] command DB_PROFILE.$cmd command: update { update: "micro_service", ordered: false, writeConcern:

{ w: "majority" }

, $db: "DB_PROFILE" } numYields:0 reslen:260 locks:{ ParallelBatchWriterMode: { acquireCount:

{ r: 1 }

}, ReplicationStateTransition: { acquireCount:

{ w: 1 }

}, Global: { acquireCount:

{ w: 1 }

}, Database: { acquireCount:

{ w: 1 }

}, Collection: { acquireCount:

{ w: 1 }

}, Mutex: { acquireCount:

{ r: 1 }

} } flowControl:{ acquireCount: 1, timeAcquiringMicros: 8 } storage:{} protocol:op_query 193ms
2022-06-21T11:58:49.182+0000 I  COMMAND  [conn21986] command DB_SUBSCRIBER.$cmd command: update { update: "subscriber_service", ordered: false, writeConcern:

{ w: "majority" }

, $db: "DB_SUBSCRIBER" } numYields:0 reslen:260 locks:{ ParallelBatchWriterMode: { acquireCount:

{ r: 1 }

}, ReplicationStateTransition: { acquireCount:

{ w: 1 }

}, Global: { acquireCount:

{ w: 1 }

}, Database: { acquireCount:

{ w: 1 }

}, Collection: { acquireCount:

{ w: 1 }

}, Mutex: { acquireCount:

{ r: 1 }

} } flowControl:{ acquireCount: 1, timeAcquiringMicros: 2 } storage:{ data:

{ bytesRead: 128080, timeReadingMicros: 330 }

} protocol:op_query 194ms
2022-06-21T11:58:49.183+0000 I  COMMAND  [conn15402] command DB_SUBSCRIBER.$cmd command: update { update: "subscriber_service_kpi_15min_history", ordered: false, writeConcern:

{ w: "majority" }

, $db: "DB_SUBSCRIBER" } numYields:0 reslen:260 locks:{ ParallelBatchWriterMode: { acquireCount:

{ r: 1 }

}, ReplicationStateTransition: { acquireCount:

{ w: 1 }

}, Global: { acquireCount:

{ w: 1 }

}, Database: { acquireCount:

{ w: 1 }

}, Collection: { acquireCount:

{ w: 1 }

}, Mutex: { acquireCount:

{ r: 2 }

} } flowControl:{ acquireCount: 1, timeAcquiringMicros: 2 } storage:{} protocol:op_query 194ms
2022-06-21T11:58:49.211+0000 F  -        [WTCheckpointThread] Got signal: 6 (Aborted).
 0x55884f8e5881 0x55884f8e4dbe 0x55884f8e4e56 0x7fef124ef980 0x7fef1212ae87 0x7fef1212c7f1 0x55884dc93600 0x55884d9c1f50 0x55884de496fb 0x55884d9d3122 0x55884d9d3589 0x55884df30cf9 0x55884de2b21c 0x55884de2c4ee 0x55884deaf08d 0x55884de5c2a4 0x55884de5ebcd 0x55884de60156 0x55884d9d2728 0x55884dd8caa4 0x55884f7acd90 0x55884f7ad3ce 0x55884fa0e09f 0x7fef124e46db 0x7fef1220d61f
----- BEGIN BACKTRACE -----

{"backtrace":[\{"b":"55884CF53000","o":"2992881","s":"_ZN5mongo15printStackTraceERSo"}

,{"b":"55884CF53000","o":"2991DBE"},{"b":"55884CF53000","o":"2991E56"},{"b":"7FEF124DD000","o":"12980"},{"b":"7FEF120EC000","o":"3EE87","s":"gsignal"},{"b":"7FEF120EC000","o":"407F1","s":"abort"},{"b":"55884CF53000","o":"D40600","s":"ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"55884CF53000","o":"A6EF50"},{"b":"55884CF53000","o":"EF66FB"},{"b":"55884CF53000","o":"A80122","s":"wt_err_func"},{"b":"55884CF53000","o":"A80589","s":"wt_panic"},{"b":"55884CF53000","o":"FDDCF9","s":"wt_block_checkpoint"},{"b":"55884CF53000","o":"ED821C"},{"b":"55884CF53000","o":"ED94EE","s":"wt_reconcile"},{"b":"55884CF53000","o":"F5C08D","s":"wt_sync_file"},{"b":"55884CF53000","o":"F092A4"},{"b":"55884CF53000","o":"F0BBCD"},{"b":"55884CF53000","o":"F0D156","s":"_wt_txn_checkpoint"},{"b":"55884CF53000","o":"A7F728"},{"b":"55884CF53000","o":"E39AA4","s":"_ZN5mongo18WiredTigerKVEngine26WiredTigerCheckpointThread3runEv"},{"b":"55884CF53000","o":"2859D90","s":"_ZN5mongo13BackgroundJob7jobBodyEv"},{"b":"55884CF53000","o":"285A3CE"},{"b":"55884CF53000","o":"2ABB09F"},{"b":"7FEF124DD000","o":"76DB"},{"b":"7FEF120EC000","o":"12161F","s":"clone"}],"processInfo":{ "mongodbVersion" : "4.2.20", "gitVersion" : "15c0712952c356cb711c13a42cb3bca8617d4ebc", "compiledModules" : [], "uname" :

{ "sysname" : "Linux", "release" : "4.15.0-176-generic", "version" : "#185-Ubuntu SMP Tue Mar 29 17:40:04 UTC 2022", "machine" : "x86_64" }

, "somap" : [ { "b" : "55884CF53000", "elfType" : 3, "buildId" : "EC22436D24BD41324558A58015C26F8711C26419" }, { "b" : "7FFD0514E000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "4BA8BE578511382D0AEC3A10C107C8BDD0BFB52A" }, { "b" : "7FEF13A30000", "path" : "/usr/lib/x86_64-linux-gnu/libcurl.so.4", "elfType" : 3, "buildId" : "7C9EF9CA1C1C9F9925ECF63E033DA8859BC1E108" }, { "b" : "7FEF13816000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "052338C7AE2BAC6E1E5C0EE5924F7560FD9E025B" }, { "b" : "7FEF1334B000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1", "elfType" : 3, "buildId" : "C518099E5DE8D6AD61A166C63F4254D6E2DA3A93" }, { "b" : "7FEF130BE000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.1", "elfType" : 3, "buildId" : "CA742A427E7AEF089B39C4D773C20EA9E074CE8E" }, { "b" : "7FEF12EBA000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "52E7F9CD225B05DCEE4DE1BA87004D670DF130D2" }, { "b" : "7FEF12CB2000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "35E6D93BDF9DA528ABB754DCF2FA8D09DFCB8BF8" }, { "b" : "7FEF12914000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "D3522C3C754AC7C2E933A71E636F0C0520449263" }, { "b" : "7FEF126FC000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "679F3AE11120EC7C483BC9295345D836F5C104F7" }, { "b" : "7FEF124DD000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "8B60B9F8D9A27F2264A0D8E6C0768B3B02C29926" }, { "b" : "7FEF120EC000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "71F0F3074A929E519E85F6A5C03A7D1FD976BFE4" }, { "b" : "7FEF13CB2000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "02AA5DB099824E560F98F6E2E3F6848B053BFEED" }, { "b" : "7FEF11EC7000", "path" : "/usr/lib/x86_64-linux-gnu/libnghttp2.so.14", "elfType" : 3, "buildId" : "4F00E5207693FDC249DA42EC6472ACA6A7B929AE" }, { "b" : "7FEF11CAA000", "path" : "/usr/lib/x86_64-linux-gnu/libidn2.so.0", "elfType" : 3, "buildId" : "EE6E9462BA2491F4EE8C4E52C3323274A9366614" }, { "b" : "7FEF11A8E000", "path" : "/usr/lib/x86_64-linux-gnu/librtmp.so.1", "elfType" : 3, "buildId" : "69465D8AA6B19086ABF2455A703F9168BF82A69F" }, { "b" : "7FEF11880000", "path" : "/usr/lib/x86_64-linux-gnu/libpsl.so.5", "elfType" : 3, "buildId" : "CDAF1F1946846941F9D06414EC8C812D131A168E" }, { "b" : "7FEF11635000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "9FD187FE6579DB3B458E021CCAC0ECCAFC66B1B7" }, { "b" : "7FEF113E3000", "path" : "/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2", "elfType" : 3, "buildId" : "3455E5726038F673FC9C90385079A2113FE4F55E" }, { "b" : "7FEF111D5000", "path" : "/usr/lib/x86_64-linux-gnu/liblber-2.4.so.2", "elfType" : 3, "buildId" : "5F842A7CA8F5C84B338CCCB703F88F020057C77E" }, { "b" : "7FEF10FB8000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "6517F816C946C49AD246642F79493CD0CA2629DB" }, { "b" : "7FEF10C3A000", "path" : "/usr/lib/x86_64-linux-gnu/libunistring.so.2", "elfType" : 3, "buildId" : "0E2784298E7D3F4D894FE130ACEFA77C3E624F72" }, { "b" : "7FEF108D4000", "path" : "/usr/lib/x86_64-linux-gnu/libgnutls.so.30", "elfType" : 3, "buildId" : "DA3C78836D1C97B4158870FB2A84817C0653D8E9" }, { "b" : "7FEF1069E000", "path" : "/usr/lib/x86_64-linux-gnu/libhogweed.so.4", "elfType" : 3, "buildId" : "372EE4A2E4BB08F1A2B01AE486F88E7466ADD389" }, { "b" : "7FEF10468000", "path" : "/usr/lib/x86_64-linux-gnu/libnettle.so.6", "elfType" : 3, "buildId" : "433053B348A412F5CF22D07F699D412AD753FBE5" }, { "b" : "7FEF101E7000", "path" : "/usr/lib/x86_64-linux-gnu/libgmp.so.10", "elfType" : 3, "buildId" : "D40EA9B5EC5BC46799E4A412319617BD38BE9341" }, { "b" : "7FEF0FF11000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.3", "elfType" : 3, "buildId" : "320112EE17C7F1886DBEB108429063B22D996E9E" }, { "b" : "7FEF0FCDF000", "path" : "/usr/lib/x86_64-linux-gnu/libk5crypto.so.3", "elfType" : 3, "buildId" : "729F89DD92E3441FA99F8682533F7B92EE48880D" }, { "b" : "7FEF0FADB000", "path" : "/lib/x86_64-linux-gnu/libcom_err.so.2", "elfType" : 3, "buildId" : "17107881DF65C66B4C6D38CAB37C285FA44663BD" }, { "b" : "7FEF0F8D0000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5support.so.0", "elfType" : 3, "buildId" : "EAE217D2281E12BCD43675C73C59DF29D7167C66" }, { "b" : "7FEF0F6B5000", "path" : "/usr/lib/x86_64-linux-gnu/libsasl2.so.2", "elfType" : 3, "buildId" : "AA489C2B43FF4886958E90DFBD9DB3825D8F6A9A" }, { "b" : "7FEF0F474000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi.so.3", "elfType" : 3, "buildId" : "A1A98DB481968073636BBAECB561A3EA8ED198AE" }, { "b" : "7FEF0F145000", "path" : "/usr/lib/x86_64-linux-gnu/libp11-kit.so.0", "elfType" : 3, "buildId" : "8CFF559D91E3376842A1BA41CCF74EB9DECD159B" }, { "b" : "7FEF0EF32000", "path" : "/usr/lib/x86_64-linux-gnu/libtasn1.so.6", "elfType" : 3, "buildId" : "6036B89A3BB671B32E01464C0C82BFA016186352" }, { "b" : "7FEF0ED2E000", "path" : "/lib/x86_64-linux-gnu/libkeyutils.so.1", "elfType" : 3, "buildId" : "CCF39E49333249E43F465986B782D687686D9BBB" }, { "b" : "7FEF0EB25000", "path" : "/usr/lib/x86_64-linux-gnu/libheimntlm.so.0", "elfType" : 3, "buildId" : "C2376C5B831991591F1A67B976758185F86896D8" }, { "b" : "7FEF0E898000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.26", "elfType" : 3, "buildId" : "69BDEE5FA0FEEDF317308BE850F78761861D520A" }, { "b" : "7FEF0E5F6000", "path" : "/usr/lib/x86_64-linux-gnu/libasn1.so.8", "elfType" : 3, "buildId" : "315D74995AAA32DE4D15BA25F335066988B1B230" }, { "b" : "7FEF0E3C0000", "path" : "/usr/lib/x86_64-linux-gnu/libhcrypto.so.4", "elfType" : 3, "buildId" : "6673972A1C24A89EBAFBAE696188A4CB26C6DDEB" }, { "b" : "7FEF0E1AA000", "path" : "/usr/lib/x86_64-linux-gnu/libroken.so.18", "elfType" : 3, "buildId" : "430827C33259C12248CF44B91A9A9821114376F5" }, { "b" : "7FEF0DFA2000", "path" : "/usr/lib/x86_64-linux-gnu/libffi.so.6", "elfType" : 3, "buildId" : "3555B5F599C9787DFDDBF9E8DF6F706B9044D985" }, { "b" : "7FEF0DD79000", "path" : "/usr/lib/x86_64-linux-gnu/libwind.so.0", "elfType" : 3, "buildId" : "93A0931B1C2818F0EA224CE6FE5E31E84A9B55BB" }, { "b" : "7FEF0DB6A000", "path" : "/usr/lib/x86_64-linux-gnu/libheimbase.so.1", "elfType" : 3, "buildId" : "669D4CCE42FA4382796EFFCF0C16F459F4382C4C" }, { "b" : "7FEF0D920000", "path" : "/usr/lib/x86_64-linux-gnu/libhx509.so.5", "elfType" : 3, "buildId" : "4B80C543356EE0AF9039EFE7C9EA1CC1F74C426A" }, { "b" : "7FEF0D617000", "path" : "/usr/lib/x86_64-linux-gnu/libsqlite3.so.0", "elfType" : 3, "buildId" : "1E52E04B0722C6CFB8E534FB8DA45DDF95B46C19" }, { "b" : "7FEF0D3DF000", "path" : "/lib/x86_64-linux-gnu/libcrypt.so.1", "elfType" : 3, "buildId" : "A8BAF433206E7FA4E8C524EC445162ECBA55A2F4" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55884f8e5881]
 mongod(+0x2991DBE) [0x55884f8e4dbe]
 mongod(+0x2991E56) [0x55884f8e4e56]
 libpthread.so.0(+0x12980) [0x7fef124ef980]
 libc.so.6(gsignal+0xC7) [0x7fef1212ae87]
 libc.so.6(abort+0x141) [0x7fef1212c7f1]
 mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x55884dc93600]
 mongod(+0xA6EF50) [0x55884d9c1f50]
 mongod(+0xEF66FB) [0x55884de496fb]
 mongod(__wt_err_func+0x90) [0x55884d9d3122]
 mongod(__wt_panic+0x39) [0x55884d9d3589]
 mongod(__wt_block_checkpoint+0x359) [0x55884df30cf9]
 mongod(+0xED821C) [0x55884de2b21c]
 mongod(__wt_reconcile+0xC1E) [0x55884de2c4ee]
 mongod(__wt_sync_file+0x4ED) [0x55884deaf08d]
 mongod(+0xF092A4) [0x55884de5c2a4]
 mongod(+0xF0BBCD) [0x55884de5ebcd]
 mongod(__wt_txn_checkpoint+0x256) [0x55884de60156]
 mongod(+0xA7F728) [0x55884d9d2728]
 mongod(_ZN5mongo18WiredTigerKVEngine26WiredTigerCheckpointThread3runEv+0x4D4) [0x55884dd8caa4]
 mongod(_ZN5mongo13BackgroundJob7jobBodyEv+0xA0) [0x55884f7acd90]
 mongod(+0x285A3CE) [0x55884f7ad3ce]
 mongod(+0x2ABB09F) [0x55884fa0e09f]
 libpthread.so.0(+0x76DB) [0x7fef124e46db]
 libc.so.6(clone+0x3F) [0x7fef1220d61f]
-----  END BACKTRACE  -----



 Comments   
Comment by Edwin Zhou [ 04/Oct/22 ]

We haven’t heard back from you for some time, so I’m going to close this ticket. If this is still an issue for you, please provide additional information and we will reopen the ticket.

Comment by Edwin Zhou [ 20/Sep/22 ]

We still need additional information to diagnose the problem. If this is still an issue for you, would you please archive (tar or zip) and upload to the secure upload portal?

For each node in the replica set spanning a time period that includes the incident, would you please archive (tar or zip) and upload to that link:

  • the mongod logs
  • the $dbpath/diagnostic.data directory (the contents are described here)
  • the /var/log/messages file covering the time of this log line

Best,
Edwin

Comment by Edwin Zhou [ 30/Aug/22 ]

Hi dani.nelaturi@gmail.com,

Thank you for the follow up and I apologize for the delay in my response.

Disk utilization usually refers to the activity on the disk. Do you mean that the space used on the disk is at 40%?

If behavior is still a problem for you, I've created a secure upload portal for you to upload additional diagnostic information. Files uploaded to this portal are hosted on Box, are visible only to MongoDB employees, and are routinely deleted after some time.

For each node in the replica set spanning a time period that includes the incident, would you please archive (tar or zip) and upload to that link:

  • the mongod logs
  • the $dbpath/diagnostic.data directory (the contents are described here)
  • the /var/log/messages file covering the time of this log line

Best,
Edwin

Comment by Daniel Raj [ 07/Jul/22 ]

Hi Edwin

 

Thanks for your quick revert. Initially I was also under the "No Space left" impression.

But however as mentioned

a) disk utilization was just 40% and

b) after the "No Space left" warning there were lot more successful events.

 

So im not sure if it crashed because of disk issue.

 

Thank You

Daniel Raj.

Comment by Edwin Zhou [ 06/Jul/22 ]

Hi dani.nelaturi@gmail.com,

Thank you for your report and information surrounding this issue. Looking at the log line you submitted, it seems that the device running mongodb has run out of storage space and fasserted with

No space left on device

.

You may resolve this by ensuring that MongoDB is using a data directory that has enough disk space to operate.

Best,
Edwin

Generated at Thu Feb 08 06:09:05 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.