[SERVER-50601] server abort, crash Created: 28/Aug/20  Updated: 27/Oct/23  Resolved: 03/Sep/20

Status: Closed
Project: Core Server
Component/s: Replication, WiredTiger
Affects Version/s: 3.6.18
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: sergio d Assignee: Dmitry Agranat
Resolution: Works as Designed Votes: 0
Labels: FA_50853
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

2020-08-27T23:04:38.584-0300 I ASIO     [NetworkInterfaceASIO-RS-0] Ending idle connection to host 10.28.60.30:27017 because the pool meets constraints; 2 connections to that host remain open
2020-08-27T23:04:40.650-0300 E STORAGE  [WTCheckpointThread] WiredTiger error (5) [1598580280:650869][61752:0x7f1d78a94700], file:index-3579-5458593001146091026.wt, WT_SESSION.checkpoint: __posix_sync, 108: /mongo/data/db/index-3579-5458593001146091026.wt: handle-sync: fdatasync: Input/output error Raw: [1598580280:650869][61752:0x7f1d78a94700], file:index-3579-5458593001146091026.wt, WT_SESSION.checkpoint: __posix_sync, 108: /mongo/data/db/index-3579-5458593001146091026.wt: handle-sync: fdatasync: Input/output error
2020-08-27T23:04:40.651-0300 E STORAGE  [WTCheckpointThread] WiredTiger error (-31804) [1598580280:651092][61752:0x7f1d78a94700], file:index-3579-5458593001146091026.wt, WT_SESSION.checkpoint: __wt_panic, 523: the process must exit and restart: WT_PANIC: WiredTiger library panic Raw: [1598580280:651092][61752:0x7f1d78a94700], file:index-3579-5458593001146091026.wt, WT_SESSION.checkpoint: __wt_panic, 523: the process must exit and restart: WT_PANIC: WiredTiger library panic
2020-08-27T23:04:40.651-0300 F -        [WTCheckpointThread] Fatal Assertion 50853 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 420
2020-08-27T23:04:40.651-0300 F -        [WTCheckpointThread] 
 
***aborting after fassert() failure
 
 
2020-08-27T23:04:40.655-0300 F -        [conn3013] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 74
2020-08-27T23:04:40.655-0300 F -        [conn3013] 
 
***aborting after fassert() failure
 
 
2020-08-27T23:04:40.835-0300 F -        [WTCheckpointThread] Got signal: 6 (Aborted).
 
 0x56461fa32ea1 0x56461fa320b9 0x56461fa3259d 0x7f1d7ee510c0 0x7f1d7ead3fcf 0x7f1d7ead53fa 0x56461e134062 0x56461e210596 0x56461e281f69 0x56461e0cf6ac 0x56461e0cfacc 0x56461e24c243 0x56461e3372f4 0x56461e294ee1 0x56461e295d93 0x56461e27b47a 0x56461e1f4a1f 0x56461f926c41 0x56461fb42010 0x7f1d7ee47494 0x7f1d7eb89aff
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"56461D798000","o":"229AEA1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"56461D798000","o":"229A0B9"},{"b":"56461D798000","o":"229A59D"},{"b":"7F1D7EE40000","o":"110C0"},{"b":"7F1D7EAA1000","o":"32FCF","s":"gsignal"},{"b":"7F1D7EAA1000","o":"343FA","s":"abort"},{"b":"56461D798000","o":"99C062","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"56461D798000","o":"A78596"},{"b":"56461D798000","o":"AE9F69"},{"b":"56461D798000","o":"9376AC","s":"__wt_err_func"},{"b":"56461D798000","o":"937ACC","s":"__wt_panic"},{"b":"56461D798000","o":"AB4243"},{"b":"56461D798000","o":"B9F2F4"},{"b":"56461D798000","o":"AFCEE1"},{"b":"56461D798000","o":"AFDD93","s":"__wt_txn_checkpoint"},{"b":"56461D798000","o":"AE347A"},{"b":"56461D798000","o":"A5CA1F","s":"_ZN5mongo18WiredTigerKVEngine26WiredTigerCheckpointThread3runEv"},{"b":"56461D798000","o":"218EC41","s":"_ZN5mongo13BackgroundJob7jobBodyEv"},{"b":"56461D798000","o":"23AA010"},{"b":"7F1D7EE40000","o":"7494"},{"b":"7F1D7EAA1000","o":"E8AFF","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.6.19", "gitVersion" : "41b289ff734a926e784d6ab42c3129f59f40d5b4", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.9.0-4-amd64", "version" : "#1 SMP Debian 4.9.51-1 (2017-09-28)", "machine" : "x86_64" }, "somap" : [ { "b" : "56461D798000", "elfType" : 3, "buildId" : "751BC9A42F8C992E57E9FF15D15223692095C959" }, { "b" : "7FFCB75BB000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "49A8CECB342FEC1F6FF31B2D8D6FD7CE11CD3E57" }, { "b" : "7F1D80083000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "1F8BBD45EFD498F52135C7F5B4F856577D5A4997" }, { "b" : "7F1D7FBF0000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1", "elfType" : 3, "buildId" : "2CFE882A331D7857E9CE1B5DE3255E6DA76EF899" }, { "b" : "7F1D7F984000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.1", "elfType" : 3, "buildId" : "E2AA3B39763D943F56B3BD05C8E36E639BA95E12" }, { "b" : "7F1D7F780000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "6A5D98612129B8186F21E800AFDFAAA627082F46" }, { "b" : "7F1D7F578000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "FE41526A83999F2FE9D0F8AADCD61D03A92CBB70" }, { "b" : "7F1D7F274000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "90DA054E12EA1A53EE0CBB5BB5E65F7069AEEE44" }, { "b" : "7F1D7F05D000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "036DAE71A7197C847FC5B720634642B922C74398" }, { "b" : "7F1D7EE40000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "968DF33F83963B559243653D74D27D89605BED02" }, { "b" : "7F1D7EAA1000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "79450F6E36287865D093EA209B85A222209925FF" }, { "b" : "7F1D8029A000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "6F150F33B150D6A81E26A425DD47D713D00F2D29" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x56461fa32ea1]
 mongod(+0x229A0B9) [0x56461fa320b9]
 mongod(+0x229A59D) [0x56461fa3259d]
 libpthread.so.0(+0x110C0) [0x7f1d7ee510c0]
 libc.so.6(gsignal+0xCF) [0x7f1d7ead3fcf]
 libc.so.6(abort+0x16A) [0x7f1d7ead53fa]
 mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x56461e134062]
 mongod(+0xA78596) [0x56461e210596]
 mongod(+0xAE9F69) [0x56461e281f69]
 mongod(__wt_err_func+0x90) [0x56461e0cf6ac]
 mongod(__wt_panic+0x3F) [0x56461e0cfacc]
 mongod(+0xAB4243) [0x56461e24c243]
 mongod(+0xB9F2F4) [0x56461e3372f4]
 mongod(+0xAFCEE1) [0x56461e294ee1]
 mongod(__wt_txn_checkpoint+0x1C3) [0x56461e295d93]
 mongod(+0xAE347A) [0x56461e27b47a]
 mongod(_ZN5mongo18WiredTigerKVEngine26WiredTigerCheckpointThread3runEv+0x23F) [0x56461e1f4a1f]
 mongod(_ZN5mongo13BackgroundJob7jobBodyEv+0x131) [0x56461f926c41]
 mongod(+0x23AA010) [0x56461fb42010]
 libpthread.so.0(+0x7494) [0x7f1d7ee47494]
 libc.so.6(clone+0x3F) [0x7f1d7eb89aff]
-----  END BACKTRACE  -----



 Comments   
Comment by sergio d [ 03/Sep/20 ]

Thank you, do you have any reference documents on how to resolve this HW issue?

Comment by Dmitry Agranat [ 03/Sep/20 ]

Hi sergioduarte.dba@gmail.com it looks like you've attached data from another member, which did not experienced this issue. In any case, Fatal Assertion 50853 indicates a HW issue which you'll need to address. After the HW issue is addressed, you can resync this node from another healthy member.

Comment by sergio d [ 02/Sep/20 ]

Hi Dima,

Done. The file /var/ log/dmesg does not exist

Regards,
Sérgio

Comment by Dmitry Agranat [ 30/Aug/20 ]

Hi sergioduarte.dba@gmail.com,

Would you please archive (tar or zip) the full mongod.log files and the $dbpath/diagnostic.data directory (the contents are described here) and upload them to this support uploader location?

Also, please upload the syslog and the output from the /var/log/dmesg.

Files uploaded to this portal are visible only to MongoDB employees and are routinely deleted after some time.

Thanks,
Dima

Generated at Thu Feb 08 05:23:05 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.