[SERVER-44885] Mongodb always restart Created: 29/Nov/19  Updated: 27/Oct/23  Resolved: 02/Dec/19

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Kelany Mahmoud Assignee: Dmitry Agranat
Resolution: Community Answered Votes: 0
Labels: FA_50853
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File screenshot-1.png    
Backwards Compatibility: Fully Compatible
Operating System: ALL
Steps To Reproduce:
  • I tried to convert it to single not cluster it's OK
  • I tried to mongodump data and restore again after 64 collections it crashes.
Participants:

 Description   

 

Log

2019-11-29T17:23:49.231+0300 I CONTROL  [main] ***** SERVER RESTARTED *****
2019-11-29T17:23:49.243+0300 I CONTROL  [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
2019-11-29T17:23:49.273+0300 I CONTROL  [initandlisten] MongoDB starting : pid=54040 port=27017 dbpath=/var/lib/mongo 64-bit host=abd01.iaae.com
2019-11-29T17:23:49.273+0300 I CONTROL  [initandlisten] db version v4.0.13
2019-11-29T17:23:49.273+0300 I CONTROL  [initandlisten] git version: bda366f0b0e432ca143bc41da54d8732bd8d03c0
2019-11-29T17:23:49.273+0300 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2019-11-29T17:23:49.274+0300 I CONTROL  [initandlisten] allocator: tcmalloc
2019-11-29T17:23:49.274+0300 I CONTROL  [initandlisten] modules: enterprise
2019-11-29T17:23:49.274+0300 I CONTROL  [initandlisten] build environment:
2019-11-29T17:23:49.274+0300 I CONTROL  [initandlisten]     distmod: rhel70
2019-11-29T17:23:49.274+0300 I CONTROL  [initandlisten]     distarch: x86_64
2019-11-29T17:23:49.274+0300 I CONTROL  [initandlisten]     target_arch: x86_64
2019-11-29T17:23:49.274+0300 I CONTROL  [initandlisten] options: { config: "/var/lib/mongo/automation-mongod.conf", net: { bindIp: "0.0.0.0", port: 27017 }, processManagement: { fork: true, timeZoneInfo: "/usr/share/zoneinfo" }, replication: { enableMajorityReadConcern: false, replSetName: "ABD_DB" }, storage: { dbPath: "/var/lib/mongo", engine: "wiredTiger", journal: { enabled: true }, wiredTiger: { engineConfig: { directoryForIndexes: true } } }, systemLog: { destination: "file", logAppend: true, path: "/var/logs/mongodb/mongod.log" } }
2019-11-29T17:23:49.275+0300 W STORAGE  [initandlisten] Detected unclean shutdown - /var/lib/mongo/mongod.lock is not empty.
2019-11-29T17:23:49.275+0300 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2019-11-29T17:23:49.275+0300 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=9204M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2019-11-29T17:23:56.264+0300 I STORAGE  [initandlisten] WiredTiger message [1575037436:264044][54040:0x7f393540bbc0], txn-recover: Main recovery loop: starting at 214/77890688 to 224/256
2019-11-29T17:23:56.264+0300 I STORAGE  [initandlisten] WiredTiger message [1575037436:264885][54040:0x7f393540bbc0], txn-recover: Recovering log 214 through 224
2019-11-29T17:23:57.317+0300 I STORAGE  [initandlisten] WiredTiger message [1575037437:317200][54040:0x7f393540bbc0], file:index/27--2264927244963632737.wt, txn-recover: Recovering log 215 through 224
2019-11-29T17:24:00.567+0300 I STORAGE  [initandlisten] WiredTiger message [1575037440:567920][54040:0x7f393540bbc0], file:index/27--2264927244963632737.wt, txn-recover: Recovering log 216 through 224
2019-11-29T17:24:03.876+0300 I STORAGE  [initandlisten] WiredTiger message [1575037443:876820][54040:0x7f393540bbc0], file:sizeStorer.wt, txn-recover: Recovering log 217 through 224
2019-11-29T17:24:07.063+0300 I STORAGE  [initandlisten] WiredTiger message [1575037447:63731][54040:0x7f393540bbc0], file:index/37--2264927244963632737.wt, txn-recover: Recovering log 218 through 224
2019-11-29T17:24:10.335+0300 I STORAGE  [initandlisten] WiredTiger message [1575037450:335047][54040:0x7f393540bbc0], file:index/39--2264927244963632737.wt, txn-recover: Recovering log 219 through 224
2019-11-29T17:24:13.615+0300 I STORAGE  [initandlisten] WiredTiger message [1575037453:615731][54040:0x7f393540bbc0], file:index/39--2264927244963632737.wt, txn-recover: Recovering log 220 through 224
2019-11-29T17:24:16.715+0300 I STORAGE  [initandlisten] WiredTiger message [1575037456:715549][54040:0x7f393540bbc0], file:index/39--2264927244963632737.wt, txn-recover: Recovering log 221 through 224
2019-11-29T17:24:19.522+0300 I STORAGE  [initandlisten] WiredTiger message [1575037459:522673][54040:0x7f393540bbc0], file:index/43--2264927244963632737.wt, txn-recover: Recovering log 222 through 224
2019-11-29T17:24:19.524+0300 I STORAGE  [initandlisten] WiredTiger message [1575037459:524626][54040:0x7f393540bbc0], file:index/43--2264927244963632737.wt, txn-recover: Recovering log 223 through 224
2019-11-29T17:24:19.624+0300 I STORAGE  [initandlisten] WiredTiger message [1575037459:624240][54040:0x7f393540bbc0], file:index/43--2264927244963632737.wt, txn-recover: Recovering log 224 through 224
2019-11-29T17:24:19.704+0300 I STORAGE  [initandlisten] WiredTiger message [1575037459:704648][54040:0x7f393540bbc0], file:index/43--2264927244963632737.wt, txn-recover: Set global recovery timestamp: 0
2019-11-29T17:24:20.811+0300 E STORAGE  [initandlisten] WiredTiger error (5) [1575037460:811484][54040:0x7f393540bbc0], file:index/39--2264927244963632737.wt, WT_SESSION.checkpoint: __posix_sync, 99: /var/lib/mongo/index/39--2264927244963632737.wt: handle-sync: fdatasync: Input/output error Raw: [1575037460:811484][54040:0x7f393540bbc0], file:index/39--2264927244963632737.wt, WT_SESSION.checkpoint: __posix_sync, 99: /var/lib/mongo/index/39--2264927244963632737.wt: handle-sync: fdatasync: Input/output error
2019-11-29T17:24:20.811+0300 E STORAGE  [initandlisten] WiredTiger error (-31804) [1575037460:811606][54040:0x7f393540bbc0], file:index/39--2264927244963632737.wt, WT_SESSION.checkpoint: __wt_panic, 494: the process must exit and restart: WT_PANIC: WiredTiger library panic Raw: [1575037460:811606][54040:0x7f393540bbc0], file:index/39--2264927244963632737.wt, WT_SESSION.checkpoint: __wt_panic, 494: the process must exit and restart: WT_PANIC: WiredTiger library panic
2019-11-29T17:24:20.811+0300 F -        [initandlisten] Fatal Assertion 50853 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 409
2019-11-29T17:24:20.811+0300 F -        [initandlisten]***aborting after fassert() failure
2019-11-29T17:24:20.836+0300 F -        [initandlisten] Got signal: 6 (Aborted).
 0x5606bfde7ef1 0x5606bfde7109 0x5606bfde75ed 0x7f3931fed5d0 0x7f3931c47207 0x7f3931c488f8 0x5606be307079 0x5606be739206 0x5606be7a3951 0x5606be294791 0x5606be294bab 0x5606be7764b3 0x5606be8739a4 0x5606be7b6459 0x5606be7b734b 0x5606be79cbda 0x5606be7bce49 0x5606be74f397 0x5606be74bca8 0x5606be70ed2a 0x5606be714efb 0x5606be6f6ed2 0x5606beb73b3a 0x5606be37b448 0x5606be37f67f 0x5606be308c09 0x7f3931c333d5 0x5606be379d2f
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"5606BD8BC000","o":"252BEF1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"5606BD8BC000","o":"252B109"},{"b":"5606BD8BC000","o":"252B5ED"},{"b":"7F3931FDE000","o":"F5D0"},{"b":"7F3931C11000","o":"36207","s":"gsignal"},{"b":"7F3931C11000","o":"378F8","s":"abort"},{"b":"5606BD8BC000","o":"A4B079","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"5606BD8BC000","o":"E7D206"},{"b":"5606BD8BC000","o":"EE7951"},{"b":"5606BD8BC000","o":"9D8791","s":"__wt_err_func"},{"b":"5606BD8BC000","o":"9D8BAB","s":"__wt_panic"},{"b":"5606BD8BC000","o":"EBA4B3"},{"b":"5606BD8BC000","o":"FB79A4"},{"b":"5606BD8BC000","o":"EFA459"},{"b":"5606BD8BC000","o":"EFB34B","s":"__wt_txn_checkpoint"},{"b":"5606BD8BC000","o":"EE0BDA"},{"b":"5606BD8BC000","o":"F00E49","s":"__wt_txn_recover"},{"b":"5606BD8BC000","o":"E93397","s":"__wt_connection_workers"},{"b":"5606BD8BC000","o":"E8FCA8","s":"wiredtiger_open"},{"b":"5606BD8BC000","o":"E52D2A","s":"_ZN5mongo18WiredTigerKVEngine15_openWiredTigerERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_"},{"b":"5606BD8BC000","o":"E58EFB","s":"_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mmbbbb"},{"b":"5606BD8BC000","o":"E3AED2"},{"b":"5606BD8BC000","o":"12B7B3A","s":"_ZN5mongo23initializeStorageEngineEPNS_14ServiceContextENS_22StorageEngineInitFlagsE"},{"b":"5606BD8BC000","o":"ABF448"},{"b":"5606BD8BC000","o":"AC367F","s":"_ZN5mongo11mongoDbMainEiPPcS1_"},{"b":"5606BD8BC000","o":"A4CC09","s":"main"},{"b":"7F3931C11000","o":"223D5","s":"__libc_start_main"},{"b":"5606BD8BC000","o":"ABDD2F"}],"processInfo":{ "mongodbVersion" : "4.0.13", "gitVersion" : "bda366f0b0e432ca143bc41da54d8732bd8d03c0", "compiledModules" : [ "enterprise" ], "uname" : { "sysname" : "Linux", "release" : "3.10.0-957.12.2.el7.x86_64", "version" : "#1 SMP Tue May 14 21:24:32 UTC 2019", "machine" : "x86_64" }, "somap" : [ { "b" : "5606BD8BC000", "elfType" : 3, "buildId" : "1E4C3CE7F1CD1060AE96009AB424CC5BDB1D9503" }, { "b" : "7FFF0D6E3000", "elfType" : 3, "buildId" : "D149FABD4DF969958863221F946E26C130432B47" }, { "b" : "7F3934FC5000", "path" : "/lib64/libldap-2.4.so.2", "elfType" : 3, "buildId" : "F1FADDDE0D21D5F4E2DCADEDD3B85B6E7AAC9883" }, { "b" : "7F3934DB6000", "path" : "/lib64/liblber-2.4.so.2", "elfType" : 3, "buildId" : "3192C56CD451E18EB9F29CB045432BA9C738DD29" }, { "b" : "7F39348FD000", "path" : "/lib64/libnetsnmpmibs.so.31", "elfType" : 3, "buildId" : "26B6C17A2D031E0B3BD83D24D463E613A8671F91" }, { "b" : "7F39346EE000", "path" : "/lib64/libsensors.so.4", "elfType" : 3, "buildId" : "DA4D8D3702D3F0ED057441AC263709F2ED01157B" }, { "b" : "7F39344EA000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "357693C8F1F49D93010C4E31529C07CDD2BD3D08" }, { "b" : "7F3934282000", "path" : "/lib64/librpm.so.3", "elfType" : 3, "buildId" : "BA9B5DF3BBE4764C0BA2AD6B9BBD5E10512BC082" }, { "b" : "7F3934055000", "path" : "/lib64/librpmio.so.3", "elfType" : 3, "buildId" : "069D6EB6C16272B4A4428CAC4087799F18318AA5" }, { "b" : "7F3933DE7000", "path" : "/lib64/libnetsnmpagent.so.31", "elfType" : 3, "buildId" : "F1EEC61134F8FD1553560FAB87D01014AD18D32B" }, { "b" : "7F3933BDC000", "path" : "/lib64/libwrap.so.0", "elfType" : 3, "buildId" : "8C4AA46577D3AA7EBF8338BDFAECC6586EF29906" }, { "b" : "7F39338DA000", "path" : "/lib64/libnetsnmp.so.31", "elfType" : 3, "buildId" : "D443C7B944B5C1E0E09122F430D48099E22619EF" }, { "b" : "7F3933668000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "AEF5E6F2240B55F90E9DF76CFBB8B9D9F5286583" }, { "b" : "7F3933206000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "3593FA778645A59EA272DBBB59D318C60940E792" }, { "b" : "7F3932FE9000", "path" : "/lib64/libsasl2.so.3", "elfType" : 3, "buildId" : "E2F2017F821DD1B9D307DA1A9B8014F2941AEB7B" }, { "b" : "7F3932D9C000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "BCC30853830CD911E58700591830DF51ABCBD7BA" }, { "b" : "7F3932B33000", "path" : "/lib64/libcurl.so.4", "elfType" : 3, "buildId" : "9114859D3C4BEC47A03CA321EE367DCA799638CD" }, { "b" : "7F3932831000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "5B14BE4D749631673523A61074C10959D50F5455" }, { "b" : "7F3932618000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "C444AE61E7CBB716FD9C18A0B46A7FE8F4FCF3E5" }, { "b" : "7F3932410000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "EFDE2029C9A4A20BE5B8D8AE7E6551FF9B5755D2" }, { "b" : "7F39321FA000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "179F202998E429AA1215907F6D4C5C1BB9C90136" }, { "b" : "7F3931FDE000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "96900CB0FF25B26F2BBDF247DE1408242E4773D8" }, { "b" : "7F3931C11000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "EB9F22A3891E5FD3494DFD9ED199E20AE71BB08D" }, { "b" : "7F393521A000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "A527FE72908703C5972AE384E78D1850D1881EE7" }, { "b" : "7F39319BF000", "path" : "/lib64/libssl3.so", "elfType" : 3, "buildId" : "2E28F6A705F2ECEA8460D4716D5D1C24B5DDA5E4" }, { "b" : "7F3931798000", "path" : "/lib64/libsmime3.so", "elfType" : 3, "buildId" : "8D0B4010959C321022DF9CE239277A9D7B34A76A" }, { "b" : "7F393146B000", "path" : "/lib64/libnss3.so", "elfType" : 3, "buildId" : "F5A64BB37FA3972E545EF459A51310F0AB56FA56" }, { "b" : "7F393123B000", "path" : "/lib64/libnssutil3.so", "elfType" : 3, "buildId" : "E0705772325A52C3372FFFB8BDE5F786E2E200D6" }, { "b" : "7F3931037000", "path" : "/lib64/libplds4.so", "elfType" : 3, "buildId" : "084D2194302908913F68B9DCD27DE46FA5B50522" }, { "b" : "7F3930E32000", "path" : "/lib64/libplc4.so", "elfType" : 3, "buildId" : "799B28AD9A5460D78376E2C11260F2E858B95DE3" }, { "b" : "7F3930BF4000", "path" : "/lib64/libnspr4.so", "elfType" : 3, "buildId" : "DE762A28174110911B273E175D54F222B313CFE0" }, { "b" : "7F3930866000", "path" : "/usr/lib64/perl5/CORE/libperl.so", "elfType" : 3, "buildId" : "E2C3C10A756404CC8888CD6CA8DFAD26064EF3CB" }, { "b" : "7F393064C000", "path" : "/lib64/libnsl.so.1", "elfType" : 3, "buildId" : "B1DC715EF11A8ACF2176AE6D34914328F9DBE9CC" }, { "b" : "7F3930415000", "path" : "/lib64/libcrypt.so.1", "elfType" : 3, "buildId" : "740CAD898E29E1F3B73A323CCEC4A7C88911647F" }, { "b" : "7F3930212000", "path" : "/lib64/libutil.so.1", "elfType" : 3, "buildId" : "BFB22A176189006BBF996A7664CCC6BDB2EB9BD9" }, { "b" : "7F3930002000", "path" : "/lib64/libbz2.so.1", "elfType" : 3, "buildId" : "0C85C0386F0CF41EA39969CF7F58A558D1AD3235" }, { "b" : "7F392FDEC000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "B9D5F73428BD6AD68C96986B57BEA3B7CEDB9745" }, { "b" : "7F392FBD4000", "path" : "/lib64/libelf.so.1", "elfType" : 3, "buildId" : "A22999760AD7CF97A32E5A78C34238D56034F9DD" }, { "b" : "7F392F9AE000", "path" : "/lib64/liblzma.so.5", "elfType" : 3, "buildId" : "3B2C97C1937B73A69C412A96D0810C43DF0C6F54" }, { "b" : "7F392F7A4000", "path" : "/lib64/libpopt.so.0", "elfType" : 3, "buildId" : "7AE00165FBAF6920DD5AED6905820DDBAB589E84" }, { "b" : "7F392F57D000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "D2DD4DA3FDE1477D25BFFF80F3A25FDB541A8179" }, { "b" : "7F392F378000", "path" : "/lib64/libcap.so.2", "elfType" : 3, "buildId" : "D594598CEBC1D6DC4ECE2ABF4C29398FDB00587E" }, { "b" : "7F392F16F000", "path" : "/lib64/libacl.so.1", "elfType" : 3, "buildId" : "7F39882FC0B80BE53790C2EAC307D39F7DE1AD6E" }, { "b" : "7F392EF41000", "path" : "/lib64/liblua-5.1.so", "elfType" : 3, "buildId" : "BDD4B9CFC1D3F31D3A5A430D2F9080E020C5B0BA" }, { "b" : "7F392EB82000", "path" : "/lib64/libdb-5.3.so", "elfType" : 3, "buildId" : "746F78257330663AD584998981E6C4D0BF49C940" }, { "b" : "7F392E959000", "path" : "/lib64/libaudit.so.1", "elfType" : 3, "buildId" : "1B298E71E97B742F93F2F6D8FC7BDDA655CDF5EF" }, { "b" : "7F392E670000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "45BAB0BB455BDFA960FDA22E4124CF17B67CC930" }, { "b" : "7F392E46C000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "B4BE1023D9606A88169DF411BF94AF417D7BA1A0" }, { "b" : "7F392E239000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "A9B3906192687CC45D483AE3C58C8AF745A6726A" }, { "b" : "7F392E029000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "94B3BCB669126166B77CDCE6092679A6AA2004C8" }, { "b" : "7F392DE25000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "2E01D5AC08C1280D013AAB96B292AC58BC30A263" }, { "b" : "7F392DBF2000", "path" : "/lib64/libidn.so.11", "elfType" : 3, "buildId" : "2B77BBEFFF65E94F3E0B71A4E89BEB68C4B476C5" }, { "b" : "7F392D9C8000", "path" : "/lib64/libssh2.so.1", "elfType" : 3, "buildId" : "689404B6B895EACFAD65BA16E07E5BF0004F5E0C" }, { "b" : "7F392D7C5000", "path" : "/lib64/libfreebl3.so", "elfType" : 3, "buildId" : "B758881F4B6AF6C28C07A1A57713CBD2144628D4" }, { "b" : "7F392D563000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "9CA3D11F018BEEB719CDB34BE800BF1641350D0A" }, { "b" : "7F392D35E000", "path" : "/lib64/libattr.so.1", "elfType" : 3, "buildId" : "2617ECC6738047E207AE3ADD990BD6A34D11B265" }, { "b" : "7F392D158000", "path" : "/lib64/libcap-ng.so.0", "elfType" : 3, "buildId" : "43578677DF613E9D58128ED4AE0C344FBC1E44C0" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x5606bfde7ef1]
 mongod(+0x252B109) [0x5606bfde7109]
 mongod(+0x252B5ED) [0x5606bfde75ed]
 libpthread.so.0(+0xF5D0) [0x7f3931fed5d0]
 libc.so.6(gsignal+0x37) [0x7f3931c47207]
 libc.so.6(abort+0x148) [0x7f3931c488f8]
 mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x5606be307079]
 mongod(+0xE7D206) [0x5606be739206]
 mongod(+0xEE7951) [0x5606be7a3951]
 mongod(__wt_err_func+0x90) [0x5606be294791]
 mongod(__wt_panic+0x39) [0x5606be294bab]
 mongod(+0xEBA4B3) [0x5606be7764b3]
 mongod(+0xFB79A4) [0x5606be8739a4]
 mongod(+0xEFA459) [0x5606be7b6459]
 mongod(__wt_txn_checkpoint+0x1DB) [0x5606be7b734b]
 mongod(+0xEE0BDA) [0x5606be79cbda]
 mongod(__wt_txn_recover+0x939) [0x5606be7bce49]
 mongod(__wt_connection_workers+0x37) [0x5606be74f397]
 mongod(wiredtiger_open+0x1A88) [0x5606be74bca8]
 mongod(_ZN5mongo18WiredTigerKVEngine15_openWiredTigerERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_+0xAA) [0x5606be70ed2a]
 mongod(_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mmbbbb+0x67B) [0x5606be714efb]
 mongod(+0xE3AED2) [0x5606be6f6ed2]
 mongod(_ZN5mongo23initializeStorageEngineEPNS_14ServiceContextENS_22StorageEngineInitFlagsE+0x56A) [0x5606beb73b3a]
 mongod(+0xABF448) [0x5606be37b448]
 mongod(_ZN5mongo11mongoDbMainEiPPcS1_+0xF0F) [0x5606be37f67f]
 mongod(main+0x9) [0x5606be308c09]
 libc.so.6(__libc_start_main+0xF5) [0x7f3931c333d5]
 mongod(+0xABDD2F) [0x5606be379d2f]
-----  END BACKTRACE  -----



 Comments   
Comment by Dmitry Agranat [ 02/Dec/19 ]

Hi kelany@afaqy.com,

I will go ahead and close this ticket as the issue is not related to a bug in MongoDB.

Regards,
Dima

Comment by Dmitry Agranat [ 02/Dec/19 ]

kelany@afaqy.com, if the backup was taken on the HW which had this issue in the past, the issue will persist after the restore. Same goes for mongodump/mongorestore. After your HW issue is resolved, we recommend resyncing this server from another (healthy) member of your replica set.

Comment by Kelany Mahmoud [ 02/Dec/19 ]

dmitry.agranat@mongodb.com Thanks for your support, I will check this problem with the infrastructure team.
But we retrieved a backup on 25 Nov but still, the problem exists, also I tried to mongodump and mongorestore also the same problem happened after restoring 5 GB from the data.

the following is the hard disk spaces allocated to this VM

Comment by Dmitry Agranat [ 01/Dec/19 ]

Hi kelany@afaqy.com,

This is one of the many errors we can see:

Nov 26 11:53:14 abdafaqydb02 kernel: device-mapper: thin: No free metadata blocks
Nov 26 11:53:14 abdafaqydb02 kernel: device-mapper: thin: 253:12: switching pool to read-only mode
Nov 26 11:53:19 abdafaqydb02 lvm[4501]: WARNING: Thin pool centos--afqmngo--indexes-pool00-tpool metadata is now 100.00% full.
Nov 26 11:55:21 abdafaqydb02 kernel: Buffer I/O error on dev dm-14, logical block 1017872, lost async page write
Nov 26 11:55:21 abdafaqydb02 kernel: Buffer I/O error on dev dm-14, logical block 1017873, lost async page write
Nov 26 11:55:21 abdafaqydb02 kernel: Buffer I/O error on dev dm-14, logical block 1017874, lost async page write
Nov 26 11:55:21 abdafaqydb02 kernel: Buffer I/O error on dev dm-14, logical block 1017875, lost async page write
Nov 26 11:55:21 abdafaqydb02 kernel: Buffer I/O error on dev dm-14, logical block 1017876, lost async page write
Nov 26 11:55:21 abdafaqydb02 kernel: Buffer I/O error on dev dm-14, logical block 1017877, lost async page write
Nov 26 11:55:21 abdafaqydb02 kernel: Buffer I/O error on dev dm-14, logical block 1017878, lost async page write
Nov 26 11:55:21 abdafaqydb02 kernel: Buffer I/O error on dev dm-14, logical block 1017879, lost async page write
Nov 26 11:55:21 abdafaqydb02 kernel: Buffer I/O error on dev dm-14, logical block 1017880, lost async page write
Nov 26 11:55:21 abdafaqydb02 kernel: Buffer I/O error on dev dm-14, logical block 1017881, lost async page write
Nov 26 11:55:21 abdafaqydb02 abrt-hook-ccpp: Process 6829 (mongod) of user 988 killed by SIGABRT - dumping core

This error might be due to either of these:

  • damaged disk platters/cells
  • connection problems (ie: bad cable, a "lost" iSCSI target, etc)
  • thinly provisioned block device whose parent pool space was exhausted

Thanks,
Dima

Comment by Kelany Mahmoud [ 01/Dec/19 ]

I have uploaded them

Comment by Dmitry Agranat [ 01/Dec/19 ]

Thanks kelany. I see you've just uploaded mongoD log but I do not see the requested syslog and dmesg logs. Please let us know when syslog and dmesg logs are uploaded.

Comment by Mahmoud Kelany [ 01/Dec/19 ]

I have uploaded the log

Best Regards

[image:
https://lh3.googleusercontent.com/mystWdY8foFPd_G9wsU6UNMimRc-Iqd2k7dmv8uKMm0if1zDWDQIJvcMxOqEY81Ltm4rugkdTnwzRi0uwA6RnyMjfZNyFuoY6mDSuir05CqLHem8uYxxZqAg9iPk8okH2IGfejfh]

Mahmoud Kelany

Software Operations Manager
Mob: +966-58-1500033

Tel: +966-9200-10519 Ext.: 1033
Head Office: Olaya St, Riyadh, Saudi Arabia
www.afaqy.com <http://www.afaqy.com/>

[image:
https://lh5.googleusercontent.com/T4Uhb67M9tdain-6AfKAJAIMOsxyBotrICCyaSht2Np0q8DZr3Ez9mmV2qs_PjR2gdBo8f-1uaC1fVBDWsLGwRnhdLArcpUueYlUXakwz1fatFPQb3t6wBEXbv0dfNzsOy0j8eJa]
<https://twitter.com/AFAQYIT> [image:
https://lh3.googleusercontent.com/_l9XPJ3HAr96pUniwWLGujFge111gwThCHPFmVNnirMgyeIwLjxGUDn_6LFa-y01g3tDkR7thDFynnpi0rwROefrjUYrQMKAjoFGoxCnv4o-vIgq2IeIeYUyCzn9TznaN79UB1-H]
<https://www.facebook.com/AFAQYIT/> [image:
https://lh5.googleusercontent.com/wlptnPx9kiczSA5Q4V9AcLw1GPBWdhw_41eZme9nsIa1lSSlWmyChf6lptaJI5Ke11_Q5OBU5acLRmAsTumFC1xKGBu7Oixy1yUPxqR-1s1c7F_UAwiiTPv7oLpQv-8r8EQupQct]
<https://www.linkedin.com/company/afaqy-co/> [image:
https://lh5.googleusercontent.com/92AU1CIU1llzvvz_wOtfw0dt-MKCQUEQTCqWk1am0JNbbynX62CA-ZNBy4fDaPUDAL-CwcfjzbQHx-RhezYFeNQhEmpoqlsX16lg7si-Nei7WFBhEyEDoKNBx6vLMk3fuPLTFfF1]
<https://www.youtube.com/channel/UCUxFcDYsuUYwfkBhJc6vKyg>

On Sun, Dec 1, 2019 at 2:41 PM Dmitry Agranat (Jira) <jira@mongodb.org>

Comment by Dmitry Agranat [ 01/Dec/19 ]

Hi kelany@afaqy.com,

The error you posted fdatasync: Input/output error usually indicates a hardware issue or a corrupted filesystem. Please upload syslog and dmesg to this secure location so that we could investigate.

Thanks,
Dima

Generated at Thu Feb 08 05:07:16 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.