[SERVER-37082] Cannot start mongod or --repair (caused by unclean shutdown) Created: 12/Sep/18 Updated: 12/Sep/18 Resolved: 12/Sep/18 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | WiredTiger |
| Affects Version/s: | 3.6.2 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Chloe Chen | Assignee: | Nick Brewer |
| Resolution: | Done | Votes: | 0 |
| Labels: | envm, rpo, trct, wtc | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
| Operating System: | Linux |
| Participants: |
| Description |
|
Hello! We have a production server that has crashed unexpectedly and we can't get it to start because of a error on WiredTiger.wt Running ./mongod --dbpath /data/db Get me: 2018-09-11T16:51:00.892-0700 I CONTROL [initandlisten] MongoDB starting : pid=10897 port=27017 dbpath=/slowfs/vginfra3/chloec/usage55/db 64-bit host=vgzeburt55 } ***aborting after fassert() failure 2018-09-11T16:51:00.961-0700 F - [initandlisten] Got signal: 6 (Aborted). 0x555557702681 0x555557701899 0x555557701d7d 0x7ffff70905e0 0x7ffff6cf31f7 0x7ffff6cf48e8 0x555555ec97bf 0x555555f91c5e 0x555555e67917 0x555555e67b33 0x555555e67e5c 0x555555fbd5d9 0x555555fbbc59 0x555555fa189a 0x555555fee4ab 0x555555fee9cd 0x555555feec8c 0x5555560572f2 0x555555fe5238 0x555555fbb32e 0x555555fbb40b 0x555555fa0604 0x555555f75fb9 0x555555f5a5b4 0x55555612aee7 0x555555e636c7 0x555555f393bc 0x555555ecb489 0x7ffff6cdfc05 0x555555f28d11 ,{"b":"555555554000","o":"21AD899"},{"b":"555555554000","o":"21ADD7D"},{"b":"7FFFF7081000","o":"F5E0"},{"b":"7FFFF6CBE000","o":"351F7","s":"gsignal"},{"b":"7FFFF6CBE000","o":"368E8","s":"abort"},{"b":"555555554000","o":"9757BF","s":"ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"555555554000","o":"A3DC5E"},{"b":"555555554000","o":"913917","s":"wt_eventv"},{"b":"555555554000","o":"913B33","s":"wt_err"},{"b":"555555554000","o":"913E5C","s":"wt_panic"},{"b":"555555554000","o":"A695D9","s":"wt_turtle_read"},{"b":"555555554000","o":"A67C59","s":"wt_metadata_search"},{"b":"555555554000","o":"A4D89A","s":"wt_conn_dhandle_open"},{"b":"555555554000","o":"A9A4AB","s":"wt_session_get_dhandle"},{"b":"555555554000","o":"A9A9CD","s":"wt_session_get_dhandle"},{"b":"555555554000","o":"A9AC8C","s":"wt_session_get_btree_ckpt"},{"b":"555555554000","o":"B032F2","s":"wt_curfile_open"},{"b":"555555554000","o":"A91238"},{"b":"555555554000","o":"A6732E","s":"wt_metadata_cursor_open"},{"b":"555555554000","o":"A6740B","s":"wt_metadata_cursor"},{"b":"555555554000","o":"A4C604","s":"wiredtiger_open"},{"b":"555555554000","o":"A21FB9","s":"_ZN5mongo18WiredTigerKVEngineC1ERKNSt7cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"},{"b":"555555554000","o":"A065B4"},{"b":"555555554000","o":"BD6EE7","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"555555554000","o":"90F6C7"},{"b":"555555554000","o":"9E53BC","s":"_ZN5mongo11mongoDbMainEiPPcS1"},{"b":"555555554000","o":"977489","s":"main"},{"b":"7FFFF6CBE000","o":"21C05","s":"__libc_start_main"},{"b":"555555554000","o":"9D4D11"}],"processInfo":{ "mongodbVersion" : "3.6.2", "gitVersion" : "489d177dbd0f0420a8ca04d39fd78d0a2c539420", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.0-693.11.6.el7.x86_64", "version" : "#1 SMP Thu Jan 4 01:06:37 UTC 2018", "machine" : "x86_64" }, "somap" : [ { "b" : "555555554000", "elfType" : 3, "buildId" : "454A81637B4013AE082538A57B34F8A42B39277A" }, { "b" : "7FFFF88FA000", "elfType" : 3, "buildId" : "33DEC63F3B0D3EE9ABDAC478FF3E7F1F43FAF9DE" }, { "b" : "7FFFF7BC1000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "FF4E72F4E574E143330FB3C66DB51613B0EC65EA" }, { "b" : "7FFFF79B9000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "6D322588B36D2617C03C0F3B93677E62FCFFDA81" }, { "b" : "7FFFF77B5000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "1E42EBFB272D37B726F457D6FE3C33D2B094BB69" }, { "b" : "7FFFF74B3000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "808BD35686C193F218A5AAAC6194C49214CFF379" }, { "b" : "7FFFF729D000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "408B46E291B2D4C9612E27C0509D165D7E186D40" }, { "b" : "7FFFF7081000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "A48D21B2578A8381FBD8857802EAA660504248DC" }, { "b" : "7FFFF6CBE000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "95FF02A4BEBABC573C7827A66D447F7BABDDAA44" }, { "b" : "7FFFF7DDB000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "22FA66DA7D14C88BF36C69454A357E5F1DEFAE4E" } ] }}
Giving --repair doesn't seem to make much a difference. I've seen several other cases in Jira, but seems like each case required a manual investigation. Thanks! |
| Comments |
| Comment by Nick Brewer [ 12/Sep/18 ] | |
|
chloec Glad to hear you were able to get it working from a backup. Some suggestions to keep in mind for the future:
-Nick | |
| Comment by Chloe Chen [ 12/Sep/18 ] | |
|
It's a VM. Gladly we just found we have data backup before. But still thanks for the help. I will keep you suggestions in mind.
Thanks, Chloe | |
| Comment by Nick Brewer [ 12/Sep/18 ] | |
|
chloec A blank .turtle file indicates corruption - as this file contain metadata that is used to interpret other WiredTiger files, we will not be able to perform a repair if the file is blank. In this case, your best option is to utilize any available backups you have. Based on this line, I assume you're using slowfs:
I'm not familiar with how MongoDB performs with slowfs, however it's worth noting that MongoDB requires fync on directories. With that in mind, you may want to ensure you're using the appropriate FsyncStrategy option for slowfs, as outlined on its GitHub. For tracking purposes, could you clarify whether this machine is a VM, native, container, etc? Thanks, | |
| Comment by Chloe Chen [ 12/Sep/18 ] | |
|
My WiredTiger.turtle files is blank... I am using mongodb-linux-x86_64-3.6.2 on CentOS7.3
| |
| Comment by Nick Brewer [ 12/Sep/18 ] | |
|
chloec To perform a repair attempt, we'd need both the WiredTiger.wt and WiredTiger.turtle files. Additionally, we'd need to confirm:
Thanks, |