[SERVER-26237] Mongodb failed to start after a disk 100% full problem Created: 22/Sep/16  Updated: 13/Aug/18  Resolved: 19/Nov/16

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.2.3
Fix Version/s: None

Type: Bug Priority: Critical - P2
Reporter: Paddy Zhu [X] Assignee: Ramon Fernandez Marina
Resolution: Duplicate Votes: 0
Labels: envns, rdi, rps, trcf, wtc
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File WiredTiger.turtle     File WiredTiger.wt     Text File mongodb.log     File repair_attempt.tgz    
Issue Links:
Duplicate
Operating System: Linux
Participants:

 Description   

My server had a 'No space left on device' problem because our disk was running out of inode early today.

But after we fixed the disk problem, our mongodb was still unable to start or repair.

I desperately need your help : (

And the log information as follow:

2016-07-25T17:29:53.558+0800 I CONTROL  [initandlisten] MongoDB starting : pid=10992 port=27017 dbpath=/ick/data2 64-bit host=localhost.localdomain
2016-07-25T17:29:53.558+0800 I CONTROL  [initandlisten] db version v3.2.3
2016-07-25T17:29:53.558+0800 I CONTROL  [initandlisten] git version: b326ba837cf6f49d65c2f85e1b70f6f31ece7937
2016-07-25T17:29:53.558+0800 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2016-07-25T17:29:53.558+0800 I CONTROL  [initandlisten] allocator: tcmalloc
2016-07-25T17:29:53.558+0800 I CONTROL  [initandlisten] modules: none
2016-07-25T17:29:53.558+0800 I CONTROL  [initandlisten] build environment:
2016-07-25T17:29:53.558+0800 I CONTROL  [initandlisten]     distmod: rhel62
2016-07-25T17:29:53.558+0800 I CONTROL  [initandlisten]     distarch: x86_64
2016-07-25T17:29:53.558+0800 I CONTROL  [initandlisten]     target_arch: x86_64
2016-07-25T17:29:53.558+0800 I CONTROL  [initandlisten] options: { repair: true, storage: { dbPath: "/ick/data2" } }
2016-07-25T17:29:53.847+0800 I -        [initandlisten] Detected data files in /ick/data2 created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2016-07-25T17:29:53.847+0800 I STORAGE  [initandlisten] Detected WT journal files.  Running recovery from last checkpoint.
2016-07-25T17:29:53.847+0800 I STORAGE  [initandlisten] journal to nojournal transition config: create,cache_size=18G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2016-07-25T17:29:53.898+0800 E STORAGE  [initandlisten] WiredTiger (0) [1469438993:898124][10992:0x7f69edc18d40], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 196608: block header checksum of 675114611 doesn't match expected checksum of 1687733155
2016-07-25T17:29:53.898+0800 E STORAGE  [initandlisten] WiredTiger (0) [1469438993:898190][10992:0x7f69edc18d40], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
2016-07-25T17:29:53.898+0800 E STORAGE  [initandlisten] WiredTiger (-31804) [1469438993:898209][10992:0x7f69edc18d40], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2016-07-25T17:29:53.898+0800 I -        [initandlisten] Fatal Assertion 28558
2016-07-25T17:29:53.898+0800 I -        [initandlisten] 
 
***aborting after fassert() failure
 
 
2016-07-25T17:29:53.946+0800 F -        [initandlisten] Got signal: 6 (Aborted).
 
 0x130e1e2 0x130d119 0x130d922 0x3ee060f7e0 0x3ee0232625 0x3ee0233e05 0x1295b32 0x108b3e3 0x1a57e3c 0x1a57fdd 0x1a583c4 0x199067c 0x1990bf3 0x198dbd7 0x19919de 0x19ae1a8 0x19e2140 0x1a56bfb 0x1a57117 0x1a5726b 0x19f02f8 0x1a54415 0x1a20b7f 0x1a20c5e 0x19df078 0x107322e 0x106f6a3 0xf99268 0x974701 0x97815d 0x3ee021ed5d 0x972279
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"F0E1E2","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"F0D119"},{"b":"400000","o":"F0D922"},{"b":"3EE0600000","o":"F7E0"},{"b":"3EE0200000","o":"32625","s":"gsignal"},{"b":"3EE0200000","o":"33E05","s":"abort"},{"b":"400000","o":"E95B32","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"C8B3E3"},{"b":"400000","o":"1657E3C","s":"__wt_eventv"},{"b":"400000","o":"1657FDD","s":"__wt_err"},{"b":"400000","o":"16583C4","s":"__wt_panic"},{"b":"400000","o":"159067C","s":"__wt_block_extlist_read"},{"b":"400000","o":"1590BF3","s":"__wt_block_extlist_read_avail"},{"b":"400000","o":"158DBD7","s":"__wt_block_checkpoint_load"},{"b":"400000","o":"15919DE"},{"b":"400000","o":"15AE1A8","s":"__wt_btree_open"},{"b":"400000","o":"15E2140","s":"__wt_conn_btree_open"},{"b":"400000","o":"1656BFB","s":"__wt_session_get_btree"},{"b":"400000","o":"1657117","s":"__wt_session_get_btree"},{"b":"400000","o":"165726B","s":"__wt_session_get_btree_ckpt"},{"b":"400000","o":"15F02F8","s":"__wt_curfile_open"},{"b":"400000","o":"1654415"},{"b":"400000","o":"1620B7F","s":"__wt_metadata_cursor_open"},{"b":"400000","o":"1620C5E","s":"__wt_metadata_cursor"},{"b":"400000","o":"15DF078","s":"wiredtiger_open"},{"b":"400000","o":"C7322E","s":"_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb"},{"b":"400000","o":"C6F6A3"},{"b":"400000","o":"B99268","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"574701","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"57815D","s":"main"},{"b":"3EE0200000","o":"1ED5D","s":"__libc_start_main"},{"b":"400000","o":"572279"}],"processInfo":{ "mongodbVersion" : "3.2.3", "gitVersion" : "b326ba837cf6f49d65c2f85e1b70f6f31ece7937", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "2.6.32-573.22.1.el6.x86_64", "version" : "#1 SMP Wed Mar 23 03:35:39 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "B389239A9108B604FCA797D0F968E253E7BF9F5E" }, { "b" : "7FFEC8C76000", "elfType" : 3, "buildId" : "2F78F7B8A7307DD9C340F3CA735BE2CAA9C157D1" }, { "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "B84C31B86733DE212F6886FE6F55630FE56180A9" }, { "path" : "/usr/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "A30A68D2F579614CBEA988BDAAC20CD56D8C48FC" }, { "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "95159178F1A4A3DBDC7819FBEA2C80E5FCDD6BAC" }, { "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "29B61382141595ECBA6576232E44F2310C3AAB72" }, { "path" : "/usr/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "5D42D8781F5FF4C06B4F839B85B21786AA44488D" }, { "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "989FE3A42CA8CEBDCC185A743896F23A0CF537ED" }, { "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "9A6E4BDFA184364D81F7DFD789474C3FB8F98A00" }, { "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "C56DD1B811FC0D9263248EBB308C73FCBCD80FC1" }, { "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "8E6FA4C4B0594C355C1B90C1D49990368C81A040" }, { "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "959C5E10A47EE8A633E7681B64B4B9F74E242ED5" }, { "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "441FA45097A11508E50D55A3D1FF169BF2BE7C62" }, { "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "F62622218875795666E08B92D176A50791183EEC" }, { "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "152E2C18A7A2145021A8A879A01A82EE134E3946" }, { "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "B8DEDADC140347276164C729418C7A37B7224135" }, { "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "5FA8E5038EC04A774AF72A9BB62DC86E1049C4D6" }, { "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "4BDFC7A19C1F328EB4FCFBCE7A1E27606928610D" }, { "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "AF374BAFB7F5B139A0B431D3F06D82014AFF3251" }, { "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "C39D7FFB49DFB1B55AD09D1D711AD802123F6623" }, { "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "E6798A06BEE17CF102BBA44FD512FF8B805CEAF1" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x130e1e2]
 mongod(+0xF0D119) [0x130d119]
 mongod(+0xF0D922) [0x130d922]
 libpthread.so.0(+0xF7E0) [0x3ee060f7e0]
 libc.so.6(gsignal+0x35) [0x3ee0232625]
 libc.so.6(abort+0x175) [0x3ee0233e05]
 mongod(_ZN5mongo13fassertFailedEi+0x82) [0x1295b32]
 mongod(+0xC8B3E3) [0x108b3e3]
 mongod(__wt_eventv+0x40C) [0x1a57e3c]
 mongod(__wt_err+0x8D) [0x1a57fdd]
 mongod(__wt_panic+0x24) [0x1a583c4]
 mongod(__wt_block_extlist_read+0x6C) [0x199067c]
 mongod(__wt_block_extlist_read_avail+0x33) [0x1990bf3]
 mongod(__wt_block_checkpoint_load+0x3B7) [0x198dbd7]
 mongod(+0x15919DE) [0x19919de]
 mongod(__wt_btree_open+0xC68) [0x19ae1a8]
 mongod(__wt_conn_btree_open+0x140) [0x19e2140]
 mongod(__wt_session_get_btree+0xEB) [0x1a56bfb]
 mongod(__wt_session_get_btree+0x607) [0x1a57117]
 mongod(__wt_session_get_btree_ckpt+0xAB) [0x1a5726b]
 mongod(__wt_curfile_open+0x1E8) [0x19f02f8]
 mongod(+0x1654415) [0x1a54415]
 mongod(__wt_metadata_cursor_open+0x5F) [0x1a20b7f]
 mongod(__wt_metadata_cursor+0x7E) [0x1a20c5e]
 mongod(wiredtiger_open+0x15F8) [0x19df078]
 mongod(_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb+0x45E) [0x107322e]
 mongod(+0xC6F6A3) [0x106f6a3]
 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x598) [0xf99268]
 mongod(_ZN5mongo13initAndListenEi+0x321) [0x974701]
 mongod(main+0x15D) [0x97815d]
 libc.so.6(__libc_start_main+0xFD) [0x3ee021ed5d]
 mongod(+0x572279) [0x972279]
-----  END BACKTRACE  -----



 Comments   
Comment by Kelsey Schubert [ 19/Nov/16 ]

Hi Paddyzhu,

The likely cause of this issue was fixed in WT-2897, and the fix is included in MongoDB 3.2.10 and later. Since this issue has been resolved, I am closing this ticket as a duplicate.

Kind regards,
Thomas

Comment by Ramon Fernandez Marina [ 03/Oct/16 ]

Apologies for the radio silence Paddyzhu.

The goal behind looking at your data files was to see if this issue had corrupted them, or if only the repaired files were affected. If you haven't experienced any further issues and your data integrity hasn't been affected (you can run validate(), but note it's resource-intensive) then we don't need your files.

Can you please confirm that you're not experiencing any issues that could be due to data corruption? If you are, then you should upload your database files.

Thanks,
Ramón.

Comment by Paddy Zhu [X] [ 23/Sep/16 ]

Hi everyone,

We replaced our files in /usr/local/mongodb/data with your repair_attempt.zip and the issue is fixed!
Thank you so much.

And since it is done do you still want our contents of /usr/local/mongodb/data ? If so I will upload it for you : )

Comment by Paddy Zhu [X] [ 23/Sep/16 ]

Thank you!

But It seems like I might just post the wrong log since I mixed other's with my own one.

please check the following one and our mongodb.log I uploaded

2016-09-22T15:34:24.570+0800 I CONTROL  [initandlisten] MongoDB starting : pid=1885 port=27017 dbpath=/usr/local/mongodb/da                                             ta 64-bit host=iZ625ztxpiuZ
2016-09-22T15:34:24.570+0800 I CONTROL  [initandlisten] db version v3.2.7
2016-09-22T15:34:24.570+0800 I CONTROL  [initandlisten] git version: 4249c1d2b5999ebbf1fdf3bc0e0e3b3ff5c0aaf2
2016-09-22T15:34:24.570+0800 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2016-09-22T15:34:24.570+0800 I CONTROL  [initandlisten] allocator: tcmalloc
2016-09-22T15:34:24.570+0800 I CONTROL  [initandlisten] modules: none
2016-09-22T15:34:24.570+0800 I CONTROL  [initandlisten] build environment:
2016-09-22T15:34:24.570+0800 I CONTROL  [initandlisten]     distmod: rhel62
2016-09-22T15:34:24.570+0800 I CONTROL  [initandlisten]     distarch: x86_64
2016-09-22T15:34:24.570+0800 I CONTROL  [initandlisten]     target_arch: x86_64
2016-09-22T15:34:24.570+0800 I CONTROL  [initandlisten] options: { repair: true, storage: { dbPath: "/usr/local/mongodb/dat                                             a", engine: "wiredTiger" } }
2016-09-22T15:34:24.593+0800 I STORAGE  [initandlisten] Detected WT journal files.  Running recovery from last checkpoint.
2016-09-22T15:34:24.593+0800 I STORAGE  [initandlisten] journal to nojournal transition config: create,cache_size=4G,sessio                                             n_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compre                                             ssor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2016-09-22T15:34:24.615+0800 E STORAGE  [initandlisten] WiredTiger (0) [1474529664:615769][1885:0x7fd02f352d40], file:Wired                                             Tiger.wt, connection: read checksum error for 4096B block at offset 12288: block header checksum of 1717798939 doesn't matc                                             h expected checksum of 3759426581
2016-09-22T15:34:24.615+0800 E STORAGE  [initandlisten] WiredTiger (0) [1474529664:615802][1885:0x7fd02f352d40], file:Wired                                             Tiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
2016-09-22T15:34:24.615+0800 E STORAGE  [initandlisten] WiredTiger (-31804) [1474529664:615813][1885:0x7fd02f352d40], file:                                             WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2016-09-22T15:34:24.615+0800 I -        [initandlisten] Fatal Assertion 28558
2016-09-22T15:34:24.615+0800 I -        [initandlisten]
 
***aborting after fassert() failure
 
 
2016-09-22T15:34:24.635+0800 F -        [initandlisten] Got signal: 6 (Aborted).
 
 0x1337cb2 0x1336be9 0x13373f2 0x393e60f7e0 0x393e2325e5 0x393e233dc5 0x12bea52 0x10af8d3 0x1a9f62c 0x1a9faed 0x1a9fed4 0x1                                             9d18fc 0x19d1e73 0x19cee15 0x19d2eb9 0x19ef92c 0x1a257c0 0x1a9e357 0x1a9e899 0x1a9e9bb 0x1a33db8 0x1a9b515 0x1a660ef 0x1a66                                             1ee 0x1a223a1 0x109741f 0x1093893 0xfbad38 0x9970dd 0x99acbd 0x393e21ed1d 0x9937e9
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"F37CB2","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"F36BE9"},{"b":"400000","o                                             ":"F373F2"},{"b":"393E600000","o":"F7E0"},{"b":"393E200000","o":"325E5","s":"gsignal"},{"b":"393E200000","o":"33DC5","s":"a                                             bort"},{"b":"400000","o":"EBEA52","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"CAF8D3"},{"b":"400000","o":"169F62C"                                             ,"s":"__wt_eventv"},{"b":"400000","o":"169FAED","s":"__wt_err"},{"b":"400000","o":"169FED4","s":"__wt_panic"},{"b":"400000"                                             ,"o":"15D18FC","s":"__wt_block_extlist_read"},{"b":"400000","o":"15D1E73","s":"__wt_block_extlist_read_avail"},{"b":"400000                                             ","o":"15CEE15","s":"__wt_block_checkpoint_load"},{"b":"400000","o":"15D2EB9"},{"b":"400000","o":"15EF92C","s":"__wt_btree_                                             open"},{"b":"400000","o":"16257C0","s":"__wt_conn_btree_open"},{"b":"400000","o":"169E357","s":"__wt_session_get_btree"},{"                                             b":"400000","o":"169E899","s":"__wt_session_get_btree"},{"b":"400000","o":"169E9BB","s":"__wt_session_get_btree_ckpt"},{"b"                                             :"400000","o":"1633DB8","s":"__wt_curfile_open"},{"b":"400000","o":"169B515"},{"b":"400000","o":"16660EF","s":"__wt_metadat                                             a_cursor_open"},{"b":"400000","o":"16661EE","s":"__wt_metadata_cursor"},{"b":"400000","o":"16223A1","s":"wiredtiger_open"},                                             {"b":"400000","o":"C9741F","s":"_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb"},{"b":"400000","o":"C93893"},{"b":"400000",                                             "o":"BBAD38","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"5970DD","s":"_ZN5mo                                             ngo13initAndListenEi"},{"b":"400000","o":"59ACBD","s":"main"},{"b":"393E200000","o":"1ED1D","s":"__libc_start_main"},{"b":"                                             400000","o":"5937E9"}],"processInfo":{ "mongodbVersion" : "3.2.7", "gitVersion" : "4249c1d2b5999ebbf1fdf3bc0e0e3b3ff5c0aaf2                                             ", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "2.6.32-642.1.1.el6.x86_64", "version" : "#1 SMP Tu                                             e May 31 21:57:07 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "D6BB40ACB6CF                                             EAA43C3A651DDD8E547FCFE19166" }, { "b" : "7FFD6B8EE000", "elfType" : 3, "buildId" : "7F719AF91EE951B4FCB6647E7868F95F766A61                                             6B" }, { "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "7C5A504A21B221F299B1C45B9ED9C2340AEC6AEB" }, { "pa                                             th" : "/usr/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "3E83FD82B1AE56360B270189F636423FB3016594" }, { "path" : "/l                                             ib64/librt.so.1", "elfType" : 3, "buildId" : "86FE5BC1F46B8F8AA9A7A479FF991900DB93F720" }, { "path" : "/lib64/libdl.so.2",                                              "elfType" : 3, "buildId" : "045D39F19533291EBD72D0EE0247F9D49BE2521E" }, { "path" : "/lib64/libm.so.6", "elfType" : 3, "bui                                             ldId" : "B60EF3FCE5D4D2D8BAD2585D5CAAA1167B35DBFD" }, { "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "9A6E4B                                             DFA184364D81F7DFD789474C3FB8F98A00" }, { "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "B3BD4C596D72FCBE460                                             7C86FEEC14F47B46D0DCC" }, { "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "F14F3F5B55FC325FDFA7FB6F22F73CA1C69773                                             79" }, { "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "DB04BDF55FE66BADD1081D833C17B4881E0EEDB9" }, {                                              "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "9B852585C66329AA02EFB28497E652A40F538E78" }, { "path" :                                              "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "EF3AACAFD6BF71BB861F194C1559153FB0B020E2" }, { "path" : "/lib64/libcom_                                             err.so.2", "elfType" : 3, "buildId" : "152E2C18A7A2145021A8A879A01A82EE134E3946" }, { "path" : "/lib64/libk5crypto.so.3", "                                             elfType" : 3, "buildId" : "DDE6774979156442185836150FC0785170F8001F" }, { "path" : "/lib64/libz.so.1", "elfType" : 3, "buil                                             dId" : "5FA8E5038EC04A774AF72A9BB62DC86E1049C4D6" }, { "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "A                                             23DAFBCE170763BF1E836A8B26113F9CD20E0DA" }, { "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "AF374BAFB7F5B                                             139A0B431D3F06D82014AFF3251" }, { "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "DDF6449707FD4C49DDE32A293EE                                             E9AC218BFC460" }, { "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "B4576BE308DDCF7BC31F7304E4734C3D846D0236                                             " } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x1337cb2]
 mongod(+0xF36BE9) [0x1336be9]
 mongod(+0xF373F2) [0x13373f2]
 libpthread.so.0(+0xF7E0) [0x393e60f7e0]
 libc.so.6(gsignal+0x35) [0x393e2325e5]
 libc.so.6(abort+0x175) [0x393e233dc5]
 mongod(_ZN5mongo13fassertFailedEi+0x82) [0x12bea52]
 mongod(+0xCAF8D3) [0x10af8d3]
 mongod(__wt_eventv+0x42C) [0x1a9f62c]
 mongod(__wt_err+0x8D) [0x1a9faed]
 mongod(__wt_panic+0x24) [0x1a9fed4]
 mongod(__wt_block_extlist_read+0x6C) [0x19d18fc]
 mongod(__wt_block_extlist_read_avail+0x33) [0x19d1e73]
 mongod(__wt_block_checkpoint_load+0x3C5) [0x19cee15]
 mongod(+0x15D2EB9) [0x19d2eb9]
 mongod(__wt_btree_open+0xC7C) [0x19ef92c]
 mongod(__wt_conn_btree_open+0x140) [0x1a257c0]
 mongod(__wt_session_get_btree+0xE7) [0x1a9e357]
 mongod(__wt_session_get_btree+0x629) [0x1a9e899]
 mongod(__wt_session_get_btree_ckpt+0xAB) [0x1a9e9bb]
 mongod(__wt_curfile_open+0x218) [0x1a33db8]
 mongod(+0x169B515) [0x1a9b515]
 mongod(__wt_metadata_cursor_open+0x5F) [0x1a660ef]
 mongod(__wt_metadata_cursor+0x7E) [0x1a661ee]
 mongod(wiredtiger_open+0x1541) [0x1a223a1]
 mongod(_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb+0x46F) [0x109741f]
 mongod(+0xC93893) [0x1093893]
 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x598) [0xfbad38]
 mongod(_ZN5mongo13initAndListenEi+0x3DD) [0x9970dd]
 mongod(main+0x15D) [0x99acbd]
 libc.so.6(__libc_start_main+0xFD) [0x393e21ed1d]
 mongod(+0x5937E9) [0x9937e9]
-----  END BACKTRACE  -----
Aborted

Comment by Ramon Fernandez Marina [ 22/Sep/16 ]

Paddyzhu, after further internal discussion we'd like to examine your database files to make sure you haven't found a new bug. Is it possible for you to upload the contents of /ick/data2 via this private, secure upload portal?

Also, can you please upload the full log file for this node? I want to see if there are other clues before the failed restart that can help us determine what happened here.

Thanks,
Ramón.

Comment by Ramon Fernandez Marina [ 22/Sep/16 ]

Paddyzhu, I've run a repair attempt at the files you uploaded and attached them to this ticket. Please extract them in /ick/data2 and try to start mongod again, but note that there's no guarantee that this will work – I'd strongly recommend you use a replica set and keep good backups. I'd also recommend you upgrade to 3.2.9 (the latest 3.2 patch release at the time of this writing) as there have been many fixes to MongoDB 3.2 since 3.2.3.

Thanks,
Ramón.

Generated at Thu Feb 08 04:11:31 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.