|
Good day Kelsey,
thanks for quick reply, it really meter.
I can provide additional info and files if needed, It`s really important for us.
service did not start even with new files provided in recovery attempt.., with this files even mongod with --recover flag end up with error..
with new files:
service restart attempt:
maxim@ColeteElastic:~$ sudo service mongod restart
|
maxim@ColeteElastic:~$ sudo service mongod status
|
● mongod.service - High-performance, schema-free document-oriented database
|
Loaded: loaded (/lib/systemd/system/mongod.service; enabled; vendor preset: enabled)
|
Active: failed (Result: exit-code) since Fri 2018-04-20 08:12:41 EEST; 4s ago
|
Docs: https://docs.mongodb.org/manual
|
Process: 1409 ExecStart=/usr/bin/mongod --config /etc/mongod.conf (code=exited, status=14)
|
Main PID: 1409 (code=exited, status=14)
|
|
Apr 20 08:12:41 ColeteElastic systemd[1]: Started High-performance, schema-free document-oriented database.
|
Apr 20 08:12:41 ColeteElastic systemd[1]: mongod.service: Main process exited, code=exited, status=14/n/a
|
Apr 20 08:12:41 ColeteElastic systemd[1]: mongod.service: Unit entered failed state.
|
Apr 20 08:12:41 ColeteElastic systemd[1]: mongod.service: Failed with result 'exit-code'.
|
log for restart attempt:
2018-04-20T08:20:23.620+0300 I CONTROL [main] ***** SERVER RESTARTED *****
2018-04-20T08:20:23.627+0300 I CONTROL [initandlisten] MongoDB starting : pid=1280 port=27017 dbpath=/var/lib/mongodb 64-bit host=ColeteElastic
2018-04-20T08:20:23.627+0300 I CONTROL [initandlisten] db version v3.4.7
2018-04-20T08:20:23.627+0300 I CONTROL [initandlisten] git version: cf38c1b8a0a8dca4a11737581beafef4fe120bcd
2018-04-20T08:20:23.627+0300 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2g 1 Mar 2016
2018-04-20T08:20:23.627+0300 I CONTROL [initandlisten] allocator: tcmalloc
2018-04-20T08:20:23.627+0300 I CONTROL [initandlisten] modules: none
2018-04-20T08:20:23.627+0300 I CONTROL [initandlisten] build environment:
2018-04-20T08:20:23.627+0300 I CONTROL [initandlisten] distmod: ubuntu1604
2018-04-20T08:20:23.627+0300 I CONTROL [initandlisten] distarch: x86_64
2018-04-20T08:20:23.627+0300 I CONTROL [initandlisten] target_arch: x86_64
2018-04-20T08:20:23.627+0300 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net:
{ bindIp: "0.0.0.0", port: 27017 }
, storage: { dbPath: "/var/lib/mongodb", journal:
{ enabled: true }
}, systemLog:
{ destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" }
}
2018-04-20T08:20:23.673+0300 I - [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2018-04-20T08:20:23.673+0300 I STORAGE [initandlisten]
2018-04-20T08:20:23.673+0300 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2018-04-20T08:20:23.673+0300 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
2018-04-20T08:20:23.673+0300 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=3475M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2018-04-20T08:20:23.683+0300 E STORAGE [initandlisten] WiredTiger error (0) [1524201623:683179][1280:0x7fbe3fa44d00], file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value
2018-04-20T08:20:23.683+0300 E STORAGE [initandlisten] WiredTiger error (-31804) [1524201623:683230][1280:0x7fbe3fa44d00], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2018-04-20T08:20:23.683+0300 I - [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
2018-04-20T08:20:23.683+0300 I - [initandlisten]
***aborting after fassert() failure
2018-04-20T08:20:23.711+0300 F - [initandlisten] Got signal: 6 (Aborted).
0x5649bc824ea1 0x5649bc8240b9 0x5649bc82459d 0x7fbe3e649390 0x7fbe3e2a3428 0x7fbe3e2a502a 0x5649bbad3733 0x5649bc538c26 0x5649bbadd9cc 0x5649bbaddbf1 0x5649bbadde49 0x5649bd1cbafc 0x5649bd1ca269 0x5649bd17ed03 0x5649bd20965b 0x5649bd209b9d 0x5649bd209e5c 0x5649bd18d4f1 0x5649bd1ffa08 0x5649bd1c99ae 0x5649bd1c9a8b 0x5649bd17b05d 0x5649bc51da6f 0x5649bc5162ec 0x5649bc409a00 0x5649bbabf95e 0x5649bbadf1f6 0x7fbe3e28e830 0x5649bbb3de09
----- BEGIN BACKTRACE -----
{"backtrace":[
{"b":"5649BB2B9000","o":"156BEA1","s":"_ZN5mongo15printStackTraceERSo"}
,
{"b":"5649BB2B9000","o":"156B0B9"}
,
{"b":"5649BB2B9000","o":"156B59D"}
,
{"b":"7FBE3E638000","o":"11390"}
,
{"b":"7FBE3E26E000","o":"35428","s":"gsignal"}
,
{"b":"7FBE3E26E000","o":"3702A","s":"abort"}
,
{"b":"5649BB2B9000","o":"81A733","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"}
,
{"b":"5649BB2B9000","o":"127FC26"}
,
{"b":"5649BB2B9000","o":"8249CC","s":"__wt_eventv"}
,
{"b":"5649BB2B9000","o":"824BF1","s":"__wt_err"}
,
{"b":"5649BB2B9000","o":"824E49","s":"__wt_panic"}
,
{"b":"5649BB2B9000","o":"1F12AFC","s":"__wt_turtle_read"}
,
{"b":"5649BB2B9000","o":"1F11269","s":"__wt_metadata_search"}
,
{"b":"5649BB2B9000","o":"1EC5D03","s":"__wt_conn_btree_open"}
,
{"b":"5649BB2B9000","o":"1F5065B","s":"__wt_session_get_btree"}
,
{"b":"5649BB2B9000","o":"1F50B9D","s":"__wt_session_get_btree"}
,
{"b":"5649BB2B9000","o":"1F50E5C","s":"__wt_session_get_btree_ckpt"}
,
{"b":"5649BB2B9000","o":"1ED44F1","s":"__wt_curfile_open"}
,
{"b":"5649BB2B9000","o":"1F46A08"}
,
{"b":"5649BB2B9000","o":"1F109AE","s":"__wt_metadata_cursor_open"}
,
{"b":"5649BB2B9000","o":"1F10A8B","s":"__wt_metadata_cursor"}
,
{"b":"5649BB2B9000","o":"1EC205D","s":"wiredtiger_open"}
,
{"b":"5649BB2B9000","o":"1264A6F","s":"_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"}
,
{"b":"5649BB2B9000","o":"125D2EC"}
,
{"b":"5649BB2B9000","o":"1150A00","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"}
,
{"b":"5649BB2B9000","o":"80695E"}
,
{"b":"5649BB2B9000","o":"8261F6","s":"main"}
,
{"b":"7FBE3E26E000","o":"20830","s":"__libc_start_main"}
,
{"b":"5649BB2B9000","o":"884E09","s":"_start"}
],"processInfo":{ "mongodbVersion" : "3.4.7", "gitVersion" : "cf38c1b8a0a8dca4a11737581beafef4fe120bcd", "compiledModules" : [], "uname" :
{ "sysname" : "Linux", "release" : "4.4.0-92-generic", "version" : "#115-Ubuntu SMP Thu Aug 10 09:04:33 UTC 2017", "machine" : "x86_64" }
, "somap" : [
{ "b" : "5649BB2B9000", "elfType" : 3, "buildId" : "7E08C88DF63C9DEE479A2B1C6C7E11D8651F1184" }
,
{ "b" : "7FFC78F9B000", "elfType" : 3, "buildId" : "F23E2C79BCC8E97B12E1BB62A1BF196F8423FC40" }
,
{ "b" : "7FBE3F5C4000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "675F454AD6FD0B6CA2E41127C7B98079DA37F7B6" }
,
{ "b" : "7FBE3F180000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "2DA08A7E5BF610030DD33B70DB951399626B7496" }
,
{ "b" : "7FBE3EF78000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "F951C1E0765FCAE48F82CAFE35D1ADD36D6C9AF9" }
,
{ "b" : "7FBE3ED74000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "0FC788F0861846257B5F1773FBD438E95DFC1032" }
,
{ "b" : "7FBE3EA6B000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "FF7A33D389E756CA381A8189291A968EA5E1F4F8" }
,
{ "b" : "7FBE3E855000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }
,
{ "b" : "7FBE3E638000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "27F189EF8DB8C3734C6A678E6EF3CB0B206D58B2" }
,
{ "b" : "7FBE3E26E000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "088A6E00A1814622219F346B41E775B8DD46C518" }
,
{ "b" : "7FBE3F82D000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9157F205547F0EB588E2AB1F2F120B74253A43EA" }
] }}
mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x5649bc824ea1]
mongod(+0x156B0B9) [0x5649bc8240b9]
mongod(+0x156B59D) [0x5649bc82459d]
libpthread.so.0(+0x11390) [0x7fbe3e649390]
libc.so.6(gsignal+0x38) [0x7fbe3e2a3428]
libc.so.6(abort+0x16A) [0x7fbe3e2a502a]
mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x5649bbad3733]
mongod(+0x127FC26) [0x5649bc538c26]
mongod(__wt_eventv+0x3D7) [0x5649bbadd9cc]
mongod(__wt_err+0x9D) [0x5649bbaddbf1]
mongod(__wt_panic+0x24) [0x5649bbadde49]
mongod(__wt_turtle_read+0x26C) [0x5649bd1cbafc]
mongod(__wt_metadata_search+0x99) [0x5649bd1ca269]
mongod(__wt_conn_btree_open+0x73) [0x5649bd17ed03]
mongod(__wt_session_get_btree+0xFB) [0x5649bd20965b]
mongod(__wt_session_get_btree+0x63D) [0x5649bd209b9d]
mongod(__wt_session_get_btree_ckpt+0x14C) [0x5649bd209e5c]
mongod(__wt_curfile_open+0x161) [0x5649bd18d4f1]
mongod(+0x1F46A08) [0x5649bd1ffa08]
mongod(__wt_metadata_cursor_open+0x6E) [0x5649bd1c99ae]
mongod(__wt_metadata_cursor+0x4B) [0x5649bd1c9a8b]
mongod(wiredtiger_open+0x164D) [0x5649bd17b05d]
mongod(ZN5mongo18WiredTigerKVEngineC2ERKNSt7_cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0x70F) [0x5649bc51da6f]
mongod(+0x125D2EC) [0x5649bc5162ec]
mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x6B0) [0x5649bc409a00]
mongod(+0x80695E) [0x5649bbabf95e]
mongod(main+0x966) [0x5649bbadf1f6]
libc.so.6(__libc_start_main+0xF0) [0x7fbe3e28e830]
mongod(_start+0x29) [0x5649bbb3de09]
----- END BACKTRACE -----
|
recovery attempt:
|
maxim@ColeteElastic:~$ sudo mongod --recover --dbpath /var/lib/mongodb
|
Error parsing command line: unrecognised option '--recover'
|
try 'mongod --help' for more information
|
maxim@ColeteElastic:~$ sudo mongod --repair --dbpath /var/lib/mongodb
|
2018-04-20T08:23:42.138+0300 I CONTROL [initandlisten] MongoDB starting : pid=1325 port=27017 dbpath=/var/lib/mongodb 64-bit host=ColeteElastic
|
2018-04-20T08:23:42.138+0300 I CONTROL [initandlisten] db version v3.4.7
|
2018-04-20T08:23:42.138+0300 I CONTROL [initandlisten] git version: cf38c1b8a0a8dca4a11737581beafef4fe120bcd
|
2018-04-20T08:23:42.138+0300 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2g 1 Mar 2016
|
2018-04-20T08:23:42.138+0300 I CONTROL [initandlisten] allocator: tcmalloc
|
2018-04-20T08:23:42.138+0300 I CONTROL [initandlisten] modules: none
|
2018-04-20T08:23:42.139+0300 I CONTROL [initandlisten] build environment:
|
2018-04-20T08:23:42.139+0300 I CONTROL [initandlisten] distmod: ubuntu1604
|
2018-04-20T08:23:42.139+0300 I CONTROL [initandlisten] distarch: x86_64
|
2018-04-20T08:23:42.139+0300 I CONTROL [initandlisten] target_arch: x86_64
|
2018-04-20T08:23:42.139+0300 I CONTROL [initandlisten] options: { repair: true, storage: { dbPath: "/var/lib/mongodb" } }
|
2018-04-20T08:23:42.190+0300 I - [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
|
2018-04-20T08:23:42.190+0300 I STORAGE [initandlisten]
|
2018-04-20T08:23:42.190+0300 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
|
2018-04-20T08:23:42.190+0300 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
|
2018-04-20T08:23:42.190+0300 I STORAGE [initandlisten] Detected WT journal files. Running recovery from last checkpoint.
|
2018-04-20T08:23:42.190+0300 I STORAGE [initandlisten] journal to nojournal transition config: create,cache_size=3475M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
|
2018-04-20T08:23:42.201+0300 E STORAGE [initandlisten] WiredTiger error (0) [1524201822:201258][1325:0x7f6da9aa1d00], file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value
|
2018-04-20T08:23:42.201+0300 E STORAGE [initandlisten] WiredTiger error (-31804) [1524201822:201340][1325:0x7f6da9aa1d00], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
|
2018-04-20T08:23:42.201+0300 I - [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
|
2018-04-20T08:23:42.201+0300 I - [initandlisten]
|
|
***aborting after fassert() failure
|
|
|
2018-04-20T08:23:42.230+0300 F - [initandlisten] Got signal: 6 (Aborted).
|
|
0x55874a5a9ea1 0x55874a5a90b9 0x55874a5a959d 0x7f6da86a6390 0x7f6da8300428 0x7f6da830202a 0x558749858733 0x55874a2bdc26 0x5587498629cc 0x558749862bf1 0x558749862e49 0x55874af50afc 0x55874af4f269 0x55874af03d03 0x55874af8e65b 0x55874af8eb9d 0x55874af8ee5c 0x55874af124f1 0x55874af84a08 0x55874af4e9ae 0x55874af4ea8b 0x55874af0005d 0x55874a2a2e09 0x55874a29b2ec 0x55874a18ea00 0x55874984495e 0x5587498641f6 0x7f6da82eb830 0x5587498c2e09
|
----- BEGIN BACKTRACE -----
|
{"backtrace":[{"b":"55874903E000","o":"156BEA1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55874903E000","o":"156B0B9"},{"b":"55874903E000","o":"156B59D"},{"b":"7F6DA8695000","o":"11390"},{"b":"7F6DA82CB000","o":"35428","s":"gsignal"},{"b":"7F6DA82CB000","o":"3702A","s":"abort"},{"b":"55874903E000","o":"81A733","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"55874903E000","o":"127FC26"},{"b":"55874903E000","o":"8249CC","s":"__wt_eventv"},{"b":"55874903E000","o":"824BF1","s":"__wt_err"},{"b":"55874903E000","o":"824E49","s":"__wt_panic"},{"b":"55874903E000","o":"1F12AFC","s":"__wt_turtle_read"},{"b":"55874903E000","o":"1F11269","s":"__wt_metadata_search"},{"b":"55874903E000","o":"1EC5D03","s":"__wt_conn_btree_open"},{"b":"55874903E000","o":"1F5065B","s":"__wt_session_get_btree"},{"b":"55874903E000","o":"1F50B9D","s":"__wt_session_get_btree"},{"b":"55874903E000","o":"1F50E5C","s":"__wt_session_get_btree_ckpt"},{"b":"55874903E000","o":"1ED44F1","s":"__wt_curfile_open"},{"b":"55874903E000","o":"1F46A08"},{"b":"55874903E000","o":"1F109AE","s":"__wt_metadata_cursor_open"},{"b":"55874903E000","o":"1F10A8B","s":"__wt_metadata_cursor"},{"b":"55874903E000","o":"1EC205D","s":"wiredtiger_open"},{"b":"55874903E000","o":"1264E09","s":"_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"},{"b":"55874903E000","o":"125D2EC"},{"b":"55874903E000","o":"1150A00","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"55874903E000","o":"80695E"},{"b":"55874903E000","o":"8261F6","s":"main"},{"b":"7F6DA82CB000","o":"20830","s":"__libc_start_main"},{"b":"55874903E000","o":"884E09","s":"_start"}],"processInfo":{ "mongodbVersion" : "3.4.7", "gitVersion" : "cf38c1b8a0a8dca4a11737581beafef4fe120bcd", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-92-generic", "version" : "#115-Ubuntu SMP Thu Aug 10 09:04:33 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "b" : "55874903E000", "elfType" : 3, "buildId" : "7E08C88DF63C9DEE479A2B1C6C7E11D8651F1184" }, { "b" : "7FFDDA3FC000", "elfType" : 3, "buildId" : "F23E2C79BCC8E97B12E1BB62A1BF196F8423FC40" }, { "b" : "7F6DA9621000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "675F454AD6FD0B6CA2E41127C7B98079DA37F7B6" }, { "b" : "7F6DA91DD000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "2DA08A7E5BF610030DD33B70DB951399626B7496" }, { "b" : "7F6DA8FD5000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "F951C1E0765FCAE48F82CAFE35D1ADD36D6C9AF9" }, { "b" : "7F6DA8DD1000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "0FC788F0861846257B5F1773FBD438E95DFC1032" }, { "b" : "7F6DA8AC8000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "FF7A33D389E756CA381A8189291A968EA5E1F4F8" }, { "b" : "7F6DA88B2000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7F6DA8695000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "27F189EF8DB8C3734C6A678E6EF3CB0B206D58B2" }, { "b" : "7F6DA82CB000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "088A6E00A1814622219F346B41E775B8DD46C518" }, { "b" : "7F6DA988A000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9157F205547F0EB588E2AB1F2F120B74253A43EA" } ] }}
|
mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55874a5a9ea1]
|
mongod(+0x156B0B9) [0x55874a5a90b9]
|
mongod(+0x156B59D) [0x55874a5a959d]
|
libpthread.so.0(+0x11390) [0x7f6da86a6390]
|
libc.so.6(gsignal+0x38) [0x7f6da8300428]
|
libc.so.6(abort+0x16A) [0x7f6da830202a]
|
mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x558749858733]
|
mongod(+0x127FC26) [0x55874a2bdc26]
|
mongod(__wt_eventv+0x3D7) [0x5587498629cc]
|
mongod(__wt_err+0x9D) [0x558749862bf1]
|
mongod(__wt_panic+0x24) [0x558749862e49]
|
mongod(__wt_turtle_read+0x26C) [0x55874af50afc]
|
mongod(__wt_metadata_search+0x99) [0x55874af4f269]
|
mongod(__wt_conn_btree_open+0x73) [0x55874af03d03]
|
mongod(__wt_session_get_btree+0xFB) [0x55874af8e65b]
|
mongod(__wt_session_get_btree+0x63D) [0x55874af8eb9d]
|
mongod(__wt_session_get_btree_ckpt+0x14C) [0x55874af8ee5c]
|
mongod(__wt_curfile_open+0x161) [0x55874af124f1]
|
mongod(+0x1F46A08) [0x55874af84a08]
|
mongod(__wt_metadata_cursor_open+0x6E) [0x55874af4e9ae]
|
mongod(__wt_metadata_cursor+0x4B) [0x55874af4ea8b]
|
mongod(wiredtiger_open+0x164D) [0x55874af0005d]
|
mongod(_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0xAA9) [0x55874a2a2e09]
|
mongod(+0x125D2EC) [0x55874a29b2ec]
|
mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x6B0) [0x55874a18ea00]
|
mongod(+0x80695E) [0x55874984495e]
|
mongod(main+0x966) [0x5587498641f6]
|
libc.so.6(__libc_start_main+0xF0) [0x7f6da82eb830]
|
mongod(_start+0x29) [0x5587498c2e09]
|
----- END BACKTRACE -----
|
Aborted (core dumped)
|
logs after recovery attempt:
2018-04-20T08:20:23.620+0300 I CONTROL [main] ***** SERVER RESTARTED *****
|
2018-04-20T08:20:23.627+0300 I CONTROL [initandlisten] MongoDB starting : pid=1280 port=27017 dbpath=/var/lib/mongodb 64-bit host=ColeteElastic
|
2018-04-20T08:20:23.627+0300 I CONTROL [initandlisten] db version v3.4.7
|
2018-04-20T08:20:23.627+0300 I CONTROL [initandlisten] git version: cf38c1b8a0a8dca4a11737581beafef4fe120bcd
|
2018-04-20T08:20:23.627+0300 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2g 1 Mar 2016
|
2018-04-20T08:20:23.627+0300 I CONTROL [initandlisten] allocator: tcmalloc
|
2018-04-20T08:20:23.627+0300 I CONTROL [initandlisten] modules: none
|
2018-04-20T08:20:23.627+0300 I CONTROL [initandlisten] build environment:
|
2018-04-20T08:20:23.627+0300 I CONTROL [initandlisten] distmod: ubuntu1604
|
2018-04-20T08:20:23.627+0300 I CONTROL [initandlisten] distarch: x86_64
|
2018-04-20T08:20:23.627+0300 I CONTROL [initandlisten] target_arch: x86_64
|
2018-04-20T08:20:23.627+0300 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "0.0.0.0", port: 27017 }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
|
2018-04-20T08:20:23.673+0300 I - [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
|
2018-04-20T08:20:23.673+0300 I STORAGE [initandlisten]
|
2018-04-20T08:20:23.673+0300 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
|
2018-04-20T08:20:23.673+0300 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
|
2018-04-20T08:20:23.673+0300 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=3475M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
|
2018-04-20T08:20:23.683+0300 E STORAGE [initandlisten] WiredTiger error (0) [1524201623:683179][1280:0x7fbe3fa44d00], file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value
|
2018-04-20T08:20:23.683+0300 E STORAGE [initandlisten] WiredTiger error (-31804) [1524201623:683230][1280:0x7fbe3fa44d00], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
|
2018-04-20T08:20:23.683+0300 I - [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
|
2018-04-20T08:20:23.683+0300 I - [initandlisten]
|
|
***aborting after fassert() failure
|
|
|
2018-04-20T08:20:23.711+0300 F - [initandlisten] Got signal: 6 (Aborted).
|
|
0x5649bc824ea1 0x5649bc8240b9 0x5649bc82459d 0x7fbe3e649390 0x7fbe3e2a3428 0x7fbe3e2a502a 0x5649bbad3733 0x5649bc538c26 0x5649bbadd9cc 0x5649bbaddbf1 0x5649bbadde49 0x5649bd1cbafc 0x5649bd1ca269 0x5649bd17ed03 0x5649bd20965b 0x5649bd209b9d 0x5649bd209e5c 0x5649bd18d4f1 0x5649bd1ffa08 0x5649bd1c99ae 0x5649bd1c9a8b 0x5649bd17b05d 0x5649bc51da6f 0x5649bc5162ec 0x5649bc409a00 0x5649bbabf95e 0x5649bbadf1f6 0x7fbe3e28e830 0x5649bbb3de09
|
----- BEGIN BACKTRACE -----
|
{"backtrace":[{"b":"5649BB2B9000","o":"156BEA1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"5649BB2B9000","o":"156B0B9"},{"b":"5649BB2B9000","o":"156B59D"},{"b":"7FBE3E638000","o":"11390"},{"b":"7FBE3E26E000","o":"35428","s":"gsignal"},{"b":"7FBE3E26E000","o":"3702A","s":"abort"},{"b":"5649BB2B9000","o":"81A733","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"5649BB2B9000","o":"127FC26"},{"b":"5649BB2B9000","o":"8249CC","s":"__wt_eventv"},{"b":"5649BB2B9000","o":"824BF1","s":"__wt_err"},{"b":"5649BB2B9000","o":"824E49","s":"__wt_panic"},{"b":"5649BB2B9000","o":"1F12AFC","s":"__wt_turtle_read"},{"b":"5649BB2B9000","o":"1F11269","s":"__wt_metadata_search"},{"b":"5649BB2B9000","o":"1EC5D03","s":"__wt_conn_btree_open"},{"b":"5649BB2B9000","o":"1F5065B","s":"__wt_session_get_btree"},{"b":"5649BB2B9000","o":"1F50B9D","s":"__wt_session_get_btree"},{"b":"5649BB2B9000","o":"1F50E5C","s":"__wt_session_get_btree_ckpt"},{"b":"5649BB2B9000","o":"1ED44F1","s":"__wt_curfile_open"},{"b":"5649BB2B9000","o":"1F46A08"},{"b":"5649BB2B9000","o":"1F109AE","s":"__wt_metadata_cursor_open"},{"b":"5649BB2B9000","o":"1F10A8B","s":"__wt_metadata_cursor"},{"b":"5649BB2B9000","o":"1EC205D","s":"wiredtiger_open"},{"b":"5649BB2B9000","o":"1264A6F","s":"_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"},{"b":"5649BB2B9000","o":"125D2EC"},{"b":"5649BB2B9000","o":"1150A00","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"5649BB2B9000","o":"80695E"},{"b":"5649BB2B9000","o":"8261F6","s":"main"},{"b":"7FBE3E26E000","o":"20830","s":"__libc_start_main"},{"b":"5649BB2B9000","o":"884E09","s":"_start"}],"processInfo":{ "mongodbVersion" : "3.4.7", "gitVersion" : "cf38c1b8a0a8dca4a11737581beafef4fe120bcd", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-92-generic", "version" : "#115-Ubuntu SMP Thu Aug 10 09:04:33 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "b" : "5649BB2B9000", "elfType" : 3, "buildId" : "7E08C88DF63C9DEE479A2B1C6C7E11D8651F1184" }, { "b" : "7FFC78F9B000", "elfType" : 3, "buildId" : "F23E2C79BCC8E97B12E1BB62A1BF196F8423FC40" }, { "b" : "7FBE3F5C4000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "675F454AD6FD0B6CA2E41127C7B98079DA37F7B6" }, { "b" : "7FBE3F180000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "2DA08A7E5BF610030DD33B70DB951399626B7496" }, { "b" : "7FBE3EF78000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "F951C1E0765FCAE48F82CAFE35D1ADD36D6C9AF9" }, { "b" : "7FBE3ED74000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "0FC788F0861846257B5F1773FBD438E95DFC1032" }, { "b" : "7FBE3EA6B000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "FF7A33D389E756CA381A8189291A968EA5E1F4F8" }, { "b" : "7FBE3E855000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7FBE3E638000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "27F189EF8DB8C3734C6A678E6EF3CB0B206D58B2" }, { "b" : "7FBE3E26E000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "088A6E00A1814622219F346B41E775B8DD46C518" }, { "b" : "7FBE3F82D000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9157F205547F0EB588E2AB1F2F120B74253A43EA" } ] }}
|
mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x5649bc824ea1]
|
mongod(+0x156B0B9) [0x5649bc8240b9]
|
mongod(+0x156B59D) [0x5649bc82459d]
|
libpthread.so.0(+0x11390) [0x7fbe3e649390]
|
libc.so.6(gsignal+0x38) [0x7fbe3e2a3428]
|
libc.so.6(abort+0x16A) [0x7fbe3e2a502a]
|
mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x5649bbad3733]
|
mongod(+0x127FC26) [0x5649bc538c26]
|
mongod(__wt_eventv+0x3D7) [0x5649bbadd9cc]
|
mongod(__wt_err+0x9D) [0x5649bbaddbf1]
|
mongod(__wt_panic+0x24) [0x5649bbadde49]
|
mongod(__wt_turtle_read+0x26C) [0x5649bd1cbafc]
|
mongod(__wt_metadata_search+0x99) [0x5649bd1ca269]
|
mongod(__wt_conn_btree_open+0x73) [0x5649bd17ed03]
|
mongod(__wt_session_get_btree+0xFB) [0x5649bd20965b]
|
mongod(__wt_session_get_btree+0x63D) [0x5649bd209b9d]
|
mongod(__wt_session_get_btree_ckpt+0x14C) [0x5649bd209e5c]
|
mongod(__wt_curfile_open+0x161) [0x5649bd18d4f1]
|
mongod(+0x1F46A08) [0x5649bd1ffa08]
|
mongod(__wt_metadata_cursor_open+0x6E) [0x5649bd1c99ae]
|
mongod(__wt_metadata_cursor+0x4B) [0x5649bd1c9a8b]
|
mongod(wiredtiger_open+0x164D) [0x5649bd17b05d]
|
mongod(_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0x70F) [0x5649bc51da6f]
|
mongod(+0x125D2EC) [0x5649bc5162ec]
|
mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x6B0) [0x5649bc409a00]
|
mongod(+0x80695E) [0x5649bbabf95e]
|
mongod(main+0x966) [0x5649bbadf1f6]
|
libc.so.6(__libc_start_main+0xF0) [0x7fbe3e28e830]
|
mongod(_start+0x29) [0x5649bbb3de09]
|
----- END BACKTRACE -----
|
|