[SERVER-34560] file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value Created: 19/Apr/18  Updated: 27/Jul/18  Resolved: 23/Apr/18

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.4.7
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Maxim [X] Assignee: Kelsey Schubert
Resolution: Done Votes: 0
Labels: envns, rge, rpu, trct, wtc
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: HTML File WiredTiger     File WiredTiger.lock     File WiredTiger.turtle     File WiredTiger.wt     File WiredTigerLAS.wt     File repair_attempt.tar.gz    
Operating System: Linux
Participants:

 Description   

service mongod unexpectedly changed state from "active" to "failed". repair tool did not help.

log file output:

2018-04-19T05:24:30.688+0300 I CONTROL  [main] ***** SERVER RESTARTED *****
2018-04-19T05:24:30.696+0300 I CONTROL  [initandlisten] MongoDB starting : pid=2648 port=27017 dbpath=/var/lib/mongodb 64-bit host=ColeteElastic
2018-04-19T05:24:30.697+0300 I CONTROL  [initandlisten] db version v3.4.7
2018-04-19T05:24:30.697+0300 I CONTROL  [initandlisten] git version: cf38c1b8a0a8dca4a11737581beafef4fe120bcd
2018-04-19T05:24:30.697+0300 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
2018-04-19T05:24:30.697+0300 I CONTROL  [initandlisten] allocator: tcmalloc
2018-04-19T05:24:30.697+0300 I CONTROL  [initandlisten] modules: none
2018-04-19T05:24:30.697+0300 I CONTROL  [initandlisten] build environment:
2018-04-19T05:24:30.697+0300 I CONTROL  [initandlisten]     distmod: ubuntu1604
2018-04-19T05:24:30.697+0300 I CONTROL  [initandlisten]     distarch: x86_64
2018-04-19T05:24:30.697+0300 I CONTROL  [initandlisten]     target_arch: x86_64
2018-04-19T05:24:30.697+0300 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "0.0.0.0", port: 27017 }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2018-04-19T05:24:30.697+0300 W -        [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty.
2018-04-19T05:24:30.751+0300 I -        [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2018-04-19T05:24:30.751+0300 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2018-04-19T05:24:30.751+0300 I STORAGE  [initandlisten] 
2018-04-19T05:24:30.751+0300 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2018-04-19T05:24:30.751+0300 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2018-04-19T05:24:30.751+0300 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=3475M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2018-04-19T05:24:30.762+0300 E STORAGE  [initandlisten] WiredTiger error (0) [1524104670:761987][2648:0x7f15a3c44d00], file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value
2018-04-19T05:24:30.762+0300 E STORAGE  [initandlisten] WiredTiger error (-31804) [1524104670:762051][2648:0x7f15a3c44d00], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2018-04-19T05:24:30.762+0300 I -        [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
2018-04-19T05:24:30.762+0300 I -        [initandlisten] 
 
***aborting after fassert() failure
 
 
2018-04-19T05:24:30.794+0300 F -        [initandlisten] Got signal: 6 (Aborted).
 
 0x556257693ea1 0x5562576930b9 0x55625769359d 0x7f15a2849390 0x7f15a24a3428 0x7f15a24a502a 0x556256942733 0x5562573a7c26 0x55625694c9cc 0x55625694cbf1 0x55625694ce49 0x55625803aafc 0x556258039269 0x556257fedd03 0x55625807865b 0x556258078b9d 0x556258078e5c 0x556257ffc4f1 0x55625806ea08 0x5562580389ae 0x556258038a8b 0x556257fea05d 0x55625738ca6f 0x5562573852ec 0x556257278a00 0x55625692e95e 0x55625694e1f6 0x7f15a248e830 0x5562569ace09
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"556256128000","o":"156BEA1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"556256128000","o":"156B0B9"},{"b":"556256128000","o":"156B59D"},{"b":"7F15A2838000","o":"11390"},{"b":"7F15A246E000","o":"35428","s":"gsignal"},{"b":"7F15A246E000","o":"3702A","s":"abort"},{"b":"556256128000","o":"81A733","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"556256128000","o":"127FC26"},{"b":"556256128000","o":"8249CC","s":"__wt_eventv"},{"b":"556256128000","o":"824BF1","s":"__wt_err"},{"b":"556256128000","o":"824E49","s":"__wt_panic"},{"b":"556256128000","o":"1F12AFC","s":"__wt_turtle_read"},{"b":"556256128000","o":"1F11269","s":"__wt_metadata_search"},{"b":"556256128000","o":"1EC5D03","s":"__wt_conn_btree_open"},{"b":"556256128000","o":"1F5065B","s":"__wt_session_get_btree"},{"b":"556256128000","o":"1F50B9D","s":"__wt_session_get_btree"},{"b":"556256128000","o":"1F50E5C","s":"__wt_session_get_btree_ckpt"},{"b":"556256128000","o":"1ED44F1","s":"__wt_curfile_open"},{"b":"556256128000","o":"1F46A08"},{"b":"556256128000","o":"1F109AE","s":"__wt_metadata_cursor_open"},{"b":"556256128000","o":"1F10A8B","s":"__wt_metadata_cursor"},{"b":"556256128000","o":"1EC205D","s":"wiredtiger_open"},{"b":"556256128000","o":"1264A6F","s":"_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"},{"b":"556256128000","o":"125D2EC"},{"b":"556256128000","o":"1150A00","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"556256128000","o":"80695E"},{"b":"556256128000","o":"8261F6","s":"main"},{"b":"7F15A246E000","o":"20830","s":"__libc_start_main"},{"b":"556256128000","o":"884E09","s":"_start"}],"processInfo":{ "mongodbVersion" : "3.4.7", "gitVersion" : "cf38c1b8a0a8dca4a11737581beafef4fe120bcd", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-92-generic", "version" : "#115-Ubuntu SMP Thu Aug 10 09:04:33 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "b" : "556256128000", "elfType" : 3, "buildId" : "7E08C88DF63C9DEE479A2B1C6C7E11D8651F1184" }, { "b" : "7FFD781FB000", "elfType" : 3, "buildId" : "F23E2C79BCC8E97B12E1BB62A1BF196F8423FC40" }, { "b" : "7F15A37C4000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "675F454AD6FD0B6CA2E41127C7B98079DA37F7B6" }, { "b" : "7F15A3380000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "2DA08A7E5BF610030DD33B70DB951399626B7496" }, { "b" : "7F15A3178000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "F951C1E0765FCAE48F82CAFE35D1ADD36D6C9AF9" }, { "b" : "7F15A2F74000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "0FC788F0861846257B5F1773FBD438E95DFC1032" }, { "b" : "7F15A2C6B000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "FF7A33D389E756CA381A8189291A968EA5E1F4F8" }, { "b" : "7F15A2A55000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7F15A2838000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "27F189EF8DB8C3734C6A678E6EF3CB0B206D58B2" }, { "b" : "7F15A246E000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "088A6E00A1814622219F346B41E775B8DD46C518" }, { "b" : "7F15A3A2D000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9157F205547F0EB588E2AB1F2F120B74253A43EA" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x556257693ea1]
 mongod(+0x156B0B9) [0x5562576930b9]
 mongod(+0x156B59D) [0x55625769359d]
 libpthread.so.0(+0x11390) [0x7f15a2849390]
 libc.so.6(gsignal+0x38) [0x7f15a24a3428]
 libc.so.6(abort+0x16A) [0x7f15a24a502a]
 mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x556256942733]
 mongod(+0x127FC26) [0x5562573a7c26]
 mongod(__wt_eventv+0x3D7) [0x55625694c9cc]
 mongod(__wt_err+0x9D) [0x55625694cbf1]
 mongod(__wt_panic+0x24) [0x55625694ce49]
 mongod(__wt_turtle_read+0x26C) [0x55625803aafc]
 mongod(__wt_metadata_search+0x99) [0x556258039269]
 mongod(__wt_conn_btree_open+0x73) [0x556257fedd03]
 mongod(__wt_session_get_btree+0xFB) [0x55625807865b]
 mongod(__wt_session_get_btree+0x63D) [0x556258078b9d]
 mongod(__wt_session_get_btree_ckpt+0x14C) [0x556258078e5c]
 mongod(__wt_curfile_open+0x161) [0x556257ffc4f1]
 mongod(+0x1F46A08) [0x55625806ea08]
 mongod(__wt_metadata_cursor_open+0x6E) [0x5562580389ae]
 mongod(__wt_metadata_cursor+0x4B) [0x556258038a8b]
 mongod(wiredtiger_open+0x164D) [0x556257fea05d]
 mongod(_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0x70F) [0x55625738ca6f]
 mongod(+0x125D2EC) [0x5562573852ec]
 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x6B0) [0x556257278a00]
 mongod(+0x80695E) [0x55625692e95e]
 mongod(main+0x966) [0x55625694e1f6]
 libc.so.6(__libc_start_main+0xF0) [0x7f15a248e830]
 mongod(_start+0x29) [0x5562569ace09]
-----  END BACKTRACE  -----

(sudo mongod --repair --dbpath /var/log/mongodb/ --storageEngine wiredTiger)
--reapir output:

2018-04-19T05:48:43.620+0300 I CONTROL  [initandlisten] MongoDB starting : pid=2675 port=27017 dbpath=/var/log/mongodb/ 64-bit host=ColeteElastic
2018-04-19T05:48:43.620+0300 I CONTROL  [initandlisten] db version v3.4.7
2018-04-19T05:48:43.620+0300 I CONTROL  [initandlisten] git version: cf38c1b8a0a8dca4a11737581beafef4fe120bcd
2018-04-19T05:48:43.620+0300 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
2018-04-19T05:48:43.620+0300 I CONTROL  [initandlisten] allocator: tcmalloc
2018-04-19T05:48:43.621+0300 I CONTROL  [initandlisten] modules: none
2018-04-19T05:48:43.621+0300 I CONTROL  [initandlisten] build environment:
2018-04-19T05:48:43.621+0300 I CONTROL  [initandlisten]     distmod: ubuntu1604
2018-04-19T05:48:43.621+0300 I CONTROL  [initandlisten]     distarch: x86_64
2018-04-19T05:48:43.621+0300 I CONTROL  [initandlisten]     target_arch: x86_64
2018-04-19T05:48:43.621+0300 I CONTROL  [initandlisten] options: { repair: true, storage: { dbPath: "/var/log/mongodb/", engine: "wiredTiger" } }
2018-04-19T05:48:43.674+0300 I STORAGE  [initandlisten] 
2018-04-19T05:48:43.674+0300 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2018-04-19T05:48:43.674+0300 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2018-04-19T05:48:43.674+0300 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=3475M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),,log=(enabled=false),
2018-04-19T05:48:43.748+0300 I STORAGE  [initandlisten] Repairing size cache
2018-04-19T05:48:43.749+0300 I STORAGE  [initandlisten] Verify succeeded on uri table:sizeStorer. Not salvaging.
2018-04-19T05:48:43.749+0300 I STORAGE  [initandlisten] Repairing catalog metadata
2018-04-19T05:48:43.750+0300 I STORAGE  [initandlisten] Verify succeeded on uri table:_mdb_catalog. Not salvaging.
2018-04-19T05:48:43.765+0300 I CONTROL  [initandlisten] 
2018-04-19T05:48:43.765+0300 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2018-04-19T05:48:43.765+0300 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2018-04-19T05:48:43.765+0300 I CONTROL  [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
2018-04-19T05:48:43.765+0300 I CONTROL  [initandlisten] 
2018-04-19T05:48:43.766+0300 I CONTROL  [initandlisten] 
2018-04-19T05:48:43.766+0300 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2018-04-19T05:48:43.766+0300 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2018-04-19T05:48:43.766+0300 I CONTROL  [initandlisten] 
2018-04-19T05:48:43.766+0300 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2018-04-19T05:48:43.766+0300 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2018-04-19T05:48:43.766+0300 I CONTROL  [initandlisten] 
2018-04-19T05:48:43.766+0300 I STORAGE  [initandlisten] finished checking dbs
2018-04-19T05:48:43.766+0300 I NETWORK  [initandlisten] shutdown: going to close listening sockets...
2018-04-19T05:48:43.766+0300 I NETWORK  [initandlisten] removing socket file: /tmp/mongodb-27017.sock
2018-04-19T05:48:43.766+0300 I NETWORK  [initandlisten] shutdown: going to flush diaglog...
2018-04-19T05:48:43.766+0300 I STORAGE  [initandlisten] WiredTigerKVEngine shutting down
2018-04-19T05:48:43.793+0300 I STORAGE  [initandlisten] shutdown: removing fs lock...
2018-04-19T05:48:43.793+0300 I CONTROL  [initandlisten] now exiting
2018-04-19T05:48:43.793+0300 I CONTROL  [initandlisten] shutting down with code:0

more info:
What kind of underlying storage mechanism are you using? Are the storage devices attached locally or over the network? Are the disks SSDs or HDDs? What kind of RAID and/or volume management system are you using?

  • storage device attached over network, HDD, RAID (NetApp native)

Would you please check the integrity of your disks?

  • yes

Has the database always been running this version of MongoDB? If not please describe the upgrade/downgrade cycles the database has been through.
-yes, always this version of mongo

Have you manipulated (copied or moved) the underlying database files? If so, was mongod running?

  • no

Have you ever restored this instance from backups?

  • no

What method do you use to create backups?

  • there was no back-ups at the moment.

When was the underlying filesystem last checked and is it currently marked clean?

  • never


 Comments   
Comment by Kelsey Schubert [ 23/Apr/18 ]

Hi Durbala,

Unfortunately, manual inspection indicates that the WiredTiger.turtle file has been zeroed. In this case, it is not possible successfully restart the mongod with these files and our best recommendation would be to resync the affected node or restore from a backup if possible.

To prevent this type of problem in the future please take note of the following guidelines to help mitigate any issues related to unreliable storage layers or server failures.

Thank you,
Kelsey

Comment by Maxim [X] [ 20/Apr/18 ]

Good day Kelsey,
thanks for quick reply, it really meter.

I can provide additional info and files if needed, It`s really important for us.

service did not start even with new files provided in recovery attempt.., with this files even mongod with --recover flag end up with error..
with new files:
service restart attempt:

maxim@ColeteElastic:~$ sudo service mongod restart
maxim@ColeteElastic:~$ sudo service mongod status
● mongod.service - High-performance, schema-free document-oriented database
   Loaded: loaded (/lib/systemd/system/mongod.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Fri 2018-04-20 08:12:41 EEST; 4s ago
     Docs: https://docs.mongodb.org/manual
  Process: 1409 ExecStart=/usr/bin/mongod --config /etc/mongod.conf (code=exited, status=14)
 Main PID: 1409 (code=exited, status=14)
 
Apr 20 08:12:41 ColeteElastic systemd[1]: Started High-performance, schema-free document-oriented database.
Apr 20 08:12:41 ColeteElastic systemd[1]: mongod.service: Main process exited, code=exited, status=14/n/a
Apr 20 08:12:41 ColeteElastic systemd[1]: mongod.service: Unit entered failed state.
Apr 20 08:12:41 ColeteElastic systemd[1]: mongod.service: Failed with result 'exit-code'.

log for restart attempt:
2018-04-20T08:20:23.620+0300 I CONTROL [main] ***** SERVER RESTARTED *****
2018-04-20T08:20:23.627+0300 I CONTROL [initandlisten] MongoDB starting : pid=1280 port=27017 dbpath=/var/lib/mongodb 64-bit host=ColeteElastic
2018-04-20T08:20:23.627+0300 I CONTROL [initandlisten] db version v3.4.7
2018-04-20T08:20:23.627+0300 I CONTROL [initandlisten] git version: cf38c1b8a0a8dca4a11737581beafef4fe120bcd
2018-04-20T08:20:23.627+0300 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2g 1 Mar 2016
2018-04-20T08:20:23.627+0300 I CONTROL [initandlisten] allocator: tcmalloc
2018-04-20T08:20:23.627+0300 I CONTROL [initandlisten] modules: none
2018-04-20T08:20:23.627+0300 I CONTROL [initandlisten] build environment:
2018-04-20T08:20:23.627+0300 I CONTROL [initandlisten] distmod: ubuntu1604
2018-04-20T08:20:23.627+0300 I CONTROL [initandlisten] distarch: x86_64
2018-04-20T08:20:23.627+0300 I CONTROL [initandlisten] target_arch: x86_64
2018-04-20T08:20:23.627+0300 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net:

{ bindIp: "0.0.0.0", port: 27017 }

, storage: { dbPath: "/var/lib/mongodb", journal:

{ enabled: true }

}, systemLog:

{ destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" }

}
2018-04-20T08:20:23.673+0300 I - [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2018-04-20T08:20:23.673+0300 I STORAGE [initandlisten]
2018-04-20T08:20:23.673+0300 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2018-04-20T08:20:23.673+0300 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
2018-04-20T08:20:23.673+0300 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=3475M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2018-04-20T08:20:23.683+0300 E STORAGE [initandlisten] WiredTiger error (0) [1524201623:683179][1280:0x7fbe3fa44d00], file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value
2018-04-20T08:20:23.683+0300 E STORAGE [initandlisten] WiredTiger error (-31804) [1524201623:683230][1280:0x7fbe3fa44d00], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2018-04-20T08:20:23.683+0300 I - [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
2018-04-20T08:20:23.683+0300 I - [initandlisten]

***aborting after fassert() failure

2018-04-20T08:20:23.711+0300 F - [initandlisten] Got signal: 6 (Aborted).

0x5649bc824ea1 0x5649bc8240b9 0x5649bc82459d 0x7fbe3e649390 0x7fbe3e2a3428 0x7fbe3e2a502a 0x5649bbad3733 0x5649bc538c26 0x5649bbadd9cc 0x5649bbaddbf1 0x5649bbadde49 0x5649bd1cbafc 0x5649bd1ca269 0x5649bd17ed03 0x5649bd20965b 0x5649bd209b9d 0x5649bd209e5c 0x5649bd18d4f1 0x5649bd1ffa08 0x5649bd1c99ae 0x5649bd1c9a8b 0x5649bd17b05d 0x5649bc51da6f 0x5649bc5162ec 0x5649bc409a00 0x5649bbabf95e 0x5649bbadf1f6 0x7fbe3e28e830 0x5649bbb3de09
----- BEGIN BACKTRACE -----
{"backtrace":[

{"b":"5649BB2B9000","o":"156BEA1","s":"_ZN5mongo15printStackTraceERSo"}

,

{"b":"5649BB2B9000","o":"156B0B9"}

,

{"b":"5649BB2B9000","o":"156B59D"}

,

{"b":"7FBE3E638000","o":"11390"}

,

{"b":"7FBE3E26E000","o":"35428","s":"gsignal"}

,

{"b":"7FBE3E26E000","o":"3702A","s":"abort"}

,

{"b":"5649BB2B9000","o":"81A733","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"}

,

{"b":"5649BB2B9000","o":"127FC26"}

,

{"b":"5649BB2B9000","o":"8249CC","s":"__wt_eventv"}

,

{"b":"5649BB2B9000","o":"824BF1","s":"__wt_err"}

,

{"b":"5649BB2B9000","o":"824E49","s":"__wt_panic"}

,

{"b":"5649BB2B9000","o":"1F12AFC","s":"__wt_turtle_read"}

,

{"b":"5649BB2B9000","o":"1F11269","s":"__wt_metadata_search"}

,

{"b":"5649BB2B9000","o":"1EC5D03","s":"__wt_conn_btree_open"}

,

{"b":"5649BB2B9000","o":"1F5065B","s":"__wt_session_get_btree"}

,

{"b":"5649BB2B9000","o":"1F50B9D","s":"__wt_session_get_btree"}

,

{"b":"5649BB2B9000","o":"1F50E5C","s":"__wt_session_get_btree_ckpt"}

,

{"b":"5649BB2B9000","o":"1ED44F1","s":"__wt_curfile_open"}

,

{"b":"5649BB2B9000","o":"1F46A08"}

,

{"b":"5649BB2B9000","o":"1F109AE","s":"__wt_metadata_cursor_open"}

,

{"b":"5649BB2B9000","o":"1F10A8B","s":"__wt_metadata_cursor"}

,

{"b":"5649BB2B9000","o":"1EC205D","s":"wiredtiger_open"}

,

{"b":"5649BB2B9000","o":"1264A6F","s":"_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"}

,

{"b":"5649BB2B9000","o":"125D2EC"}

,

{"b":"5649BB2B9000","o":"1150A00","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"}

,

{"b":"5649BB2B9000","o":"80695E"}

,

{"b":"5649BB2B9000","o":"8261F6","s":"main"}

,

{"b":"7FBE3E26E000","o":"20830","s":"__libc_start_main"}

,

{"b":"5649BB2B9000","o":"884E09","s":"_start"}

],"processInfo":{ "mongodbVersion" : "3.4.7", "gitVersion" : "cf38c1b8a0a8dca4a11737581beafef4fe120bcd", "compiledModules" : [], "uname" :

{ "sysname" : "Linux", "release" : "4.4.0-92-generic", "version" : "#115-Ubuntu SMP Thu Aug 10 09:04:33 UTC 2017", "machine" : "x86_64" }

, "somap" : [

{ "b" : "5649BB2B9000", "elfType" : 3, "buildId" : "7E08C88DF63C9DEE479A2B1C6C7E11D8651F1184" }

,

{ "b" : "7FFC78F9B000", "elfType" : 3, "buildId" : "F23E2C79BCC8E97B12E1BB62A1BF196F8423FC40" }

,

{ "b" : "7FBE3F5C4000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "675F454AD6FD0B6CA2E41127C7B98079DA37F7B6" }

,

{ "b" : "7FBE3F180000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "2DA08A7E5BF610030DD33B70DB951399626B7496" }

,

{ "b" : "7FBE3EF78000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "F951C1E0765FCAE48F82CAFE35D1ADD36D6C9AF9" }

,

{ "b" : "7FBE3ED74000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "0FC788F0861846257B5F1773FBD438E95DFC1032" }

,

{ "b" : "7FBE3EA6B000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "FF7A33D389E756CA381A8189291A968EA5E1F4F8" }

,

{ "b" : "7FBE3E855000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }

,

{ "b" : "7FBE3E638000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "27F189EF8DB8C3734C6A678E6EF3CB0B206D58B2" }

,

{ "b" : "7FBE3E26E000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "088A6E00A1814622219F346B41E775B8DD46C518" }

,

{ "b" : "7FBE3F82D000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9157F205547F0EB588E2AB1F2F120B74253A43EA" }

] }}
mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x5649bc824ea1]
mongod(+0x156B0B9) [0x5649bc8240b9]
mongod(+0x156B59D) [0x5649bc82459d]
libpthread.so.0(+0x11390) [0x7fbe3e649390]
libc.so.6(gsignal+0x38) [0x7fbe3e2a3428]
libc.so.6(abort+0x16A) [0x7fbe3e2a502a]
mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x5649bbad3733]
mongod(+0x127FC26) [0x5649bc538c26]
mongod(__wt_eventv+0x3D7) [0x5649bbadd9cc]
mongod(__wt_err+0x9D) [0x5649bbaddbf1]
mongod(__wt_panic+0x24) [0x5649bbadde49]
mongod(__wt_turtle_read+0x26C) [0x5649bd1cbafc]
mongod(__wt_metadata_search+0x99) [0x5649bd1ca269]
mongod(__wt_conn_btree_open+0x73) [0x5649bd17ed03]
mongod(__wt_session_get_btree+0xFB) [0x5649bd20965b]
mongod(__wt_session_get_btree+0x63D) [0x5649bd209b9d]
mongod(__wt_session_get_btree_ckpt+0x14C) [0x5649bd209e5c]
mongod(__wt_curfile_open+0x161) [0x5649bd18d4f1]
mongod(+0x1F46A08) [0x5649bd1ffa08]
mongod(__wt_metadata_cursor_open+0x6E) [0x5649bd1c99ae]
mongod(__wt_metadata_cursor+0x4B) [0x5649bd1c9a8b]
mongod(wiredtiger_open+0x164D) [0x5649bd17b05d]
mongod(ZN5mongo18WiredTigerKVEngineC2ERKNSt7_cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0x70F) [0x5649bc51da6f]
mongod(+0x125D2EC) [0x5649bc5162ec]
mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x6B0) [0x5649bc409a00]
mongod(+0x80695E) [0x5649bbabf95e]
mongod(main+0x966) [0x5649bbadf1f6]
libc.so.6(__libc_start_main+0xF0) [0x7fbe3e28e830]
mongod(_start+0x29) [0x5649bbb3de09]
----- END BACKTRACE -----

 
recovery attempt:
maxim@ColeteElastic:~$ sudo mongod --recover --dbpath /var/lib/mongodb
Error parsing command line: unrecognised option '--recover'
try 'mongod --help' for more information
maxim@ColeteElastic:~$ sudo mongod --repair --dbpath /var/lib/mongodb
2018-04-20T08:23:42.138+0300 I CONTROL  [initandlisten] MongoDB starting : pid=1325 port=27017 dbpath=/var/lib/mongodb 64-bit host=ColeteElastic
2018-04-20T08:23:42.138+0300 I CONTROL  [initandlisten] db version v3.4.7
2018-04-20T08:23:42.138+0300 I CONTROL  [initandlisten] git version: cf38c1b8a0a8dca4a11737581beafef4fe120bcd
2018-04-20T08:23:42.138+0300 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
2018-04-20T08:23:42.138+0300 I CONTROL  [initandlisten] allocator: tcmalloc
2018-04-20T08:23:42.138+0300 I CONTROL  [initandlisten] modules: none
2018-04-20T08:23:42.139+0300 I CONTROL  [initandlisten] build environment:
2018-04-20T08:23:42.139+0300 I CONTROL  [initandlisten]     distmod: ubuntu1604
2018-04-20T08:23:42.139+0300 I CONTROL  [initandlisten]     distarch: x86_64
2018-04-20T08:23:42.139+0300 I CONTROL  [initandlisten]     target_arch: x86_64
2018-04-20T08:23:42.139+0300 I CONTROL  [initandlisten] options: { repair: true, storage: { dbPath: "/var/lib/mongodb" } }
2018-04-20T08:23:42.190+0300 I -        [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2018-04-20T08:23:42.190+0300 I STORAGE  [initandlisten]
2018-04-20T08:23:42.190+0300 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2018-04-20T08:23:42.190+0300 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2018-04-20T08:23:42.190+0300 I STORAGE  [initandlisten] Detected WT journal files.  Running recovery from last checkpoint.
2018-04-20T08:23:42.190+0300 I STORAGE  [initandlisten] journal to nojournal transition config: create,cache_size=3475M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2018-04-20T08:23:42.201+0300 E STORAGE  [initandlisten] WiredTiger error (0) [1524201822:201258][1325:0x7f6da9aa1d00], file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value
2018-04-20T08:23:42.201+0300 E STORAGE  [initandlisten] WiredTiger error (-31804) [1524201822:201340][1325:0x7f6da9aa1d00], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2018-04-20T08:23:42.201+0300 I -        [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
2018-04-20T08:23:42.201+0300 I -        [initandlisten]
 
***aborting after fassert() failure
 
 
2018-04-20T08:23:42.230+0300 F -        [initandlisten] Got signal: 6 (Aborted).
 
 0x55874a5a9ea1 0x55874a5a90b9 0x55874a5a959d 0x7f6da86a6390 0x7f6da8300428 0x7f6da830202a 0x558749858733 0x55874a2bdc26 0x5587498629cc 0x558749862bf1 0x558749862e49 0x55874af50afc 0x55874af4f269 0x55874af03d03 0x55874af8e65b 0x55874af8eb9d 0x55874af8ee5c 0x55874af124f1 0x55874af84a08 0x55874af4e9ae 0x55874af4ea8b 0x55874af0005d 0x55874a2a2e09 0x55874a29b2ec 0x55874a18ea00 0x55874984495e 0x5587498641f6 0x7f6da82eb830 0x5587498c2e09
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"55874903E000","o":"156BEA1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55874903E000","o":"156B0B9"},{"b":"55874903E000","o":"156B59D"},{"b":"7F6DA8695000","o":"11390"},{"b":"7F6DA82CB000","o":"35428","s":"gsignal"},{"b":"7F6DA82CB000","o":"3702A","s":"abort"},{"b":"55874903E000","o":"81A733","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"55874903E000","o":"127FC26"},{"b":"55874903E000","o":"8249CC","s":"__wt_eventv"},{"b":"55874903E000","o":"824BF1","s":"__wt_err"},{"b":"55874903E000","o":"824E49","s":"__wt_panic"},{"b":"55874903E000","o":"1F12AFC","s":"__wt_turtle_read"},{"b":"55874903E000","o":"1F11269","s":"__wt_metadata_search"},{"b":"55874903E000","o":"1EC5D03","s":"__wt_conn_btree_open"},{"b":"55874903E000","o":"1F5065B","s":"__wt_session_get_btree"},{"b":"55874903E000","o":"1F50B9D","s":"__wt_session_get_btree"},{"b":"55874903E000","o":"1F50E5C","s":"__wt_session_get_btree_ckpt"},{"b":"55874903E000","o":"1ED44F1","s":"__wt_curfile_open"},{"b":"55874903E000","o":"1F46A08"},{"b":"55874903E000","o":"1F109AE","s":"__wt_metadata_cursor_open"},{"b":"55874903E000","o":"1F10A8B","s":"__wt_metadata_cursor"},{"b":"55874903E000","o":"1EC205D","s":"wiredtiger_open"},{"b":"55874903E000","o":"1264E09","s":"_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"},{"b":"55874903E000","o":"125D2EC"},{"b":"55874903E000","o":"1150A00","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"55874903E000","o":"80695E"},{"b":"55874903E000","o":"8261F6","s":"main"},{"b":"7F6DA82CB000","o":"20830","s":"__libc_start_main"},{"b":"55874903E000","o":"884E09","s":"_start"}],"processInfo":{ "mongodbVersion" : "3.4.7", "gitVersion" : "cf38c1b8a0a8dca4a11737581beafef4fe120bcd", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-92-generic", "version" : "#115-Ubuntu SMP Thu Aug 10 09:04:33 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "b" : "55874903E000", "elfType" : 3, "buildId" : "7E08C88DF63C9DEE479A2B1C6C7E11D8651F1184" }, { "b" : "7FFDDA3FC000", "elfType" : 3, "buildId" : "F23E2C79BCC8E97B12E1BB62A1BF196F8423FC40" }, { "b" : "7F6DA9621000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "675F454AD6FD0B6CA2E41127C7B98079DA37F7B6" }, { "b" : "7F6DA91DD000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "2DA08A7E5BF610030DD33B70DB951399626B7496" }, { "b" : "7F6DA8FD5000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "F951C1E0765FCAE48F82CAFE35D1ADD36D6C9AF9" }, { "b" : "7F6DA8DD1000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "0FC788F0861846257B5F1773FBD438E95DFC1032" }, { "b" : "7F6DA8AC8000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "FF7A33D389E756CA381A8189291A968EA5E1F4F8" }, { "b" : "7F6DA88B2000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7F6DA8695000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "27F189EF8DB8C3734C6A678E6EF3CB0B206D58B2" }, { "b" : "7F6DA82CB000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "088A6E00A1814622219F346B41E775B8DD46C518" }, { "b" : "7F6DA988A000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9157F205547F0EB588E2AB1F2F120B74253A43EA" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55874a5a9ea1]
 mongod(+0x156B0B9) [0x55874a5a90b9]
 mongod(+0x156B59D) [0x55874a5a959d]
 libpthread.so.0(+0x11390) [0x7f6da86a6390]
 libc.so.6(gsignal+0x38) [0x7f6da8300428]
 libc.so.6(abort+0x16A) [0x7f6da830202a]
 mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x558749858733]
 mongod(+0x127FC26) [0x55874a2bdc26]
 mongod(__wt_eventv+0x3D7) [0x5587498629cc]
 mongod(__wt_err+0x9D) [0x558749862bf1]
 mongod(__wt_panic+0x24) [0x558749862e49]
 mongod(__wt_turtle_read+0x26C) [0x55874af50afc]
 mongod(__wt_metadata_search+0x99) [0x55874af4f269]
 mongod(__wt_conn_btree_open+0x73) [0x55874af03d03]
 mongod(__wt_session_get_btree+0xFB) [0x55874af8e65b]
 mongod(__wt_session_get_btree+0x63D) [0x55874af8eb9d]
 mongod(__wt_session_get_btree_ckpt+0x14C) [0x55874af8ee5c]
 mongod(__wt_curfile_open+0x161) [0x55874af124f1]
 mongod(+0x1F46A08) [0x55874af84a08]
 mongod(__wt_metadata_cursor_open+0x6E) [0x55874af4e9ae]
 mongod(__wt_metadata_cursor+0x4B) [0x55874af4ea8b]
 mongod(wiredtiger_open+0x164D) [0x55874af0005d]
 mongod(_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0xAA9) [0x55874a2a2e09]
 mongod(+0x125D2EC) [0x55874a29b2ec]
 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x6B0) [0x55874a18ea00]
 mongod(+0x80695E) [0x55874984495e]
 mongod(main+0x966) [0x5587498641f6]
 libc.so.6(__libc_start_main+0xF0) [0x7f6da82eb830]
 mongod(_start+0x29) [0x5587498c2e09]
-----  END BACKTRACE  -----
Aborted (core dumped)

logs after recovery attempt:

2018-04-20T08:20:23.620+0300 I CONTROL  [main] ***** SERVER RESTARTED *****
2018-04-20T08:20:23.627+0300 I CONTROL  [initandlisten] MongoDB starting : pid=1280 port=27017 dbpath=/var/lib/mongodb 64-bit host=ColeteElastic
2018-04-20T08:20:23.627+0300 I CONTROL  [initandlisten] db version v3.4.7
2018-04-20T08:20:23.627+0300 I CONTROL  [initandlisten] git version: cf38c1b8a0a8dca4a11737581beafef4fe120bcd
2018-04-20T08:20:23.627+0300 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
2018-04-20T08:20:23.627+0300 I CONTROL  [initandlisten] allocator: tcmalloc
2018-04-20T08:20:23.627+0300 I CONTROL  [initandlisten] modules: none
2018-04-20T08:20:23.627+0300 I CONTROL  [initandlisten] build environment:
2018-04-20T08:20:23.627+0300 I CONTROL  [initandlisten]     distmod: ubuntu1604
2018-04-20T08:20:23.627+0300 I CONTROL  [initandlisten]     distarch: x86_64
2018-04-20T08:20:23.627+0300 I CONTROL  [initandlisten]     target_arch: x86_64
2018-04-20T08:20:23.627+0300 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "0.0.0.0", port: 27017 }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2018-04-20T08:20:23.673+0300 I -        [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2018-04-20T08:20:23.673+0300 I STORAGE  [initandlisten]
2018-04-20T08:20:23.673+0300 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2018-04-20T08:20:23.673+0300 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2018-04-20T08:20:23.673+0300 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=3475M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2018-04-20T08:20:23.683+0300 E STORAGE  [initandlisten] WiredTiger error (0) [1524201623:683179][1280:0x7fbe3fa44d00], file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value
2018-04-20T08:20:23.683+0300 E STORAGE  [initandlisten] WiredTiger error (-31804) [1524201623:683230][1280:0x7fbe3fa44d00], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2018-04-20T08:20:23.683+0300 I -        [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
2018-04-20T08:20:23.683+0300 I -        [initandlisten]
 
***aborting after fassert() failure
 
 
2018-04-20T08:20:23.711+0300 F -        [initandlisten] Got signal: 6 (Aborted).
 
 0x5649bc824ea1 0x5649bc8240b9 0x5649bc82459d 0x7fbe3e649390 0x7fbe3e2a3428 0x7fbe3e2a502a 0x5649bbad3733 0x5649bc538c26 0x5649bbadd9cc 0x5649bbaddbf1 0x5649bbadde49 0x5649bd1cbafc 0x5649bd1ca269 0x5649bd17ed03 0x5649bd20965b 0x5649bd209b9d 0x5649bd209e5c 0x5649bd18d4f1 0x5649bd1ffa08 0x5649bd1c99ae 0x5649bd1c9a8b 0x5649bd17b05d 0x5649bc51da6f 0x5649bc5162ec 0x5649bc409a00 0x5649bbabf95e 0x5649bbadf1f6 0x7fbe3e28e830 0x5649bbb3de09
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"5649BB2B9000","o":"156BEA1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"5649BB2B9000","o":"156B0B9"},{"b":"5649BB2B9000","o":"156B59D"},{"b":"7FBE3E638000","o":"11390"},{"b":"7FBE3E26E000","o":"35428","s":"gsignal"},{"b":"7FBE3E26E000","o":"3702A","s":"abort"},{"b":"5649BB2B9000","o":"81A733","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"5649BB2B9000","o":"127FC26"},{"b":"5649BB2B9000","o":"8249CC","s":"__wt_eventv"},{"b":"5649BB2B9000","o":"824BF1","s":"__wt_err"},{"b":"5649BB2B9000","o":"824E49","s":"__wt_panic"},{"b":"5649BB2B9000","o":"1F12AFC","s":"__wt_turtle_read"},{"b":"5649BB2B9000","o":"1F11269","s":"__wt_metadata_search"},{"b":"5649BB2B9000","o":"1EC5D03","s":"__wt_conn_btree_open"},{"b":"5649BB2B9000","o":"1F5065B","s":"__wt_session_get_btree"},{"b":"5649BB2B9000","o":"1F50B9D","s":"__wt_session_get_btree"},{"b":"5649BB2B9000","o":"1F50E5C","s":"__wt_session_get_btree_ckpt"},{"b":"5649BB2B9000","o":"1ED44F1","s":"__wt_curfile_open"},{"b":"5649BB2B9000","o":"1F46A08"},{"b":"5649BB2B9000","o":"1F109AE","s":"__wt_metadata_cursor_open"},{"b":"5649BB2B9000","o":"1F10A8B","s":"__wt_metadata_cursor"},{"b":"5649BB2B9000","o":"1EC205D","s":"wiredtiger_open"},{"b":"5649BB2B9000","o":"1264A6F","s":"_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"},{"b":"5649BB2B9000","o":"125D2EC"},{"b":"5649BB2B9000","o":"1150A00","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"5649BB2B9000","o":"80695E"},{"b":"5649BB2B9000","o":"8261F6","s":"main"},{"b":"7FBE3E26E000","o":"20830","s":"__libc_start_main"},{"b":"5649BB2B9000","o":"884E09","s":"_start"}],"processInfo":{ "mongodbVersion" : "3.4.7", "gitVersion" : "cf38c1b8a0a8dca4a11737581beafef4fe120bcd", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-92-generic", "version" : "#115-Ubuntu SMP Thu Aug 10 09:04:33 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "b" : "5649BB2B9000", "elfType" : 3, "buildId" : "7E08C88DF63C9DEE479A2B1C6C7E11D8651F1184" }, { "b" : "7FFC78F9B000", "elfType" : 3, "buildId" : "F23E2C79BCC8E97B12E1BB62A1BF196F8423FC40" }, { "b" : "7FBE3F5C4000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "675F454AD6FD0B6CA2E41127C7B98079DA37F7B6" }, { "b" : "7FBE3F180000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "2DA08A7E5BF610030DD33B70DB951399626B7496" }, { "b" : "7FBE3EF78000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "F951C1E0765FCAE48F82CAFE35D1ADD36D6C9AF9" }, { "b" : "7FBE3ED74000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "0FC788F0861846257B5F1773FBD438E95DFC1032" }, { "b" : "7FBE3EA6B000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "FF7A33D389E756CA381A8189291A968EA5E1F4F8" }, { "b" : "7FBE3E855000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7FBE3E638000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "27F189EF8DB8C3734C6A678E6EF3CB0B206D58B2" }, { "b" : "7FBE3E26E000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "088A6E00A1814622219F346B41E775B8DD46C518" }, { "b" : "7FBE3F82D000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9157F205547F0EB588E2AB1F2F120B74253A43EA" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x5649bc824ea1]
 mongod(+0x156B0B9) [0x5649bc8240b9]
 mongod(+0x156B59D) [0x5649bc82459d]
 libpthread.so.0(+0x11390) [0x7fbe3e649390]
 libc.so.6(gsignal+0x38) [0x7fbe3e2a3428]
 libc.so.6(abort+0x16A) [0x7fbe3e2a502a]
 mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x5649bbad3733]
 mongod(+0x127FC26) [0x5649bc538c26]
 mongod(__wt_eventv+0x3D7) [0x5649bbadd9cc]
 mongod(__wt_err+0x9D) [0x5649bbaddbf1]
 mongod(__wt_panic+0x24) [0x5649bbadde49]
 mongod(__wt_turtle_read+0x26C) [0x5649bd1cbafc]
 mongod(__wt_metadata_search+0x99) [0x5649bd1ca269]
 mongod(__wt_conn_btree_open+0x73) [0x5649bd17ed03]
 mongod(__wt_session_get_btree+0xFB) [0x5649bd20965b]
 mongod(__wt_session_get_btree+0x63D) [0x5649bd209b9d]
 mongod(__wt_session_get_btree_ckpt+0x14C) [0x5649bd209e5c]
 mongod(__wt_curfile_open+0x161) [0x5649bd18d4f1]
 mongod(+0x1F46A08) [0x5649bd1ffa08]
 mongod(__wt_metadata_cursor_open+0x6E) [0x5649bd1c99ae]
 mongod(__wt_metadata_cursor+0x4B) [0x5649bd1c9a8b]
 mongod(wiredtiger_open+0x164D) [0x5649bd17b05d]
 mongod(_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0x70F) [0x5649bc51da6f]
 mongod(+0x125D2EC) [0x5649bc5162ec]
 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x6B0) [0x5649bc409a00]
 mongod(+0x80695E) [0x5649bbabf95e]
 mongod(main+0x966) [0x5649bbadf1f6]
 libc.so.6(__libc_start_main+0xF0) [0x7fbe3e28e830]
 mongod(_start+0x29) [0x5649bbb3de09]
-----  END BACKTRACE  -----

Comment by Kelsey Schubert [ 19/Apr/18 ]

Hi Durbala,

Thank you for your report. I've attached a repair attempt, repair_attempt.tar.gz, of the files you provided. Please extract these files and replace them in your $dbpath and let us know if it resolves the issue. If you are still seeing errors after replacing these files, please provide the complete logs from the affected node so that we can further investigate.

Thank you,
Kelsey

Generated at Thu Feb 08 04:37:05 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.