2018-05-30T17:51:15.785+0000 I CONTROL [main] ***** SERVER RESTARTED ***** 2018-05-30T17:51:15.803+0000 I CONTROL [initandlisten] MongoDB starting : pid=4246 port=27017 dbpath=/var/mongodata 64-bit host=apiquotausage-rs0-2 2018-05-30T17:51:15.803+0000 I CONTROL [initandlisten] db version v3.6.5 2018-05-30T17:51:15.803+0000 I CONTROL [initandlisten] git version: a20ecd3e3a174162052ff99913bc2ca9a839d618 2018-05-30T17:51:15.803+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.0-fips 29 Mar 2010 2018-05-30T17:51:15.803+0000 I CONTROL [initandlisten] allocator: tcmalloc 2018-05-30T17:51:15.803+0000 I CONTROL [initandlisten] modules: none 2018-05-30T17:51:15.803+0000 I CONTROL [initandlisten] build environment: 2018-05-30T17:51:15.803+0000 I CONTROL [initandlisten] distmod: amazon 2018-05-30T17:51:15.803+0000 I CONTROL [initandlisten] distarch: x86_64 2018-05-30T17:51:15.803+0000 I CONTROL [initandlisten] target_arch: x86_64 2018-05-30T17:51:15.803+0000 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIpAll: true, port: 27017 }, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid" }, replication: { replSetName: "rs0" }, security: { authorization: "enabled", keyFile: "/var/mongodata/apiquotausage-mongo-key" }, storage: { dbPath: "/var/mongodata", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } } 2018-05-30T17:51:15.803+0000 W - [initandlisten] Detected unclean shutdown - /var/mongodata/mongod.lock is not empty. 2018-05-30T17:51:15.804+0000 I - [initandlisten] Detected data files in /var/mongodata created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'. 2018-05-30T17:51:15.804+0000 W STORAGE [initandlisten] Recovering data from the last clean checkpoint. 2018-05-30T17:51:15.804+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=7512M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),cache_cursors=false,log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress), 2018-05-30T17:51:16.630+0000 I STORAGE [initandlisten] WiredTiger message [1527702676:630298][4246:0x7f5fef122ac0], txn-recover: Main recovery loop: starting at 1140/46566144 2018-05-30T17:51:16.635+0000 I STORAGE [initandlisten] WiredTiger message [1527702676:635983][4246:0x7f5fef122ac0], txn-recover: Recovering log 1140 through 1141 2018-05-30T17:51:16.719+0000 I STORAGE [initandlisten] WiredTiger message [1527702676:719007][4246:0x7f5fef122ac0], file:collection-4--1985707126079834107.wt, txn-recover: Recovering log 1141 through 1141 2018-05-30T17:51:16.772+0000 I STORAGE [initandlisten] WiredTiger message [1527702676:772163][4246:0x7f5fef122ac0], file:collection-4--1985707126079834107.wt, txn-recover: Set global recovery timestamp: 0 2018-05-30T17:51:17.981+0000 I STORAGE [initandlisten] Starting WiredTigerRecordStoreThread local.oplog.rs 2018-05-30T17:51:17.981+0000 I STORAGE [initandlisten] The size storer reports that the oplog contains 4464049 records totaling to 1413821696 bytes 2018-05-30T17:51:17.982+0000 I STORAGE [initandlisten] Sampling from the oplog between May 28 15:29:16:5 and May 30 17:02:47:45 to determine where to place markers for truncation 2018-05-30T17:51:17.982+0000 I STORAGE [initandlisten] Taking 264 samples and assuming that each section of oplog contains approximately 168881 records totaling to 53486783 bytes 2018-05-30T17:51:18.208+0000 I STORAGE [initandlisten] Placing a marker at optime May 28 17:35:41:31 2018-05-30T17:51:18.208+0000 I STORAGE [initandlisten] Placing a marker at optime May 28 19:41:49:4 2018-05-30T17:51:18.208+0000 I STORAGE [initandlisten] Placing a marker at optime May 28 22:40:48:43 2018-05-30T17:51:18.208+0000 I STORAGE [initandlisten] Placing a marker at optime May 29 00:49:01:3 2018-05-30T17:51:18.208+0000 I STORAGE [initandlisten] Placing a marker at optime May 29 01:42:07:8 2018-05-30T17:51:18.208+0000 I STORAGE [initandlisten] Placing a marker at optime May 29 02:47:16:52 2018-05-30T17:51:18.208+0000 I STORAGE [initandlisten] Placing a marker at optime May 29 04:38:36:7 2018-05-30T17:51:18.208+0000 I STORAGE [initandlisten] Placing a marker at optime May 29 05:58:57:65 2018-05-30T17:51:18.208+0000 I STORAGE [initandlisten] Placing a marker at optime May 29 07:34:01:16 2018-05-30T17:51:18.208+0000 I STORAGE [initandlisten] Placing a marker at optime May 29 09:33:32:11 2018-05-30T17:51:18.208+0000 I STORAGE [initandlisten] Placing a marker at optime May 29 11:44:20:3 2018-05-30T17:51:18.208+0000 I STORAGE [initandlisten] Placing a marker at optime May 29 13:59:18:2 2018-05-30T17:51:18.208+0000 I STORAGE [initandlisten] Placing a marker at optime May 29 16:00:08:24 2018-05-30T17:51:18.208+0000 I STORAGE [initandlisten] Placing a marker at optime May 29 18:30:13:27 2018-05-30T17:51:18.208+0000 I STORAGE [initandlisten] Placing a marker at optime May 29 20:22:30:2 2018-05-30T17:51:18.208+0000 I STORAGE [initandlisten] Placing a marker at optime May 29 23:09:13:19 2018-05-30T17:51:18.208+0000 I STORAGE [initandlisten] Placing a marker at optime May 30 00:43:08:2 2018-05-30T17:51:18.208+0000 I STORAGE [initandlisten] Placing a marker at optime May 30 02:03:27:61 2018-05-30T17:51:18.208+0000 I STORAGE [initandlisten] Placing a marker at optime May 30 04:27:23:16 2018-05-30T17:51:18.208+0000 I STORAGE [initandlisten] Placing a marker at optime May 30 06:15:52:6 2018-05-30T17:51:18.208+0000 I STORAGE [initandlisten] Placing a marker at optime May 30 07:28:20:9 2018-05-30T17:51:18.208+0000 I STORAGE [initandlisten] Placing a marker at optime May 30 09:02:20:24 2018-05-30T17:51:18.208+0000 I STORAGE [initandlisten] Placing a marker at optime May 30 10:55:33:50 2018-05-30T17:51:18.208+0000 I STORAGE [initandlisten] Placing a marker at optime May 30 12:49:04:29 2018-05-30T17:51:18.208+0000 I STORAGE [initandlisten] Placing a marker at optime May 30 16:01:18:5 2018-05-30T17:51:18.208+0000 I STORAGE [initandlisten] Placing a marker at optime May 30 16:49:56:8 2018-05-30T17:51:18.242+0000 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory '/var/mongodata/diagnostic.data' 2018-05-30T17:51:18.242+0000 I REPL [initandlisten] Did not find local voted for document at startup. 2018-05-30T17:51:18.243+0000 I REPL [initandlisten] Rollback ID is 1 2018-05-30T17:51:18.244+0000 I REPL [initandlisten] Starting recovery oplog application at the appliedThrough: { ts: Timestamp(1527698369, 38), t: 128 } 2018-05-30T17:51:18.244+0000 I REPL [initandlisten] Replaying stored operations from { : Timestamp(1527698369, 38) } (exclusive) to { : Timestamp(1527699767, 45) } (inclusive). 2018-05-30T17:51:19.004+0000 I FTDC [ftdc] Unclean full-time diagnostic data capture shutdown detected, found interim file, some metrics may have been lost. OK {noformat} I left the secondary in this state for a few hours to see if it would eventually get there as I was observing high CPU and disk usage. After about two hours the following additional information was output in the log: {noformat}2018-05-30T19:39:54.852+0000 E STORAGE [initandlisten] WiredTiger error (28) [1527709194:850754][4246:0x7f5fef122ac0], file:WiredTigerLAS.wt, WT_SESSION.commit_transaction: /var/mongodata/WiredTigerLAS.wt: handle-write: pwrite: failed to write 794624 bytes at offset 105847193600: No space left on device 2018-05-30T19:39:55.036+0000 E STORAGE [thread1] WiredTiger error (28) [1527709195:35810][4246:0x7f5fe8569700], file:WiredTigerLAS.wt, eviction-server: /var/mongodata/WiredTigerLAS.wt: handle-write: pwrite: failed to write 794624 bytes at offset 105847193600: No space left on device 2018-05-30T19:39:55.036+0000 E STORAGE [thread1] WiredTiger error (28) [1527709195:36383][4246:0x7f5fe8569700], eviction-server: cache eviction thread error: No space left on device 2018-05-30T19:39:55.036+0000 E STORAGE [thread1] WiredTiger error (-31804) [1527709195:36407][4246:0x7f5fe8569700], eviction-server: the process must exit and restart: WT_PANIC: WiredTiger library panic 2018-05-30T19:39:55.037+0000 F - [thread1] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 366 2018-05-30T19:39:55.037+0000 F - [thread1] ***aborting after fassert() failure 2018-05-30T19:39:55.045+0000 E STORAGE [thread2] WiredTiger error (-31804) [1527709195:44990][4246:0x7f5fe9d6c700], eviction-server: cache eviction thread error: WT_PANIC: WiredTiger library panic 2018-05-30T19:39:55.045+0000 F - [thread2] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 366 2018-05-30T19:39:55.045+0000 F - [thread2] ***aborting after fassert() failure 2018-05-30T19:39:55.065+0000 F - [initandlisten] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64 2018-05-30T19:39:55.065+0000 F - [initandlisten] ***aborting after fassert() failure 2018-05-30T19:39:55.093+0000 F - [thread1] Got signal: 6 (Aborted). 0x55b9769b4221 0x55b9769b3439 0x55b9769b391d 0x7f5fedaf85a0 0x7f5fed75a1f7 0x7f5fed75b8e8 0x55b975107395 0x55b9751d4a3e 0x55b975241691 0x55b9750a4d09 0x55b9750a5032 0x55b9751fcc95 0x55b97524db79 0x7f5fedaf0de5 0x7f5fed81d30d ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"55B974778000","o":"223C221","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55B974778000","o":"223B439"},{"b":"55B974778000","o":"223B91D"},{"b":"7F5FEDAE9000","o":"F5A0"},{"b":"7F5FED725000","o":"351F7","s":"gsignal"},{"b":"7F5FED725000","o":"368E8","s":"abort"},{"b":"55B974778000","o":"98F395","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"55B974778000","o":"A5CA3E"},{"b":"55B974778000","o":"AC9691"},{"b":"55B974778000","o":"92CD09","s":"__wt_err"},{"b":"55B974778000","o":"92D032","s":"__wt_panic"},{"b":"55B974778000","o":"A84C95","s":"__wt_evict_thread_run"},{"b":"55B974778000","o":"AD5B79"},{"b":"7F5FEDAE9000","o":"7DE5"},{"b":"7F5FED725000","o":"F830D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.6.5", "gitVersion" : "a20ecd3e3a174162052ff99913bc2ca9a839d618", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.9.93-41.60.amzn1.x86_64", "version" : "#1 SMP Fri Apr 13 21:58:27 UTC 2018", "machine" : "x86_64" }, "somap" : [ { "b" : "55B974778000", "elfType" : 3, "buildId" : "8428531F0F991A7F1859B560D1B2369D1FF658C0" }, { "b" : "7FFFC390F000", "elfType" : 3, "buildId" : "74A3162241B05E2E564BBAA6890CD918EF609244" }, { "b" : "7F5FEECF8000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "0BED68388AE751ACB77DA9FBB6B63CAA5E4622A9" }, { "b" : "7F5FEE89A000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "B71E5A22C6224C3A54A0CDAAA0C23F81EADE5C5F" }, { "b" : "7F5FEE629000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "FAADF98D6634B218EF976D389BCEA90C97FB8CD2" }, { "b" : "7F5FEE425000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "2CC91EC12043BC0357CD502BBB4D5A4940BFFB1B" }, { "b" : "7F5FEE21D000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "BA23B2F2723B7C9BF5FFE9A8AF9E50219C03EB0E" }, { "b" : "7F5FEDF1B000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "31404F7BCB6B5A880B25E0007EA5C73BBEC87E16" }, { "b" : "7F5FEDD05000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "AC58019512A5359B077D2610DEA4AD6CF14CAC53" }, { "b" : "7F5FEDAE9000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "4136F256787CE855F91B70793154EBCFCE047CFA" }, { "b" : "7F5FED725000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "FB411B5AB3944BD21963101275C0F9EB3F943BDC" }, { "b" : "7F5FEEF12000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "CC0DF41E2210C483EFF9C95EDB5F2AE014752C70" }, { "b" : "7F5FED50F000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "89C6AF118B6B4FB6A73AE1813E2C8BDD722956D1" }, { "b" : "7F5FED2C2000", "path" : "/usr/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "6FBBD34B86296FDF883FE5122017EC5CD3F98ED7" }, { "b" : "7F5FECFDA000", "path" : "/usr/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "76429E6FD408BBB675798D6458F2735383710D0B" }, { "b" : "7F5FECDD7000", "path" : "/usr/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "5C01209C5AE1B1714F19B07EB58F2A1274B69DC8" }, { "b" : "7F5FECBA4000", "path" : "/usr/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "5B2A76F1EF91EDAA0494BE680CADAFE6489326E1" }, { "b" : "7F5FEC996000", "path" : "/usr/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "3ACB59488C6D8DE0A1F4F1B0C290A570D9E42F3D" }, { "b" : "7F5FEC793000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "37A58210FA50C91E09387765408A92909468D25B" }, { "b" : "7F5FEC572000", "path" : "/usr/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "F5054DC94443326819FBF3065CFDF5E4726F57EE" } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55b9769b4221] mongod(+0x223B439) [0x55b9769b3439] mongod(+0x223B91D) [0x55b9769b391d] libpthread.so.0(+0xF5A0) [0x7f5fedaf85a0] libc.so.6(gsignal+0x37) [0x7f5fed75a1f7] libc.so.6(abort+0x148) [0x7f5fed75b8e8] mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x55b975107395] mongod(+0xA5CA3E) [0x55b9751d4a3e] mongod(+0xAC9691) [0x55b975241691] mongod(__wt_err+0x9D) [0x55b9750a4d09] mongod(__wt_panic+0x33) [0x55b9750a5032] mongod(__wt_evict_thread_run+0x2EE5) [0x55b9751fcc95] mongod(+0xAD5B79) [0x55b97524db79] libpthread.so.0(+0x7DE5) [0x7f5fedaf0de5] libc.so.6(clone+0x6D) [0x7f5fed81d30d] ----- END BACKTRACE -----