2020-07-07T14:12:09.114-0500 I CONTROL [main] ***** SERVER RESTARTED ***** 2020-07-07T14:12:09.138-0500 I CONTROL [initandlisten] MongoDB starting : pid=14231 port=27017 dbpath=/data1/mongodb/data 64-bit host=abrtp-mongo1a.marketo.org 2020-07-07T14:12:09.138-0500 I CONTROL [initandlisten] db version v3.2.1 2020-07-07T14:12:09.138-0500 I CONTROL [initandlisten] git version: a14d55980c2cdc565d4704a7e3ad37e4e535c1b2 2020-07-07T14:12:09.138-0500 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013 2020-07-07T14:12:09.138-0500 I CONTROL [initandlisten] allocator: tcmalloc 2020-07-07T14:12:09.138-0500 I CONTROL [initandlisten] modules: none 2020-07-07T14:12:09.138-0500 I CONTROL [initandlisten] build environment: 2020-07-07T14:12:09.138-0500 I CONTROL [initandlisten] distmod: rhel62 2020-07-07T14:12:09.138-0500 I CONTROL [initandlisten] distarch: x86_64 2020-07-07T14:12:09.138-0500 I CONTROL [initandlisten] target_arch: x86_64 2020-07-07T14:12:09.138-0500 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { maxIncomingConnections: 3000, port: 27017, unixDomainSocket: { enabled: true, pathPrefix: "/data1/mongodb/run" }, wireObjectCheck: true }, processManagement: { fork: true, pidFilePath: "/data1/mongodb/run/mongod.pid" }, replication: { oplogSizeMB: 500000, replSetName: "abrtprs1", secondaryIndexPrefetch: "all" }, security: { authorization: "disabled" }, setParameter: { enableLocalhostAuthBypass: "false" }, storage: { dbPath: "/data1/mongodb/data", directoryPerDB: true, engine: "wiredTiger", journal: { commitIntervalMs: 100, enabled: true }, mmapv1: { nsSize: 16 }, syncPeriodSecs: 5.0 }, systemLog: { destination: "file", logAppend: true, path: "/data1/mongodb/log/mongod.log" } } 2020-07-07T14:12:09.139-0500 W - [initandlisten] Detected unclean shutdown - /data1/mongodb/data/mongod.lock is not empty. 2020-07-07T14:12:09.139-0500 W STORAGE [initandlisten] Recovering data from the last clean checkpoint. 2020-07-07T14:12:09.139-0500 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=150G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=5,log_size=2GB),statistics_log=(wait=0), 2020-07-07T14:12:27.777-0500 I STORAGE [initandlisten] Starting WiredTigerRecordStoreThread local.oplog.rs 2020-07-07T14:12:27.777-0500 I STORAGE [initandlisten] The size storer reports that the oplog contains 477317526 records totaling to 526224495609 bytes 2020-07-07T14:12:27.777-0500 I STORAGE [initandlisten] Sampling from the oplog between Jun 26 09:59:13:1c6 and Jul 3 05:15:08:230 to determine where to place markers for truncation 2020-07-07T14:12:27.777-0500 I STORAGE [initandlisten] Taking 1003 samples and assuming that each section of oplog contains approximately 4755611 records totaling to 5242881024 bytes 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 26 10:59:22:19b 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 26 11:49:07:5c 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 26 13:37:41:18d 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 26 15:39:42:124 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 26 17:19:46:379 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 26 18:50:56:1a9 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 26 20:42:10:17b 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 26 22:36:33:537 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 26 23:33:46:22f 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 27 01:07:32:41c 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 27 02:45:53:339 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 27 04:15:10:243 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 27 06:04:22:259 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 27 07:40:59:26e 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 27 09:12:42:232 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 27 10:51:46:116 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 27 12:13:51:22 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 27 14:01:08:48 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 27 15:31:24:219 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 27 17:24:51:5e7 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 27 19:04:52:38b 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 27 20:38:10:558 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 27 22:08:47:93 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 27 23:56:19:101 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 28 01:29:37:cd 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 28 03:20:42:10 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 28 04:37:52:126 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 28 06:27:46:a4 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 28 07:57:01:28d 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 28 09:07:38:49d 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 28 10:18:05:25b 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 28 11:58:41:406 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 28 13:22:51:4c 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 28 15:06:29:223 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 28 16:35:55:1eb 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 28 18:11:27:ab 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 28 19:32:36:580 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 28 20:52:49:1e4 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 28 22:10:23:9 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 28 23:37:05:4db 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 29 00:55:22:2d4 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 29 02:29:58:f6 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 29 03:41:55:2fa5 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 29 05:01:18:28 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 29 06:27:50:21b 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 29 07:44:28:32d 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 29 09:14:23:16 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 29 10:34:23:4b 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 29 11:46:01:44d 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 29 13:23:27:1bc 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 29 14:44:51:b 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 29 16:41:07:69 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 29 18:26:33:430 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 29 19:02:18:52b 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 29 19:37:22:2bf 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 29 20:37:06:369 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 29 21:50:18:15b 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 30 00:02:48:c1 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 30 02:33:57:a1 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 30 04:07:48:2c0 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 30 05:45:57:359 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 30 07:18:06:150 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 30 08:19:58:423 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 30 09:26:23:1a6 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 30 10:28:14:3f0 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 30 11:28:03:2e4 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 30 12:31:29:f9 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 30 13:19:37:8e 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 30 14:18:38:49 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 30 15:10:21:5c 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 30 15:55:05:53 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 30 17:02:08:14d 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 30 18:52:35:2f8 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 30 20:18:36:65 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 30 21:43:53:223 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jun 30 23:31:53:2d 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jul 1 02:17:18:87 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jul 1 05:02:36:24f 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jul 1 06:53:46:149 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jul 1 08:40:37:2aa 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jul 1 09:41:03:1a8 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jul 1 11:31:39:cc 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jul 1 13:22:25:1a8 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jul 1 15:18:49:1ea 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jul 1 17:20:11:1af 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jul 1 19:54:41:62 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jul 1 21:37:24:7d 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jul 2 00:42:57:17c 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jul 2 03:06:42:1fc 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jul 2 05:37:33:150 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jul 2 07:55:43:7d 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jul 2 09:40:55:68 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jul 2 11:33:33:15e 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jul 2 13:48:08:20f 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jul 2 15:50:01:1bb 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jul 2 18:45:14:109 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jul 2 21:06:20:22 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jul 3 00:07:40:2d 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jul 3 03:04:18:277 2020-07-07T14:12:36.238-0500 I STORAGE [initandlisten] Placing a marker at optime Jul 3 04:30:35:369 2020-07-07T14:13:34.196-0500 W STORAGE [initandlisten] Detected configuration for non-active storage engine mmapv1 when current storage engine is wiredTiger 2020-07-07T14:13:34.229-0500 I CONTROL [initandlisten] 2020-07-07T14:13:34.229-0500 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'. 2020-07-07T14:13:34.229-0500 I CONTROL [initandlisten] ** We suggest setting it to 'never' 2020-07-07T14:13:34.229-0500 I CONTROL [initandlisten] 2020-07-07T14:13:34.229-0500 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'. 2020-07-07T14:13:34.229-0500 I CONTROL [initandlisten] ** We suggest setting it to 'never' 2020-07-07T14:13:34.229-0500 I CONTROL [initandlisten] 2020-07-07T14:13:54.920-0500 I NETWORK [HostnameCanonicalizationWorker] Starting hostname canonicalization worker 2020-07-07T14:13:54.920-0500 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory '/data1/mongodb/data/diagnostic.data' 2020-07-07T14:13:54.920-0500 I NETWORK [initandlisten] waiting for connections on port 27017 2020-07-07T14:13:54.922-0500 I REPL [ReplicationExecutor] New replica set config in use: { _id: "abrtprs1", version: 21, protocolVersion: 1, members: [ { _id: 0, host: "abrtp-mongo1a.marketo.org:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 0.5, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 3, host: "abrtp-mongo1b.marketo.org:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 4, host: "abrtp-mongo1c.marketo.org:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 0.5, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 } } } 2020-07-07T14:13:54.922-0500 I REPL [ReplicationExecutor] This node is abrtp-mongo1a.marketo.org:27017 in the config 2020-07-07T14:13:54.922-0500 I REPL [ReplicationExecutor] transition to STARTUP2 2020-07-07T14:13:54.922-0500 I REPL [ReplicationExecutor] Starting replication applier threads 2020-07-07T14:13:54.923-0500 I REPL [ReplicationExecutor] transition to RECOVERING 2020-07-07T14:13:54.923-0500 I ASIO [NetworkInterfaceASIO-Replication-0] Successfully connected to abrtp-mongo1b.marketo.org:27017 2020-07-07T14:13:54.924-0500 I ASIO [NetworkInterfaceASIO-Replication-0] Successfully connected to abrtp-mongo1c.marketo.org:27017 2020-07-07T14:13:54.924-0500 I REPL [ReplicationExecutor] Member abrtp-mongo1b.marketo.org:27017 is now in state PRIMARY 2020-07-07T14:13:54.924-0500 I REPL [ReplicationExecutor] Member abrtp-mongo1c.marketo.org:27017 is now in state SECONDARY 2020-07-07T14:13:54.972-0500 I NETWORK [initandlisten] connection accepted from 10.1.11.118:46164 #1 (1 connection now open) 2020-07-07T14:13:54.980-0500 I NETWORK [initandlisten] connection accepted from 10.1.15.95:56810 #2 (2 connections now open) 2020-07-07T14:13:55.878-0500 I NETWORK [initandlisten] connection accepted from 10.1.15.99:55919 #3 (3 connections now open) 2020-07-07T14:13:56.348-0500 I NETWORK [initandlisten] connection accepted from 10.1.15.94:42730 #4 (4 connections now open) 2020-07-07T14:13:56.561-0500 I NETWORK [initandlisten] connection accepted from 10.1.11.119:37822 #5 (5 connections now open) 2020-07-07T14:13:57.745-0500 I NETWORK [initandlisten] connection accepted from 127.0.0.1:60932 #6 (6 connections now open) 2020-07-07T14:13:58.127-0500 I NETWORK [initandlisten] connection accepted from 10.1.15.98:44853 #7 (7 connections now open) 2020-07-07T14:13:58.422-0500 I NETWORK [initandlisten] connection accepted from 10.1.11.119:37928 #8 (8 connections now open) 2020-07-07T14:13:58.501-0500 I NETWORK [initandlisten] connection accepted from 10.1.15.99:56005 #9 (9 connections now open) 2020-07-07T14:13:59.174-0500 I NETWORK [initandlisten] connection accepted from 10.1.11.118:46402 #10 (10 connections now open) 2020-07-07T14:13:59.543-0500 I NETWORK [initandlisten] connection accepted from 10.1.15.98:44898 #11 (11 connections now open) 2020-07-07T14:14:00.924-0500 I REPL [ReplicationExecutor] syncing from: abrtp-mongo1c.marketo.org:27017 2020-07-07T14:14:00.925-0500 I REPL [SyncSourceFeedback] setting syncSourceFeedback to abrtp-mongo1c.marketo.org:27017 2020-07-07T14:14:00.926-0500 I ASIO [NetworkInterfaceASIO-BGSync-0] Successfully connected to abrtp-mongo1c.marketo.org:27017 2020-07-07T14:14:01.605-0500 I NETWORK [initandlisten] connection accepted from 127.0.0.1:60936 #12 (12 connections now open) 2020-07-07T14:14:02.366-0500 I NETWORK [initandlisten] connection accepted from 127.0.0.1:60938 #13 (13 connections now open) 2020-07-07T14:14:02.702-0500 E STORAGE [repl writer worker 14] WiredTiger (22) [1594149242:702281][14231:0x7f7fb495a700], file:dx_account_765/index-166665--5020176994984769950.wt, WT_SESSION.open_cursor: live.avail: existing range 34836480-34861056 overlaps with merge range 34852864-34869248: Invalid argument 2020-07-07T14:14:02.702-0500 E STORAGE [repl writer worker 14] WiredTiger (-31804) [1594149242:702331][14231:0x7f7fb495a700], file:dx_account_765/index-166665--5020176994984769950.wt, WT_SESSION.open_cursor: the process must exit and restart: WT_PANIC: WiredTiger library panic 2020-07-07T14:14:02.702-0500 I - [repl writer worker 14] Fatal Assertion 28558 2020-07-07T14:14:02.702-0500 I - [repl writer worker 6] Fatal Assertion 28559 2020-07-07T14:14:02.702-0500 I - [repl writer worker 14] ***aborting after fassert() failure 2020-07-07T14:14:02.702-0500 I - [repl writer worker 6] ***aborting after fassert() failure 2020-07-07T14:14:02.702-0500 I - [repl writer worker 10] Fatal Assertion 28559 2020-07-07T14:14:02.702-0500 I - [repl writer worker 10] ***aborting after fassert() failure 2020-07-07T14:14:02.714-0500 I - [WTJournalFlusher] Fatal Assertion 28559 2020-07-07T14:14:02.714-0500 I - [WTJournalFlusher] ***aborting after fassert() failure 2020-07-07T14:14:02.721-0500 F - [repl writer worker 14] Got signal: 6 (Aborted). 0x13075f2 0x1306539 0x1306d42 0x7f7fd77a87e0 0x7f7fd7437625 0x7f7fd7438e05 0x128f4d2 0x1084f43 0x1a4d28c 0x1a4d42d 0x1a4d814 0x198929d 0x198a424 0x198a673 0x1987787 0x198b43e 0x19a7518 0x19d8c90 0x1a4c08b 0x1a4c586 0x1a4c6bb 0x19e6f58 0x1a49b55 0x19fcaeb 0x1a499d9 0x1a49f69 0x107f6a4 0x107bbf0 0x1063ef0 0xc844af 0xaccaf9 0xad0e9f 0xad0f40 0xaaf8d5 0xaafc41 0xaafdeb 0xe98e7d 0xf2f47e 0xf292f0 0xf2b076 0xf2bc16 0xf2edbb 0x1299d9e 0x129a6f9 0x129b250 0x1abe690 0x7f7fd77a0aa1 0x7f7fd74ed93d ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"400000","o":"F075F2"},{"b":"400000","o":"F06539"},{"b":"400000","o":"F06D42"},{"b":"7F7FD7799000","o":"F7E0"},{"b":"7F7FD7405000","o":"32625"},{"b":"7F7FD7405000","o":"33E05"},{"b":"400000","o":"E8F4D2"},{"b":"400000","o":"C84F43"},{"b":"400000","o":"164D28C"},{"b":"400000","o":"164D42D"},{"b":"400000","o":"164D814"},{"b":"400000","o":"158929D"},{"b":"400000","o":"158A424"},{"b":"400000","o":"158A673"},{"b":"400000","o":"1587787"},{"b":"400000","o":"158B43E"},{"b":"400000","o":"15A7518"},{"b":"400000","o":"15D8C90"},{"b":"400000","o":"164C08B"},{"b":"400000","o":"164C586"},{"b":"400000","o":"164C6BB"},{"b":"400000","o":"15E6F58"},{"b":"400000","o":"1649B55"},{"b":"400000","o":"15FCAEB"},{"b":"400000","o":"16499D9"},{"b":"400000","o":"1649F69"},{"b":"400000","o":"C7F6A4"},{"b":"400000","o":"C7BBF0"},{"b":"400000","o":"C63EF0"},{"b":"400000","o":"8844AF"},{"b":"400000","o":"6CCAF9"},{"b":"400000","o":"6D0E9F"},{"b":"400000","o":"6D0F40"},{"b":"400000","o":"6AF8D5"},{"b":"400000","o":"6AFC41"},{"b":"400000","o":"6AFDEB"},{"b":"400000","o":"A98E7D"},{"b":"400000","o":"B2F47E"},{"b":"400000","o":"B292F0"},{"b":"400000","o":"B2B076"},{"b":"400000","o":"B2BC16"},{"b":"400000","o":"B2EDBB"},{"b":"400000","o":"E99D9E"},{"b":"400000","o":"E9A6F9"},{"b":"400000","o":"E9B250"},{"b":"400000","o":"16BE690"},{"b":"7F7FD7799000","o":"7AA1"},{"b":"7F7FD7405000","o":"E893D"}],"processInfo":{ "mongodbVersion" : "3.2.1", "gitVersion" : "a14d55980c2cdc565d4704a7e3ad37e4e535c1b2", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "2.6.32-573.7.1.el6.x86_64", "version" : "#1 SMP Thu Sep 10 13:42:16 EDT 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "C8721EDBA2A3AED1EC436921AEA52A4B63C72F30" }, { "b" : "7FFC2C0B7000", "elfType" : 3, "buildId" : "2426D85978796C7ED259CDC601A7C310C339A21C" }, { "b" : "7F7FD8946000", "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "7C5A504A21B221F299B1C45B9ED9C2340AEC6AEB" }, { "b" : "7F7FD8562000", "path" : "/usr/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "2CF03CE94B9388A10544E4EF073450851A4D6AEB" }, { "b" : "7F7FD835A000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "95159178F1A4A3DBDC7819FBEA2C80E5FCDD6BAC" }, { "b" : "7F7FD8156000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "29B61382141595ECBA6576232E44F2310C3AAB72" }, { "b" : "7F7FD7E50000", "path" : "/usr/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "28AF9321EBEA9D172CA43E11A60E02D0F7014870" }, { "b" : "7F7FD7BCC000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "989FE3A42CA8CEBDCC185A743896F23A0CF537ED" }, { "b" : "7F7FD79B6000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "2AC15B051D1B8B53937E3341EA931D0E96F745D9" }, { "b" : "7F7FD7799000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "C56DD1B811FC0D9263248EBB308C73FCBCD80FC1" }, { "b" : "7F7FD7405000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "A1DB9754D1F523A6F16ADA929D6764A133DC6FA2" }, { "b" : "7F7FD8BB2000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "959C5E10A47EE8A633E7681B64B4B9F74E242ED5" }, { "b" : "7F7FD71C1000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "0C72521270790A1BD52C8F6B989EEA5A575085BF" }, { "b" : "7F7FD6EDA000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "DC11D5D89BDC77FF242481122D51E5A08DB60DA8" }, { "b" : "7F7FD6CD6000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "13FFCD68952B7715DDF34C9321D82E3041EA9006" }, { "b" : "7F7FD6AAA000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "15782495E3AF093E67DDAE9A86436FFC6B3CC4D3" }, { "b" : "7F7FD6894000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "D053BB4FF0C2FC983842F81598813B9B931AD0D1" }, { "b" : "7F7FD6689000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "44A3A1C1891B4C8170C3DB80E7117A022E5EECD0" }, { "b" : "7F7FD6486000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "3BCCABE75DC61BBA81AAE45D164E26EF4F9F55DB" }, { "b" : "7F7FD626C000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "C39D7FFB49DFB1B55AD09D1D711AD802123F6623" }, { "b" : "7F7FD604D000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "2D0F26E648D9661ABD83ED8B4BBE8F2CFA50393B" } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x13075f2] mongod(+0xF06539) [0x1306539] mongod(+0xF06D42) [0x1306d42] libpthread.so.0(+0xF7E0) [0x7f7fd77a87e0] libc.so.6(gsignal+0x35) [0x7f7fd7437625] libc.so.6(abort+0x175) [0x7f7fd7438e05] mongod(_ZN5mongo13fassertFailedEi+0x82) [0x128f4d2] mongod(+0xC84F43) [0x1084f43] mongod(__wt_eventv+0x40C) [0x1a4d28c] mongod(__wt_err+0x8D) [0x1a4d42d] mongod(__wt_panic+0x24) [0x1a4d814] mongod(+0x158929D) [0x198929d] mongod(__wt_block_extlist_read+0x394) [0x198a424] mongod(__wt_block_extlist_read_avail+0x33) [0x198a673] mongod(__wt_block_checkpoint_load+0x3B7) [0x1987787] mongod(+0x158B43E) [0x198b43e] mongod(__wt_btree_open+0xC68) [0x19a7518] mongod(__wt_conn_btree_open+0x140) [0x19d8c90] mongod(__wt_session_get_btree+0xEB) [0x1a4c08b] mongod(__wt_session_get_btree+0x5E6) [0x1a4c586] mongod(__wt_session_get_btree_ckpt+0xAB) [0x1a4c6bb] mongod(__wt_curfile_open+0x208) [0x19e6f58] mongod(+0x1649B55) [0x1a49b55] mongod(__wt_curtable_open+0x34B) [0x19fcaeb] mongod(+0x16499D9) [0x1a499d9] mongod(+0x1649F69) [0x1a49f69] mongod(_ZN5mongo17WiredTigerSession9getCursorERKSsmb+0x94) [0x107f6a4] mongod(_ZN5mongo16WiredTigerCursorC1ERKSsmbPNS_16OperationContextE+0x50) [0x107bbf0] mongod(_ZN5mongo15WiredTigerIndex6insertEPNS_16OperationContextERKNS_7BSONObjERKNS_8RecordIdEb+0xA0) [0x1063ef0] mongod(_ZN5mongo17IndexAccessMethod6insertEPNS_16OperationContextERKNS_7BSONObjERKNS_8RecordIdERKNS_19InsertDeleteOptionsEPl+0x18F) [0xc844af] mongod(_ZN5mongo12IndexCatalog21_indexFilteredRecordsEPNS_16OperationContextEPNS_17IndexCatalogEntryERKSt6vectorINS_10BsonRecordESaIS6_EE+0x109) [0xaccaf9] mongod(_ZN5mongo12IndexCatalog13_indexRecordsEPNS_16OperationContextEPNS_17IndexCatalogEntryERKSt6vectorINS_10BsonRecordESaIS6_EE+0x11F) [0xad0e9f] mongod(_ZN5mongo12IndexCatalog12indexRecordsEPNS_16OperationContextERKSt6vectorINS_10BsonRecordESaIS4_EE+0x80) [0xad0f40] mongod(_ZN5mongo10Collection16_insertDocumentsEPNS_16OperationContextEN9__gnu_cxx17__normal_iteratorIPKNS_7BSONObjESt6vectorIS5_SaIS5_EEEESB_b+0x325) [0xaaf8d5] mongod(_ZN5mongo10Collection15insertDocumentsEPNS_16OperationContextEN9__gnu_cxx17__normal_iteratorIPKNS_7BSONObjESt6vectorIS5_SaIS5_EEEESB_bb+0x1B1) [0xaafc41] mongod(_ZN5mongo10Collection14insertDocumentEPNS_16OperationContextERKNS_7BSONObjEbb+0x6B) [0xaafdeb] mongod(_ZN5mongo4repl21applyOperation_inlockEPNS_16OperationContextEPNS_8DatabaseERKNS_7BSONObjEb+0xF1D) [0xe98e7d] mongod(_ZNSt17_Function_handlerIFN5mongo6StatusEPNS0_16OperationContextEPNS0_8DatabaseERKNS0_7BSONObjEbEPS9_E9_M_invokeERKSt9_Any_dataS3_S5_S8_b+0x1E) [0xf2f47e] mongod(+0xB292F0) [0xf292f0] mongod(_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEbSt8functionIFNS_6StatusES3_PNS_8DatabaseES6_bEES7_IFS8_S3_S6_EES7_IFvvEE+0x336) [0xf2b076] mongod(_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEb+0xE6) [0xf2bc16] mongod(_ZN5mongo4repl14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x9B) [0xf2edbb] mongod(_ZN5mongo10ThreadPool10_doOneTaskEPSt11unique_lockISt5mutexE+0x11E) [0x1299d9e] mongod(_ZN5mongo10ThreadPool13_consumeTasksEv+0xA9) [0x129a6f9] mongod(_ZN5mongo10ThreadPool17_workerThreadBodyEPS0_RKSs+0x100) [0x129b250] mongod(+0x16BE690) [0x1abe690] libpthread.so.0(+0x7AA1) [0x7f7fd77a0aa1] libc.so.6(clone+0x6D) [0x7f7fd74ed93d] ----- END BACKTRACE -----