2018-06-05T13:09:43.798+0000 I CONTROL [conn160] pid=14221 port=27018 64-bit host=cmodb889.abc.com 2018-06-05T13:09:43.798+0000 I CONTROL [conn160] Replica Set Config: { _id: "shard0005", version: 9, protocolVersion: 1, members: [ { _id: 0, host: "cmodb881.abc.com:27018", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 2.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 1, host: "cmodb889.abc.com:27018", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 2, host: "cmodb875.abc.com:27027", arbiterOnly: true, buildIndexes: true, hidden: false, priority: 0.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 3, host: "cmodb899.abc.com:27027", arbiterOnly: true, buildIndexes: true, hidden: false, priority: 0.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, catchUpTimeoutMillis: -1, catchUpTakeoverDelayMillis: 30000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 }, replicaSetId: ObjectId('5b0ec21ab9273eb1421a2e09') } } 2018-06-05T13:09:43.798+0000 I CONTROL [conn160] Replica Set Member State: SECONDARY 2018-06-05T13:09:43.798+0000 I CONTROL [conn160] db version v3.6.5 2018-06-05T13:09:43.798+0000 I CONTROL [conn160] git version: a20ecd3e3a174162052ff99913bc2ca9a839d618 2018-06-05T13:09:43.798+0000 I CONTROL [conn160] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013 2018-06-05T13:09:43.798+0000 I CONTROL [conn160] allocator: tcmalloc 2018-06-05T13:09:43.798+0000 I CONTROL [conn160] modules: none 2018-06-05T13:09:43.798+0000 I CONTROL [conn160] build environment: 2018-06-05T13:09:43.798+0000 I CONTROL [conn160] distmod: rhel70 2018-06-05T13:09:43.798+0000 I CONTROL [conn160] distarch: x86_64 2018-06-05T13:09:43.798+0000 I CONTROL [conn160] target_arch: x86_64 2018-06-05T13:09:43.798+0000 I CONTROL [conn160] options: { config: "/etc/mongod.conf", net: { bindIpAll: true, port: 27018, ssl: { CAFile: "…", PEMKeyFile: "...", allowInvalidCertificates: true, mode: "preferSSL" } }, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid" }, replication: { replSetName: "shard0005" }, security: { clusterAuthMode: "x509" }, sharding: { clusterRole: "shardsvr" }, storage: { dbPath: "/data/db", directoryPerDB: true, wiredTiger: { engineConfig: { cacheSizeGB: 45.0, directoryForIndexes: true } } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } } 2018-06-05T13:09:43.801+0000 I NETWORK [conn160] end connection 10.120.10.29:39486 (49 connections now open) 2018-06-05T13:10:00.011+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:11:00.013+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:12:00.048+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:13:00.010+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:13:53.580+0000 I ASIO [NetworkInterfaceASIO-RS-0] Ending idle connection to host cmodb881.abc.com:27018 because the pool meets constraints; 2 connections to that host remain open 2018-06-05T13:14:00.010+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:14:12.946+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to cmodb881.abc.com:27018 2018-06-05T13:14:12.970+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to cmodb881.abc.com:27018, took 24ms (3 connections now open to cmodb881.abc.com:27018) 2018-06-05T13:14:40.880+0000 I SHARDING [thread38] Refreshing chunks for collection config.system.sessions based on version 1|0||5b0eb8b9c81053954f8d7232 2018-06-05T13:14:40.887+0000 I SHARDING [ShardServerCatalogCacheLoader-1169] Refresh for collection config.system.sessions took 7 ms and found version 1|0||5b0eb8b9c81053954f8d7232 2018-06-05T13:15:00.013+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:16:00.012+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:17:00.010+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:18:00.010+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:18:32.811+0000 I ASIO [NetworkInterfaceASIO-RS-0] Ending idle connection to host cmodb881.abc.com:27018 because the pool meets constraints; 2 connections to that host remain open 2018-06-05T13:18:43.605+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to cmodb881.abc.com:27018 2018-06-05T13:18:43.629+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to cmodb881.abc.com:27018, took 24ms (3 connections now open to cmodb881.abc.com:27018) 2018-06-05T13:19:00.010+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:19:40.880+0000 I SHARDING [thread38] Refreshing chunks for collection config.system.sessions based on version 1|0||5b0eb8b9c81053954f8d7232 2018-06-05T13:19:40.899+0000 I SHARDING [ShardServerCatalogCacheLoader-1170] Refresh for collection config.system.sessions took 18 ms and found version 1|0||5b0eb8b9c81053954f8d7232 2018-06-05T13:20:00.016+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:21:00.011+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:22:00.016+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:22:57.882+0000 I COMMAND [repl writer worker 5] CMD: drop M_20180604.tmp.mrs.uimsi_d_1528201798_4 2018-06-05T13:22:57.882+0000 I STORAGE [repl writer worker 5] dropCollection: M_20180604.tmp.mrs.uimsi_d_1528201798_4 (251885d1-90ea-4355-a320-cc2fb117aefb) - renaming to drop-pending collection: M_20180604.system.drop.1528204977i12587t6.tmp.mrs.uimsi_d_1528201798_4 with drop optime { ts: Timestamp(1528204977, 12587), t: 6 } 2018-06-05T13:22:57.893+0000 I REPL [replication-415] Completing collection drop for M_20180604.system.drop.1528204977i12587t6.tmp.mrs.uimsi_d_1528201798_4 with drop optime { ts: Timestamp(1528204977, 12587), t: 6 } (notification optime: { ts: Timestamp(1528204977, 12587), t: 6 }) 2018-06-05T13:22:57.893+0000 I STORAGE [replication-415] Finishing collection drop for M_20180604.system.drop.1528204977i12587t6.tmp.mrs.uimsi_d_1528201798_4 (251885d1-90ea-4355-a320-cc2fb117aefb). 2018-06-05T13:23:00.009+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:24:00.009+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:24:40.880+0000 I SHARDING [thread38] Refreshing chunks for collection config.system.sessions based on version 1|0||5b0eb8b9c81053954f8d7232 2018-06-05T13:24:40.888+0000 I SHARDING [ShardServerCatalogCacheLoader-1171] Refresh for collection config.system.sessions took 8 ms and found version 1|0||5b0eb8b9c81053954f8d7232 2018-06-05T13:25:00.010+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:26:00.012+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:27:00.010+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:28:00.011+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:28:01.989+0000 I ASIO [NetworkInterfaceASIO-RS-0] Ending idle connection to host cmodb881.abc.com:27018 because the pool meets constraints; 2 connections to that host remain open 2018-06-05T13:29:00.010+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:29:10.593+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to cmodb881.abc.com:27018 2018-06-05T13:29:10.616+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to cmodb881.abc.com:27018, took 23ms (3 connections now open to cmodb881.abc.com:27018) 2018-06-05T13:29:40.880+0000 I SHARDING [thread38] Refreshing chunks for collection config.system.sessions based on version 1|0||5b0eb8b9c81053954f8d7232 2018-06-05T13:29:40.901+0000 I SHARDING [ShardServerCatalogCacheLoader-1172] Refresh for collection config.system.sessions took 20 ms and found version 1|0||5b0eb8b9c81053954f8d7232 2018-06-05T13:30:00.013+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:31:00.014+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:32:00.016+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:33:00.010+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:34:00.014+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:34:40.880+0000 I SHARDING [thread38] Refreshing chunks for collection config.system.sessions based on version 1|0||5b0eb8b9c81053954f8d7232 2018-06-05T13:34:40.887+0000 I SHARDING [ShardServerCatalogCacheLoader-1173] Refresh for collection config.system.sessions took 7 ms and found version 1|0||5b0eb8b9c81053954f8d7232 2018-06-05T13:35:00.010+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:36:00.010+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:37:00.013+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:37:45.283+0000 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Connecting to cmodb874.abc.com:27019 2018-06-05T13:37:45.306+0000 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to cmodb874.abc.com:27019, took 23ms (1 connections now open to cmodb874.abc.com:27019) 2018-06-05T13:38:00.011+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:38:19.706+0000 I ASIO [NetworkInterfaceASIO-RS-0] Ending idle connection to host cmodb881.abc.com:27018 because the pool meets constraints; 2 connections to that host remain open 2018-06-05T13:39:00.011+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:39:04.690+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to cmodb881.abc.com:27018 2018-06-05T13:39:04.714+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to cmodb881.abc.com:27018, took 24ms (3 connections now open to cmodb881.abc.com:27018) 2018-06-05T13:39:40.880+0000 I SHARDING [thread38] Refreshing chunks for collection config.system.sessions based on version 1|0||5b0eb8b9c81053954f8d7232 2018-06-05T13:39:40.887+0000 I SHARDING [ShardServerCatalogCacheLoader-1174] Refresh for collection config.system.sessions took 7 ms and found version 1|0||5b0eb8b9c81053954f8d7232 2018-06-05T13:40:00.009+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:41:00.013+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:42:00.010+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:43:00.010+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:44:00.008+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:44:40.880+0000 I SHARDING [thread39] Refreshing chunks for collection config.system.sessions based on version 1|0||5b0eb8b9c81053954f8d7232 2018-06-05T13:44:40.887+0000 I SHARDING [ShardServerCatalogCacheLoader-1175] Refresh for collection config.system.sessions took 7 ms and found version 1|0||5b0eb8b9c81053954f8d7232 2018-06-05T13:45:00.013+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:46:00.015+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:47:00.009+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:48:00.008+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:48:37.244+0000 I ASIO [NetworkInterfaceASIO-RS-0] Ending idle connection to host cmodb881.abc.com:27018 because the pool meets constraints; 2 connections to that host remain open 2018-06-05T13:48:57.247+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to cmodb881.abc.com:27018 2018-06-05T13:48:57.275+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to cmodb881.abc.com:27018, took 28ms (3 connections now open to cmodb881.abc.com:27018) 2018-06-05T13:49:00.013+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:49:40.880+0000 I SHARDING [thread39] Refreshing chunks for collection config.system.sessions based on version 1|0||5b0eb8b9c81053954f8d7232 2018-06-05T13:49:40.888+0000 I SHARDING [ShardServerCatalogCacheLoader-1176] Refresh for collection config.system.sessions took 7 ms and found version 1|0||5b0eb8b9c81053954f8d7232 2018-06-05T13:50:00.010+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:51:00.019+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:52:00.008+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:53:00.015+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:53:05.433+0000 I STORAGE [repl writer worker 3] createCollection: config.cache.chunks.M_20180604.uimsi_d_a3 with provided UUID: 21135217-78cb-4acd-875e-45ef0a375ffe 2018-06-05T13:53:05.445+0000 I INDEX [repl writer worker 1] build index on: config.cache.chunks.M_20180604.uimsi_d_a3 properties: { v: 2, key: { lastmod: 1 }, name: "lastmod_1", ns: "config.cache.chunks.M_20180604.uimsi_d_a3" } 2018-06-05T13:53:05.445+0000 I INDEX [repl writer worker 1] building index using bulk method; build may temporarily use up to 500 megabytes of RAM 2018-06-05T13:53:05.446+0000 I INDEX [repl writer worker 1] build index done. scanned 0 total records. 0 secs 2018-06-05T13:53:05.462+0000 I STORAGE [repl writer worker 7] createCollection: M_20180604.tmp.mr.tmp.uimsi_d_a3_4 with provided UUID: fe38a916-efde-41c0-aeab-61205502b8df 2018-06-05T13:53:28.945+0000 I COMMAND [repl writer worker 13] renameCollection: renaming collection fe38a916-efde-41c0-aeab-61205502b8df from M_20180604.tmp.mr.tmp.uimsi_d_a3_4 to M_20180604.uimsi_d_a3 2018-06-05T13:53:49.613+0000 I INDEX [repl index builder 314] build index on: M_20180604.uimsi_d_a3 properties: { v: 2, key: { _id.imsi: 1.0 }, name: "_id.imsi_1", background: true, ns: "M_20180604.uimsi_d_a3" } 2018-06-05T13:53:49.637+0000 I INDEX [repl index builder 315] build index on: M_20180604.uimsi_d_a3 properties: { v: 2, key: { _id.hpmn: 1.0, _id.vpmn: 1.0 }, name: "_id.hpmn_1__id.vpmn_1", background: true, ns: "M_20180604.uimsi_d_a3" } 2018-06-05T13:53:49.653+0000 I INDEX [repl index builder 316] build index on: M_20180604.uimsi_d_a3 properties: { v: 2, key: { _id.vpmn: 1.0, _id.hpmn: 1.0 }, name: "_id.vpmn_1__id.hpmn_1", background: true, ns: "M_20180604.uimsi_d_a3" } 2018-06-05T13:53:52.000+0000 I - [repl index builder 315] Index Build (background): 259500/655788 39% 2018-06-05T13:53:52.000+0000 I - [repl index builder 316] Index Build (background): 258700/655788 39% 2018-06-05T13:53:52.001+0000 I - [repl index builder 314] Index Build (background): 298000/655788 45% 2018-06-05T13:53:54.013+0000 I INDEX [repl index builder 314] build index done. scanned 655788 total records. 4 secs 2018-06-05T13:53:55.000+0000 I - [repl index builder 315] Index Build (background): 610600/655788 93% 2018-06-05T13:53:55.001+0000 I - [repl index builder 316] Index Build (background): 618600/655788 94% 2018-06-05T13:53:55.296+0000 I INDEX [repl index builder 316] build index done. scanned 655788 total records. 5 secs 2018-06-05T13:53:55.307+0000 I INDEX [repl index builder 315] build index done. scanned 655788 total records. 5 secs 2018-06-05T13:54:00.010+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:54:40.880+0000 I SHARDING [thread39] Refreshing chunks for collection config.system.sessions based on version 1|0||5b0eb8b9c81053954f8d7232 2018-06-05T13:54:40.888+0000 I SHARDING [ShardServerCatalogCacheLoader-1177] Refresh for collection config.system.sessions took 7 ms and found version 1|0||5b0eb8b9c81053954f8d7232 2018-06-05T13:55:00.010+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:56:00.012+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:57:00.011+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:58:00.013+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:58:28.315+0000 I ASIO [NetworkInterfaceASIO-RS-0] Ending idle connection to host cmodb881.abc.com:27018 because the pool meets constraints; 2 connections to that host remain open 2018-06-05T13:58:49.772+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to cmodb881.abc.com:27018 2018-06-05T13:58:49.796+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to cmodb881.abc.com:27018, took 23ms (3 connections now open to cmodb881.abc.com:27018) 2018-06-05T13:59:00.012+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T13:59:40.880+0000 I SHARDING [thread38] Refreshing chunks for collection config.system.sessions based on version 1|0||5b0eb8b9c81053954f8d7232 2018-06-05T13:59:40.887+0000 I SHARDING [ShardServerCatalogCacheLoader-1178] Refresh for collection config.system.sessions took 7 ms and found version 1|0||5b0eb8b9c81053954f8d7232 2018-06-05T14:00:00.026+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:01:00.012+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:02:00.013+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:03:00.010+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:03:19.359+0000 I ASIO [NetworkInterfaceASIO-RS-0] Ending idle connection to host cmodb881.abc.com:27018 because the pool meets constraints; 2 connections to that host remain open 2018-06-05T14:03:58.848+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to cmodb881.abc.com:27018 2018-06-05T14:03:58.871+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to cmodb881.abc.com:27018, took 23ms (3 connections now open to cmodb881.abc.com:27018) 2018-06-05T14:04:00.016+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:04:40.880+0000 I SHARDING [thread38] Refreshing chunks for collection config.system.sessions based on version 1|0||5b0eb8b9c81053954f8d7232 2018-06-05T14:04:40.887+0000 I SHARDING [ShardServerCatalogCacheLoader-1179] Refresh for collection config.system.sessions took 7 ms and found version 1|0||5b0eb8b9c81053954f8d7232 2018-06-05T14:05:00.013+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:06:00.015+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:07:00.014+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:07:45.554+0000 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Connecting to cmodb874.abc.com:27019 2018-06-05T14:07:45.573+0000 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to cmodb874.abc.com:27019, took 19ms (1 connections now open to cmodb874.abc.com:27019) 2018-06-05T14:08:00.013+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:08:47.743+0000 I ASIO [NetworkInterfaceASIO-RS-0] Ending idle connection to host cmodb881.abc.com:27018 because the pool meets constraints; 2 connections to that host remain open 2018-06-05T14:09:00.009+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:09:18.090+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to cmodb881.abc.com:27018 2018-06-05T14:09:18.113+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to cmodb881.abc.com:27018, took 24ms (3 connections now open to cmodb881.abc.com:27018) 2018-06-05T14:09:40.880+0000 I SHARDING [thread38] Refreshing chunks for collection config.system.sessions based on version 1|0||5b0eb8b9c81053954f8d7232 2018-06-05T14:09:40.887+0000 I SHARDING [ShardServerCatalogCacheLoader-1180] Refresh for collection config.system.sessions took 7 ms and found version 1|0||5b0eb8b9c81053954f8d7232 2018-06-05T14:10:00.012+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:11:00.070+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:12:00.010+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:13:00.009+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:13:47.295+0000 I ASIO [NetworkInterfaceASIO-RS-0] Ending idle connection to host cmodb881.abc.com:27018 because the pool meets constraints; 2 connections to that host remain open 2018-06-05T14:14:00.010+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:14:10.093+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to cmodb881.abc.com:27018 2018-06-05T14:14:10.116+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to cmodb881.abc.com:27018, took 23ms (3 connections now open to cmodb881.abc.com:27018) 2018-06-05T14:14:40.880+0000 I SHARDING [thread38] Refreshing chunks for collection config.system.sessions based on version 1|0||5b0eb8b9c81053954f8d7232 2018-06-05T14:14:40.887+0000 I SHARDING [ShardServerCatalogCacheLoader-1181] Refresh for collection config.system.sessions took 7 ms and found version 1|0||5b0eb8b9c81053954f8d7232 2018-06-05T14:15:00.012+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:16:00.011+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:16:15.638+0000 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Connecting to cmodb874.abc.com:27019 2018-06-05T14:16:15.662+0000 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to cmodb874.abc.com:27019, took 24ms (1 connections now open to cmodb874.abc.com:27019) 2018-06-05T14:17:00.019+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:17:22.032+0000 I STORAGE [repl writer worker 14] createCollection: M_20180602.uimsi_d_a3 with provided UUID: feb46c4e-28c0-4799-b9d0-391aae8bf0e2 2018-06-05T14:17:22.055+0000 I STORAGE [repl writer worker 6] createCollection: M_20180602.tmp.uimsi_d_a3 with provided UUID: 16e14cce-7a36-4e3d-850c-0e630dedaea3 2018-06-05T14:17:22.157+0000 I STORAGE [repl writer worker 3] createCollection: M_20180602.tmp.mr.uimsi_d_5 with provided UUID: 33bf3bea-1e62-4303-943e-2590644ee37c 2018-06-05T14:18:00.012+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:18:43.951+0000 I ASIO [NetworkInterfaceASIO-RS-0] Ending idle connection to host cmodb881.abc.com:27018 because the pool meets constraints; 2 connections to that host remain open 2018-06-05T14:18:44.476+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to cmodb881.abc.com:27018 2018-06-05T14:18:44.499+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to cmodb881.abc.com:27018, took 23ms (3 connections now open to cmodb881.abc.com:27018) 2018-06-05T14:19:00.010+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:19:40.880+0000 I SHARDING [thread38] Refreshing chunks for collection config.system.sessions based on version 1|0||5b0eb8b9c81053954f8d7232 2018-06-05T14:19:40.888+0000 I SHARDING [ShardServerCatalogCacheLoader-1182] Refresh for collection config.system.sessions took 7 ms and found version 1|0||5b0eb8b9c81053954f8d7232 2018-06-05T14:20:00.014+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:21:00.015+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:22:00.010+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:23:00.011+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:23:49.889+0000 I ASIO [NetworkInterfaceASIO-RS-0] Ending idle connection to host cmodb881.abc.com:27018 because the pool meets constraints; 2 connections to that host remain open 2018-06-05T14:23:52.620+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to cmodb881.abc.com:27018 2018-06-05T14:23:52.638+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to cmodb881.abc.com:27018, took 19ms (3 connections now open to cmodb881.abc.com:27018) 2018-06-05T14:24:00.010+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:24:40.880+0000 I SHARDING [thread38] Refreshing chunks for collection config.system.sessions based on version 1|0||5b0eb8b9c81053954f8d7232 2018-06-05T14:24:40.900+0000 I SHARDING [ShardServerCatalogCacheLoader-1183] Refresh for collection config.system.sessions took 19 ms and found version 1|0||5b0eb8b9c81053954f8d7232 2018-06-05T14:25:00.012+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:26:00.012+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:27:00.010+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:28:00.011+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:29:00.010+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:29:40.880+0000 I SHARDING [thread38] Refreshing chunks for collection config.system.sessions based on version 1|0||5b0eb8b9c81053954f8d7232 2018-06-05T14:29:40.888+0000 I SHARDING [ShardServerCatalogCacheLoader-1184] Refresh for collection config.system.sessions took 7 ms and found version 1|0||5b0eb8b9c81053954f8d7232 2018-06-05T14:30:00.012+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:31:00.010+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:32:00.009+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:33:00.010+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:33:15.549+0000 I ASIO [NetworkInterfaceASIO-RS-0] Ending idle connection to host cmodb881.abc.com:27018 because the pool meets constraints; 2 connections to that host remain open 2018-06-05T14:33:39.411+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to cmodb881.abc.com:27018 2018-06-05T14:33:39.437+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to cmodb881.abc.com:27018, took 26ms (3 connections now open to cmodb881.abc.com:27018) 2018-06-05T14:34:00.014+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:34:40.880+0000 I SHARDING [thread39] Refreshing chunks for collection config.system.sessions based on version 1|0||5b0eb8b9c81053954f8d7232 2018-06-05T14:34:40.887+0000 I SHARDING [ShardServerCatalogCacheLoader-1185] Refresh for collection config.system.sessions took 7 ms and found version 1|0||5b0eb8b9c81053954f8d7232 2018-06-05T14:35:00.013+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:36:00.009+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:37:00.011+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:38:00.010+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:39:00.008+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:39:40.880+0000 I SHARDING [thread39] Refreshing chunks for collection config.system.sessions based on version 1|0||5b0eb8b9c81053954f8d7232 2018-06-05T14:39:40.888+0000 I SHARDING [ShardServerCatalogCacheLoader-1186] Refresh for collection config.system.sessions took 7 ms and found version 1|0||5b0eb8b9c81053954f8d7232 2018-06-05T14:40:00.010+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:41:00.015+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:42:00.011+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:42:51.562+0000 I ASIO [NetworkInterfaceASIO-RS-0] Ending idle connection to host cmodb881.abc.com:27018 because the pool meets constraints; 2 connections to that host remain open 2018-06-05T14:43:00.008+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:43:49.917+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to cmodb881.abc.com:27018 2018-06-05T14:43:49.936+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to cmodb881.abc.com:27018, took 19ms (3 connections now open to cmodb881.abc.com:27018) 2018-06-05T14:44:00.013+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:44:16.287+0000 I COMMAND [repl writer worker 4] renameCollection: renaming collection 33bf3bea-1e62-4303-943e-2590644ee37c from M_20180602.tmp.mr.uimsi_d_5 to M_20180602.tmp.mrs.uimsi_d_1528208242_6 2018-06-05T14:44:40.880+0000 I SHARDING [thread39] Refreshing chunks for collection config.system.sessions based on version 1|0||5b0eb8b9c81053954f8d7232 2018-06-05T14:44:40.888+0000 I SHARDING [ShardServerCatalogCacheLoader-1187] Refresh for collection config.system.sessions took 7 ms and found version 1|0||5b0eb8b9c81053954f8d7232 2018-06-05T14:45:00.013+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:45:02.375+0000 I STORAGE [repl writer worker 0] createCollection: config.cache.chunks.M_20180602.tmp.uimsi_d_a3 with provided UUID: e22370a7-78de-4356-b10f-4397258a1f6f 2018-06-05T14:45:02.399+0000 I INDEX [repl writer worker 14] build index on: config.cache.chunks.M_20180602.tmp.uimsi_d_a3 properties: { v: 2, key: { lastmod: 1 }, name: "lastmod_1", ns: "config.cache.chunks.M_20180602.tmp.uimsi_d_a3" } 2018-06-05T14:45:02.399+0000 I INDEX [repl writer worker 14] building index using bulk method; build may temporarily use up to 500 megabytes of RAM 2018-06-05T14:45:02.400+0000 I INDEX [repl writer worker 14] build index done. scanned 0 total records. 0 secs 2018-06-05T14:45:03.169+0000 I COMMAND [repl writer worker 4] CMD: drop M_20180602.tmp.mr.uimsi_d_6 2018-06-05T14:45:03.169+0000 I STORAGE [repl writer worker 4] dropCollection: M_20180602.tmp.mr.uimsi_d_6 (16e14cce-7a36-4e3d-850c-0e630dedaea3) - renaming to drop-pending collection: M_20180602.system.drop.1528209903i23797t6.tmp.mr.uimsi_d_6 with drop optime { ts: Timestamp(1528209903, 23797), t: 6 } 2018-06-05T14:45:03.245+0000 I REPL [replication-423] Completing collection drop for M_20180602.system.drop.1528209903i23797t6.tmp.mr.uimsi_d_6 with drop optime { ts: Timestamp(1528209903, 23797), t: 6 } (notification optime: { ts: Timestamp(1528209903, 23797), t: 6 }) 2018-06-05T14:45:03.245+0000 I STORAGE [replication-423] Finishing collection drop for M_20180602.system.drop.1528209903i23797t6.tmp.mr.uimsi_d_6 (16e14cce-7a36-4e3d-850c-0e630dedaea3). 2018-06-05T14:46:00.017+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:46:57.816+0000 I COMMAND [repl writer worker 9] CMD: drop M_20180602.tmp.mrs.uimsi_d_1528208242_6 2018-06-05T14:46:57.816+0000 I STORAGE [repl writer worker 9] dropCollection: M_20180602.tmp.mrs.uimsi_d_1528208242_6 (33bf3bea-1e62-4303-943e-2590644ee37c) - renaming to drop-pending collection: M_20180602.system.drop.1528210018i4369t6.tmp.mrs.uimsi_d_1528208242_6 with drop optime { ts: Timestamp(1528210018, 4369), t: 6 } 2018-06-05T14:46:57.846+0000 I REPL [replication-425] Completing collection drop for M_20180602.system.drop.1528210018i4369t6.tmp.mrs.uimsi_d_1528208242_6 with drop optime { ts: Timestamp(1528210018, 4369), t: 6 } (notification optime: { ts: Timestamp(1528210018, 4369), t: 6 }) 2018-06-05T14:46:57.847+0000 I STORAGE [replication-425] Finishing collection drop for M_20180602.system.drop.1528210018i4369t6.tmp.mrs.uimsi_d_1528208242_6 (33bf3bea-1e62-4303-943e-2590644ee37c). 2018-06-05T14:46:58.016+0000 I STORAGE [repl writer worker 15] createCollection: M_20180602.tmp.mr.tmp.uimsi_d_a3_7 with provided UUID: 6ef141fe-4cf6-435e-9eac-eb48c7db63b9 2018-06-05T14:46:59.222+0000 I COMMAND [repl writer worker 5] renameCollection: renaming collection 6ef141fe-4cf6-435e-9eac-eb48c7db63b9 from M_20180602.tmp.mr.tmp.uimsi_d_a3_7 to M_20180602.tmp.mrs.tmp.uimsi_d_a3_1528210017_7 2018-06-05T14:47:00.012+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:48:00.036+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:49:00.011+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:49:40.881+0000 I SHARDING [thread39] Refreshing chunks for collection config.system.sessions based on version 1|0||5b0eb8b9c81053954f8d7232 2018-06-05T14:49:40.888+0000 I SHARDING [ShardServerCatalogCacheLoader-1188] Refresh for collection config.system.sessions took 7 ms and found version 1|0||5b0eb8b9c81053954f8d7232 2018-06-05T14:50:00.010+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:51:00.016+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:52:00.012+0000 I ACCESS [conn1] Successfully authenticated as principal service_monitor on admin 2018-06-05T14:52:53.464+0000 F REPL [repl writer worker 3] writer worker caught exception: NamespaceNotFound: Failed to apply operation due to missing collection (16e14cce-7a36-4e3d-850c-0e630dedaea3): { ts: Timestamp(1528210373, 667), t: 6, h: 285159385916138329, v: 2, op: "i", ns: "M_20180602.tmp.uimsi_d_a3", ui: UUID("16e14cce-7a36-4e3d-850c-0e630dedaea3"), fromMigrate: true, wall: new Date(1528210373457), o: { _id: { utc: new Date(1527897600000), imsi: "46e783793e1a0491a44cd352be6608835596895d1da3e5197d2cf288f6e4b85a", vpmn: "TZAVC", hpmn: "KENSA" }, value: { msisdn: "2e045846194e8240ff95e78d3d8ad9358182f8c068cfd1b0d73711e18e4e19b1", imei: undefined, hsai_s: 14.0, hul_s: 13.0, hcl_s: 14.0, hcl_f: 13.0, hmtfw_s: 1.0, hmtfw_t: 2.0, hprn_s: 4.0 } } } on: { ts: Timestamp(1528210373, 667), t: 6, h: 285159385916138329, v: 2, op: "i", ns: "M_20180602.tmp.uimsi_d_a3", ui: UUID("16e14cce-7a36-4e3d-850c-0e630dedaea3"), fromMigrate: true, wall: new Date(1528210373457), o: { _id: { utc: new Date(1527897600000), imsi: "46e783793e1a0491a44cd352be6608835596895d1da3e5197d2cf288f6e4b85a", vpmn: "TZAVC", hpmn: "KENSA" }, value: { msisdn: "2e045846194e8240ff95e78d3d8ad9358182f8c068cfd1b0d73711e18e4e19b1", imei: undefined, hsai_s: 14.0, hul_s: 13.0, hcl_s: 14.0, hcl_f: 13.0, hmtfw_s: 1.0, hmtfw_t: 2.0, hprn_s: 4.0 } } } 2018-06-05T14:52:53.464+0000 F - [repl writer worker 3] Fatal assertion 16359 NamespaceNotFound: Failed to apply operation due to missing collection (16e14cce-7a36-4e3d-850c-0e630dedaea3): { ts: Timestamp(1528210373, 667), t: 6, h: 285159385916138329, v: 2, op: "i", ns: "M_20180602.tmp.uimsi_d_a3", ui: UUID("16e14cce-7a36-4e3d-850c-0e630dedaea3"), fromMigrate: true, wall: new Date(1528210373457), o: { _id: { utc: new Date(1527897600000), imsi: "46e783793e1a0491a44cd352be6608835596895d1da3e5197d2cf288f6e4b85a", vpmn: "TZAVC", hpmn: "KENSA" }, value: { msisdn: "2e045846194e8240ff95e78d3d8ad9358182f8c068cfd1b0d73711e18e4e19b1", imei: undefined, hsai_s: 14.0, hul_s: 13.0, hcl_s: 14.0, hcl_f: 13.0, hmtfw_s: 1.0, hmtfw_t: 2.0, hprn_s: 4.0 } } } at src/mongo/db/repl/sync_tail.cpp 1204 2018-06-05T14:52:53.464+0000 F - [repl writer worker 3] ***aborting after fassert() failure Service does a restart automatically but can not resync again: 2018-06-05T14:53:25.363+0000 I CONTROL [main] ***** SERVER RESTARTED ***** 2018-06-05T14:53:25.373+0000 I CONTROL [initandlisten] MongoDB starting : pid=4264 port=27018 dbpath=/data/db 64-bit host=cmodb889.abc.com 2018-06-05T14:53:25.373+0000 I CONTROL [initandlisten] db version v3.6.5 2018-06-05T14:53:25.373+0000 I CONTROL [initandlisten] git version: a20ecd3e3a174162052ff99913bc2ca9a839d618 2018-06-05T14:53:25.373+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013 2018-06-05T14:53:25.373+0000 I CONTROL [initandlisten] allocator: tcmalloc 2018-06-05T14:53:25.373+0000 I CONTROL [initandlisten] modules: none 2018-06-05T14:53:25.373+0000 I CONTROL [initandlisten] build environment: 2018-06-05T14:53:25.373+0000 I CONTROL [initandlisten] distmod: rhel70 2018-06-05T14:53:25.373+0000 I CONTROL [initandlisten] distarch: x86_64 2018-06-05T14:53:25.373+0000 I CONTROL [initandlisten] target_arch: x86_64 2018-06-05T14:53:25.373+0000 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIpAll: true, port: 27018, ssl: { CAFile: "…", PEMKeyFile: "...", allowInvalidCertificates: true, mode: "preferSSL" } }, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid" }, replication: { replSetName: "shard0005" }, security: { clusterAuthMode: "x509" }, sharding: { clusterRole: "shardsvr" }, storage: { dbPath: "/data/db", directoryPerDB: true, wiredTiger: { engineConfig: { cacheSizeGB: 45.0, directoryForIndexes: true } } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } } 2018-06-05T14:53:25.373+0000 W - [initandlisten] Detected unclean shutdown - /data/db/mongod.lock is not empty. 2018-06-05T14:53:25.374+0000 I - [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'. 2018-06-05T14:53:25.374+0000 W STORAGE [initandlisten] Recovering data from the last clean checkpoint. 2018-06-05T14:53:25.374+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=46080M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),cache_cursors=false,log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress), 2018-06-05T14:53:26.146+0000 I STORAGE [initandlisten] WiredTiger message [1528210406:146029][4264:0x7f3f521c3b00], txn-recover: Main recovery loop: starting at 3762/15511168 2018-06-05T14:53:26.146+0000 I STORAGE [initandlisten] WiredTiger message [1528210406:146437][4264:0x7f3f521c3b00], txn-recover: Recovering log 3762 through 3763 2018-06-05T14:53:26.244+0000 I STORAGE [initandlisten] WiredTiger message [1528210406:244687][4264:0x7f3f521c3b00], file:config/index/685--9035914618061057276.wt, txn-recover: Recovering log 3763 through 3763 2018-06-05T14:53:26.307+0000 I STORAGE [initandlisten] WiredTiger message [1528210406:307615][4264:0x7f3f521c3b00], file:config/index/685--9035914618061057276.wt, txn-recover: Set global recovery timestamp: 0 2018-06-05T14:53:27.131+0000 I STORAGE [initandlisten] Starting WiredTigerRecordStoreThread local.oplog.rs 2018-06-05T14:53:27.131+0000 I STORAGE [initandlisten] The size storer reports that the oplog contains 120980897 records totaling to 50568091580 bytes 2018-06-05T14:53:27.131+0000 I STORAGE [initandlisten] Sampling from the oplog between Jun 5 06:14:41:6145 and Jun 5 14:52:53:667 to determine where to place markers for truncation 2018-06-05T14:53:27.131+0000 I STORAGE [initandlisten] Taking 992 samples and assuming that each section of oplog contains approximately 1219195 records totaling to 509604127 bytes 2018-06-05T14:53:36.601+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 06:20:38:1285 2018-06-05T14:53:36.601+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 06:28:46:6376 2018-06-05T14:53:36.601+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 06:35:21:24734 2018-06-05T14:53:36.601+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 06:44:27:7309 2018-06-05T14:53:36.601+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 06:50:06:8945 2018-06-05T14:53:36.601+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 06:54:23:7389 2018-06-05T14:53:36.601+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 06:59:28:29702 2018-06-05T14:53:36.601+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 07:05:34:29367 2018-06-05T14:53:36.601+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 07:10:02:13498 2018-06-05T14:53:36.601+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 07:16:11:10805 2018-06-05T14:53:36.601+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 07:20:20:14064 2018-06-05T14:53:36.601+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 07:29:24:27127 2018-06-05T14:53:36.601+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 07:33:06:3431 2018-06-05T14:53:36.601+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 07:36:04:26286 2018-06-05T14:53:36.601+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 07:39:53:22455 2018-06-05T14:53:36.601+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 07:42:58:7292 2018-06-05T14:53:36.601+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 07:46:02:36249 2018-06-05T14:53:36.601+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 07:47:54:9905 2018-06-05T14:53:36.601+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 07:52:00:39822 2018-06-05T14:53:36.601+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 07:55:21:39640 2018-06-05T14:53:36.601+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 07:58:39:13119 2018-06-05T14:53:36.601+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 08:02:03:12241 2018-06-05T14:53:36.601+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 08:04:00:19457 2018-06-05T14:53:36.601+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 08:06:21:6422 2018-06-05T14:53:36.601+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 08:09:49:29848 2018-06-05T14:53:36.601+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 08:12:45:11671 2018-06-05T14:53:36.601+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 08:16:09:11855 2018-06-05T14:53:36.601+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 08:20:40:9092 2018-06-05T14:53:36.601+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 08:24:04:29837 2018-06-05T14:53:36.601+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 08:26:38:43195 2018-06-05T14:53:36.601+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 08:29:17:21385 2018-06-05T14:53:36.601+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 08:32:49:17787 2018-06-05T14:53:36.601+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 08:35:00:17545 2018-06-05T14:53:36.601+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 08:40:24:9645 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 08:44:38:28773 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 08:48:06:11323 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 08:52:11:8240 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 08:56:48:19375 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 09:00:31:3225 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 09:05:47:8707 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 09:12:17:1290 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 09:19:15:4704 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 09:22:26:3353 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 09:26:22:20728 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 09:31:32:15784 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 09:35:54:14362 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 09:41:34:23459 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 09:44:50:9221 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 09:50:10:42218 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 09:55:00:27154 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 09:59:53:7826 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 10:05:36:19728 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 10:10:38:25630 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 10:15:37:19055 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 10:20:24:32116 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 10:25:02:8497 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 10:29:36:5859 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 10:35:00:6590 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 10:40:12:9007 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 10:45:12:38219 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 10:50:53:2618 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 10:58:58:2272 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 11:04:43:7832 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 11:10:27:5768 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 11:19:36:23972 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 11:25:15:15253 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 11:30:35:13834 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 11:36:20:10902 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 11:44:53:22370 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 11:51:52:8304 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 11:59:30:29337 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 12:07:58:7590 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 12:14:01:2222 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 12:19:49:29195 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 12:26:33:19151 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 12:34:18:32279 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 12:37:39:10112 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 12:41:21:14148 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 12:45:55:892 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 12:50:25:1138 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 12:54:30:7998 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 13:01:53:12026 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 13:05:19:31990 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 13:14:18:22625 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 13:21:58:60982 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 13:29:46:25900 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 13:39:23:28964 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 13:46:28:214 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 13:53:23:5097 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 13:59:42:25590 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 14:09:16:24688 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 14:17:08:1099 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 14:22:03:8147 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 14:29:51:30799 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 14:35:35:25225 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 14:39:48:238 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 14:44:06:40454 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 14:46:27:29899 2018-06-05T14:53:36.602+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 14:50:50:6365 2018-06-05T14:53:36.612+0000 I CONTROL [initandlisten] 2018-06-05T14:53:36.612+0000 I CONTROL [initandlisten] ** WARNING: soft rlimits too low. rlimits set to 4096 processes, 32000 files. Number of processes should be at least 16000 : 0.5 times number of files. 2018-06-05T14:53:37.341+0000 I SHARDING [initandlisten] initializing sharding state with: { _id: "shardIdentity", configsvrConnectionString: "rsconfig/cmodb872.abc.com:27019,cmodb873.abc.com:27019,cmodb874.abc.com:27019", shardName: "shard0005", clusterId: ObjectId('5b0ea92df92c698c9cf871e9') } 2018-06-05T14:53:37.341+0000 I SHARDING [initandlisten] first cluster operation detected, adding sharding hook to enable versioning and authentication to remote servers 2018-06-05T14:53:37.342+0000 I NETWORK [initandlisten] Starting new replica set monitor for rsconfig/cmodb872.abc.com:27019,cmodb873.abc.com:27019,cmodb874.abc.com:27019 2018-06-05T14:53:37.343+0000 I SHARDING [thread1] creating distributed lock ping thread for process cmodb889.abc.com:27018:1528210417:-2242984646922103534 (sleeping for 30000ms) 2018-06-05T14:53:37.343+0000 I SHARDING [initandlisten] initialized sharding components for secondary node. 2018-06-05T14:53:37.365+0000 I NETWORK [monitoring keys for HMAC] Successfully connected to cmodb872.abc.com:27019 (1 connections now open to cmodb872.abc.com:27019 with a 5 second timeout) 2018-06-05T14:53:37.369+0000 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Connecting to cmodb872.abc.com:27019 2018-06-05T14:53:37.369+0000 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Connecting to cmodb872.abc.com:27019 2018-06-05T14:53:37.371+0000 I NETWORK [shard registry reload] Successfully connected to cmodb873.abc.com:27019 (1 connections now open to cmodb873.abc.com:27019 with a 5 second timeout) 2018-06-05T14:53:37.375+0000 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to cmodb874.abc.com:27019 (1 connections now open to cmodb874.abc.com:27019 with a 5 second timeout) 2018-06-05T14:53:37.388+0000 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to cmodb872.abc.com:27019, took 19ms (2 connections now open to cmodb872.abc.com:27019) 2018-06-05T14:53:37.388+0000 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Connecting to cmodb872.abc.com:27019 2018-06-05T14:53:37.392+0000 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to cmodb872.abc.com:27019, took 23ms (3 connections now open to cmodb872.abc.com:27019) 2018-06-05T14:53:37.397+0000 I NETWORK [shard registry reload] Starting new replica set monitor for shard0000/cmodb876.abc.com:27018,cmodb884.abc.com:27018 2018-06-05T14:53:37.397+0000 I NETWORK [shard registry reload] Starting new replica set monitor for shard0011/cmodb865.abc.com:27018,cmodb869.abc.com:27018 2018-06-05T14:53:37.398+0000 I NETWORK [shard registry reload] Starting new replica set monitor for shard0001/cmodb877.abc.com:27018,cmodb885.abc.com:27018 2018-06-05T14:53:37.398+0000 I NETWORK [shard registry reload] Starting new replica set monitor for shard0002/cmodb878.abc.com:27018,cmodb886.abc.com:27018 2018-06-05T14:53:37.398+0000 I NETWORK [shard registry reload] Starting new replica set monitor for shard0003/cmodb879.abc.com:27018,cmodb887.abc.com:27018 2018-06-05T14:53:37.398+0000 I NETWORK [shard registry reload] Starting new replica set monitor for shard0004/cmodb880.abc.com:27018,cmodb888.abc.com:27018 2018-06-05T14:53:37.398+0000 I NETWORK [shard registry reload] Starting new replica set monitor for shard0005/cmodb881.abc.com:27018,cmodb889.abc.com:27018 2018-06-05T14:53:37.398+0000 I NETWORK [shard registry reload] Starting new replica set monitor for shard0006/cmodb882.abc.com:27018,cmodb890.abc.com:27018 2018-06-05T14:53:37.398+0000 I NETWORK [shard registry reload] Starting new replica set monitor for shard0007/cmodb883.abc.com:27018,cmodb891.abc.com:27018 2018-06-05T14:53:37.398+0000 I NETWORK [shard registry reload] Starting new replica set monitor for shard0008/cmodb862.abc.com:27018,cmodb866.abc.com:27018 2018-06-05T14:53:37.398+0000 I NETWORK [shard registry reload] Starting new replica set monitor for shard0009/cmodb863.abc.com:27018,cmodb867.abc.com:27018 2018-06-05T14:53:37.398+0000 I NETWORK [shard registry reload] Starting new replica set monitor for shard0010/cmodb864.abc.com:27018,cmodb868.abc.com:27018 2018-06-05T14:53:37.398+0000 I NETWORK [shard registry reload] Starting new replica set monitor for shard0012/cmodb854.abc.com:27018,cmodb858.abc.com:27018 2018-06-05T14:53:37.398+0000 I NETWORK [shard registry reload] Starting new replica set monitor for shard0013/cmodb855.abc.com:27018,cmodb859.abc.com:27018 2018-06-05T14:53:37.398+0000 I NETWORK [shard registry reload] Starting new replica set monitor for shard0014/cmodb856.abc.com:27018,cmodb860.abc.com:27018 2018-06-05T14:53:37.398+0000 I NETWORK [shard registry reload] Starting new replica set monitor for shard0015/cmodb857.abc.com:27018,cmodb861.abc.com:27018 2018-06-05T14:53:37.399+0000 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: LockStateChangeFailed: findAndModify query predicate didn't match any lock document 2018-06-05T14:53:37.415+0000 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to cmodb872.abc.com:27019, took 27ms (3 connections now open to cmodb872.abc.com:27019) 2018-06-05T14:53:37.424+0000 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to cmodb876.abc.com:27018 (1 connections now open to cmodb876.abc.com:27018 with a 5 second timeout) 2018-06-05T14:53:37.432+0000 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to cmodb884.abc.com:27018 (1 connections now open to cmodb884.abc.com:27018 with a 5 second timeout) 2018-06-05T14:53:37.459+0000 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to cmodb865.abc.com:27018 (1 connections now open to cmodb865.abc.com:27018 with a 5 second timeout) 2018-06-05T14:53:37.469+0000 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to cmodb869.abc.com:27018 (1 connections now open to cmodb869.abc.com:27018 with a 5 second timeout) 2018-06-05T14:53:37.491+0000 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to cmodb877.abc.com:27018 (1 connections now open to cmodb877.abc.com:27018 with a 5 second timeout) 2018-06-05T14:53:37.499+0000 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to cmodb885.abc.com:27018 (1 connections now open to cmodb885.abc.com:27018 with a 5 second timeout) 2018-06-05T14:53:37.526+0000 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to cmodb878.abc.com:27018 (1 connections now open to cmodb878.abc.com:27018 with a 5 second timeout) 2018-06-05T14:53:37.534+0000 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to cmodb886.abc.com:27018 (1 connections now open to cmodb886.abc.com:27018 with a 5 second timeout) 2018-06-05T14:53:37.556+0000 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to cmodb879.abc.com:27018 (1 connections now open to cmodb879.abc.com:27018 with a 5 second timeout) 2018-06-05T14:53:37.563+0000 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to cmodb887.abc.com:27018 (1 connections now open to cmodb887.abc.com:27018 with a 5 second timeout) 2018-06-05T14:53:37.591+0000 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to cmodb880.abc.com:27018 (1 connections now open to cmodb880.abc.com:27018 with a 5 second timeout) 2018-06-05T14:53:37.600+0000 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to cmodb888.abc.com:27018 (1 connections now open to cmodb888.abc.com:27018 with a 5 second timeout) 2018-06-05T14:53:37.627+0000 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to cmodb881.abc.com:27018 (1 connections now open to cmodb881.abc.com:27018 with a 5 second timeout) 2018-06-05T14:53:37.631+0000 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 10.120.10.29:27018, in(checking socket for error after poll), reason: Connection refused 2018-06-05T14:53:37.663+0000 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to cmodb882.abc.com:27018 (1 connections now open to cmodb882.abc.com:27018 with a 5 second timeout) 2018-06-05T14:53:37.673+0000 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to cmodb890.abc.com:27018 (1 connections now open to cmodb890.abc.com:27018 with a 5 second timeout) 2018-06-05T14:53:37.695+0000 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to cmodb883.abc.com:27018 (1 connections now open to cmodb883.abc.com:27018 with a 5 second timeout) 2018-06-05T14:53:37.702+0000 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to cmodb891.abc.com:27018 (1 connections now open to cmodb891.abc.com:27018 with a 5 second timeout) 2018-06-05T14:53:37.729+0000 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to cmodb862.abc.com:27018 (1 connections now open to cmodb862.abc.com:27018 with a 5 second timeout) 2018-06-05T14:53:37.737+0000 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to cmodb866.abc.com:27018 (1 connections now open to cmodb866.abc.com:27018 with a 5 second timeout) 2018-06-05T14:53:37.758+0000 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to cmodb863.abc.com:27018 (1 connections now open to cmodb863.abc.com:27018 with a 5 second timeout) 2018-06-05T14:53:37.765+0000 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to cmodb867.abc.com:27018 (1 connections now open to cmodb867.abc.com:27018 with a 5 second timeout) 2018-06-05T14:53:37.798+0000 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to cmodb864.abc.com:27018 (1 connections now open to cmodb864.abc.com:27018 with a 5 second timeout) 2018-06-05T14:53:37.807+0000 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to cmodb868.abc.com:27018 (1 connections now open to cmodb868.abc.com:27018 with a 5 second timeout) 2018-06-05T14:53:37.839+0000 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to cmodb854.abc.com:27018 (1 connections now open to cmodb854.abc.com:27018 with a 5 second timeout) 2018-06-05T14:53:37.849+0000 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to cmodb858.abc.com:27018 (1 connections now open to cmodb858.abc.com:27018 with a 5 second timeout) 2018-06-05T14:53:37.881+0000 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to cmodb855.abc.com:27018 (1 connections now open to cmodb855.abc.com:27018 with a 5 second timeout) 2018-06-05T14:53:37.890+0000 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to cmodb859.abc.com:27018 (1 connections now open to cmodb859.abc.com:27018 with a 5 second timeout) 2018-06-05T14:53:37.917+0000 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to cmodb856.abc.com:27018 (1 connections now open to cmodb856.abc.com:27018 with a 5 second timeout) 2018-06-05T14:53:37.926+0000 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to cmodb860.abc.com:27018 (1 connections now open to cmodb860.abc.com:27018 with a 5 second timeout) 2018-06-05T14:53:37.958+0000 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to cmodb857.abc.com:27018 (1 connections now open to cmodb857.abc.com:27018 with a 5 second timeout) 2018-06-05T14:53:37.968+0000 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to cmodb861.abc.com:27018 (1 connections now open to cmodb861.abc.com:27018 with a 5 second timeout) 2018-06-05T14:53:39.398+0000 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory '/data/db/diagnostic.data' 2018-06-05T14:53:39.400+0000 I REPL [initandlisten] Did not find local voted for document at startup. 2018-06-05T14:53:39.401+0000 I REPL [initandlisten] Rollback ID is 1 2018-06-05T14:53:39.403+0000 I REPL [initandlisten] Removing unapplied entries starting at: { : Timestamp(1528210373, 667) } 2018-06-05T14:53:39.403+0000 I REPL [initandlisten] Starting recovery oplog application at the appliedThrough: { ts: Timestamp(1528210373, 666), t: 6 } 2018-06-05T14:53:39.403+0000 I REPL [initandlisten] No oplog entries to apply for recovery. appliedThrough is at the top of the oplog. 2018-06-05T14:53:39.404+0000 I NETWORK [initandlisten] waiting for connections on port 27018 ssl 2018-06-05T14:53:39.508+0000 I REPL [replexec-0] New replica set config in use: { _id: "shard0005", version: 9, protocolVersion: 1, members: [ { _id: 0, host: "cmodb881.abc.com:27018", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 2.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 1, host: "cmodb889.abc.com:27018", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 2, host: "cmodb875.abc.com:27027", arbiterOnly: true, buildIndexes: true, hidden: false, priority: 0.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 3, host: "cmodb899.abc.com:27027", arbiterOnly: true, buildIndexes: true, hidden: false, priority: 0.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, catchUpTimeoutMillis: -1, catchUpTakeoverDelayMillis: 30000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 }, replicaSetId: ObjectId('5b0ec21ab9273eb1421a2e09') } } 2018-06-05T14:53:39.508+0000 I REPL [replexec-0] This node is cmodb889.abc.com:27018 in the config 2018-06-05T14:53:39.508+0000 I REPL [replexec-0] transition to STARTUP2 from STARTUP 2018-06-05T14:53:39.508+0000 I REPL [replexec-0] Starting replication storage threads 2018-06-05T14:53:39.508+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to cmodb875.abc.com:27027 2018-06-05T14:53:39.509+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to cmodb881.abc.com:27018 2018-06-05T14:53:39.509+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to cmodb899.abc.com:27027 2018-06-05T14:53:39.509+0000 I REPL [replexec-0] transition to RECOVERING from STARTUP2 2018-06-05T14:53:39.509+0000 I REPL [replexec-0] Starting replication fetcher thread 2018-06-05T14:53:39.509+0000 I REPL [replexec-0] Starting replication applier thread 2018-06-05T14:53:39.509+0000 I REPL [replexec-0] Starting replication reporter thread 2018-06-05T14:53:39.510+0000 I REPL [rsSync] transition to SECONDARY from RECOVERING 2018-06-05T14:53:39.533+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Successfully connected to cmodb881.abc.com:27018, took 25ms (1 connections now open to cmodb881.abc.com:27018) 2018-06-05T14:53:39.536+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Successfully connected to cmodb875.abc.com:27027, took 28ms (1 connections now open to cmodb875.abc.com:27027) 2018-06-05T14:53:39.537+0000 I REPL [replexec-2] Member cmodb881.abc.com:27018 is now in state PRIMARY 2018-06-05T14:53:39.541+0000 I REPL [replexec-1] Member cmodb875.abc.com:27027 is now in state ARBITER 2018-06-05T14:53:39.571+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Successfully connected to cmodb899.abc.com:27027, took 63ms (1 connections now open to cmodb899.abc.com:27027) 2018-06-05T14:53:39.583+0000 I REPL [replexec-0] Member cmodb899.abc.com:27027 is now in state ARBITER 2018-06-05T14:53:40.006+0000 I FTDC [ftdc] Unclean full-time diagnostic data capture shutdown detected, found interim file, some metrics may have been lost. OK 2018-06-05T14:53:40.038+0000 I NETWORK [listener] connection accepted from 10.120.10.29:39936 #1 (1 connection now open) 2018-06-05T14:53:40.259+0000 I NETWORK [listener] connection accepted from 10.120.10.15:45972 #2 (2 connections now open) 2018-06-05T14:53:40.269+0000 I NETWORK [conn2] received client metadata from 10.120.10.15:45972 conn2: { driver: { name: "NetworkInterfaceASIO-Replication", version: "3.6.5" }, os: { type: "Linux", name: "CentOS Linux release 7.5.1804 (Core) ", architecture: "x86_64", version: "Kernel 3.10.0-862.3.2.el7.x86_64" } } 2018-06-05T14:53:40.272+0000 I ACCESS [conn2] authenticate db: $external { authenticate: 1, mechanism: "MONGODB-X509", user: "…", $db: "$external" } 2018-06-05T14:53:40.284+0000 I NETWORK [listener] connection accepted from 10.120.10.21:57226 #3 (3 connections now open) 2018-06-05T14:53:40.295+0000 I NETWORK [conn3] received client metadata from 10.120.10.21:57226 conn3: { driver: { name: "NetworkInterfaceASIO-Replication", version: "3.6.5" }, os: { type: "Linux", name: "CentOS Linux release 7.5.1804 (Core) ", architecture: "x86_64", version: "Kernel 3.10.0-862.3.2.el7.x86_64" } } 2018-06-05T14:53:40.299+0000 I ACCESS [conn3] authenticate db: $external { authenticate: 1, mechanism: "MONGODB-X509", user: "…", $db: "$external" } 2018-06-05T14:53:40.510+0000 I REPL [rsBackgroundSync] sync source candidate: cmodb881.abc.com:27018 2018-06-05T14:53:40.510+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to cmodb881.abc.com:27018 2018-06-05T14:53:40.529+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to cmodb881.abc.com:27018, took 19ms (1 connections now open to cmodb881.abc.com:27018) 2018-06-05T14:53:40.535+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to cmodb881.abc.com:27018 2018-06-05T14:53:40.553+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to cmodb881.abc.com:27018, took 18ms (2 connections now open to cmodb881.abc.com:27018) 2018-06-05T14:53:40.744+0000 I NETWORK [listener] connection accepted from 10.20.160.130:46372 #4 (4 connections now open) 2018-06-05T14:53:40.773+0000 I NETWORK [conn4] received client metadata from 10.20.160.130:46372 conn4: { driver: { name: "NetworkInterfaceASIO-Replication", version: "3.6.5" }, os: { type: "Linux", name: "CentOS Linux release 7.5.1804 (Core) ", architecture: "x86_64", version: "Kernel 3.10.0-862.3.2.el7.x86_64" } } 2018-06-05T14:53:40.784+0000 I ACCESS [conn4] authenticate db: $external { authenticate: 1, mechanism: "MONGODB-X509", user: "…", $db: "$external" } 2018-06-05T14:53:40.871+0000 F REPL [repl writer worker 10] writer worker caught exception: NamespaceNotFound: Failed to apply operation due to missing collection (16e14cce-7a36-4e3d-850c-0e630dedaea3): { ts: Timestamp(1528210373, 684), t: 6, h: -5869453682661485531, v: 2, op: "i", ns: "M_20180602.tmp.uimsi_d_a3", ui: UUID("16e14cce-7a36-4e3d-850c-0e630dedaea3"), fromMigrate: true, wall: new Date(1528210373457), o: { _id: { utc: new Date(1527897600000), imsi: "46e7c101c004f89624c28e364b783ee821361149287cc59d271c982feac1cc8f", vpmn: "SGPM1", hpmn: "CHNCT" }, value: { msisdn: undefined, imei: undefined, vsai_f: 2.0, vgsai_f: 2.0 } } } on: { ts: Timestamp(1528210373, 684), t: 6, h: -5869453682661485531, v: 2, op: "i", ns: "M_20180602.tmp.uimsi_d_a3", ui: UUID("16e14cce-7a36-4e3d-850c-0e630dedaea3"), fromMigrate: true, wall: new Date(1528210373457), o: { _id: { utc: new Date(1527897600000), imsi: "46e7c101c004f89624c28e364b783ee821361149287cc59d271c982feac1cc8f", vpmn: "SGPM1", hpmn: "CHNCT" }, value: { msisdn: undefined, imei: undefined, vsai_f: 2.0, vgsai_f: 2.0 } } } 2018-06-05T14:53:40.871+0000 F - [repl writer worker 10] Fatal assertion 16359 NamespaceNotFound: Failed to apply operation due to missing collection (16e14cce-7a36-4e3d-850c-0e630dedaea3): { ts: Timestamp(1528210373, 684), t: 6, h: -5869453682661485531, v: 2, op: "i", ns: "M_20180602.tmp.uimsi_d_a3", ui: UUID("16e14cce-7a36-4e3d-850c-0e630dedaea3"), fromMigrate: true, wall: new Date(1528210373457), o: { _id: { utc: new Date(1527897600000), imsi: "46e7c101c004f89624c28e364b783ee821361149287cc59d271c982feac1cc8f", vpmn: "SGPM1", hpmn: "CHNCT" }, value: { msisdn: undefined, imei: undefined, vsai_f: 2.0, vgsai_f: 2.0 } } } at src/mongo/db/repl/sync_tail.cpp 1204 2018-06-05T14:53:40.871+0000 F REPL [repl writer worker 15] writer worker caught exception: NamespaceNotFound: Failed to apply operation due to missing collection (16e14cce-7a36-4e3d-850c-0e630dedaea3): { ts: Timestamp(1528210373, 697), t: 6, h: 140069530794967834, v: 2, op: "i", ns: "M_20180602.tmp.uimsi_d_a3", ui: UUID("16e14cce-7a36-4e3d-850c-0e630dedaea3"), fromMigrate: true, wall: new Date(1528210373457), o: { _id: { utc: new Date(1527897600000), imsi: "46e7e2ad3736701706d1c233286baa456e2128dcb15066f622938dfb28060ac0", vpmn: "GBRVF", hpmn: "SWEIQ" }, value: { msisdn: "da2c302c05b2417c9aeb2b8cf3e196700ef6d9120cae1aa5538fb4fb59d8cb07", imei: undefined, hsai_s: 5.0, hgsai_s: 4.0, hul_s: 1.0, hgul_s: 1.0 } } } on: { ts: Timestamp(1528210373, 697), t: 6, h: 140069530794967834, v: 2, op: "i", ns: "M_20180602.tmp.uimsi_d_a3", ui: UUID("16e14cce-7a36-4e3d-850c-0e630dedaea3"), fromMigrate: true, wall: new Date(1528210373457), o: { _id: { utc: new Date(1527897600000), imsi: "46e7e2ad3736701706d1c233286baa456e2128dcb15066f622938dfb28060ac0", vpmn: "GBRVF", hpmn: "SWEIQ" }, value: { msisdn: "da2c302c05b2417c9aeb2b8cf3e196700ef6d9120cae1aa5538fb4fb59d8cb07", imei: undefined, hsai_s: 5.0, hgsai_s: 4.0, hul_s: 1.0, hgul_s: 1.0 } } } 2018-06-05T14:53:40.871+0000 F - [repl writer worker 15] Fatal assertion 16359 NamespaceNotFound: Failed to apply operation due to missing collection (16e14cce-7a36-4e3d-850c-0e630dedaea3): { ts: Timestamp(1528210373, 697), t: 6, h: 140069530794967834, v: 2, op: "i", ns: "M_20180602.tmp.uimsi_d_a3", ui: UUID("16e14cce-7a36-4e3d-850c-0e630dedaea3"), fromMigrate: true, wall: new Date(1528210373457), o: { _id: { utc: new Date(1527897600000), imsi: "46e7e2ad3736701706d1c233286baa456e2128dcb15066f622938dfb28060ac0", vpmn: "GBRVF", hpmn: "SWEIQ" }, value: { msisdn: "da2c302c05b2417c9aeb2b8cf3e196700ef6d9120cae1aa5538fb4fb59d8cb07", imei: undefined, hsai_s: 5.0, hgsai_s: 4.0, hul_s: 1.0, hgul_s: 1.0 } } } at src/mongo/db/repl/sync_tail.cpp 1204 2018-06-05T14:53:40.871+0000 F - [repl writer worker 15] ***aborting after fassert() failure 2018-06-05T14:53:54.623+0000 I CONTROL [main] ***** SERVER RESTARTED ***** 2018-06-05T14:53:54.631+0000 I CONTROL [initandlisten] MongoDB starting : pid=6158 port=27018 dbpath=/data/db 64-bit host=cmodb889.abc.com 2018-06-05T14:53:54.631+0000 I CONTROL [initandlisten] db version v3.6.5 2018-06-05T14:53:54.631+0000 I CONTROL [initandlisten] git version: a20ecd3e3a174162052ff99913bc2ca9a839d618 2018-06-05T14:53:54.631+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013 2018-06-05T14:53:54.631+0000 I CONTROL [initandlisten] allocator: tcmalloc 2018-06-05T14:53:54.631+0000 I CONTROL [initandlisten] modules: none 2018-06-05T14:53:54.631+0000 I CONTROL [initandlisten] build environment: 2018-06-05T14:53:54.631+0000 I CONTROL [initandlisten] distmod: rhel70 2018-06-05T14:53:54.631+0000 I CONTROL [initandlisten] distarch: x86_64 2018-06-05T14:53:54.631+0000 I CONTROL [initandlisten] target_arch: x86_64 2018-06-05T14:53:54.631+0000 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIpAll: true, port: 27018, ssl: { CAFile: "…", PEMKeyFile: "…", allowInvalidCertificates: true, mode: "preferSSL" } }, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid" }, replication: { replSetName: "shard0005" }, security: { clusterAuthMode: "x509" }, sharding: { clusterRole: "shardsvr" }, storage: { dbPath: "/data/db", directoryPerDB: true, wiredTiger: { engineConfig: { cacheSizeGB: 45.0, directoryForIndexes: true } } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } } 2018-06-05T14:53:54.631+0000 W - [initandlisten] Detected unclean shutdown - /data/db/mongod.lock is not empty. 2018-06-05T14:53:54.632+0000 I - [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'. 2018-06-05T14:53:54.632+0000 W STORAGE [initandlisten] Recovering data from the last clean checkpoint. 2018-06-05T14:53:54.632+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=46080M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),cache_cursors=false,log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress), 2018-06-05T14:53:55.336+0000 I STORAGE [initandlisten] WiredTiger message [1528210435:336569][6158:0x7f8d5cb69b00], txn-recover: Main recovery loop: starting at 3763/256 2018-06-05T14:53:55.336+0000 I STORAGE [initandlisten] WiredTiger message [1528210435:336918][6158:0x7f8d5cb69b00], txn-recover: Recovering log 3763 through 3764 2018-06-05T14:53:55.539+0000 I STORAGE [initandlisten] WiredTiger message [1528210435:539864][6158:0x7f8d5cb69b00], file:local/collection/27-3489778355046627000.wt, txn-recover: Recovering log 3764 through 3764 2018-06-05T14:53:55.596+0000 I STORAGE [initandlisten] WiredTiger message [1528210435:596616][6158:0x7f8d5cb69b00], file:local/collection/27-3489778355046627000.wt, txn-recover: Set global recovery timestamp: 0 2018-06-05T14:53:55.814+0000 I STORAGE [initandlisten] Starting WiredTigerRecordStoreThread local.oplog.rs 2018-06-05T14:53:55.814+0000 I STORAGE [initandlisten] The size storer reports that the oplog contains 120980897 records totaling to 50568091580 bytes 2018-06-05T14:53:55.814+0000 I STORAGE [initandlisten] Sampling from the oplog between Jun 5 06:14:41:6145 and Jun 5 14:52:54:28469 to determine where to place markers for truncation 2018-06-05T14:53:55.814+0000 I STORAGE [initandlisten] Taking 992 samples and assuming that each section of oplog contains approximately 1219195 records totaling to 509604127 bytes 2018-06-05T14:53:57.434+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 06:24:25:16488 2018-06-05T14:53:57.434+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 06:35:07:1972 2018-06-05T14:53:57.434+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 06:48:15:23265 2018-06-05T14:53:57.434+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 06:54:19:21824 2018-06-05T14:53:57.434+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 07:01:40:12086 2018-06-05T14:53:57.434+0000 I STORAGE [initandlisten] Placing a marker at optime Jun 5 07:09:53:11899