2021-06-10T13:00:03.484+0000 I NETWORK [listener] connection accepted from 10.27.0.14:60526 #87082 (21 connections now open) 2021-06-10T13:00:03.484+0000 I NETWORK [conn87082] received client metadata from 10.27.0.14:60526 conn87082: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T13:00:03.492+0000 I ACCESS [conn87082] Successfully authenticated as principal __system on local from client 10.27.0.14:60526 2021-06-10T13:01:36.308+0000 I NETWORK [conn87078] end connection 10.27.0.12:58168 (20 connections now open) 2021-06-10T13:01:36.356+0000 I NETWORK [listener] connection accepted from 10.27.0.12:58188 #87083 (21 connections now open) 2021-06-10T13:01:36.356+0000 I NETWORK [conn87083] received client metadata from 10.27.0.12:58188 conn87083: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T13:01:36.365+0000 I ACCESS [conn87083] Successfully authenticated as principal __system on local from client 10.27.0.12:58188 2021-06-10T13:04:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Dropping all pooled connections to 10.27.0.12:25002 due to ShutdownInProgress: Pool for 10.27.0.12:25002 has expired. 2021-06-10T13:04:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Dropping all pooled connections to 10.27.0.13:25002 due to ShutdownInProgress: Pool for 10.27.0.13:25002 has expired. 2021-06-10T13:04:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Dropping all pooled connections to 10.27.0.14:25002 due to ShutdownInProgress: Pool for 10.27.0.14:25002 has expired. 2021-06-10T13:04:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Connecting to 10.27.0.13:25002 2021-06-10T13:04:59.116+0000 I CONNPOOL [TaskExecutorPool-0] Connecting to 10.27.0.12:25002 2021-06-10T13:04:59.116+0000 I CONNPOOL [TaskExecutorPool-0] Connecting to 10.27.0.14:25002 2021-06-10T13:07:03.493+0000 I NETWORK [conn87082] end connection 10.27.0.14:60526 (20 connections now open) 2021-06-10T13:07:16.361+0000 I NETWORK [conn87070] end connection 10.27.0.12:58148 (19 connections now open) 2021-06-10T13:08:46.422+0000 I NETWORK [listener] connection accepted from 10.27.0.12:58270 #87087 (20 connections now open) 2021-06-10T13:08:46.422+0000 I NETWORK [conn87087] received client metadata from 10.27.0.12:58270 conn87087: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T13:08:46.431+0000 I ACCESS [conn87087] Successfully authenticated as principal __system on local from client 10.27.0.12:58270 2021-06-10T13:09:33.537+0000 I NETWORK [listener] connection accepted from 10.27.0.14:60552 #87088 (21 connections now open) 2021-06-10T13:09:33.537+0000 I NETWORK [conn87088] received client metadata from 10.27.0.14:60552 conn87088: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T13:09:33.545+0000 I ACCESS [conn87088] Successfully authenticated as principal __system on local from client 10.27.0.14:60552 2021-06-10T13:11:36.390+0000 I NETWORK [conn87083] end connection 10.27.0.12:58188 (20 connections now open) 2021-06-10T13:11:36.400+0000 I NETWORK [listener] connection accepted from 10.27.0.12:58274 #87089 (21 connections now open) 2021-06-10T13:11:36.400+0000 I NETWORK [conn87089] received client metadata from 10.27.0.12:58274 conn87089: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T13:11:36.409+0000 I ACCESS [conn87089] Successfully authenticated as principal __system on local from client 10.27.0.12:58274 2021-06-10T13:11:46.471+0000 I NETWORK [listener] connection accepted from 10.27.0.16:54648 #87090 (22 connections now open) 2021-06-10T13:11:46.471+0000 I NETWORK [conn87090] received client metadata from 10.27.0.16:54648 conn87090: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T13:11:46.482+0000 I ACCESS [conn87090] Successfully authenticated as principal __system on local from client 10.27.0.16:54648 2021-06-10T13:12:46.483+0000 I NETWORK [conn87090] end connection 10.27.0.16:54648 (21 connections now open) 2021-06-10T13:14:46.691+0000 I NETWORK [listener] connection accepted from 10.27.0.16:54660 #87091 (22 connections now open) 2021-06-10T13:14:46.691+0000 I NETWORK [conn87091] received client metadata from 10.27.0.16:54660 conn87091: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T13:14:46.703+0000 I ACCESS [conn87091] Successfully authenticated as principal __system on local from client 10.27.0.16:54660 2021-06-10T13:14:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Dropping all pooled connections to 10.27.0.12:25002 due to ShutdownInProgress: Pool for 10.27.0.12:25002 has expired. 2021-06-10T13:14:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Dropping all pooled connections to 10.27.0.13:25002 due to ShutdownInProgress: Pool for 10.27.0.13:25002 has expired. 2021-06-10T13:14:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Dropping all pooled connections to 10.27.0.14:25002 due to ShutdownInProgress: Pool for 10.27.0.14:25002 has expired. 2021-06-10T13:14:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Connecting to 10.27.0.13:25002 2021-06-10T13:14:59.116+0000 I CONNPOOL [TaskExecutorPool-0] Connecting to 10.27.0.12:25002 2021-06-10T13:14:59.116+0000 I CONNPOOL [TaskExecutorPool-0] Connecting to 10.27.0.14:25002 2021-06-10T13:15:18.848+0000 I NETWORK [conn87077] end connection 10.27.0.14:60508 (21 connections now open) 2021-06-10T13:15:46.703+0000 I NETWORK [conn87091] end connection 10.27.0.16:54660 (20 connections now open) 2021-06-10T13:17:18.873+0000 I NETWORK [listener] connection accepted from 10.27.0.14:60642 #87095 (21 connections now open) 2021-06-10T13:17:18.873+0000 I NETWORK [conn87095] received client metadata from 10.27.0.14:60642 conn87095: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T13:17:18.881+0000 I ACCESS [conn87095] Successfully authenticated as principal __system on local from client 10.27.0.14:60642 2021-06-10T13:19:03.583+0000 I NETWORK [conn87088] end connection 10.27.0.14:60552 (20 connections now open) 2021-06-10T13:19:33.608+0000 I NETWORK [listener] connection accepted from 10.27.0.14:60646 #87096 (21 connections now open) 2021-06-10T13:19:33.608+0000 I NETWORK [conn87096] received client metadata from 10.27.0.14:60646 conn87096: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T13:19:33.616+0000 I ACCESS [conn87096] Successfully authenticated as principal __system on local from client 10.27.0.14:60646 2021-06-10T13:22:36.446+0000 I NETWORK [conn87089] end connection 10.27.0.12:58274 (20 connections now open) 2021-06-10T13:23:06.472+0000 I NETWORK [listener] connection accepted from 10.27.0.12:58300 #87097 (21 connections now open) 2021-06-10T13:23:06.472+0000 I NETWORK [conn87097] received client metadata from 10.27.0.12:58300 conn87097: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T13:23:06.481+0000 I ACCESS [conn87097] Successfully authenticated as principal __system on local from client 10.27.0.12:58300 2021-06-10T13:24:33.616+0000 I NETWORK [conn87096] end connection 10.27.0.14:60646 (20 connections now open) 2021-06-10T13:27:16.488+0000 I NETWORK [conn87087] end connection 10.27.0.12:58270 (19 connections now open) 2021-06-10T13:28:06.481+0000 I NETWORK [conn87097] end connection 10.27.0.12:58300 (18 connections now open) 2021-06-10T13:29:33.637+0000 I NETWORK [listener] connection accepted from 10.27.0.14:60670 #87098 (19 connections now open) 2021-06-10T13:29:33.637+0000 I NETWORK [conn87098] received client metadata from 10.27.0.14:60670 conn87098: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T13:29:33.645+0000 I ACCESS [conn87098] Successfully authenticated as principal __system on local from client 10.27.0.14:60670 2021-06-10T13:29:48.926+0000 I NETWORK [conn87095] end connection 10.27.0.14:60642 (18 connections now open) 2021-06-10T13:29:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Dropping all pooled connections to 10.27.0.12:25002 due to ShutdownInProgress: Pool for 10.27.0.12:25002 has expired. 2021-06-10T13:29:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Dropping all pooled connections to 10.27.0.13:25002 due to ShutdownInProgress: Pool for 10.27.0.13:25002 has expired. 2021-06-10T13:29:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Dropping all pooled connections to 10.27.0.14:25002 due to ShutdownInProgress: Pool for 10.27.0.14:25002 has expired. 2021-06-10T13:29:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Connecting to 10.27.0.13:25002 2021-06-10T13:29:59.116+0000 I CONNPOOL [TaskExecutorPool-0] Connecting to 10.27.0.12:25002 2021-06-10T13:29:59.116+0000 I CONNPOOL [TaskExecutorPool-0] Connecting to 10.27.0.14:25002 2021-06-10T13:30:36.491+0000 I NETWORK [listener] connection accepted from 10.27.0.12:58322 #87102 (19 connections now open) 2021-06-10T13:30:36.492+0000 I NETWORK [conn87102] received client metadata from 10.27.0.12:58322 conn87102: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T13:30:36.501+0000 I ACCESS [conn87102] Successfully authenticated as principal __system on local from client 10.27.0.12:58322 2021-06-10T13:31:46.538+0000 I NETWORK [listener] connection accepted from 10.27.0.12:58326 #87103 (20 connections now open) 2021-06-10T13:31:46.538+0000 I NETWORK [conn87103] received client metadata from 10.27.0.12:58326 conn87103: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T13:31:46.547+0000 I ACCESS [conn87103] Successfully authenticated as principal __system on local from client 10.27.0.12:58326 2021-06-10T13:33:37.644+0000 I NETWORK [listener] connection accepted from 10.27.0.16:54718 #87104 (21 connections now open) 2021-06-10T13:33:37.644+0000 I NETWORK [conn87104] received client metadata from 10.27.0.16:54718 conn87104: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T13:33:37.656+0000 I ACCESS [conn87104] Successfully authenticated as principal __system on local from client 10.27.0.16:54718 2021-06-10T13:33:48.936+0000 I NETWORK [listener] connection accepted from 10.27.0.14:60684 #87105 (22 connections now open) 2021-06-10T13:33:48.936+0000 I NETWORK [conn87105] received client metadata from 10.27.0.14:60684 conn87105: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T13:33:48.945+0000 I ACCESS [conn87105] Successfully authenticated as principal __system on local from client 10.27.0.14:60684 2021-06-10T13:34:29.407+0000 I NETWORK [conn18] end connection 10.27.0.16:46138 (21 connections now open) 2021-06-10T13:34:29.408+0000 I NETWORK [conn87059] end connection 10.27.0.16:54458 (20 connections now open) 2021-06-10T13:34:29.408+0000 I NETWORK [conn87104] end connection 10.27.0.16:54718 (19 connections now open) 2021-06-10T13:34:33.645+0000 I NETWORK [conn87098] end connection 10.27.0.14:60670 (18 connections now open) 2021-06-10T13:34:42.163+0000 I NETWORK [listener] connection accepted from 10.27.0.16:54722 #87106 (19 connections now open) 2021-06-10T13:34:42.164+0000 I NETWORK [conn87106] received client metadata from 10.27.0.16:54722 conn87106: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T13:34:42.298+0000 I ACCESS [conn87106] Successfully authenticated as principal __system on local from client 10.27.0.16:54722 2021-06-10T13:34:42.300+0000 I NETWORK [listener] connection accepted from 10.27.0.16:54736 #87107 (20 connections now open) 2021-06-10T13:34:42.300+0000 I NETWORK [conn87107] received client metadata from 10.27.0.16:54736 conn87107: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T13:34:42.339+0000 I ACCESS [conn87107] Successfully authenticated as principal __system on local from client 10.27.0.16:54736 2021-06-10T13:34:42.480+0000 I NETWORK [listener] connection accepted from 10.27.0.16:54760 #87108 (21 connections now open) 2021-06-10T13:34:42.480+0000 I NETWORK [conn87108] received client metadata from 10.27.0.16:54760 conn87108: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T13:34:42.494+0000 I ACCESS [conn87108] Successfully authenticated as principal __system on local from client 10.27.0.16:54760 2021-06-10T13:35:19.914+0000 I NETWORK [listener] connection accepted from 10.27.0.15:37374 #87109 (22 connections now open) 2021-06-10T13:35:19.914+0000 I NETWORK [conn87109] received client metadata from 10.27.0.15:37374 conn87109: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T13:35:19.929+0000 I ACCESS [conn87109] Successfully authenticated as principal __system on local from client 10.27.0.15:37374 2021-06-10T13:35:42.481+0000 I NETWORK [conn87107] end connection 10.27.0.16:54736 (21 connections now open) 2021-06-10T13:36:19.915+0000 I NETWORK [conn87071] end connection 10.27.0.15:37226 (20 connections now open) 2021-06-10T13:36:33.679+0000 I NETWORK [listener] connection accepted from 10.27.0.14:60696 #87110 (21 connections now open) 2021-06-10T13:36:33.679+0000 I NETWORK [conn87110] received client metadata from 10.27.0.14:60696 conn87110: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T13:36:33.687+0000 I ACCESS [conn87110] Successfully authenticated as principal __system on local from client 10.27.0.14:60696 2021-06-10T13:36:46.547+0000 I NETWORK [conn87103] end connection 10.27.0.12:58326 (20 connections now open) 2021-06-10T13:37:02.647+0000 I NETWORK [conn87106] end connection 10.27.0.16:54722 (19 connections now open) 2021-06-10T13:37:02.647+0000 I NETWORK [conn87108] end connection 10.27.0.16:54760 (18 connections now open) 2021-06-10T13:37:59.296+0000 I NETWORK [listener] connection accepted from 10.27.0.16:54794 #87111 (19 connections now open) 2021-06-10T13:37:59.297+0000 I NETWORK [conn87111] received client metadata from 10.27.0.16:54794 conn87111: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T13:37:59.432+0000 I ACCESS [conn87111] Successfully authenticated as principal __system on local from client 10.27.0.16:54794 2021-06-10T13:37:59.623+0000 I NETWORK [listener] connection accepted from 10.27.0.16:54822 #87112 (20 connections now open) 2021-06-10T13:37:59.623+0000 I NETWORK [conn87112] received client metadata from 10.27.0.16:54822 conn87112: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T13:37:59.645+0000 I ACCESS [conn87112] Successfully authenticated as principal __system on local from client 10.27.0.16:54822 2021-06-10T13:37:59.686+0000 I NETWORK [listener] connection accepted from 10.27.0.16:54826 #87113 (21 connections now open) 2021-06-10T13:37:59.687+0000 I NETWORK [conn87113] received client metadata from 10.27.0.16:54826 conn87113: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T13:37:59.701+0000 I ACCESS [conn87113] Successfully authenticated as principal __system on local from client 10.27.0.16:54826 2021-06-10T13:38:16.557+0000 I NETWORK [listener] connection accepted from 10.27.0.12:58340 #87114 (22 connections now open) 2021-06-10T13:38:16.557+0000 I NETWORK [conn87114] received client metadata from 10.27.0.12:58340 conn87114: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T13:38:16.565+0000 I ACCESS [conn87114] Successfully authenticated as principal __system on local from client 10.27.0.12:58340 2021-06-10T13:38:59.701+0000 I NETWORK [conn87113] end connection 10.27.0.16:54826 (21 connections now open) 2021-06-10T13:39:30.109+0000 I NETWORK [listener] connection accepted from 10.27.0.15:37384 #87115 (22 connections now open) 2021-06-10T13:39:30.110+0000 I NETWORK [conn87115] received client metadata from 10.27.0.15:37384 conn87115: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T13:39:30.122+0000 I ACCESS [conn87115] Successfully authenticated as principal __system on local from client 10.27.0.15:37384 2021-06-10T13:39:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Dropping all pooled connections to 10.27.0.12:25002 due to ShutdownInProgress: Pool for 10.27.0.12:25002 has expired. 2021-06-10T13:39:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Dropping all pooled connections to 10.27.0.13:25002 due to ShutdownInProgress: Pool for 10.27.0.13:25002 has expired. 2021-06-10T13:39:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Dropping all pooled connections to 10.27.0.14:25002 due to ShutdownInProgress: Pool for 10.27.0.14:25002 has expired. 2021-06-10T13:39:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Connecting to 10.27.0.13:25002 2021-06-10T13:39:59.116+0000 I CONNPOOL [TaskExecutorPool-0] Connecting to 10.27.0.12:25002 2021-06-10T13:39:59.116+0000 I CONNPOOL [TaskExecutorPool-0] Connecting to 10.27.0.14:25002 2021-06-10T13:40:30.123+0000 I NETWORK [conn87109] end connection 10.27.0.15:37374 (21 connections now open) 2021-06-10T13:46:48.990+0000 I NETWORK [conn87105] end connection 10.27.0.14:60684 (20 connections now open) 2021-06-10T13:47:49.001+0000 I NETWORK [listener] connection accepted from 10.27.0.14:60724 #87119 (21 connections now open) 2021-06-10T13:47:49.001+0000 I NETWORK [conn87119] received client metadata from 10.27.0.14:60724 conn87119: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T13:47:49.009+0000 I ACCESS [conn87119] Successfully authenticated as principal __system on local from client 10.27.0.14:60724 2021-06-10T13:49:03.699+0000 I NETWORK [conn87110] end connection 10.27.0.14:60696 (20 connections now open) 2021-06-10T13:49:33.733+0000 I NETWORK [listener] connection accepted from 10.27.0.14:60728 #87120 (21 connections now open) 2021-06-10T13:49:33.733+0000 I NETWORK [conn87120] received client metadata from 10.27.0.14:60728 conn87120: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T13:49:33.741+0000 I ACCESS [conn87120] Successfully authenticated as principal __system on local from client 10.27.0.14:60728 2021-06-10T13:49:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Dropping all pooled connections to 10.27.0.12:25002 due to ShutdownInProgress: Pool for 10.27.0.12:25002 has expired. 2021-06-10T13:49:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Dropping all pooled connections to 10.27.0.13:25002 due to ShutdownInProgress: Pool for 10.27.0.13:25002 has expired. 2021-06-10T13:49:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Dropping all pooled connections to 10.27.0.14:25002 due to ShutdownInProgress: Pool for 10.27.0.14:25002 has expired. 2021-06-10T13:49:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Connecting to 10.27.0.13:25002 2021-06-10T13:49:59.116+0000 I CONNPOOL [TaskExecutorPool-0] Connecting to 10.27.0.14:25002 2021-06-10T13:49:59.116+0000 I CONNPOOL [TaskExecutorPool-0] Connecting to 10.27.0.12:25002 2021-06-10T13:51:16.616+0000 I NETWORK [conn87114] end connection 10.27.0.12:58340 (20 connections now open) 2021-06-10T13:51:16.642+0000 I NETWORK [listener] connection accepted from 10.27.0.12:58366 #87124 (21 connections now open) 2021-06-10T13:51:16.643+0000 I NETWORK [conn87124] received client metadata from 10.27.0.12:58366 conn87124: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T13:51:16.652+0000 I ACCESS [conn87124] Successfully authenticated as principal __system on local from client 10.27.0.12:58366 2021-06-10T13:54:33.741+0000 I NETWORK [conn87120] end connection 10.27.0.14:60728 (20 connections now open) 2021-06-10T13:54:33.766+0000 I NETWORK [listener] connection accepted from 10.27.0.14:60748 #87125 (21 connections now open) 2021-06-10T13:54:33.766+0000 I NETWORK [conn87125] received client metadata from 10.27.0.14:60748 conn87125: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T13:54:33.774+0000 I ACCESS [conn87125] Successfully authenticated as principal __system on local from client 10.27.0.14:60748 2021-06-10T13:55:06.557+0000 I NETWORK [conn87102] end connection 10.27.0.12:58322 (20 connections now open) 2021-06-10T13:57:49.046+0000 I NETWORK [conn87119] end connection 10.27.0.14:60724 (19 connections now open) 2021-06-10T13:58:19.071+0000 I NETWORK [listener] connection accepted from 10.27.0.14:60756 #87126 (20 connections now open) 2021-06-10T13:58:19.071+0000 I NETWORK [conn87126] received client metadata from 10.27.0.14:60756 conn87126: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T13:58:19.079+0000 I ACCESS [conn87126] Successfully authenticated as principal __system on local from client 10.27.0.14:60756 2021-06-10T13:59:36.641+0000 I NETWORK [listener] connection accepted from 10.27.0.12:58388 #87127 (21 connections now open) 2021-06-10T13:59:36.641+0000 I NETWORK [conn87127] received client metadata from 10.27.0.12:58388 conn87127: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T13:59:36.650+0000 I ACCESS [conn87127] Successfully authenticated as principal __system on local from client 10.27.0.12:58388 2021-06-10T13:59:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Dropping all pooled connections to 10.27.0.12:25002 due to ShutdownInProgress: Pool for 10.27.0.12:25002 has expired. 2021-06-10T13:59:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Dropping all pooled connections to 10.27.0.13:25002 due to ShutdownInProgress: Pool for 10.27.0.13:25002 has expired. 2021-06-10T13:59:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Dropping all pooled connections to 10.27.0.14:25002 due to ShutdownInProgress: Pool for 10.27.0.14:25002 has expired. 2021-06-10T13:59:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Dropping all pooled connections to 10.27.0.11:25001 due to ShutdownInProgress: Pool for 10.27.0.11:25001 has expired. 2021-06-10T13:59:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Dropping all pooled connections to 10.27.0.12:25001 due to ShutdownInProgress: Pool for 10.27.0.12:25001 has expired. 2021-06-10T13:59:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Dropping all pooled connections to 10.27.0.14:25001 due to ShutdownInProgress: Pool for 10.27.0.14:25001 has expired. 2021-06-10T13:59:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Connecting to 10.27.0.11:25001 2021-06-10T13:59:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Connecting to 10.27.0.13:25002 2021-06-10T13:59:59.116+0000 I CONNPOOL [TaskExecutorPool-0] Connecting to 10.27.0.14:25002 2021-06-10T13:59:59.116+0000 I CONNPOOL [TaskExecutorPool-0] Connecting to 10.27.0.12:25002 2021-06-10T13:59:59.116+0000 I CONNPOOL [TaskExecutorPool-0] Connecting to 10.27.0.12:25001 2021-06-10T13:59:59.116+0000 I CONNPOOL [TaskExecutorPool-0] Connecting to 10.27.0.14:25001 2021-06-10T14:03:16.663+0000 I NETWORK [conn87124] end connection 10.27.0.12:58366 (20 connections now open) 2021-06-10T14:05:41.483+0000 I NETWORK [listener] connection accepted from 10.27.0.15:37426 #87134 (21 connections now open) 2021-06-10T14:05:41.483+0000 I NETWORK [conn87134] received client metadata from 10.27.0.15:37426 conn87134: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T14:05:41.498+0000 I ACCESS [conn87134] Successfully authenticated as principal __system on local from client 10.27.0.15:37426 2021-06-10T14:06:41.497+0000 I NETWORK [conn87115] end connection 10.27.0.15:37384 (20 connections now open) 2021-06-10T14:07:06.661+0000 I NETWORK [conn87127] end connection 10.27.0.12:58388 (19 connections now open) 2021-06-10T14:07:06.688+0000 I NETWORK [listener] connection accepted from 10.27.0.12:58462 #87135 (20 connections now open) 2021-06-10T14:07:06.688+0000 I NETWORK [conn87135] received client metadata from 10.27.0.12:58462 conn87135: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T14:07:06.697+0000 I ACCESS [conn87135] Successfully authenticated as principal __system on local from client 10.27.0.12:58462 2021-06-10T14:07:16.723+0000 I NETWORK [listener] connection accepted from 10.27.0.12:58464 #87136 (21 connections now open) 2021-06-10T14:07:16.723+0000 I NETWORK [conn87136] received client metadata from 10.27.0.12:58464 conn87136: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T14:07:16.732+0000 I ACCESS [conn87136] Successfully authenticated as principal __system on local from client 10.27.0.12:58464 2021-06-10T14:07:49.115+0000 I NETWORK [conn87126] end connection 10.27.0.14:60756 (20 connections now open) 2021-06-10T14:09:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Dropping all pooled connections to 10.27.0.12:25002 due to ShutdownInProgress: Pool for 10.27.0.12:25002 has expired. 2021-06-10T14:09:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Dropping all pooled connections to 10.27.0.13:25002 due to ShutdownInProgress: Pool for 10.27.0.13:25002 has expired. 2021-06-10T14:09:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Dropping all pooled connections to 10.27.0.14:25002 due to ShutdownInProgress: Pool for 10.27.0.14:25002 has expired. 2021-06-10T14:09:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Connecting to 10.27.0.13:25002 2021-06-10T14:09:59.116+0000 I CONNPOOL [TaskExecutorPool-0] Connecting to 10.27.0.12:25002 2021-06-10T14:09:59.116+0000 I CONNPOOL [TaskExecutorPool-0] Connecting to 10.27.0.14:25002 2021-06-10T14:13:27.865+0000 I NETWORK [conn25] end connection 10.27.0.15:51948 (19 connections now open) 2021-06-10T14:13:28.792+0000 I CONNPOOL [replexec-145] dropping unhealthy pooled connection to 10.27.0.15:23002 2021-06-10T14:13:28.792+0000 I CONNPOOL [Replication] Connecting to 10.27.0.15:23002 2021-06-10T14:13:28.793+0000 I REPL_HB [replexec-149] Heartbeat to 10.27.0.15:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.15:23002 :: caused by :: Connection refused 2021-06-10T14:13:28.799+0000 I REPL [replexec-149] Member 10.27.0.15:23002 is now in state RS_DOWN - Error connecting to 10.27.0.15:23002 :: caused by :: Connection refused 2021-06-10T14:13:30.162+0000 I CONNPOOL [Replication] Connecting to 10.27.0.15:23002 2021-06-10T14:13:30.793+0000 I REPL_HB [replexec-149] Heartbeat to 10.27.0.15:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.15:23002 :: caused by :: Connection refused 2021-06-10T14:13:32.162+0000 I CONNPOOL [Replication] Connecting to 10.27.0.15:23002 2021-06-10T14:13:32.794+0000 I REPL_HB [replexec-145] Heartbeat to 10.27.0.15:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.15:23002 :: caused by :: Connection refused 2021-06-10T14:13:34.162+0000 I CONNPOOL [Replication] Connecting to 10.27.0.15:23002 2021-06-10T14:13:34.794+0000 I REPL_HB [replexec-145] Heartbeat to 10.27.0.15:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.15:23002 :: caused by :: Connection refused 2021-06-10T14:13:36.162+0000 I CONNPOOL [Replication] Connecting to 10.27.0.15:23002 2021-06-10T14:13:36.795+0000 I REPL_HB [replexec-149] Heartbeat to 10.27.0.15:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.15:23002 :: caused by :: Connection refused 2021-06-10T14:13:38.162+0000 I CONNPOOL [Replication] Connecting to 10.27.0.15:23002 2021-06-10T14:13:38.795+0000 I REPL_HB [replexec-149] Heartbeat to 10.27.0.15:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.15:23002 :: caused by :: Connection refused 2021-06-10T14:13:40.162+0000 I CONNPOOL [Replication] Connecting to 10.27.0.15:23002 2021-06-10T14:13:40.795+0000 I REPL_HB [replexec-149] Heartbeat to 10.27.0.15:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.15:23002 :: caused by :: Connection refused 2021-06-10T14:13:42.162+0000 I CONNPOOL [Replication] Connecting to 10.27.0.15:23002 2021-06-10T14:13:42.795+0000 I REPL_HB [replexec-149] Heartbeat to 10.27.0.15:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.15:23002 :: caused by :: Connection refused 2021-06-10T14:13:43.113+0000 I NETWORK [listener] connection accepted from 10.27.0.15:37534 #87140 (20 connections now open) 2021-06-10T14:13:43.149+0000 I ACCESS [conn87140] Successfully authenticated as principal __system on local from client 10.27.0.15:37534 2021-06-10T14:13:43.150+0000 I NETWORK [conn87140] end connection 10.27.0.15:37534 (19 connections now open) 2021-06-10T14:13:43.189+0000 I NETWORK [listener] connection accepted from 10.27.0.15:37544 #87142 (20 connections now open) 2021-06-10T14:13:43.190+0000 I NETWORK [conn87142] received client metadata from 10.27.0.15:37544 conn87142: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T14:13:43.321+0000 I ACCESS [conn87142] Successfully authenticated as principal __system on local from client 10.27.0.15:37544 2021-06-10T14:13:44.800+0000 I REPL [replexec-149] Member 10.27.0.15:23002 is now in state SECONDARY 2021-06-10T14:14:04.106+0000 I NETWORK [conn21] end connection 10.27.0.15:51946 (19 connections now open) 2021-06-10T14:14:04.107+0000 I NETWORK [conn87134] end connection 10.27.0.15:37426 (18 connections now open) 2021-06-10T14:14:17.620+0000 I NETWORK [listener] connection accepted from 10.27.0.15:37576 #87143 (19 connections now open) 2021-06-10T14:14:17.620+0000 I NETWORK [conn87143] received client metadata from 10.27.0.15:37576 conn87143: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T14:14:17.775+0000 I ACCESS [conn87143] Successfully authenticated as principal __system on local from client 10.27.0.15:37576 2021-06-10T14:14:19.848+0000 I NETWORK [listener] connection accepted from 10.27.0.15:37608 #87144 (20 connections now open) 2021-06-10T14:14:19.849+0000 I NETWORK [conn87144] received client metadata from 10.27.0.15:37608 conn87144: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T14:14:19.863+0000 I ACCESS [conn87144] Successfully authenticated as principal __system on local from client 10.27.0.15:37608 2021-06-10T14:14:19.915+0000 I NETWORK [listener] connection accepted from 10.27.0.15:37612 #87145 (21 connections now open) 2021-06-10T14:14:19.915+0000 I NETWORK [conn87145] received client metadata from 10.27.0.15:37612 conn87145: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T14:14:19.928+0000 I ACCESS [conn87145] Successfully authenticated as principal __system on local from client 10.27.0.15:37612 2021-06-10T14:15:06.697+0000 I NETWORK [conn87135] end connection 10.27.0.12:58462 (20 connections now open) 2021-06-10T14:15:19.928+0000 I NETWORK [conn87144] end connection 10.27.0.15:37608 (19 connections now open) 2021-06-10T14:16:19.212+0000 I NETWORK [listener] connection accepted from 10.27.0.14:32812 #87146 (20 connections now open) 2021-06-10T14:16:19.212+0000 I NETWORK [conn87146] received client metadata from 10.27.0.14:32812 conn87146: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T14:16:19.220+0000 I ACCESS [conn87146] Successfully authenticated as principal __system on local from client 10.27.0.14:32812 2021-06-10T14:16:36.770+0000 I NETWORK [listener] connection accepted from 10.27.0.12:58630 #87147 (21 connections now open) 2021-06-10T14:16:36.770+0000 I NETWORK [conn87147] received client metadata from 10.27.0.12:58630 conn87147: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T14:16:36.779+0000 I ACCESS [conn87147] Successfully authenticated as principal __system on local from client 10.27.0.12:58630 2021-06-10T14:19:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Dropping all pooled connections to 10.27.0.12:25002 due to ShutdownInProgress: Pool for 10.27.0.12:25002 has expired. 2021-06-10T14:19:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Dropping all pooled connections to 10.27.0.13:25002 due to ShutdownInProgress: Pool for 10.27.0.13:25002 has expired. 2021-06-10T14:19:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Dropping all pooled connections to 10.27.0.14:25002 due to ShutdownInProgress: Pool for 10.27.0.14:25002 has expired. 2021-06-10T14:19:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Connecting to 10.27.0.13:25002 2021-06-10T14:19:59.116+0000 I CONNPOOL [TaskExecutorPool-0] Connecting to 10.27.0.14:25002 2021-06-10T14:19:59.116+0000 I CONNPOOL [TaskExecutorPool-0] Connecting to 10.27.0.12:25002 2021-06-10T14:21:16.743+0000 I NETWORK [conn87136] end connection 10.27.0.12:58464 (20 connections now open) 2021-06-10T14:21:16.768+0000 I NETWORK [listener] connection accepted from 10.27.0.12:58638 #87151 (21 connections now open) 2021-06-10T14:21:16.768+0000 I NETWORK [conn87151] received client metadata from 10.27.0.12:58638 conn87151: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T14:21:16.777+0000 I ACCESS [conn87151] Successfully authenticated as principal __system on local from client 10.27.0.12:58638 2021-06-10T14:24:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Dropping all pooled connections to 10.27.0.11:25001 due to ShutdownInProgress: Pool for 10.27.0.11:25001 has expired. 2021-06-10T14:24:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Dropping all pooled connections to 10.27.0.12:25001 due to ShutdownInProgress: Pool for 10.27.0.12:25001 has expired. 2021-06-10T14:24:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Dropping all pooled connections to 10.27.0.14:25001 due to ShutdownInProgress: Pool for 10.27.0.14:25001 has expired. 2021-06-10T14:24:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Connecting to 10.27.0.11:25001 2021-06-10T14:24:59.116+0000 I CONNPOOL [TaskExecutorPool-0] Connecting to 10.27.0.14:25001 2021-06-10T14:24:59.116+0000 I CONNPOOL [TaskExecutorPool-0] Connecting to 10.27.0.12:25001 2021-06-10T14:25:29.032+0000 I NETWORK [conn87142] end connection 10.27.0.15:37544 (20 connections now open) 2021-06-10T14:25:30.795+0000 I CONNPOOL [replexec-149] dropping unhealthy pooled connection to 10.27.0.15:23002 2021-06-10T14:25:30.795+0000 I CONNPOOL [Replication] Connecting to 10.27.0.15:23002 2021-06-10T14:25:30.795+0000 I REPL_HB [replexec-145] Heartbeat to 10.27.0.15:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.15:23002 :: caused by :: Connection refused 2021-06-10T14:25:30.796+0000 I REPL [replexec-145] Member 10.27.0.15:23002 is now in state RS_DOWN - Error connecting to 10.27.0.15:23002 :: caused by :: Connection refused 2021-06-10T14:25:32.162+0000 I CONNPOOL [Replication] Connecting to 10.27.0.15:23002 2021-06-10T14:25:32.795+0000 I REPL_HB [replexec-145] Heartbeat to 10.27.0.15:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.15:23002 :: caused by :: Connection refused 2021-06-10T14:25:34.162+0000 I CONNPOOL [Replication] Connecting to 10.27.0.15:23002 2021-06-10T14:25:34.795+0000 I REPL_HB [replexec-149] Heartbeat to 10.27.0.15:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.15:23002 :: caused by :: Connection refused 2021-06-10T14:25:35.711+0000 I NETWORK [listener] connection accepted from 10.27.0.15:37664 #87155 (21 connections now open) 2021-06-10T14:25:35.746+0000 I ACCESS [conn87155] Successfully authenticated as principal __system on local from client 10.27.0.15:37664 2021-06-10T14:25:35.746+0000 I NETWORK [conn87155] end connection 10.27.0.15:37664 (20 connections now open) 2021-06-10T14:25:35.786+0000 I NETWORK [listener] connection accepted from 10.27.0.15:37672 #87156 (21 connections now open) 2021-06-10T14:25:35.786+0000 I NETWORK [conn87156] received client metadata from 10.27.0.15:37672 conn87156: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T14:25:35.912+0000 I ACCESS [conn87156] Successfully authenticated as principal __system on local from client 10.27.0.15:37672 2021-06-10T14:25:36.162+0000 I CONNPOOL [Replication] Connecting to 10.27.0.15:23002 2021-06-10T14:25:36.795+0000 I REPL [replexec-145] Member 10.27.0.15:23002 is now in state SECONDARY 2021-06-10T14:26:39.783+0000 I NETWORK [conn19] end connection 10.27.0.16:46140 (20 connections now open) 2021-06-10T14:26:39.806+0000 I REPL [replication-210] Restarting oplog query due to error: InterruptedAtShutdown: error in fetcher batch callback :: caused by :: interrupted at shutdown. Last fetched optime: { ts: Timestamp(1623335197, 4), t: 9 }. Restarts remaining: 1 2021-06-10T14:26:39.806+0000 I REPL [replication-210] Scheduled new oplog query Fetcher source: 10.27.0.16:23002 database: local query: { find: "oplog.rs", filter: { ts: { $gte: Timestamp(1623335197, 4) } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 2000, batchSize: 13981010, term: 9, readConcern: { afterClusterTime: Timestamp(0, 1) } } query metadata: { $replData: 1, $oplogQueryData: 1, $readPreference: { mode: "secondaryPreferred" } } active: 1 findNetworkTimeout: 7000ms getMoreNetworkTimeout: 10000ms shutting down?: 0 first: 1 firstCommandScheduler: RemoteCommandRetryScheduler request: RemoteCommand 74013118 -- target:10.27.0.16:23002 db:local cmd:{ find: "oplog.rs", filter: { ts: { $gte: Timestamp(1623335197, 4) } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 2000, batchSize: 13981010, term: 9, readConcern: { afterClusterTime: Timestamp(0, 1) } } active: 1 callbackHandle.valid: 1 callbackHandle.cancelled: 0 attempt: 1 retryPolicy: {type: "NoRetryPolicy"} 2021-06-10T14:26:39.806+0000 I REPL [replication-207] Error returned from oplog query (no more query restarts left): InterruptedAtShutdown: error in fetcher batch callback :: caused by :: interrupted at shutdown 2021-06-10T14:26:39.810+0000 W REPL [rsBackgroundSync] Fetcher stopped querying remote oplog with error: InterruptedAtShutdown: error in fetcher batch callback :: caused by :: interrupted at shutdown 2021-06-10T14:26:39.810+0000 I REPL [rsBackgroundSync] Clearing sync source 10.27.0.16:23002 to choose a new one. 2021-06-10T14:26:39.811+0000 I REPL [rsBackgroundSync] could not find member to sync from 2021-06-10T14:26:39.812+0000 I REPL_HB [replexec-159] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: InterruptedAtShutdown: interrupted at shutdown 2021-06-10T14:26:39.812+0000 I REPL [replexec-159] Member 10.27.0.16:23002 is now in state RS_DOWN - interrupted at shutdown 2021-06-10T14:26:40.312+0000 I CONNPOOL [replexec-155] dropping unhealthy pooled connection to 10.27.0.16:23002 2021-06-10T14:26:40.312+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:26:40.312+0000 I REPL_HB [replexec-151] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:26:40.811+0000 I REPL [rsBackgroundSync] sync source candidate: 10.27.0.11:23002 2021-06-10T14:26:40.811+0000 I CONNPOOL [RS] Connecting to 10.27.0.11:23002 2021-06-10T14:26:40.812+0000 I REPL_HB [replexec-152] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:26:40.838+0000 I REPL [rsBackgroundSync] Changed sync source from empty to 10.27.0.11:23002 2021-06-10T14:26:40.838+0000 I REPL [SyncSourceFeedback] SyncSourceFeedback error sending update to 10.27.0.16:23002: InvalidSyncSource: Sync source changed from 10.27.0.16:23002 to 10.27.0.11:23002 2021-06-10T14:26:42.148+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:26:42.813+0000 I REPL_HB [replexec-149] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:26:44.148+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:26:44.813+0000 I REPL_HB [replexec-152] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:26:46.148+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:26:46.813+0000 I REPL_HB [replexec-149] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:26:47.043+0000 I NETWORK [listener] connection accepted from 10.27.0.16:55176 #87160 (21 connections now open) 2021-06-10T14:26:47.078+0000 I ACCESS [conn87160] Successfully authenticated as principal __system on local from client 10.27.0.16:55176 2021-06-10T14:26:47.078+0000 I NETWORK [conn87160] end connection 10.27.0.16:55176 (20 connections now open) 2021-06-10T14:26:47.118+0000 I NETWORK [listener] connection accepted from 10.27.0.16:55184 #87161 (21 connections now open) 2021-06-10T14:26:47.154+0000 I NETWORK [conn87161] received client metadata from 10.27.0.16:55184 conn87161: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T14:26:47.220+0000 I ACCESS [conn87161] Successfully authenticated as principal __system on local from client 10.27.0.16:55184 2021-06-10T14:26:48.813+0000 I REPL [replexec-155] Member 10.27.0.16:23002 is now in state SECONDARY 2021-06-10T14:27:37.773+0000 I CONNPOOL [RS] Ending idle connection to host 10.27.0.16:23002 because the pool meets constraints; 1 connections to that host remain open 2021-06-10T14:27:40.093+0000 I CONNPOOL [RS] Connecting to 10.27.0.16:23002 2021-06-10T14:29:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Dropping all pooled connections to 10.27.0.12:25002 due to ShutdownInProgress: Pool for 10.27.0.12:25002 has expired. 2021-06-10T14:29:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Dropping all pooled connections to 10.27.0.13:25002 due to ShutdownInProgress: Pool for 10.27.0.13:25002 has expired. 2021-06-10T14:29:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Dropping all pooled connections to 10.27.0.14:25002 due to ShutdownInProgress: Pool for 10.27.0.14:25002 has expired. 2021-06-10T14:29:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Connecting to 10.27.0.13:25002 2021-06-10T14:29:59.116+0000 I CONNPOOL [TaskExecutorPool-0] Connecting to 10.27.0.12:25002 2021-06-10T14:29:59.116+0000 I CONNPOOL [TaskExecutorPool-0] Connecting to 10.27.0.14:25002 2021-06-10T14:31:39.806+0000 I CONNPOOL [RS] Dropping all pooled connections to 10.27.0.16:23002 due to ShutdownInProgress: Pool for 10.27.0.16:23002 has expired. 2021-06-10T14:34:41.644+0000 I NETWORK [conn87111] end connection 10.27.0.16:54794 (20 connections now open) 2021-06-10T14:34:41.646+0000 I NETWORK [conn87112] end connection 10.27.0.16:54822 (19 connections now open) 2021-06-10T14:34:42.233+0000 I NETWORK [conn87161] end connection 10.27.0.16:55184 (18 connections now open) 2021-06-10T14:34:42.813+0000 I CONNPOOL [replexec-149] dropping unhealthy pooled connection to 10.27.0.16:23002 2021-06-10T14:34:42.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:34:52.813+0000 I REPL [replexec-149] Member 10.27.0.16:23002 is now in state RS_DOWN - Couldn't get a connection within the time limit 2021-06-10T14:34:57.839+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:34:57.839+0000 I REPL_HB [replexec-149] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:34:59.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:34:59.839+0000 I REPL_HB [replexec-149] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:35:01.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:35:01.839+0000 I REPL_HB [replexec-152] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:35:03.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:35:03.839+0000 I REPL_HB [replexec-149] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:35:05.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:35:05.839+0000 I REPL_HB [replexec-152] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:35:07.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:35:07.839+0000 I REPL_HB [replexec-156] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:35:09.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:35:09.839+0000 I REPL_HB [replexec-149] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:35:11.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:35:11.839+0000 I REPL_HB [replexec-149] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:35:13.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:35:13.839+0000 I REPL_HB [replexec-149] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:35:15.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:35:15.839+0000 I REPL_HB [replexec-156] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:35:17.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:35:17.839+0000 I REPL_HB [replexec-149] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:35:19.281+0000 I NETWORK [conn87146] end connection 10.27.0.14:32812 (17 connections now open) 2021-06-10T14:35:19.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:35:19.839+0000 I REPL_HB [replexec-149] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:35:21.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:35:21.839+0000 I REPL_HB [replexec-156] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:35:23.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:35:23.839+0000 I REPL_HB [replexec-152] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:35:25.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:35:25.839+0000 I REPL_HB [replexec-156] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:35:27.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:35:27.839+0000 I REPL_HB [replexec-149] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:35:29.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:35:29.839+0000 I REPL_HB [replexec-156] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:35:31.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:35:31.839+0000 I REPL_HB [replexec-149] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:35:33.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:35:33.839+0000 I REPL_HB [replexec-152] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:35:35.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:35:35.839+0000 I REPL_HB [replexec-149] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:35:37.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:35:37.839+0000 I REPL_HB [replexec-156] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:35:39.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:35:39.839+0000 I REPL_HB [replexec-152] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:35:41.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:35:41.839+0000 I REPL_HB [replexec-156] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:35:43.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:35:43.839+0000 I REPL_HB [replexec-156] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:35:45.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:35:45.839+0000 I REPL_HB [replexec-152] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:35:47.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:35:47.839+0000 I REPL_HB [replexec-156] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:35:49.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:35:49.839+0000 I REPL_HB [replexec-156] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:35:51.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:35:51.839+0000 I REPL_HB [replexec-152] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:35:53.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:35:53.839+0000 I REPL_HB [replexec-149] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:35:55.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:35:55.839+0000 I REPL_HB [replexec-149] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:35:57.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:35:57.839+0000 I REPL_HB [replexec-156] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:35:59.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:35:59.839+0000 I REPL_HB [replexec-156] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:36:01.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:36:01.839+0000 I REPL_HB [replexec-149] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:36:03.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:36:03.839+0000 I REPL_HB [replexec-156] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:36:05.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:36:05.839+0000 I REPL_HB [replexec-149] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:36:07.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:36:07.839+0000 I REPL_HB [replexec-152] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:36:09.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:36:09.839+0000 I REPL_HB [replexec-156] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:36:11.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:36:11.839+0000 I REPL_HB [replexec-152] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:36:13.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:36:13.839+0000 I REPL_HB [replexec-149] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:36:15.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:36:15.839+0000 I REPL_HB [replexec-152] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:36:17.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:36:17.839+0000 I REPL_HB [replexec-152] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:36:19.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:36:19.839+0000 I REPL_HB [replexec-156] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:36:21.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:36:21.839+0000 I REPL_HB [replexec-149] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:36:23.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:36:23.839+0000 I REPL_HB [replexec-152] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:36:25.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:36:25.839+0000 I REPL_HB [replexec-152] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:36:27.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:36:27.839+0000 I REPL_HB [replexec-152] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:36:29.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:36:29.839+0000 I REPL_HB [replexec-149] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:36:31.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:36:31.839+0000 I REPL_HB [replexec-152] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:36:33.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:36:33.839+0000 I REPL_HB [replexec-152] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:36:35.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:36:35.839+0000 I REPL_HB [replexec-152] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:36:36.888+0000 I NETWORK [conn87147] end connection 10.27.0.12:58630 (16 connections now open) 2021-06-10T14:36:37.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:36:37.839+0000 I REPL_HB [replexec-156] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:36:39.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:36:39.839+0000 I REPL_HB [replexec-156] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:36:41.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:36:41.839+0000 I REPL_HB [replexec-156] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:36:43.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:36:43.839+0000 I REPL_HB [replexec-156] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:36:45.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:36:45.839+0000 I REPL_HB [replexec-152] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:36:47.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:36:47.839+0000 I REPL_HB [replexec-152] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:36:49.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:36:49.839+0000 I REPL_HB [replexec-156] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:36:51.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:36:51.839+0000 I REPL_HB [replexec-149] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:36:53.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:36:53.839+0000 I REPL_HB [replexec-152] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:36:55.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:36:55.839+0000 I REPL_HB [replexec-152] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:36:57.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:36:57.839+0000 I REPL_HB [replexec-149] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:36:59.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:36:59.839+0000 I REPL_HB [replexec-152] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:37:01.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:37:01.839+0000 I REPL_HB [replexec-152] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:37:03.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:37:03.839+0000 I REPL_HB [replexec-149] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:37:05.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:37:05.839+0000 I REPL_HB [replexec-156] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:37:07.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:37:07.839+0000 I REPL_HB [replexec-152] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:37:09.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:37:09.839+0000 I REPL_HB [replexec-152] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:37:11.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:37:11.839+0000 I REPL_HB [replexec-149] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:37:13.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:37:13.839+0000 I REPL_HB [replexec-152] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:37:15.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:37:15.839+0000 I REPL_HB [replexec-149] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:37:17.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:37:17.839+0000 I REPL_HB [replexec-149] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:37:19.300+0000 I NETWORK [listener] connection accepted from 10.27.0.14:34508 #87167 (17 connections now open) 2021-06-10T14:37:19.301+0000 I NETWORK [conn87167] received client metadata from 10.27.0.14:34508 conn87167: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T14:37:19.309+0000 I ACCESS [conn87167] Successfully authenticated as principal __system on local from client 10.27.0.14:34508 2021-06-10T14:37:19.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:37:19.839+0000 I REPL_HB [replexec-152] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:37:21.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:37:21.839+0000 I REPL_HB [replexec-149] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:37:23.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:37:23.839+0000 I REPL_HB [replexec-152] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:37:25.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:37:25.839+0000 I REPL_HB [replexec-156] Heartbeat to 10.27.0.16:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.16:23002 :: caused by :: Connection refused 2021-06-10T14:37:27.438+0000 I NETWORK [listener] connection accepted from 10.27.0.16:24220 #87168 (18 connections now open) 2021-06-10T14:37:27.471+0000 I ACCESS [conn87168] Successfully authenticated as principal __system on local from client 10.27.0.16:24220 2021-06-10T14:37:27.472+0000 I NETWORK [conn87168] end connection 10.27.0.16:24220 (17 connections now open) 2021-06-10T14:37:27.510+0000 I NETWORK [listener] connection accepted from 10.27.0.16:24230 #87169 (18 connections now open) 2021-06-10T14:37:27.510+0000 I NETWORK [conn87169] received client metadata from 10.27.0.16:24230 conn87169: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1160.6.1.el7.x86_64" } } 2021-06-10T14:37:27.598+0000 I ACCESS [conn87169] Successfully authenticated as principal __system on local from client 10.27.0.16:24230 2021-06-10T14:37:27.813+0000 I CONNPOOL [Replication] Connecting to 10.27.0.16:23002 2021-06-10T14:37:27.839+0000 I REPL [replexec-156] Member 10.27.0.16:23002 is now in state SECONDARY 2021-06-10T14:37:37.977+0000 I NETWORK [listener] connection accepted from 10.27.0.16:24244 #87171 (19 connections now open) 2021-06-10T14:37:37.977+0000 I NETWORK [conn87171] received client metadata from 10.27.0.16:24244 conn87171: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1160.6.1.el7.x86_64" } } 2021-06-10T14:37:38.109+0000 I ACCESS [conn87171] Successfully authenticated as principal __system on local from client 10.27.0.16:24244 2021-06-10T14:37:38.374+0000 I NETWORK [listener] connection accepted from 10.27.0.16:24278 #87172 (20 connections now open) 2021-06-10T14:37:38.374+0000 I NETWORK [conn87172] received client metadata from 10.27.0.16:24278 conn87172: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1160.6.1.el7.x86_64" } } 2021-06-10T14:37:38.419+0000 I ACCESS [conn87172] Successfully authenticated as principal __system on local from client 10.27.0.16:24278 2021-06-10T14:37:38.419+0000 I NETWORK [listener] connection accepted from 10.27.0.16:24282 #87173 (21 connections now open) 2021-06-10T14:37:38.419+0000 I NETWORK [conn87173] received client metadata from 10.27.0.16:24282 conn87173: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1160.6.1.el7.x86_64" } } 2021-06-10T14:37:38.431+0000 I ACCESS [conn87173] Successfully authenticated as principal __system on local from client 10.27.0.16:24282 2021-06-10T14:38:06.935+0000 I NETWORK [listener] connection accepted from 10.27.0.12:60744 #87174 (22 connections now open) 2021-06-10T14:38:06.935+0000 I NETWORK [conn87174] received client metadata from 10.27.0.12:60744 conn87174: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T14:38:06.944+0000 I ACCESS [conn87174] Successfully authenticated as principal __system on local from client 10.27.0.12:60744 2021-06-10T14:38:38.431+0000 I NETWORK [conn87172] end connection 10.27.0.16:24278 (21 connections now open) 2021-06-10T14:39:46.823+0000 I NETWORK [conn87151] end connection 10.27.0.12:58638 (20 connections now open) 2021-06-10T14:39:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Dropping all pooled connections to 10.27.0.12:25002 due to ShutdownInProgress: Pool for 10.27.0.12:25002 has expired. 2021-06-10T14:39:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Dropping all pooled connections to 10.27.0.13:25002 due to ShutdownInProgress: Pool for 10.27.0.13:25002 has expired. 2021-06-10T14:39:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Dropping all pooled connections to 10.27.0.14:25002 due to ShutdownInProgress: Pool for 10.27.0.14:25002 has expired. 2021-06-10T14:39:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Dropping all pooled connections to 10.27.0.11:25001 due to ShutdownInProgress: Pool for 10.27.0.11:25001 has expired. 2021-06-10T14:39:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Dropping all pooled connections to 10.27.0.12:25001 due to ShutdownInProgress: Pool for 10.27.0.12:25001 has expired. 2021-06-10T14:39:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Dropping all pooled connections to 10.27.0.14:25001 due to ShutdownInProgress: Pool for 10.27.0.14:25001 has expired. 2021-06-10T14:39:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Connecting to 10.27.0.11:25001 2021-06-10T14:39:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Connecting to 10.27.0.13:25002 2021-06-10T14:39:59.116+0000 I CONNPOOL [TaskExecutorPool-0] Connecting to 10.27.0.12:25001 2021-06-10T14:39:59.116+0000 I CONNPOOL [TaskExecutorPool-0] Connecting to 10.27.0.12:25002 2021-06-10T14:39:59.116+0000 I CONNPOOL [TaskExecutorPool-0] Connecting to 10.27.0.14:25001 2021-06-10T14:39:59.116+0000 I CONNPOOL [TaskExecutorPool-0] Connecting to 10.27.0.14:25002 2021-06-10T14:46:01.098+0000 I ELECTION [conn14] Received vote request: { replSetRequestVotes: 1, setName: "configs", dryRun: false, term: 10, candidateIndex: 1, configVersion: 3, lastCommittedOp: { ts: Timestamp(1623336358, 1), t: 9 } } 2021-06-10T14:46:01.098+0000 I ELECTION [conn14] Sending vote response: { term: 10, voteGranted: true, reason: "" } 2021-06-10T14:46:01.311+0000 I REPL [replexec-152] Member 10.27.0.11:23002 is now in state SECONDARY 2021-06-10T14:46:02.100+0000 I NETWORK [conn7] end connection 10.27.0.11:22268 (19 connections now open) 2021-06-10T14:46:02.104+0000 I REPL [replication-207] Restarting oplog query due to error: InterruptedAtShutdown: error in fetcher batch callback :: caused by :: interrupted at shutdown. Last fetched optime: { ts: Timestamp(1623336358, 1), t: 9 }. Restarts remaining: 1 2021-06-10T14:46:02.104+0000 I REPL [replication-207] Scheduled new oplog query Fetcher source: 10.27.0.11:23002 database: local query: { find: "oplog.rs", filter: { ts: { $gte: Timestamp(1623336358, 1) } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 2000, batchSize: 13981010, term: 10, readConcern: { afterClusterTime: Timestamp(0, 1) } } query metadata: { $replData: 1, $oplogQueryData: 1, $readPreference: { mode: "secondaryPreferred" } } active: 1 findNetworkTimeout: 7000ms getMoreNetworkTimeout: 10000ms shutting down?: 0 first: 1 firstCommandScheduler: RemoteCommandRetryScheduler request: RemoteCommand 74018935 -- target:10.27.0.11:23002 db:local cmd:{ find: "oplog.rs", filter: { ts: { $gte: Timestamp(1623336358, 1) } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 2000, batchSize: 13981010, term: 10, readConcern: { afterClusterTime: Timestamp(0, 1) } } active: 1 callbackHandle.valid: 1 callbackHandle.cancelled: 0 attempt: 1 retryPolicy: {type: "NoRetryPolicy"} 2021-06-10T14:46:02.104+0000 I REPL [replication-210] Error returned from oplog query (no more query restarts left): InterruptedAtShutdown: error in fetcher batch callback :: caused by :: interrupted at shutdown 2021-06-10T14:46:02.104+0000 W REPL [rsBackgroundSync] Fetcher stopped querying remote oplog with error: InterruptedAtShutdown: error in fetcher batch callback :: caused by :: interrupted at shutdown 2021-06-10T14:46:02.104+0000 I REPL [rsBackgroundSync] Clearing sync source 10.27.0.11:23002 to choose a new one. 2021-06-10T14:46:02.104+0000 I REPL [rsBackgroundSync] could not find member to sync from 2021-06-10T14:46:02.105+0000 I REPL [replexec-165] Member 10.27.0.12:23002 is now in state PRIMARY 2021-06-10T14:46:02.105+0000 I REPL_HB [replexec-162] Heartbeat to 10.27.0.11:23002 failed after 2 retries, response status: InterruptedAtShutdown: interrupted at shutdown 2021-06-10T14:46:02.105+0000 I REPL [replexec-162] Member 10.27.0.11:23002 is now in state RS_DOWN - interrupted at shutdown 2021-06-10T14:46:02.605+0000 I CONNPOOL [replexec-156] dropping unhealthy pooled connection to 10.27.0.11:23002 2021-06-10T14:46:02.605+0000 I CONNPOOL [Replication] Connecting to 10.27.0.11:23002 2021-06-10T14:46:02.605+0000 I REPL_HB [replexec-161] Heartbeat to 10.27.0.11:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.11:23002 :: caused by :: Connection refused 2021-06-10T14:46:03.105+0000 I REPL [rsBackgroundSync] sync source candidate: 10.27.0.12:23002 2021-06-10T14:46:03.105+0000 I CONNPOOL [RS] Connecting to 10.27.0.12:23002 2021-06-10T14:46:03.106+0000 I REPL_HB [replexec-164] Heartbeat to 10.27.0.11:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.11:23002 :: caused by :: Connection refused 2021-06-10T14:46:03.132+0000 I REPL [rsBackgroundSync] Changed sync source from empty to 10.27.0.12:23002 2021-06-10T14:46:03.132+0000 I REPL [SyncSourceFeedback] SyncSourceFeedback error sending update to 10.27.0.11:23002: InvalidSyncSource: Sync source changed from 10.27.0.11:23002 to 10.27.0.12:23002 2021-06-10T14:46:04.162+0000 I CONNPOOL [Replication] Connecting to 10.27.0.11:23002 2021-06-10T14:46:05.106+0000 I REPL_HB [replexec-162] Heartbeat to 10.27.0.11:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.11:23002 :: caused by :: Connection refused 2021-06-10T14:46:06.162+0000 I CONNPOOL [Replication] Connecting to 10.27.0.11:23002 2021-06-10T14:46:07.106+0000 I REPL_HB [replexec-167] Heartbeat to 10.27.0.11:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.11:23002 :: caused by :: Connection refused 2021-06-10T14:46:08.162+0000 I CONNPOOL [Replication] Connecting to 10.27.0.11:23002 2021-06-10T14:46:09.106+0000 I REPL_HB [replexec-163] Heartbeat to 10.27.0.11:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.11:23002 :: caused by :: Connection refused 2021-06-10T14:46:10.162+0000 I CONNPOOL [Replication] Connecting to 10.27.0.11:23002 2021-06-10T14:46:11.106+0000 I REPL_HB [replexec-160] Heartbeat to 10.27.0.11:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.11:23002 :: caused by :: Connection refused 2021-06-10T14:46:12.162+0000 I CONNPOOL [Replication] Connecting to 10.27.0.11:23002 2021-06-10T14:46:13.106+0000 I REPL_HB [replexec-156] Heartbeat to 10.27.0.11:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.11:23002 :: caused by :: Connection refused 2021-06-10T14:46:14.162+0000 I CONNPOOL [Replication] Connecting to 10.27.0.11:23002 2021-06-10T14:46:15.106+0000 I REPL_HB [replexec-160] Heartbeat to 10.27.0.11:23002 failed after 2 retries, response status: HostUnreachable: Error connecting to 10.27.0.11:23002 :: caused by :: Connection refused 2021-06-10T14:46:16.162+0000 I CONNPOOL [Replication] Connecting to 10.27.0.11:23002 2021-06-10T14:46:16.537+0000 I NETWORK [listener] connection accepted from 10.27.0.11:49980 #87183 (20 connections now open) 2021-06-10T14:46:16.561+0000 I ACCESS [conn87183] Successfully authenticated as principal __system on local from client 10.27.0.11:49980 2021-06-10T14:46:16.562+0000 I NETWORK [conn87183] end connection 10.27.0.11:49980 (19 connections now open) 2021-06-10T14:46:16.613+0000 I NETWORK [listener] connection accepted from 10.27.0.11:49988 #87184 (20 connections now open) 2021-06-10T14:46:16.613+0000 I NETWORK [conn87184] received client metadata from 10.27.0.11:49988 conn87184: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1160.6.1.el7.x86_64" } } 2021-06-10T14:46:16.652+0000 I ACCESS [conn87184] Successfully authenticated as principal __system on local from client 10.27.0.11:49988 2021-06-10T14:46:17.115+0000 I REPL [replexec-163] Member 10.27.0.11:23002 is now in state SECONDARY 2021-06-10T14:46:19.310+0000 I NETWORK [conn87167] end connection 10.27.0.14:34508 (19 connections now open) 2021-06-10T14:46:19.370+0000 I NETWORK [listener] connection accepted from 10.27.0.14:34722 #87186 (20 connections now open) 2021-06-10T14:46:19.370+0000 I NETWORK [conn87186] received client metadata from 10.27.0.14:34722 conn87186: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T14:46:19.378+0000 I ACCESS [conn87186] Successfully authenticated as principal __system on local from client 10.27.0.14:34722 2021-06-10T14:46:56.709+0000 I CONNPOOL [RS] Ending idle connection to host 10.27.0.11:23002 because the pool meets constraints; 1 connections to that host remain open 2021-06-10T14:47:02.811+0000 I CONNPOOL [RS] Connecting to 10.27.0.11:23002 2021-06-10T14:47:04.025+0000 I NETWORK [conn87125] end connection 10.27.0.14:60748 (19 connections now open) 2021-06-10T14:48:16.944+0000 I NETWORK [listener] connection accepted from 10.27.0.12:60886 #87188 (20 connections now open) 2021-06-10T14:48:16.944+0000 I NETWORK [conn87188] received client metadata from 10.27.0.12:60886 conn87188: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T14:48:16.953+0000 I ACCESS [conn87188] Successfully authenticated as principal __system on local from client 10.27.0.12:60886 2021-06-10T14:48:34.059+0000 I NETWORK [listener] connection accepted from 10.27.0.14:34738 #87189 (21 connections now open) 2021-06-10T14:48:34.059+0000 I NETWORK [conn87189] received client metadata from 10.27.0.14:34738 conn87189: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T14:48:34.067+0000 I ACCESS [conn87189] Successfully authenticated as principal __system on local from client 10.27.0.14:34738 2021-06-10T14:51:02.104+0000 I CONNPOOL [RS] Dropping all pooled connections to 10.27.0.11:23002 due to ShutdownInProgress: Pool for 10.27.0.11:23002 has expired. 2021-06-10T14:53:16.953+0000 I NETWORK [conn87188] end connection 10.27.0.12:60886 (20 connections now open) 2021-06-10T14:53:16.962+0000 I NETWORK [listener] connection accepted from 10.27.0.12:60898 #87190 (21 connections now open) 2021-06-10T14:53:16.962+0000 I NETWORK [conn87190] received client metadata from 10.27.0.12:60898 conn87190: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T14:53:16.971+0000 I ACCESS [conn87190] Successfully authenticated as principal __system on local from client 10.27.0.12:60898 2021-06-10T14:53:36.977+0000 I NETWORK [conn87174] end connection 10.27.0.12:60744 (20 connections now open) 2021-06-10T14:54:06.987+0000 I NETWORK [listener] connection accepted from 10.27.0.12:60900 #87191 (21 connections now open) 2021-06-10T14:54:06.987+0000 I NETWORK [conn87191] received client metadata from 10.27.0.12:60900 conn87191: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T14:54:06.996+0000 I ACCESS [conn87191] Successfully authenticated as principal __system on local from client 10.27.0.12:60900 2021-06-10T14:56:34.093+0000 I NETWORK [conn87189] end connection 10.27.0.14:34738 (20 connections now open) 2021-06-10T14:57:04.104+0000 I NETWORK [listener] connection accepted from 10.27.0.14:34756 #87192 (21 connections now open) 2021-06-10T14:57:04.104+0000 I NETWORK [conn87192] received client metadata from 10.27.0.14:34756 conn87192: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T14:57:04.112+0000 I ACCESS [conn87192] Successfully authenticated as principal __system on local from client 10.27.0.14:34756 2021-06-10T14:57:19.378+0000 I NETWORK [conn87186] end connection 10.27.0.14:34722 (20 connections now open) 2021-06-10T14:59:19.423+0000 I NETWORK [listener] connection accepted from 10.27.0.14:34760 #87193 (21 connections now open) 2021-06-10T14:59:19.423+0000 I NETWORK [conn87193] received client metadata from 10.27.0.14:34760 conn87193: { driver: { name: "NetworkInterfaceTL", version: "4.2.11" }, os: { type: "Linux", name: "CentOS Linux release 7.8.2003 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1127.19.1.el7.x86_64" } } 2021-06-10T14:59:19.431+0000 I ACCESS [conn87193] Successfully authenticated as principal __system on local from client 10.27.0.14:34760 2021-06-10T14:59:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Dropping all pooled connections to 10.27.0.12:25002 due to ShutdownInProgress: Pool for 10.27.0.12:25002 has expired. 2021-06-10T14:59:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Dropping all pooled connections to 10.27.0.13:25002 due to ShutdownInProgress: Pool for 10.27.0.13:25002 has expired. 2021-06-10T14:59:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Dropping all pooled connections to 10.27.0.14:25002 due to ShutdownInProgress: Pool for 10.27.0.14:25002 has expired. 2021-06-10T14:59:58.116+0000 I CONNPOOL [TaskExecutorPool-0] Connecting to 10.27.0.13:25002 2021-06-10T14:59:59.116+0000 I CONNPOOL [TaskExecutorPool-0] Connecting to 10.27.0.14:25002 2021-06-10T14:59:59.116+0000 I CONNPOOL [TaskExecutorPool-0] Connecting to 10.27.0.12:25002