[resmoke] 2018-03-16T17:28:04.483-0400 resmoke.py invocation: buildscripts/resmoke.py --suites=sharding_last_stable_mongos_and_mixed_shards jstests/sharding/stale_mongos_updates_and_removes.js [resmoke] 2018-03-16T17:28:04.502-0400 YAML configuration of suite sharding_last_stable_mongos_and_mixed_shards test_kind: js_test selector: roots: - jstests/sharding/stale_mongos_updates_and_removes.js executor: config: shell_options: global_vars: TestData: mongosBinVersion: last-stable shardMixedBinVersions: true skipCheckingUUIDsConsistentAcrossCluster: true nodb: '' logging: executor: format: '[%(name)s] %(asctime)s %(message)s' handlers: - class: logging.StreamHandler fixture: format: '[%(name)s] %(message)s' handlers: - class: logging.StreamHandler tests: format: '[%(name)s] %(asctime)s %(message)s' handlers: - class: logging.StreamHandler [executor] 2018-03-16T17:28:04.503-0400 Starting execution of js_tests... [executor:js_test:job0] 2018-03-16T17:28:04.504-0400 Running stale_mongos_updates_and_removes.js... ./mongo --eval MongoRunner.dataDir = "/data/db/job0/mongorunner"; TestData = new Object(); TestData.wiredTigerEngineConfigString = ""; TestData.maxPort = 20249; TestData.failIfUnterminatedProcesses = true; TestData.shardMixedBinVersions = true; TestData.storageEngine = ""; TestData.minPort = 20010; TestData.transportLayer = ""; TestData.skipCheckingUUIDsConsistentAcrossCluster = true; TestData.storageEngineCacheSizeGB = ""; TestData.wiredTigerIndexConfigString = ""; TestData.mongosBinVersion = "last-stable"; TestData.noJournal = false; TestData.testName = "stale_mongos_updates_and_removes"; TestData.serviceExecutor = ""; TestData.noJournalPrealloc = true; TestData.wiredTigerCollectionConfigString = ""; MongoRunner.dataPath = "/data/db/job0/mongorunner/"; load('jstests/libs/override_methods/validate_collections_on_shutdown.js');; load('jstests/libs/override_methods/check_uuids_consistent_across_cluster.js'); --nodb jstests/sharding/stale_mongos_updates_and_removes.js [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:04.504-0400 Starting JSTest jstests/sharding/stale_mongos_updates_and_removes.js... ./mongo --eval MongoRunner.dataDir = "/data/db/job0/mongorunner"; TestData = new Object(); TestData.wiredTigerEngineConfigString = ""; TestData.maxPort = 20249; TestData.storageEngineCacheSizeGB = ""; TestData.wiredTigerIndexConfigString = ""; TestData.mongosBinVersion = "last-stable"; TestData.noJournal = false; TestData.failIfUnterminatedProcesses = true; TestData.isMainTest = true; TestData.serviceExecutor = ""; TestData.skipCheckingUUIDsConsistentAcrossCluster = true; TestData.shardMixedBinVersions = true; TestData.storageEngine = ""; TestData.minPort = 20010; TestData.wiredTigerCollectionConfigString = ""; TestData.transportLayer = ""; TestData.noJournalPrealloc = true; TestData.numTestClients = 1; TestData.testName = "stale_mongos_updates_and_removes"; MongoRunner.dataPath = "/data/db/job0/mongorunner/"; load('jstests/libs/override_methods/validate_collections_on_shutdown.js');; load('jstests/libs/override_methods/check_uuids_consistent_across_cluster.js'); --nodb jstests/sharding/stale_mongos_updates_and_removes.js [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:04.506-0400 JSTest jstests/sharding/stale_mongos_updates_and_removes.js started with pid 24143. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:04.512-0400 MongoDB shell version v0.0.0 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:04.557-0400 true [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:04.558-0400 Starting new replica set stale_mongos_updates_and_removes-rs0 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:04.558-0400 ReplSetTest starting set [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:04.558-0400 ReplSetTest n is : 0 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:04.559-0400 { [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:04.559-0400 "useHostName" : true, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:04.559-0400 "oplogSize" : 16, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:04.559-0400 "keyFile" : undefined, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:04.559-0400 "port" : 20010, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:04.559-0400 "noprealloc" : "", [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:04.559-0400 "smallfiles" : "", [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:04.559-0400 "replSet" : "stale_mongos_updates_and_removes-rs0", [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:04.559-0400 "dbpath" : "$set-$node", [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:04.559-0400 "useHostname" : true, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:04.559-0400 "noJournalPrealloc" : undefined, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:04.559-0400 "shardsvr" : "", [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:04.559-0400 "pathOpts" : { [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:04.560-0400 "testName" : "stale_mongos_updates_and_removes", [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:04.560-0400 "shard" : 0, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:04.560-0400 "node" : 0, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:04.560-0400 "set" : "stale_mongos_updates_and_removes-rs0" [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:04.560-0400 }, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:04.560-0400 "restart" : undefined, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:04.560-0400 "setParameter" : { [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:04.560-0400 "writePeriodicNoops" : false, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:04.560-0400 "numInitialSyncAttempts" : 1, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:04.560-0400 "numInitialSyncConnectAttempts" : 60 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:04.560-0400 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:04.560-0400 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:04.560-0400 ReplSetTest Starting.... [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:04.560-0400 Resetting db path '/data/db/job0/mongorunner/stale_mongos_updates_and_removes-rs0-0' [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:04.562-0400 2018-03-16T17:28:04.561-0400 I - [js] shell: started program (sh24148): /home/janna/server/mongo/mongod --oplogSize 16 --port 20010 --noprealloc --smallfiles --replSet stale_mongos_updates_and_removes-rs0 --dbpath /data/db/job0/mongorunner/stale_mongos_updates_and_removes-rs0-0 --shardsvr --setParameter writePeriodicNoops=false --setParameter numInitialSyncAttempts=1 --setParameter numInitialSyncConnectAttempts=60 --bind_ip 0.0.0.0 --nopreallocj --setParameter enableTestCommands=1 --setParameter disableLogicalSessionCacheRefresh=true --setParameter orphanCleanupDelaySecs=1 --setParameter logComponentVerbosity={"tracking":0,"replication":{"heartbeats":2,"rollback":2}} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:04.574-0400 d20010| 2018-03-16T17:28:04.573-0400 I CONTROL [main] note: noprealloc may hurt performance in many applications [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:04.577-0400 d20010| 2018-03-16T17:28:04.577-0400 I CONTROL [initandlisten] MongoDB starting : pid=24148 port=20010 dbpath=/data/db/job0/mongorunner/stale_mongos_updates_and_removes-rs0-0 64-bit host=ubuntu [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:04.578-0400 d20010| 2018-03-16T17:28:04.577-0400 I CONTROL [initandlisten] db version v0.0.0 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:04.578-0400 d20010| 2018-03-16T17:28:04.577-0400 I CONTROL [initandlisten] git version: unknown [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:04.578-0400 d20010| 2018-03-16T17:28:04.577-0400 I CONTROL [initandlisten] allocator: tcmalloc [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:04.578-0400 d20010| 2018-03-16T17:28:04.577-0400 I CONTROL [initandlisten] modules: ninja [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:04.578-0400 d20010| 2018-03-16T17:28:04.577-0400 I CONTROL [initandlisten] build environment: [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:04.578-0400 d20010| 2018-03-16T17:28:04.577-0400 I CONTROL [initandlisten] distarch: x86_64 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:04.578-0400 d20010| 2018-03-16T17:28:04.577-0400 I CONTROL [initandlisten] target_arch: x86_64 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:04.579-0400 d20010| 2018-03-16T17:28:04.577-0400 I CONTROL [initandlisten] options: { net: { bindIp: "0.0.0.0", port: 20010 }, replication: { oplogSizeMB: 16, replSet: "stale_mongos_updates_and_removes-rs0" }, setParameter: { disableLogicalSessionCacheRefresh: "true", enableTestCommands: "1", logComponentVerbosity: "{"tracking":0,"replication":{"heartbeats":2,"rollback":2}}", numInitialSyncAttempts: "1", numInitialSyncConnectAttempts: "60", orphanCleanupDelaySecs: "1", writePeriodicNoops: "false" }, sharding: { clusterRole: "shardsvr" }, storage: { dbPath: "/data/db/job0/mongorunner/stale_mongos_updates_and_removes-rs0-0", mmapv1: { journal: { nopreallocj: true }, preallocDataFiles: false, smallFiles: true } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:04.579-0400 d20010| 2018-03-16T17:28:04.577-0400 I STORAGE [initandlisten] [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:04.579-0400 d20010| 2018-03-16T17:28:04.577-0400 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:04.579-0400 d20010| 2018-03-16T17:28:04.577-0400 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:04.580-0400 d20010| 2018-03-16T17:28:04.578-0400 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=31632M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress), [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.434-0400 d20010| 2018-03-16T17:28:05.434-0400 I STORAGE [initandlisten] WiredTiger message [1521235685:434337][24148:0x7f1ff52af9c0], txn-recover: Set global recovery timestamp: 0 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.472-0400 d20010| 2018-03-16T17:28:05.472-0400 W STORAGE [initandlisten] Detected configuration for non-active storage engine mmapv1 when current storage engine is wiredTiger [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.472-0400 d20010| 2018-03-16T17:28:05.472-0400 I CONTROL [initandlisten] [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.472-0400 d20010| 2018-03-16T17:28:05.472-0400 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.472-0400 d20010| 2018-03-16T17:28:05.472-0400 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.472-0400 d20010| 2018-03-16T17:28:05.472-0400 I CONTROL [initandlisten] [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.473-0400 d20010| 2018-03-16T17:28:05.473-0400 I CONTROL [initandlisten] [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.473-0400 d20010| 2018-03-16T17:28:05.473-0400 I CONTROL [initandlisten] ** WARNING: You are running on a NUMA machine. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.473-0400 d20010| 2018-03-16T17:28:05.473-0400 I CONTROL [initandlisten] ** We suggest launching mongod like this to avoid performance problems: [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.473-0400 d20010| 2018-03-16T17:28:05.473-0400 I CONTROL [initandlisten] ** numactl --interleave=all mongod [other options] [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.474-0400 d20010| 2018-03-16T17:28:05.473-0400 I CONTROL [initandlisten] [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.475-0400 d20010| 2018-03-16T17:28:05.475-0400 W SHARDING [initandlisten] Started with --shardsvr, but no shardIdentity document was found on disk in admin.system.version. This most likely means this server has not yet been added to a sharded cluster. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.475-0400 d20010| 2018-03-16T17:28:05.475-0400 I STORAGE [initandlisten] createCollection: local.startup_log with generated UUID: ad9c99ae-3978-429c-af75-c0c32fef555a [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.490-0400 d20010| 2018-03-16T17:28:05.490-0400 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory '/data/db/job0/mongorunner/stale_mongos_updates_and_removes-rs0-0/diagnostic.data' [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.490-0400 d20010| 2018-03-16T17:28:05.490-0400 I STORAGE [initandlisten] createCollection: local.me with generated UUID: 6a6caead-76f5-4336-b631-7052a8459947 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.505-0400 d20010| 2018-03-16T17:28:05.504-0400 I STORAGE [initandlisten] createCollection: local.replset.oplogTruncateAfterPoint with generated UUID: 3135ac74-8fe2-45ee-91f9-a621419ce5de [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.524-0400 d20010| 2018-03-16T17:28:05.524-0400 I STORAGE [initandlisten] createCollection: local.replset.minvalid with generated UUID: 90ddc4c6-7f98-47b2-9dda-43cfe7e81c20 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.539-0400 d20010| 2018-03-16T17:28:05.539-0400 I STORAGE [initandlisten] createCollection: local.replset.checkpointTimestamp with generated UUID: dd78e7aa-9762-4900-a8bc-816f562b70d4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.554-0400 d20010| 2018-03-16T17:28:05.553-0400 I REPL [initandlisten] Did not find local voted for document at startup. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.554-0400 d20010| 2018-03-16T17:28:05.553-0400 I REPL [initandlisten] Did not find local Rollback ID document at startup. Creating one. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.554-0400 d20010| 2018-03-16T17:28:05.553-0400 I STORAGE [initandlisten] createCollection: local.system.rollback.id with generated UUID: 96dc9024-6382-4344-9705-9adf1a0912a2 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.568-0400 d20010| 2018-03-16T17:28:05.568-0400 I REPL [initandlisten] Initialized the rollback ID to 1 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.569-0400 d20010| 2018-03-16T17:28:05.568-0400 I REPL [initandlisten] Did not find local replica set configuration document at startup; NoMatchingDocument: Did not find replica set configuration document in local.system.replset [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.569-0400 d20010| 2018-03-16T17:28:05.568-0400 I NETWORK [initandlisten] waiting for connections on port 20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.763-0400 d20010| 2018-03-16T17:28:05.763-0400 I NETWORK [listener] connection accepted from 127.0.0.1:33768 #1 (1 connection now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.765-0400 d20010| 2018-03-16T17:28:05.765-0400 I NETWORK [conn1] received client metadata from 127.0.0.1:33768 conn1: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.765-0400 [ connection to ubuntu:20010 ] [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.768-0400 { [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.768-0400 "replSetInitiate" : { [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.768-0400 "_id" : "stale_mongos_updates_and_removes-rs0", [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.768-0400 "members" : [ [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.768-0400 { [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.768-0400 "_id" : 0, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.769-0400 "host" : "ubuntu:20010" [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.769-0400 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.769-0400 ] [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.769-0400 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.769-0400 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.769-0400 d20010| 2018-03-16T17:28:05.768-0400 I REPL [conn1] replSetInitiate admin command received from client [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.769-0400 d20010| 2018-03-16T17:28:05.769-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56194 #3 (2 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.769-0400 d20010| 2018-03-16T17:28:05.769-0400 I REPL [conn1] replSetInitiate config object with 1 members parses ok [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.769-0400 d20010| 2018-03-16T17:28:05.769-0400 I NETWORK [conn3] end connection 127.0.0.1:56194 (1 connection now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.770-0400 d20010| 2018-03-16T17:28:05.769-0400 I REPL [conn1] ****** [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.770-0400 d20010| 2018-03-16T17:28:05.769-0400 I REPL [conn1] creating replication oplog of size: 16MB... [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.770-0400 d20010| 2018-03-16T17:28:05.769-0400 I STORAGE [conn1] createCollection: local.oplog.rs with generated UUID: 6ed85e44-1e36-4c2d-9b9a-8d1ce0befb97 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.778-0400 d20010| 2018-03-16T17:28:05.778-0400 I STORAGE [conn1] Starting WiredTigerRecordStoreThread local.oplog.rs [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.778-0400 d20010| 2018-03-16T17:28:05.778-0400 I STORAGE [conn1] The size storer reports that the oplog contains 0 records totaling to 0 bytes [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.779-0400 d20010| 2018-03-16T17:28:05.778-0400 I STORAGE [conn1] Scanning the oplog to determine where to place markers for truncation [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.819-0400 d20010| 2018-03-16T17:28:05.819-0400 I REPL [conn1] ****** [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.819-0400 d20010| 2018-03-16T17:28:05.819-0400 I STORAGE [conn1] createCollection: local.system.replset with generated UUID: 0420e425-70d4-43dd-bb22-34e72cfc2042 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.933-0400 d20010| 2018-03-16T17:28:05.933-0400 I STORAGE [conn1] createCollection: admin.system.version with provided UUID: e4936b0c-d4ee-49e2-a90a-e8c8d9c2b2fb [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.947-0400 d20010| 2018-03-16T17:28:05.947-0400 I COMMAND [conn1] setting featureCompatibilityVersion to 3.6 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.947-0400 d20010| 2018-03-16T17:28:05.947-0400 D REPL_HB [conn1] Cancelling all heartbeats. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.948-0400 d20010| 2018-03-16T17:28:05.947-0400 I REPL [conn1] New replica set config in use: { _id: "stale_mongos_updates_and_removes-rs0", version: 1, protocolVersion: 1, members: [ { _id: 0, host: "ubuntu:20010", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, catchUpTimeoutMillis: -1, catchUpTakeoverDelayMillis: 30000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 }, replicaSetId: ObjectId('5aac36e5f7e5a0b40b8e3453') } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.948-0400 d20010| 2018-03-16T17:28:05.947-0400 I REPL [conn1] This node is ubuntu:20010 in the config [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.948-0400 d20010| 2018-03-16T17:28:05.947-0400 I REPL [conn1] transition to STARTUP2 from STARTUP [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.948-0400 d20010| 2018-03-16T17:28:05.947-0400 I REPL [conn1] Starting replication storage threads [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.948-0400 d20010| 2018-03-16T17:28:05.948-0400 I REPL [conn1] transition to RECOVERING from STARTUP2 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.948-0400 d20010| 2018-03-16T17:28:05.948-0400 I REPL [conn1] Starting replication fetcher thread [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.949-0400 d20010| 2018-03-16T17:28:05.948-0400 I REPL [conn1] Starting replication applier thread [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.949-0400 d20010| 2018-03-16T17:28:05.948-0400 I REPL [conn1] Starting replication reporter thread [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.949-0400 d20010| 2018-03-16T17:28:05.948-0400 I COMMAND [conn1] command local.system.replset appName: "MongoDB Shell" command: replSetInitiate { replSetInitiate: { _id: "stale_mongos_updates_and_removes-rs0", members: [ { _id: 0.0, host: "ubuntu:20010" } ] }, $clusterTime: { clusterTime: Timestamp(0, 0), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, $db: "admin" } numYields:0 reslen:147 locks:{ Global: { acquireCount: { r: 15, w: 7, W: 2 }, acquireWaitCount: { W: 1 }, timeAcquiringMicros: { W: 15 } }, Database: { acquireCount: { r: 3, w: 4, W: 3 } }, Collection: { acquireCount: { r: 1, w: 2 } }, oplog: { acquireCount: { r: 2, w: 3 } } } protocol:op_msg 179ms [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.949-0400 d20010| 2018-03-16T17:28:05.948-0400 I REPL [rsSync] transition to SECONDARY from RECOVERING [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.950-0400 d20010| 2018-03-16T17:28:05.948-0400 I REPL [rsSync] conducting a dry run election to see if we could be elected. current term: 0 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.950-0400 d20010| 2018-03-16T17:28:05.948-0400 I REPL [replexec-0] dry election run succeeded, running for election in term 1 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.950-0400 d20010| 2018-03-16T17:28:05.949-0400 I STORAGE [replexec-0] createCollection: local.replset.election with generated UUID: d21f7f61-294b-4fdb-82fb-4645069e9664 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.970-0400 d20010| 2018-03-16T17:28:05.970-0400 I REPL [replexec-2] election succeeded, assuming primary role in term 1 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.971-0400 d20010| 2018-03-16T17:28:05.970-0400 I REPL [replexec-2] transition to PRIMARY from SECONDARY [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.971-0400 d20010| 2018-03-16T17:28:05.970-0400 D REPL_HB [replexec-2] Cancelling all heartbeats. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.971-0400 d20010| 2018-03-16T17:28:05.970-0400 I REPL [replexec-2] Entering primary catch-up mode. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:05.971-0400 d20010| 2018-03-16T17:28:05.970-0400 I REPL [replexec-2] Exited primary catch-up mode. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:07.949-0400 d20010| 2018-03-16T17:28:07.949-0400 I STORAGE [rsSync] createCollection: config.transactions with generated UUID: bfc3de8f-fa9d-4678-b55c-3db3ca4db50d [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:07.967-0400 d20010| 2018-03-16T17:28:07.967-0400 I STORAGE [rsSync] Triggering the first stable checkpoint. Initial Data: Timestamp(1521235685, 1) PrevStable: Timestamp(0, 0) CurrStable: Timestamp(1521235685, 1) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:07.968-0400 d20010| 2018-03-16T17:28:07.967-0400 I REPL [rsSync] transition to primary complete; database writes are now permitted [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:08.159-0400 Set shouldWaitForKeys from RS options: false [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:08.159-0400 Starting new replica set stale_mongos_updates_and_removes-rs1 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:08.159-0400 ReplSetTest starting set [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:08.159-0400 ReplSetTest n is : 0 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:08.159-0400 { [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:08.160-0400 "useHostName" : true, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:08.160-0400 "oplogSize" : 16, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:08.160-0400 "keyFile" : undefined, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:08.160-0400 "port" : 20011, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:08.160-0400 "noprealloc" : "", [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:08.160-0400 "smallfiles" : "", [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:08.160-0400 "replSet" : "stale_mongos_updates_and_removes-rs1", [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:08.160-0400 "dbpath" : "$set-$node", [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:08.160-0400 "useHostname" : true, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:08.160-0400 "noJournalPrealloc" : undefined, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:08.160-0400 "shardsvr" : "", [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:08.160-0400 "pathOpts" : { [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:08.161-0400 "testName" : "stale_mongos_updates_and_removes", [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:08.161-0400 "shard" : 1, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:08.161-0400 "node" : 0, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:08.161-0400 "set" : "stale_mongos_updates_and_removes-rs1" [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:08.161-0400 }, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:08.161-0400 "restart" : undefined, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:08.161-0400 "setParameter" : { [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:08.161-0400 "writePeriodicNoops" : false, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:08.161-0400 "numInitialSyncAttempts" : 1, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:08.161-0400 "numInitialSyncConnectAttempts" : 60 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:08.161-0400 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:08.161-0400 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:08.161-0400 ReplSetTest Starting.... [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:08.162-0400 Resetting db path '/data/db/job0/mongorunner/stale_mongos_updates_and_removes-rs1-0' [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:08.162-0400 2018-03-16T17:28:08.162-0400 I - [js] shell: started program (sh24218): /home/janna/server/mongo/mongod --oplogSize 16 --port 20011 --noprealloc --smallfiles --replSet stale_mongos_updates_and_removes-rs1 --dbpath /data/db/job0/mongorunner/stale_mongos_updates_and_removes-rs1-0 --shardsvr --setParameter writePeriodicNoops=false --setParameter numInitialSyncAttempts=1 --setParameter numInitialSyncConnectAttempts=60 --bind_ip 0.0.0.0 --nopreallocj --setParameter enableTestCommands=1 --setParameter disableLogicalSessionCacheRefresh=true --setParameter orphanCleanupDelaySecs=1 --setParameter logComponentVerbosity={"tracking":0,"replication":{"heartbeats":2,"rollback":2}} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:08.178-0400 d20011| 2018-03-16T17:28:08.177-0400 I CONTROL [main] note: noprealloc may hurt performance in many applications [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:08.182-0400 d20011| 2018-03-16T17:28:08.182-0400 I CONTROL [initandlisten] MongoDB starting : pid=24218 port=20011 dbpath=/data/db/job0/mongorunner/stale_mongos_updates_and_removes-rs1-0 64-bit host=ubuntu [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:08.182-0400 d20011| 2018-03-16T17:28:08.182-0400 I CONTROL [initandlisten] db version v0.0.0 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:08.182-0400 d20011| 2018-03-16T17:28:08.182-0400 I CONTROL [initandlisten] git version: unknown [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:08.182-0400 d20011| 2018-03-16T17:28:08.182-0400 I CONTROL [initandlisten] allocator: tcmalloc [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:08.183-0400 d20011| 2018-03-16T17:28:08.182-0400 I CONTROL [initandlisten] modules: ninja [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:08.183-0400 d20011| 2018-03-16T17:28:08.182-0400 I CONTROL [initandlisten] build environment: [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:08.183-0400 d20011| 2018-03-16T17:28:08.182-0400 I CONTROL [initandlisten] distarch: x86_64 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:08.183-0400 d20011| 2018-03-16T17:28:08.182-0400 I CONTROL [initandlisten] target_arch: x86_64 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:08.183-0400 d20011| 2018-03-16T17:28:08.182-0400 I CONTROL [initandlisten] options: { net: { bindIp: "0.0.0.0", port: 20011 }, replication: { oplogSizeMB: 16, replSet: "stale_mongos_updates_and_removes-rs1" }, setParameter: { disableLogicalSessionCacheRefresh: "true", enableTestCommands: "1", logComponentVerbosity: "{"tracking":0,"replication":{"heartbeats":2,"rollback":2}}", numInitialSyncAttempts: "1", numInitialSyncConnectAttempts: "60", orphanCleanupDelaySecs: "1", writePeriodicNoops: "false" }, sharding: { clusterRole: "shardsvr" }, storage: { dbPath: "/data/db/job0/mongorunner/stale_mongos_updates_and_removes-rs1-0", mmapv1: { journal: { nopreallocj: true }, preallocDataFiles: false, smallFiles: true } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:08.184-0400 d20011| 2018-03-16T17:28:08.182-0400 I STORAGE [initandlisten] [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:08.184-0400 d20011| 2018-03-16T17:28:08.182-0400 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:08.184-0400 d20011| 2018-03-16T17:28:08.182-0400 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:08.184-0400 d20011| 2018-03-16T17:28:08.182-0400 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=31632M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress), [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.042-0400 d20011| 2018-03-16T17:28:09.041-0400 I STORAGE [initandlisten] WiredTiger message [1521235689:41981][24218:0x7f41c44af9c0], txn-recover: Set global recovery timestamp: 0 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.081-0400 d20011| 2018-03-16T17:28:09.081-0400 W STORAGE [initandlisten] Detected configuration for non-active storage engine mmapv1 when current storage engine is wiredTiger [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.081-0400 d20011| 2018-03-16T17:28:09.081-0400 I CONTROL [initandlisten] [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.081-0400 d20011| 2018-03-16T17:28:09.081-0400 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.081-0400 d20011| 2018-03-16T17:28:09.081-0400 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.081-0400 d20011| 2018-03-16T17:28:09.081-0400 I CONTROL [initandlisten] [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.082-0400 d20011| 2018-03-16T17:28:09.082-0400 I CONTROL [initandlisten] [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.082-0400 d20011| 2018-03-16T17:28:09.082-0400 I CONTROL [initandlisten] ** WARNING: You are running on a NUMA machine. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.082-0400 d20011| 2018-03-16T17:28:09.082-0400 I CONTROL [initandlisten] ** We suggest launching mongod like this to avoid performance problems: [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.082-0400 d20011| 2018-03-16T17:28:09.082-0400 I CONTROL [initandlisten] ** numactl --interleave=all mongod [other options] [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.082-0400 d20011| 2018-03-16T17:28:09.082-0400 I CONTROL [initandlisten] [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.084-0400 d20011| 2018-03-16T17:28:09.084-0400 W SHARDING [initandlisten] Started with --shardsvr, but no shardIdentity document was found on disk in admin.system.version. This most likely means this server has not yet been added to a sharded cluster. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.084-0400 d20011| 2018-03-16T17:28:09.084-0400 I STORAGE [initandlisten] createCollection: local.startup_log with generated UUID: 93bf156f-a939-4de0-973d-6f9008a033fc [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.100-0400 d20011| 2018-03-16T17:28:09.099-0400 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory '/data/db/job0/mongorunner/stale_mongos_updates_and_removes-rs1-0/diagnostic.data' [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.100-0400 d20011| 2018-03-16T17:28:09.100-0400 I STORAGE [initandlisten] createCollection: local.me with generated UUID: 82f15821-72f4-4123-95c6-a40dc4b6d7d8 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.115-0400 d20011| 2018-03-16T17:28:09.115-0400 I STORAGE [initandlisten] createCollection: local.replset.oplogTruncateAfterPoint with generated UUID: 7c863a63-8a3c-453e-a8fc-7073fec2057d [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.135-0400 d20011| 2018-03-16T17:28:09.135-0400 I STORAGE [initandlisten] createCollection: local.replset.minvalid with generated UUID: 09744268-c956-402d-af12-6ca0eedeb6df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.151-0400 d20011| 2018-03-16T17:28:09.150-0400 I STORAGE [initandlisten] createCollection: local.replset.checkpointTimestamp with generated UUID: b9cac20d-e504-4527-957c-2d50129882e2 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.167-0400 d20011| 2018-03-16T17:28:09.166-0400 I REPL [initandlisten] Did not find local voted for document at startup. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.167-0400 d20011| 2018-03-16T17:28:09.166-0400 I REPL [initandlisten] Did not find local Rollback ID document at startup. Creating one. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.167-0400 d20011| 2018-03-16T17:28:09.166-0400 I STORAGE [initandlisten] createCollection: local.system.rollback.id with generated UUID: fb67bb03-55a0-4bde-9c81-492bd041136a [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.183-0400 d20011| 2018-03-16T17:28:09.182-0400 I REPL [initandlisten] Initialized the rollback ID to 1 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.183-0400 d20011| 2018-03-16T17:28:09.183-0400 I REPL [initandlisten] Did not find local replica set configuration document at startup; NoMatchingDocument: Did not find replica set configuration document in local.system.replset [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.183-0400 d20011| 2018-03-16T17:28:09.183-0400 I NETWORK [initandlisten] waiting for connections on port 20011 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.364-0400 d20011| 2018-03-16T17:28:09.364-0400 I NETWORK [listener] connection accepted from 127.0.0.1:35710 #1 (1 connection now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.365-0400 d20011| 2018-03-16T17:28:09.364-0400 I NETWORK [conn1] received client metadata from 127.0.0.1:35710 conn1: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.366-0400 [ connection to ubuntu:20011 ] [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.367-0400 { [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.367-0400 "replSetInitiate" : { [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.367-0400 "_id" : "stale_mongos_updates_and_removes-rs1", [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.368-0400 "members" : [ [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.368-0400 { [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.368-0400 "_id" : 0, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.368-0400 "host" : "ubuntu:20011" [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.368-0400 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.368-0400 ] [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.368-0400 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.368-0400 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.368-0400 d20011| 2018-03-16T17:28:09.367-0400 I REPL [conn1] replSetInitiate admin command received from client [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.368-0400 d20011| 2018-03-16T17:28:09.368-0400 I NETWORK [listener] connection accepted from 127.0.0.1:54854 #3 (2 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.369-0400 d20011| 2018-03-16T17:28:09.368-0400 I REPL [conn1] replSetInitiate config object with 1 members parses ok [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.369-0400 d20011| 2018-03-16T17:28:09.368-0400 I NETWORK [conn3] end connection 127.0.0.1:54854 (1 connection now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.369-0400 d20011| 2018-03-16T17:28:09.368-0400 I REPL [conn1] ****** [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.369-0400 d20011| 2018-03-16T17:28:09.369-0400 I REPL [conn1] creating replication oplog of size: 16MB... [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.369-0400 d20011| 2018-03-16T17:28:09.369-0400 I STORAGE [conn1] createCollection: local.oplog.rs with generated UUID: bfc0130f-bd2e-4b62-bde3-416374b3d05a [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.376-0400 d20011| 2018-03-16T17:28:09.376-0400 I STORAGE [conn1] Starting WiredTigerRecordStoreThread local.oplog.rs [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.377-0400 d20011| 2018-03-16T17:28:09.376-0400 I STORAGE [conn1] The size storer reports that the oplog contains 0 records totaling to 0 bytes [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.377-0400 d20011| 2018-03-16T17:28:09.376-0400 I STORAGE [conn1] Scanning the oplog to determine where to place markers for truncation [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.420-0400 d20011| 2018-03-16T17:28:09.420-0400 I REPL [conn1] ****** [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.420-0400 d20011| 2018-03-16T17:28:09.420-0400 I STORAGE [conn1] createCollection: local.system.replset with generated UUID: ef4cc674-0b17-4805-995b-ac797362f957 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.535-0400 d20011| 2018-03-16T17:28:09.535-0400 I STORAGE [conn1] createCollection: admin.system.version with provided UUID: c2319d89-9948-4388-b875-847347072a77 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.552-0400 d20011| 2018-03-16T17:28:09.551-0400 I COMMAND [conn1] setting featureCompatibilityVersion to 3.6 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.552-0400 d20011| 2018-03-16T17:28:09.552-0400 D REPL_HB [conn1] Cancelling all heartbeats. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.552-0400 d20011| 2018-03-16T17:28:09.552-0400 I REPL [conn1] New replica set config in use: { _id: "stale_mongos_updates_and_removes-rs1", version: 1, protocolVersion: 1, members: [ { _id: 0, host: "ubuntu:20011", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, catchUpTimeoutMillis: -1, catchUpTakeoverDelayMillis: 30000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 }, replicaSetId: ObjectId('5aac36e96a0ae3d58b6402a0') } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.553-0400 d20011| 2018-03-16T17:28:09.552-0400 I REPL [conn1] This node is ubuntu:20011 in the config [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.553-0400 d20011| 2018-03-16T17:28:09.552-0400 I REPL [conn1] transition to STARTUP2 from STARTUP [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.553-0400 d20011| 2018-03-16T17:28:09.552-0400 I REPL [conn1] Starting replication storage threads [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.553-0400 d20011| 2018-03-16T17:28:09.552-0400 I REPL [conn1] transition to RECOVERING from STARTUP2 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.553-0400 d20011| 2018-03-16T17:28:09.552-0400 I REPL [conn1] Starting replication fetcher thread [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.553-0400 d20011| 2018-03-16T17:28:09.552-0400 I REPL [conn1] Starting replication applier thread [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.553-0400 d20011| 2018-03-16T17:28:09.553-0400 I REPL [conn1] Starting replication reporter thread [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.554-0400 d20011| 2018-03-16T17:28:09.553-0400 I COMMAND [conn1] command local.system.replset appName: "MongoDB Shell" command: replSetInitiate { replSetInitiate: { _id: "stale_mongos_updates_and_removes-rs1", members: [ { _id: 0.0, host: "ubuntu:20011" } ] }, $clusterTime: { clusterTime: Timestamp(0, 0), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, $db: "admin" } numYields:0 reslen:147 locks:{ Global: { acquireCount: { r: 15, w: 7, W: 2 }, acquireWaitCount: { W: 1 }, timeAcquiringMicros: { W: 67 } }, Database: { acquireCount: { r: 3, w: 4, W: 3 } }, Collection: { acquireCount: { r: 1, w: 2 } }, oplog: { acquireCount: { r: 2, w: 3 } } } protocol:op_msg 185ms [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.554-0400 d20011| 2018-03-16T17:28:09.553-0400 I REPL [rsSync] transition to SECONDARY from RECOVERING [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.554-0400 d20011| 2018-03-16T17:28:09.553-0400 I REPL [rsSync] conducting a dry run election to see if we could be elected. current term: 0 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.554-0400 d20011| 2018-03-16T17:28:09.553-0400 I REPL [replexec-0] dry election run succeeded, running for election in term 1 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.555-0400 d20011| 2018-03-16T17:28:09.553-0400 I STORAGE [replexec-1] createCollection: local.replset.election with generated UUID: b58504cd-f1ea-455f-9915-9a8cc71acdf4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.571-0400 d20011| 2018-03-16T17:28:09.571-0400 I REPL [replexec-1] election succeeded, assuming primary role in term 1 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.571-0400 d20011| 2018-03-16T17:28:09.571-0400 I REPL [replexec-1] transition to PRIMARY from SECONDARY [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.572-0400 d20011| 2018-03-16T17:28:09.571-0400 D REPL_HB [replexec-1] Cancelling all heartbeats. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.572-0400 d20011| 2018-03-16T17:28:09.571-0400 I REPL [replexec-1] Entering primary catch-up mode. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:09.572-0400 d20011| 2018-03-16T17:28:09.571-0400 I REPL [replexec-1] Exited primary catch-up mode. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.554-0400 d20011| 2018-03-16T17:28:11.554-0400 I STORAGE [rsSync] createCollection: config.transactions with generated UUID: 9267130e-4720-431e-9017-ab9c3a98bdfd [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.572-0400 d20011| 2018-03-16T17:28:11.572-0400 I STORAGE [rsSync] Triggering the first stable checkpoint. Initial Data: Timestamp(1521235689, 1) PrevStable: Timestamp(0, 0) CurrStable: Timestamp(1521235689, 1) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.572-0400 d20011| 2018-03-16T17:28:11.572-0400 I REPL [rsSync] transition to primary complete; database writes are now permitted [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.763-0400 Set shouldWaitForKeys from RS options: false [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.765-0400 d20010| 2018-03-16T17:28:11.765-0400 I STORAGE [conn1] createCollection: admin.foo with generated UUID: e0533010-bcc6-4a60-ab22-c26f55e43252 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.789-0400 2018-03-16T17:28:11.789-0400 I NETWORK [js] Starting new replica set monitor for stale_mongos_updates_and_removes-rs0/ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.790-0400 d20010| 2018-03-16T17:28:11.789-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56212 #4 (2 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.790-0400 d20010| 2018-03-16T17:28:11.790-0400 I NETWORK [conn4] received client metadata from 127.0.0.1:56212 conn4: { driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.790-0400 2018-03-16T17:28:11.790-0400 I NETWORK [js] Successfully connected to ubuntu:20010 (1 connections now open to ubuntu:20010 with a 5 second timeout) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.792-0400 d20011| 2018-03-16T17:28:11.791-0400 I STORAGE [conn1] createCollection: admin.foo with generated UUID: 1fcb206c-6e75-4936-ae77-ada0e1ca0cc5 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.830-0400 2018-03-16T17:28:11.830-0400 I NETWORK [js] Starting new replica set monitor for stale_mongos_updates_and_removes-rs1/ubuntu:20011 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.830-0400 d20011| 2018-03-16T17:28:11.830-0400 I NETWORK [listener] connection accepted from 127.0.0.1:54858 #4 (2 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.831-0400 d20011| 2018-03-16T17:28:11.830-0400 I NETWORK [conn4] received client metadata from 127.0.0.1:54858 conn4: { driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.831-0400 2018-03-16T17:28:11.831-0400 I NETWORK [js] Successfully connected to ubuntu:20011 (1 connections now open to ubuntu:20011 with a 5 second timeout) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.831-0400 Starting new replica set stale_mongos_updates_and_removes-configRS [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.831-0400 ReplSetTest starting set [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.831-0400 ReplSetTest n is : 0 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.832-0400 { [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.832-0400 "useHostName" : true, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.832-0400 "oplogSize" : 40, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.832-0400 "keyFile" : undefined, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.832-0400 "port" : 20012, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.832-0400 "noprealloc" : "", [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.832-0400 "smallfiles" : "", [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.832-0400 "replSet" : "stale_mongos_updates_and_removes-configRS", [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.833-0400 "dbpath" : "$set-$node", [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.833-0400 "pathOpts" : { [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.833-0400 "testName" : "stale_mongos_updates_and_removes", [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.833-0400 "node" : 0, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.833-0400 "set" : "stale_mongos_updates_and_removes-configRS" [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.833-0400 }, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.833-0400 "journal" : "", [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.833-0400 "configsvr" : "", [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.833-0400 "noJournalPrealloc" : undefined, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.833-0400 "storageEngine" : "wiredTiger", [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.833-0400 "restart" : undefined, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.833-0400 "setParameter" : { [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.833-0400 "writePeriodicNoops" : false, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.834-0400 "numInitialSyncAttempts" : 1, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.834-0400 "numInitialSyncConnectAttempts" : 60 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.834-0400 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.834-0400 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.834-0400 ReplSetTest Starting.... [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.834-0400 Resetting db path '/data/db/job0/mongorunner/stale_mongos_updates_and_removes-configRS-0' [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.835-0400 2018-03-16T17:28:11.834-0400 I - [js] shell: started program (sh24291): /home/janna/server/mongo/mongod --oplogSize 40 --port 20012 --noprealloc --smallfiles --replSet stale_mongos_updates_and_removes-configRS --dbpath /data/db/job0/mongorunner/stale_mongos_updates_and_removes-configRS-0 --journal --configsvr --storageEngine wiredTiger --setParameter writePeriodicNoops=false --setParameter numInitialSyncAttempts=1 --setParameter numInitialSyncConnectAttempts=60 --bind_ip 0.0.0.0 --nopreallocj --setParameter enableTestCommands=1 --setParameter disableLogicalSessionCacheRefresh=true --setParameter orphanCleanupDelaySecs=1 --setParameter logComponentVerbosity={"tracking":0,"replication":{"heartbeats":2,"rollback":2}} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.849-0400 c20012| 2018-03-16T17:28:11.849-0400 I CONTROL [main] note: noprealloc may hurt performance in many applications [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.853-0400 c20012| 2018-03-16T17:28:11.853-0400 I CONTROL [initandlisten] MongoDB starting : pid=24291 port=20012 dbpath=/data/db/job0/mongorunner/stale_mongos_updates_and_removes-configRS-0 64-bit host=ubuntu [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.853-0400 c20012| 2018-03-16T17:28:11.853-0400 I CONTROL [initandlisten] db version v0.0.0 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.854-0400 c20012| 2018-03-16T17:28:11.853-0400 I CONTROL [initandlisten] git version: unknown [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.854-0400 c20012| 2018-03-16T17:28:11.853-0400 I CONTROL [initandlisten] allocator: tcmalloc [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.854-0400 c20012| 2018-03-16T17:28:11.853-0400 I CONTROL [initandlisten] modules: ninja [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.854-0400 c20012| 2018-03-16T17:28:11.853-0400 I CONTROL [initandlisten] build environment: [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.854-0400 c20012| 2018-03-16T17:28:11.853-0400 I CONTROL [initandlisten] distarch: x86_64 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.854-0400 c20012| 2018-03-16T17:28:11.853-0400 I CONTROL [initandlisten] target_arch: x86_64 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.855-0400 c20012| 2018-03-16T17:28:11.853-0400 I CONTROL [initandlisten] options: { net: { bindIp: "0.0.0.0", port: 20012 }, replication: { oplogSizeMB: 40, replSet: "stale_mongos_updates_and_removes-configRS" }, setParameter: { disableLogicalSessionCacheRefresh: "true", enableTestCommands: "1", logComponentVerbosity: "{"tracking":0,"replication":{"heartbeats":2,"rollback":2}}", numInitialSyncAttempts: "1", numInitialSyncConnectAttempts: "60", orphanCleanupDelaySecs: "1", writePeriodicNoops: "false" }, sharding: { clusterRole: "configsvr" }, storage: { dbPath: "/data/db/job0/mongorunner/stale_mongos_updates_and_removes-configRS-0", engine: "wiredTiger", journal: { enabled: true }, mmapv1: { journal: { nopreallocj: true }, preallocDataFiles: false, smallFiles: true } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.855-0400 c20012| 2018-03-16T17:28:11.853-0400 I STORAGE [initandlisten] [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.855-0400 c20012| 2018-03-16T17:28:11.853-0400 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.855-0400 c20012| 2018-03-16T17:28:11.853-0400 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:11.856-0400 c20012| 2018-03-16T17:28:11.853-0400 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=31632M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress), [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:12.709-0400 c20012| 2018-03-16T17:28:12.708-0400 I STORAGE [initandlisten] WiredTiger message [1521235692:708957][24291:0x7f6a34f2d9c0], txn-recover: Set global recovery timestamp: 0 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:12.750-0400 c20012| 2018-03-16T17:28:12.749-0400 W STORAGE [initandlisten] Detected configuration for non-active storage engine mmapv1 when current storage engine is wiredTiger [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:12.750-0400 c20012| 2018-03-16T17:28:12.749-0400 I CONTROL [initandlisten] [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:12.750-0400 c20012| 2018-03-16T17:28:12.749-0400 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:12.750-0400 c20012| 2018-03-16T17:28:12.749-0400 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:12.750-0400 c20012| 2018-03-16T17:28:12.749-0400 I CONTROL [initandlisten] [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:12.750-0400 c20012| 2018-03-16T17:28:12.750-0400 I CONTROL [initandlisten] [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:12.751-0400 c20012| 2018-03-16T17:28:12.750-0400 I CONTROL [initandlisten] ** WARNING: You are running on a NUMA machine. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:12.751-0400 c20012| 2018-03-16T17:28:12.750-0400 I CONTROL [initandlisten] ** We suggest launching mongod like this to avoid performance problems: [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:12.751-0400 c20012| 2018-03-16T17:28:12.750-0400 I CONTROL [initandlisten] ** numactl --interleave=all mongod [other options] [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:12.751-0400 c20012| 2018-03-16T17:28:12.750-0400 I CONTROL [initandlisten] [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:12.753-0400 c20012| 2018-03-16T17:28:12.753-0400 I STORAGE [initandlisten] createCollection: local.startup_log with generated UUID: 403ea22c-f625-4987-90e2-2fba7083dc27 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:12.770-0400 c20012| 2018-03-16T17:28:12.769-0400 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory '/data/db/job0/mongorunner/stale_mongos_updates_and_removes-configRS-0/diagnostic.data' [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:12.770-0400 c20012| 2018-03-16T17:28:12.770-0400 I SHARDING [thread1] creating distributed lock ping thread for process ConfigServer (sleeping for 30000ms) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:12.770-0400 c20012| 2018-03-16T17:28:12.770-0400 I STORAGE [initandlisten] createCollection: local.me with generated UUID: 97e84f4e-2233-4262-a9d3-827b2cc9d090 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:12.770-0400 c20012| 2018-03-16T17:28:12.770-0400 I SHARDING [shard registry reload] Periodic reload of shard registry failed :: caused by :: ReadConcernMajorityNotAvailableYet: could not get updated shard list from config server :: caused by :: Read concern majority reads are currently not possible.; will retry after 30s [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:12.787-0400 c20012| 2018-03-16T17:28:12.787-0400 I STORAGE [initandlisten] createCollection: local.replset.oplogTruncateAfterPoint with generated UUID: 170340a1-2da8-47a5-954b-1f01965ecbe2 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:12.804-0400 c20012| 2018-03-16T17:28:12.804-0400 I STORAGE [initandlisten] createCollection: local.replset.minvalid with generated UUID: 964b7eac-34aa-4a0b-a6cb-b067f58643b5 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:12.820-0400 c20012| 2018-03-16T17:28:12.820-0400 I STORAGE [initandlisten] createCollection: local.replset.checkpointTimestamp with generated UUID: 9d960db0-e157-43e7-8353-1bc39bfd0fae [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:12.837-0400 c20012| 2018-03-16T17:28:12.837-0400 I REPL [initandlisten] Did not find local voted for document at startup. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:12.838-0400 c20012| 2018-03-16T17:28:12.837-0400 I REPL [initandlisten] Did not find local Rollback ID document at startup. Creating one. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:12.838-0400 c20012| 2018-03-16T17:28:12.837-0400 I STORAGE [initandlisten] createCollection: local.system.rollback.id with generated UUID: 1c452791-01a7-4c54-98cf-6f5a017083ed [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:12.853-0400 c20012| 2018-03-16T17:28:12.853-0400 I REPL [initandlisten] Initialized the rollback ID to 1 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:12.853-0400 c20012| 2018-03-16T17:28:12.853-0400 I REPL [initandlisten] Did not find local replica set configuration document at startup; NoMatchingDocument: Did not find replica set configuration document in local.system.replset [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:12.854-0400 c20012| 2018-03-16T17:28:12.853-0400 I NETWORK [initandlisten] waiting for connections on port 20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.037-0400 c20012| 2018-03-16T17:28:13.037-0400 I NETWORK [listener] connection accepted from 127.0.0.1:41148 #1 (1 connection now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.037-0400 c20012| 2018-03-16T17:28:13.037-0400 I NETWORK [conn1] received client metadata from 127.0.0.1:41148 conn1: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.038-0400 [ connection to ubuntu:20012 ] [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.038-0400 ReplSetTest n is : 1 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.038-0400 { [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.038-0400 "useHostName" : true, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.038-0400 "oplogSize" : 40, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.038-0400 "keyFile" : undefined, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.038-0400 "port" : 20013, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.039-0400 "noprealloc" : "", [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.039-0400 "smallfiles" : "", [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.039-0400 "replSet" : "stale_mongos_updates_and_removes-configRS", [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.039-0400 "dbpath" : "$set-$node", [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.039-0400 "pathOpts" : { [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.039-0400 "testName" : "stale_mongos_updates_and_removes", [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.039-0400 "node" : 1, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.039-0400 "set" : "stale_mongos_updates_and_removes-configRS" [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.039-0400 }, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.039-0400 "journal" : "", [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.039-0400 "configsvr" : "", [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.039-0400 "noJournalPrealloc" : undefined, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.040-0400 "storageEngine" : "wiredTiger", [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.040-0400 "restart" : undefined, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.040-0400 "setParameter" : { [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.040-0400 "writePeriodicNoops" : false, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.040-0400 "numInitialSyncAttempts" : 1, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.040-0400 "numInitialSyncConnectAttempts" : 60 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.040-0400 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.040-0400 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.040-0400 ReplSetTest Starting.... [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.040-0400 Resetting db path '/data/db/job0/mongorunner/stale_mongos_updates_and_removes-configRS-1' [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.042-0400 2018-03-16T17:28:13.041-0400 I - [js] shell: started program (sh24322): /home/janna/server/mongo/mongod --oplogSize 40 --port 20013 --noprealloc --smallfiles --replSet stale_mongos_updates_and_removes-configRS --dbpath /data/db/job0/mongorunner/stale_mongos_updates_and_removes-configRS-1 --journal --configsvr --storageEngine wiredTiger --setParameter writePeriodicNoops=false --setParameter numInitialSyncAttempts=1 --setParameter numInitialSyncConnectAttempts=60 --bind_ip 0.0.0.0 --nopreallocj --setParameter enableTestCommands=1 --setParameter disableLogicalSessionCacheRefresh=true --setParameter orphanCleanupDelaySecs=1 --setParameter logComponentVerbosity={"tracking":0,"replication":{"heartbeats":2,"rollback":2}} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.056-0400 c20013| 2018-03-16T17:28:13.056-0400 I CONTROL [main] note: noprealloc may hurt performance in many applications [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.060-0400 c20013| 2018-03-16T17:28:13.060-0400 I CONTROL [initandlisten] MongoDB starting : pid=24322 port=20013 dbpath=/data/db/job0/mongorunner/stale_mongos_updates_and_removes-configRS-1 64-bit host=ubuntu [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.060-0400 c20013| 2018-03-16T17:28:13.060-0400 I CONTROL [initandlisten] db version v0.0.0 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.061-0400 c20013| 2018-03-16T17:28:13.060-0400 I CONTROL [initandlisten] git version: unknown [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.061-0400 c20013| 2018-03-16T17:28:13.060-0400 I CONTROL [initandlisten] allocator: tcmalloc [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.061-0400 c20013| 2018-03-16T17:28:13.060-0400 I CONTROL [initandlisten] modules: ninja [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.061-0400 c20013| 2018-03-16T17:28:13.060-0400 I CONTROL [initandlisten] build environment: [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.061-0400 c20013| 2018-03-16T17:28:13.060-0400 I CONTROL [initandlisten] distarch: x86_64 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.061-0400 c20013| 2018-03-16T17:28:13.060-0400 I CONTROL [initandlisten] target_arch: x86_64 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.062-0400 c20013| 2018-03-16T17:28:13.060-0400 I CONTROL [initandlisten] options: { net: { bindIp: "0.0.0.0", port: 20013 }, replication: { oplogSizeMB: 40, replSet: "stale_mongos_updates_and_removes-configRS" }, setParameter: { disableLogicalSessionCacheRefresh: "true", enableTestCommands: "1", logComponentVerbosity: "{"tracking":0,"replication":{"heartbeats":2,"rollback":2}}", numInitialSyncAttempts: "1", numInitialSyncConnectAttempts: "60", orphanCleanupDelaySecs: "1", writePeriodicNoops: "false" }, sharding: { clusterRole: "configsvr" }, storage: { dbPath: "/data/db/job0/mongorunner/stale_mongos_updates_and_removes-configRS-1", engine: "wiredTiger", journal: { enabled: true }, mmapv1: { journal: { nopreallocj: true }, preallocDataFiles: false, smallFiles: true } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.062-0400 c20013| 2018-03-16T17:28:13.060-0400 I STORAGE [initandlisten] [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.062-0400 c20013| 2018-03-16T17:28:13.060-0400 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.062-0400 c20013| 2018-03-16T17:28:13.060-0400 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.063-0400 c20013| 2018-03-16T17:28:13.060-0400 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=31632M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress), [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.916-0400 c20013| 2018-03-16T17:28:13.916-0400 I STORAGE [initandlisten] WiredTiger message [1521235693:916347][24322:0x7fc7d7a059c0], txn-recover: Set global recovery timestamp: 0 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.956-0400 c20013| 2018-03-16T17:28:13.956-0400 W STORAGE [initandlisten] Detected configuration for non-active storage engine mmapv1 when current storage engine is wiredTiger [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.956-0400 c20013| 2018-03-16T17:28:13.956-0400 I CONTROL [initandlisten] [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.956-0400 c20013| 2018-03-16T17:28:13.956-0400 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.957-0400 c20013| 2018-03-16T17:28:13.956-0400 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.957-0400 c20013| 2018-03-16T17:28:13.956-0400 I CONTROL [initandlisten] [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.957-0400 c20013| 2018-03-16T17:28:13.957-0400 I CONTROL [initandlisten] [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.957-0400 c20013| 2018-03-16T17:28:13.957-0400 I CONTROL [initandlisten] ** WARNING: You are running on a NUMA machine. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.957-0400 c20013| 2018-03-16T17:28:13.957-0400 I CONTROL [initandlisten] ** We suggest launching mongod like this to avoid performance problems: [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.957-0400 c20013| 2018-03-16T17:28:13.957-0400 I CONTROL [initandlisten] ** numactl --interleave=all mongod [other options] [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.958-0400 c20013| 2018-03-16T17:28:13.957-0400 I CONTROL [initandlisten] [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.959-0400 c20013| 2018-03-16T17:28:13.959-0400 I STORAGE [initandlisten] createCollection: local.startup_log with generated UUID: c63098a3-5c07-45a4-ac2c-a5a5f95f501b [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.975-0400 c20013| 2018-03-16T17:28:13.974-0400 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory '/data/db/job0/mongorunner/stale_mongos_updates_and_removes-configRS-1/diagnostic.data' [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.975-0400 c20013| 2018-03-16T17:28:13.975-0400 I SHARDING [thread1] creating distributed lock ping thread for process ConfigServer (sleeping for 30000ms) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.975-0400 c20013| 2018-03-16T17:28:13.975-0400 I STORAGE [initandlisten] createCollection: local.me with generated UUID: 9d30af32-b5db-4b5d-9898-79a5ab0c19dd [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.976-0400 c20013| 2018-03-16T17:28:13.975-0400 I SHARDING [shard registry reload] Periodic reload of shard registry failed :: caused by :: ReadConcernMajorityNotAvailableYet: could not get updated shard list from config server :: caused by :: Read concern majority reads are currently not possible.; will retry after 30s [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:13.990-0400 c20013| 2018-03-16T17:28:13.990-0400 I STORAGE [initandlisten] createCollection: local.replset.oplogTruncateAfterPoint with generated UUID: 0e79eaa8-b8d0-42e6-a12a-5ccfa2468c05 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.000-0400 c20013| 2018-03-16T17:28:14.000-0400 W REPL [ftdc] Rollback ID is not initialized yet. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.011-0400 c20013| 2018-03-16T17:28:14.011-0400 I STORAGE [initandlisten] createCollection: local.replset.minvalid with generated UUID: da885e38-916a-41a4-b86f-f296b314c917 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.026-0400 c20013| 2018-03-16T17:28:14.026-0400 I STORAGE [initandlisten] createCollection: local.replset.checkpointTimestamp with generated UUID: d2595055-7415-461b-84ca-bacd1861c40f [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.041-0400 c20013| 2018-03-16T17:28:14.041-0400 I REPL [initandlisten] Did not find local voted for document at startup. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.041-0400 c20013| 2018-03-16T17:28:14.041-0400 I REPL [initandlisten] Did not find local Rollback ID document at startup. Creating one. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.042-0400 c20013| 2018-03-16T17:28:14.041-0400 I STORAGE [initandlisten] createCollection: local.system.rollback.id with generated UUID: 33d58a34-e91d-442f-8bc5-364ff91c55d4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.056-0400 c20013| 2018-03-16T17:28:14.056-0400 I REPL [initandlisten] Initialized the rollback ID to 1 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.057-0400 c20013| 2018-03-16T17:28:14.056-0400 I REPL [initandlisten] Did not find local replica set configuration document at startup; NoMatchingDocument: Did not find replica set configuration document in local.system.replset [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.057-0400 c20013| 2018-03-16T17:28:14.056-0400 I NETWORK [initandlisten] waiting for connections on port 20013 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.243-0400 c20013| 2018-03-16T17:28:14.243-0400 I NETWORK [listener] connection accepted from 127.0.0.1:48852 #1 (1 connection now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.244-0400 c20013| 2018-03-16T17:28:14.243-0400 I NETWORK [conn1] received client metadata from 127.0.0.1:48852 conn1: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.244-0400 [ connection to ubuntu:20012, connection to ubuntu:20013 ] [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.244-0400 ReplSetTest n is : 2 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.245-0400 { [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.245-0400 "useHostName" : true, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.245-0400 "oplogSize" : 40, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.245-0400 "keyFile" : undefined, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.245-0400 "port" : 20014, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.245-0400 "noprealloc" : "", [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.245-0400 "smallfiles" : "", [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.245-0400 "replSet" : "stale_mongos_updates_and_removes-configRS", [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.245-0400 "dbpath" : "$set-$node", [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.245-0400 "pathOpts" : { [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.245-0400 "testName" : "stale_mongos_updates_and_removes", [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.246-0400 "node" : 2, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.246-0400 "set" : "stale_mongos_updates_and_removes-configRS" [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.246-0400 }, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.246-0400 "journal" : "", [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.246-0400 "configsvr" : "", [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.246-0400 "noJournalPrealloc" : undefined, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.246-0400 "storageEngine" : "wiredTiger", [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.246-0400 "restart" : undefined, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.246-0400 "setParameter" : { [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.246-0400 "writePeriodicNoops" : false, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.246-0400 "numInitialSyncAttempts" : 1, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.246-0400 "numInitialSyncConnectAttempts" : 60 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.247-0400 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.247-0400 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.247-0400 ReplSetTest Starting.... [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.247-0400 Resetting db path '/data/db/job0/mongorunner/stale_mongos_updates_and_removes-configRS-2' [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.248-0400 2018-03-16T17:28:14.247-0400 I - [js] shell: started program (sh24353): /home/janna/server/mongo/mongod --oplogSize 40 --port 20014 --noprealloc --smallfiles --replSet stale_mongos_updates_and_removes-configRS --dbpath /data/db/job0/mongorunner/stale_mongos_updates_and_removes-configRS-2 --journal --configsvr --storageEngine wiredTiger --setParameter writePeriodicNoops=false --setParameter numInitialSyncAttempts=1 --setParameter numInitialSyncConnectAttempts=60 --bind_ip 0.0.0.0 --nopreallocj --setParameter enableTestCommands=1 --setParameter disableLogicalSessionCacheRefresh=true --setParameter orphanCleanupDelaySecs=1 --setParameter logComponentVerbosity={"tracking":0,"replication":{"heartbeats":2,"rollback":2}} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.263-0400 c20014| 2018-03-16T17:28:14.262-0400 I CONTROL [main] note: noprealloc may hurt performance in many applications [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.267-0400 c20014| 2018-03-16T17:28:14.267-0400 I CONTROL [initandlisten] MongoDB starting : pid=24353 port=20014 dbpath=/data/db/job0/mongorunner/stale_mongos_updates_and_removes-configRS-2 64-bit host=ubuntu [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.267-0400 c20014| 2018-03-16T17:28:14.267-0400 I CONTROL [initandlisten] db version v0.0.0 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.267-0400 c20014| 2018-03-16T17:28:14.267-0400 I CONTROL [initandlisten] git version: unknown [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.267-0400 c20014| 2018-03-16T17:28:14.267-0400 I CONTROL [initandlisten] allocator: tcmalloc [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.268-0400 c20014| 2018-03-16T17:28:14.267-0400 I CONTROL [initandlisten] modules: ninja [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.268-0400 c20014| 2018-03-16T17:28:14.267-0400 I CONTROL [initandlisten] build environment: [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.268-0400 c20014| 2018-03-16T17:28:14.267-0400 I CONTROL [initandlisten] distarch: x86_64 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.268-0400 c20014| 2018-03-16T17:28:14.267-0400 I CONTROL [initandlisten] target_arch: x86_64 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.269-0400 c20014| 2018-03-16T17:28:14.267-0400 I CONTROL [initandlisten] options: { net: { bindIp: "0.0.0.0", port: 20014 }, replication: { oplogSizeMB: 40, replSet: "stale_mongos_updates_and_removes-configRS" }, setParameter: { disableLogicalSessionCacheRefresh: "true", enableTestCommands: "1", logComponentVerbosity: "{"tracking":0,"replication":{"heartbeats":2,"rollback":2}}", numInitialSyncAttempts: "1", numInitialSyncConnectAttempts: "60", orphanCleanupDelaySecs: "1", writePeriodicNoops: "false" }, sharding: { clusterRole: "configsvr" }, storage: { dbPath: "/data/db/job0/mongorunner/stale_mongos_updates_and_removes-configRS-2", engine: "wiredTiger", journal: { enabled: true }, mmapv1: { journal: { nopreallocj: true }, preallocDataFiles: false, smallFiles: true } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.269-0400 c20014| 2018-03-16T17:28:14.267-0400 I STORAGE [initandlisten] [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.269-0400 c20014| 2018-03-16T17:28:14.267-0400 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.269-0400 c20014| 2018-03-16T17:28:14.267-0400 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:14.269-0400 c20014| 2018-03-16T17:28:14.267-0400 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=31632M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress), [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.123-0400 c20014| 2018-03-16T17:28:15.122-0400 I STORAGE [initandlisten] WiredTiger message [1521235695:122788][24353:0x7f836edd29c0], txn-recover: Set global recovery timestamp: 0 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.170-0400 c20014| 2018-03-16T17:28:15.170-0400 W STORAGE [initandlisten] Detected configuration for non-active storage engine mmapv1 when current storage engine is wiredTiger [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.170-0400 c20014| 2018-03-16T17:28:15.170-0400 I CONTROL [initandlisten] [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.170-0400 c20014| 2018-03-16T17:28:15.170-0400 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.170-0400 c20014| 2018-03-16T17:28:15.170-0400 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.171-0400 c20014| 2018-03-16T17:28:15.170-0400 I CONTROL [initandlisten] [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.171-0400 c20014| 2018-03-16T17:28:15.171-0400 I CONTROL [initandlisten] [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.171-0400 c20014| 2018-03-16T17:28:15.171-0400 I CONTROL [initandlisten] ** WARNING: You are running on a NUMA machine. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.171-0400 c20014| 2018-03-16T17:28:15.171-0400 I CONTROL [initandlisten] ** We suggest launching mongod like this to avoid performance problems: [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.171-0400 c20014| 2018-03-16T17:28:15.171-0400 I CONTROL [initandlisten] ** numactl --interleave=all mongod [other options] [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.171-0400 c20014| 2018-03-16T17:28:15.171-0400 I CONTROL [initandlisten] [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.173-0400 c20014| 2018-03-16T17:28:15.173-0400 I STORAGE [initandlisten] createCollection: local.startup_log with generated UUID: 334ebb01-c6d5-42c5-8c6a-1deb6d29da1a [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.189-0400 c20014| 2018-03-16T17:28:15.188-0400 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory '/data/db/job0/mongorunner/stale_mongos_updates_and_removes-configRS-2/diagnostic.data' [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.189-0400 c20014| 2018-03-16T17:28:15.189-0400 I SHARDING [thread1] creating distributed lock ping thread for process ConfigServer (sleeping for 30000ms) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.189-0400 c20014| 2018-03-16T17:28:15.189-0400 I STORAGE [initandlisten] createCollection: local.me with generated UUID: 77d4fbec-13b7-4e7d-8f46-75fbba18fce2 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.189-0400 c20014| 2018-03-16T17:28:15.189-0400 I SHARDING [shard registry reload] Periodic reload of shard registry failed :: caused by :: ReadConcernMajorityNotAvailableYet: could not get updated shard list from config server :: caused by :: Read concern majority reads are currently not possible.; will retry after 30s [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.205-0400 c20014| 2018-03-16T17:28:15.204-0400 I STORAGE [initandlisten] createCollection: local.replset.oplogTruncateAfterPoint with generated UUID: 228894ab-4242-4c96-9614-fcd5074e3ea9 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.225-0400 c20014| 2018-03-16T17:28:15.224-0400 I STORAGE [initandlisten] createCollection: local.replset.minvalid with generated UUID: d3c51ca9-6654-4cd2-9471-8b246b80b5cc [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.240-0400 c20014| 2018-03-16T17:28:15.240-0400 I STORAGE [initandlisten] createCollection: local.replset.checkpointTimestamp with generated UUID: 079a37a7-d913-4e65-99ef-8f2907ef336e [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.256-0400 c20014| 2018-03-16T17:28:15.256-0400 I REPL [initandlisten] Did not find local voted for document at startup. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.256-0400 c20014| 2018-03-16T17:28:15.256-0400 I REPL [initandlisten] Did not find local Rollback ID document at startup. Creating one. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.256-0400 c20014| 2018-03-16T17:28:15.256-0400 I STORAGE [initandlisten] createCollection: local.system.rollback.id with generated UUID: 109f9387-272d-4e4c-a6ed-db7a89823643 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.271-0400 c20014| 2018-03-16T17:28:15.271-0400 I REPL [initandlisten] Initialized the rollback ID to 1 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.272-0400 c20014| 2018-03-16T17:28:15.271-0400 I REPL [initandlisten] Did not find local replica set configuration document at startup; NoMatchingDocument: Did not find replica set configuration document in local.system.replset [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.272-0400 c20014| 2018-03-16T17:28:15.271-0400 I NETWORK [initandlisten] waiting for connections on port 20014 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.450-0400 c20014| 2018-03-16T17:28:15.449-0400 I NETWORK [listener] connection accepted from 127.0.0.1:59988 #1 (1 connection now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.450-0400 c20014| 2018-03-16T17:28:15.450-0400 I NETWORK [conn1] received client metadata from 127.0.0.1:59988 conn1: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.450-0400 [ [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.450-0400 connection to ubuntu:20012, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.451-0400 connection to ubuntu:20013, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.451-0400 connection to ubuntu:20014 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.451-0400 ] [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.451-0400 { [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.451-0400 "replSetInitiate" : { [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.451-0400 "_id" : "stale_mongos_updates_and_removes-configRS", [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.451-0400 "members" : [ [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.451-0400 { [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.451-0400 "_id" : 0, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.451-0400 "host" : "ubuntu:20012" [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.451-0400 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.451-0400 ], [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.451-0400 "configsvr" : true [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.452-0400 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.452-0400 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.452-0400 c20012| 2018-03-16T17:28:15.451-0400 I REPL [conn1] replSetInitiate admin command received from client [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.452-0400 c20012| 2018-03-16T17:28:15.452-0400 I NETWORK [listener] connection accepted from 127.0.0.1:60602 #3 (2 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.452-0400 c20012| 2018-03-16T17:28:15.452-0400 I REPL [conn1] replSetInitiate config object with 1 members parses ok [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.452-0400 c20012| 2018-03-16T17:28:15.452-0400 I NETWORK [conn3] end connection 127.0.0.1:60602 (1 connection now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.452-0400 c20012| 2018-03-16T17:28:15.452-0400 I REPL [conn1] ****** [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.453-0400 c20012| 2018-03-16T17:28:15.452-0400 I REPL [conn1] creating replication oplog of size: 40MB... [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.453-0400 c20012| 2018-03-16T17:28:15.452-0400 I STORAGE [conn1] createCollection: local.oplog.rs with generated UUID: 4a2de09d-a013-4c41-acbf-5df34f29e319 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.461-0400 c20012| 2018-03-16T17:28:15.461-0400 I STORAGE [conn1] Starting WiredTigerRecordStoreThread local.oplog.rs [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.462-0400 c20012| 2018-03-16T17:28:15.461-0400 I STORAGE [conn1] The size storer reports that the oplog contains 0 records totaling to 0 bytes [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.462-0400 c20012| 2018-03-16T17:28:15.461-0400 I STORAGE [conn1] Scanning the oplog to determine where to place markers for truncation [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.503-0400 c20012| 2018-03-16T17:28:15.503-0400 I REPL [conn1] ****** [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.504-0400 c20012| 2018-03-16T17:28:15.503-0400 I STORAGE [conn1] createCollection: local.system.replset with generated UUID: 32e14251-bb9e-4e3a-b83a-58e9b61105a5 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.618-0400 c20012| 2018-03-16T17:28:15.618-0400 I STORAGE [conn1] createCollection: admin.system.version with provided UUID: daa3215e-f467-4b5e-bffb-efcfd121a869 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.639-0400 c20012| 2018-03-16T17:28:15.639-0400 I COMMAND [conn1] setting featureCompatibilityVersion to 4.0 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.639-0400 c20012| 2018-03-16T17:28:15.639-0400 I NETWORK [conn1] Skip closing connection for connection # 1 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.639-0400 c20012| 2018-03-16T17:28:15.639-0400 D REPL_HB [conn1] Cancelling all heartbeats. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.640-0400 c20012| 2018-03-16T17:28:15.639-0400 I REPL [conn1] New replica set config in use: { _id: "stale_mongos_updates_and_removes-configRS", version: 1, configsvr: true, protocolVersion: 1, members: [ { _id: 0, host: "ubuntu:20012", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, catchUpTimeoutMillis: -1, catchUpTakeoverDelayMillis: 30000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 }, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f') } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.640-0400 c20012| 2018-03-16T17:28:15.639-0400 I REPL [conn1] This node is ubuntu:20012 in the config [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.640-0400 c20012| 2018-03-16T17:28:15.639-0400 I REPL [conn1] transition to STARTUP2 from STARTUP [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.640-0400 c20012| 2018-03-16T17:28:15.639-0400 I REPL [conn1] Starting replication storage threads [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.640-0400 c20012| 2018-03-16T17:28:15.639-0400 I REPL [conn1] transition to RECOVERING from STARTUP2 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.640-0400 c20012| 2018-03-16T17:28:15.639-0400 I REPL [conn1] Starting replication fetcher thread [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.641-0400 c20012| 2018-03-16T17:28:15.639-0400 I REPL [conn1] Starting replication applier thread [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.641-0400 c20012| 2018-03-16T17:28:15.640-0400 I REPL [conn1] Starting replication reporter thread [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.641-0400 c20012| 2018-03-16T17:28:15.640-0400 I COMMAND [conn1] command local.system.replset appName: "MongoDB Shell" command: replSetInitiate { replSetInitiate: { _id: "stale_mongos_updates_and_removes-configRS", members: [ { _id: 0.0, host: "ubuntu:20012" } ], configsvr: true }, $db: "admin" } numYields:0 reslen:236 locks:{ Global: { acquireCount: { r: 15, w: 7, W: 2 }, acquireWaitCount: { W: 1 }, timeAcquiringMicros: { W: 102 } }, Database: { acquireCount: { r: 3, w: 4, W: 3 } }, Collection: { acquireCount: { r: 1, w: 2 } }, oplog: { acquireCount: { r: 2, w: 3 } } } protocol:op_msg 189ms [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.641-0400 c20012| 2018-03-16T17:28:15.640-0400 I REPL [rsSync] transition to SECONDARY from RECOVERING [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.641-0400 c20012| 2018-03-16T17:28:15.640-0400 I REPL [rsSync] conducting a dry run election to see if we could be elected. current term: 0 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.642-0400 c20012| 2018-03-16T17:28:15.640-0400 I REPL [replexec-0] dry election run succeeded, running for election in term 1 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.642-0400 c20012| 2018-03-16T17:28:15.641-0400 I STORAGE [replexec-0] createCollection: local.replset.election with generated UUID: 38cd896d-96d6-4cb3-aa15-e348ed2ac069 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.659-0400 c20012| 2018-03-16T17:28:15.658-0400 I REPL [replexec-0] election succeeded, assuming primary role in term 1 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.659-0400 c20012| 2018-03-16T17:28:15.658-0400 I REPL [replexec-0] transition to PRIMARY from SECONDARY [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.659-0400 c20012| 2018-03-16T17:28:15.658-0400 D REPL_HB [replexec-0] Cancelling all heartbeats. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.659-0400 c20012| 2018-03-16T17:28:15.658-0400 I REPL [replexec-0] Entering primary catch-up mode. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:15.659-0400 c20012| 2018-03-16T17:28:15.658-0400 I REPL [replexec-0] Exited primary catch-up mode. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:17.641-0400 c20012| 2018-03-16T17:28:17.641-0400 I STORAGE [rsSync] createCollection: config.chunks with generated UUID: ca74780e-951c-4639-ab72-4df24247c87b [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:17.668-0400 c20012| 2018-03-16T17:28:17.668-0400 I INDEX [rsSync] build index on: config.chunks properties: { v: 2, unique: true, key: { ns: 1, min: 1 }, name: "ns_1_min_1", ns: "config.chunks" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:17.668-0400 c20012| 2018-03-16T17:28:17.668-0400 I INDEX [rsSync] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:17.669-0400 c20012| 2018-03-16T17:28:17.669-0400 I INDEX [rsSync] build index done. scanned 0 total records. 0 secs [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:17.678-0400 c20012| 2018-03-16T17:28:17.678-0400 I INDEX [rsSync] build index on: config.chunks properties: { v: 2, unique: true, key: { ns: 1, shard: 1, min: 1 }, name: "ns_1_shard_1_min_1", ns: "config.chunks" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:17.678-0400 c20012| 2018-03-16T17:28:17.678-0400 I INDEX [rsSync] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:17.679-0400 c20012| 2018-03-16T17:28:17.678-0400 I INDEX [rsSync] build index done. scanned 0 total records. 0 secs [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:17.693-0400 c20012| 2018-03-16T17:28:17.692-0400 I INDEX [rsSync] build index on: config.chunks properties: { v: 2, unique: true, key: { ns: 1, lastmod: 1 }, name: "ns_1_lastmod_1", ns: "config.chunks" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:17.693-0400 c20012| 2018-03-16T17:28:17.692-0400 I INDEX [rsSync] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:17.693-0400 c20012| 2018-03-16T17:28:17.693-0400 I INDEX [rsSync] build index done. scanned 0 total records. 0 secs [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:17.693-0400 c20012| 2018-03-16T17:28:17.693-0400 I STORAGE [rsSync] createCollection: config.migrations with generated UUID: 7121c15b-943b-4d4b-b5dd-50f357a13814 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:17.718-0400 c20012| 2018-03-16T17:28:17.718-0400 I INDEX [rsSync] build index on: config.migrations properties: { v: 2, unique: true, key: { ns: 1, min: 1 }, name: "ns_1_min_1", ns: "config.migrations" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:17.718-0400 c20012| 2018-03-16T17:28:17.718-0400 I INDEX [rsSync] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:17.719-0400 c20012| 2018-03-16T17:28:17.718-0400 I INDEX [rsSync] build index done. scanned 0 total records. 0 secs [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:17.719-0400 c20012| 2018-03-16T17:28:17.719-0400 I STORAGE [rsSync] createCollection: config.shards with generated UUID: c13519a3-4118-49d6-8a1d-2a757c74d1f7 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:17.743-0400 c20012| 2018-03-16T17:28:17.742-0400 I INDEX [rsSync] build index on: config.shards properties: { v: 2, unique: true, key: { host: 1 }, name: "host_1", ns: "config.shards" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:17.743-0400 c20012| 2018-03-16T17:28:17.742-0400 I INDEX [rsSync] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:17.743-0400 c20012| 2018-03-16T17:28:17.743-0400 I INDEX [rsSync] build index done. scanned 0 total records. 0 secs [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:17.743-0400 c20012| 2018-03-16T17:28:17.743-0400 I STORAGE [rsSync] createCollection: config.locks with generated UUID: 437da60d-d424-4d25-bc6a-eb52c6ac482a [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:17.781-0400 c20012| 2018-03-16T17:28:17.781-0400 I INDEX [rsSync] build index on: config.locks properties: { v: 2, key: { ts: 1 }, name: "ts_1", ns: "config.locks" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:17.782-0400 c20012| 2018-03-16T17:28:17.781-0400 I INDEX [rsSync] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:17.782-0400 c20012| 2018-03-16T17:28:17.782-0400 I INDEX [rsSync] build index done. scanned 0 total records. 0 secs [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:17.791-0400 c20012| 2018-03-16T17:28:17.791-0400 I INDEX [rsSync] build index on: config.locks properties: { v: 2, key: { state: 1, process: 1 }, name: "state_1_process_1", ns: "config.locks" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:17.791-0400 c20012| 2018-03-16T17:28:17.791-0400 I INDEX [rsSync] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:17.791-0400 c20012| 2018-03-16T17:28:17.791-0400 I INDEX [rsSync] build index done. scanned 0 total records. 0 secs [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:17.792-0400 c20012| 2018-03-16T17:28:17.791-0400 I STORAGE [rsSync] createCollection: config.lockpings with generated UUID: 9c0b3109-1569-46d5-8030-c76bd8db19ee [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:17.817-0400 c20012| 2018-03-16T17:28:17.816-0400 I INDEX [rsSync] build index on: config.lockpings properties: { v: 2, key: { ping: 1 }, name: "ping_1", ns: "config.lockpings" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:17.817-0400 c20012| 2018-03-16T17:28:17.816-0400 I INDEX [rsSync] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:17.818-0400 c20012| 2018-03-16T17:28:17.817-0400 I INDEX [rsSync] build index done. scanned 0 total records. 0 secs [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:17.818-0400 c20012| 2018-03-16T17:28:17.818-0400 I STORAGE [rsSync] createCollection: config.tags with generated UUID: e213515a-d815-4811-a6df-51ef6c4f2652 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:17.843-0400 c20012| 2018-03-16T17:28:17.842-0400 I INDEX [rsSync] build index on: config.tags properties: { v: 2, unique: true, key: { ns: 1, min: 1 }, name: "ns_1_min_1", ns: "config.tags" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:17.843-0400 c20012| 2018-03-16T17:28:17.842-0400 I INDEX [rsSync] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:17.843-0400 c20012| 2018-03-16T17:28:17.843-0400 I INDEX [rsSync] build index done. scanned 0 total records. 0 secs [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:17.857-0400 c20012| 2018-03-16T17:28:17.856-0400 I INDEX [rsSync] build index on: config.tags properties: { v: 2, key: { ns: 1, tag: 1 }, name: "ns_1_tag_1", ns: "config.tags" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:17.857-0400 c20012| 2018-03-16T17:28:17.857-0400 I INDEX [rsSync] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:17.857-0400 c20012| 2018-03-16T17:28:17.857-0400 I INDEX [rsSync] build index done. scanned 0 total records. 0 secs [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:17.858-0400 c20012| 2018-03-16T17:28:17.857-0400 I STORAGE [rsSync] createCollection: config.version with generated UUID: 3b13a852-5aec-4401-9e13-c21941b9b63f [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:17.874-0400 c20012| 2018-03-16T17:28:17.874-0400 I STORAGE [rsSync] createCollection: config.transactions with generated UUID: 01eb99c7-45a1-4fab-8a52-0bfdf9e027e2 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:17.874-0400 c20012| 2018-03-16T17:28:17.874-0400 I SHARDING [Balancer] CSRS balancer is starting [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:17.890-0400 c20012| 2018-03-16T17:28:17.890-0400 I STORAGE [rsSync] Triggering the first stable checkpoint. Initial Data: Timestamp(1521235695, 1) PrevStable: Timestamp(0, 0) CurrStable: Timestamp(1521235697, 15) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:17.890-0400 c20012| 2018-03-16T17:28:17.890-0400 I REPL [rsSync] transition to primary complete; database writes are now permitted [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:17.942-0400 c20012| 2018-03-16T17:28:17.942-0400 I STORAGE [monitoring keys for HMAC] createCollection: admin.system.keys with generated UUID: 32997d36-cabe-4e06-8861-b98e77fe07ee [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:17.942-0400 c20012| 2018-03-16T17:28:17.942-0400 I SHARDING [Balancer] CSRS balancer thread is recovering [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:17.943-0400 c20012| 2018-03-16T17:28:17.942-0400 I SHARDING [Balancer] CSRS balancer thread is recovered [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.064-0400 Reconfiguring replica set to add in other nodes [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.064-0400 { [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.064-0400 "replSetReconfig" : { [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.064-0400 "_id" : "stale_mongos_updates_and_removes-configRS", [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.064-0400 "members" : [ [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.064-0400 { [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.065-0400 "_id" : 0, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.065-0400 "host" : "ubuntu:20012" [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.065-0400 }, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.065-0400 { [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.065-0400 "_id" : 1, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.065-0400 "host" : "ubuntu:20013" [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.065-0400 }, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.065-0400 { [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.065-0400 "_id" : 2, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.065-0400 "host" : "ubuntu:20014" [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.065-0400 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.065-0400 ], [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.065-0400 "configsvr" : true, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.065-0400 "settings" : { [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.066-0400 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.066-0400 }, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.066-0400 "version" : 2 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.066-0400 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.066-0400 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.066-0400 c20012| 2018-03-16T17:28:18.064-0400 I REPL [conn1] replSetReconfig admin command received from client; new config: { _id: "stale_mongos_updates_and_removes-configRS", members: [ { _id: 0.0, host: "ubuntu:20012" }, { _id: 1.0, host: "ubuntu:20013" }, { _id: 2.0, host: "ubuntu:20014" } ], configsvr: true, settings: {}, version: 2.0 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.066-0400 c20012| 2018-03-16T17:28:18.065-0400 I NETWORK [listener] connection accepted from 127.0.0.1:60604 #5 (2 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.066-0400 c20012| 2018-03-16T17:28:18.065-0400 I NETWORK [conn5] end connection 127.0.0.1:60604 (1 connection now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.067-0400 c20013| 2018-03-16T17:28:18.065-0400 I NETWORK [listener] connection accepted from 127.0.0.1:46972 #2 (2 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.067-0400 c20013| 2018-03-16T17:28:18.065-0400 I NETWORK [conn2] end connection 127.0.0.1:46972 (1 connection now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.067-0400 c20014| 2018-03-16T17:28:18.066-0400 I NETWORK [listener] connection accepted from 127.0.0.1:44394 #2 (2 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.067-0400 c20012| 2018-03-16T17:28:18.066-0400 I REPL [conn1] replSetReconfig config object with 3 members parses ok [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.067-0400 c20014| 2018-03-16T17:28:18.066-0400 I NETWORK [conn2] end connection 127.0.0.1:44394 (1 connection now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.067-0400 c20012| 2018-03-16T17:28:18.066-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to ubuntu:20013 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.067-0400 c20012| 2018-03-16T17:28:18.066-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to ubuntu:20014 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.068-0400 c20013| 2018-03-16T17:28:18.066-0400 I NETWORK [listener] connection accepted from 127.0.0.1:46976 #3 (2 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.068-0400 c20014| 2018-03-16T17:28:18.066-0400 I NETWORK [listener] connection accepted from 127.0.0.1:44398 #3 (2 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.068-0400 c20013| 2018-03-16T17:28:18.067-0400 I NETWORK [conn3] received client metadata from 127.0.0.1:46976 conn3: { driver: { name: "NetworkInterfaceASIO-Replication", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.068-0400 c20014| 2018-03-16T17:28:18.067-0400 I NETWORK [conn3] received client metadata from 127.0.0.1:44398 conn3: { driver: { name: "NetworkInterfaceASIO-Replication", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.069-0400 c20012| 2018-03-16T17:28:18.067-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Successfully connected to ubuntu:20013, took 1ms (1 connections now open to ubuntu:20013) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.069-0400 c20012| 2018-03-16T17:28:18.067-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Successfully connected to ubuntu:20014, took 1ms (1 connections now open to ubuntu:20014) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.069-0400 c20013| 2018-03-16T17:28:18.067-0400 D REPL_HB [conn3] Received heartbeat request from ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20012", fromId: 0, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235697, 23), signature: { hash: BinData(0, CF7281952076F5A660A84B3D3A7A6A19F4855210), keyId: 6533657568122765331 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.069-0400 c20013| 2018-03-16T17:28:18.067-0400 D REPL_HB [conn3] Scheduling heartbeat to ubuntu:20012 at 2018-03-16T21:28:18.067Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.070-0400 c20013| 2018-03-16T17:28:18.067-0400 D REPL_HB [conn3] Processed heartbeat from ubuntu:20012 and generated response, { ok: 1.0, state: 0, v: -2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", durableOpTime: { ts: Timestamp(0, 0), t: -1 }, opTime: { ts: Timestamp(0, 0), t: -1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.070-0400 c20014| 2018-03-16T17:28:18.067-0400 D REPL_HB [conn3] Received heartbeat request from ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20012", fromId: 0, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235697, 23), signature: { hash: BinData(0, CF7281952076F5A660A84B3D3A7A6A19F4855210), keyId: 6533657568122765331 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.070-0400 c20014| 2018-03-16T17:28:18.067-0400 D REPL_HB [conn3] Scheduling heartbeat to ubuntu:20012 at 2018-03-16T21:28:18.067Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.070-0400 c20013| 2018-03-16T17:28:18.067-0400 D REPL_HB [replexec-0] Sending heartbeat (requestId: 1) to ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: -2, hbv: 1, from: "", fromId: -1, term: 0 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.071-0400 c20014| 2018-03-16T17:28:18.067-0400 D REPL_HB [conn3] Processed heartbeat from ubuntu:20012 and generated response, { ok: 1.0, state: 0, v: -2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", durableOpTime: { ts: Timestamp(0, 0), t: -1 }, opTime: { ts: Timestamp(0, 0), t: -1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.071-0400 c20014| 2018-03-16T17:28:18.067-0400 D REPL_HB [replexec-0] Sending heartbeat (requestId: 1) to ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: -2, hbv: 1, from: "", fromId: -1, term: 0 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.071-0400 c20012| 2018-03-16T17:28:18.067-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Ending connection to host ubuntu:20014 due to bad connection status; 0 connections to that host remain open [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.071-0400 c20013| 2018-03-16T17:28:18.067-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.072-0400 c20014| 2018-03-16T17:28:18.067-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.072-0400 c20014| 2018-03-16T17:28:18.067-0400 I NETWORK [conn3] end connection 127.0.0.1:44398 (1 connection now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.072-0400 c20012| 2018-03-16T17:28:18.067-0400 D REPL_HB [replexec-0] Cancelling all heartbeats. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.073-0400 c20012| 2018-03-16T17:28:18.067-0400 I REPL [replexec-0] New replica set config in use: { _id: "stale_mongos_updates_and_removes-configRS", version: 2, configsvr: true, protocolVersion: 1, members: [ { _id: 0, host: "ubuntu:20012", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 1, host: "ubuntu:20013", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 2, host: "ubuntu:20014", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, catchUpTimeoutMillis: -1, catchUpTakeoverDelayMillis: 30000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 }, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f') } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.073-0400 c20012| 2018-03-16T17:28:18.067-0400 I REPL [replexec-0] This node is ubuntu:20012 in the config [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.073-0400 c20012| 2018-03-16T17:28:18.067-0400 D REPL_HB [replexec-0] Scheduling heartbeat to ubuntu:20013 at 2018-03-16T21:28:18.067Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.073-0400 c20012| 2018-03-16T17:28:18.067-0400 D REPL_HB [replexec-0] Scheduling heartbeat to ubuntu:20014 at 2018-03-16T21:28:18.067Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.073-0400 c20012| 2018-03-16T17:28:18.067-0400 D REPL_HB [replexec-2] Sending heartbeat (requestId: 7) to ubuntu:20013, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20012", fromId: 0, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.074-0400 c20012| 2018-03-16T17:28:18.068-0400 D REPL_HB [replexec-1] Sending heartbeat (requestId: 8) to ubuntu:20014, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20012", fromId: 0, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.074-0400 c20012| 2018-03-16T17:28:18.068-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to ubuntu:20014 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.074-0400 c20012| 2018-03-16T17:28:18.068-0400 I NETWORK [listener] connection accepted from 127.0.0.1:60614 #8 (2 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.074-0400 c20013| 2018-03-16T17:28:18.068-0400 D REPL_HB [conn3] Received heartbeat request from ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20012", fromId: 0, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235698, 1), signature: { hash: BinData(0, 39C1AEA96A684F83B4F8E41F2FCCF25802D76EFE), keyId: 6533657568122765331 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.074-0400 c20013| 2018-03-16T17:28:18.068-0400 D REPL_HB [conn3] Processed heartbeat from ubuntu:20012 and generated response, { ok: 1.0, state: 0, v: -2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", durableOpTime: { ts: Timestamp(0, 0), t: -1 }, opTime: { ts: Timestamp(0, 0), t: -1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.075-0400 c20012| 2018-03-16T17:28:18.068-0400 I NETWORK [listener] connection accepted from 127.0.0.1:60616 #9 (3 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.075-0400 c20012| 2018-03-16T17:28:18.068-0400 I NETWORK [conn8] received client metadata from 127.0.0.1:60614 conn8: { driver: { name: "NetworkInterfaceASIO-Replication", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.075-0400 c20014| 2018-03-16T17:28:18.068-0400 I NETWORK [listener] connection accepted from 127.0.0.1:44404 #4 (2 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.076-0400 c20012| 2018-03-16T17:28:18.068-0400 D REPL_HB [replexec-0] Received response to heartbeat (requestId: 7) from ubuntu:20013, { ok: 1.0, state: 0, v: -2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", durableOpTime: { ts: Timestamp(0, 0), t: -1 }, opTime: { ts: Timestamp(0, 0), t: -1 }, $replData: { term: -1, lastOpCommitted: { ts: Timestamp(0, 0), t: -1 }, lastOpVisible: { ts: Timestamp(0, 0), t: -1 }, configVersion: 1, replicaSetId: ObjectId('000000000000000000000000'), primaryIndex: -1, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235698, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(0, 0) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.076-0400 c20012| 2018-03-16T17:28:18.068-0400 I NETWORK [conn9] received client metadata from 127.0.0.1:60616 conn9: { driver: { name: "NetworkInterfaceASIO-Replication", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.076-0400 c20012| 2018-03-16T17:28:18.068-0400 I REPL [replexec-0] Member ubuntu:20013 is now in state STARTUP [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.076-0400 c20012| 2018-03-16T17:28:18.068-0400 D REPL_HB [replexec-0] Scheduling heartbeat to ubuntu:20013 at 2018-03-16T21:28:20.068Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.076-0400 c20014| 2018-03-16T17:28:18.068-0400 I NETWORK [conn4] received client metadata from 127.0.0.1:44404 conn4: { driver: { name: "NetworkInterfaceASIO-Replication", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.077-0400 c20014| 2018-03-16T17:28:18.068-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Successfully connected to ubuntu:20012, took 1ms (1 connections now open to ubuntu:20012) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.077-0400 c20012| 2018-03-16T17:28:18.068-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Successfully connected to ubuntu:20014, took 0ms (1 connections now open to ubuntu:20014) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.077-0400 c20013| 2018-03-16T17:28:18.068-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Successfully connected to ubuntu:20012, took 1ms (1 connections now open to ubuntu:20012) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.077-0400 c20014| 2018-03-16T17:28:18.068-0400 D REPL_HB [conn4] Received heartbeat request from ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20012", fromId: 0, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235698, 1), signature: { hash: BinData(0, 39C1AEA96A684F83B4F8E41F2FCCF25802D76EFE), keyId: 6533657568122765331 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.078-0400 c20012| 2018-03-16T17:28:18.068-0400 D REPL_HB [conn8] Received heartbeat request from , { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: -2, hbv: 1, from: "", fromId: -1, term: 0, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235697, 23), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.078-0400 c20014| 2018-03-16T17:28:18.068-0400 D REPL_HB [conn4] Processed heartbeat from ubuntu:20012 and generated response, { ok: 1.0, state: 0, v: -2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", durableOpTime: { ts: Timestamp(0, 0), t: -1 }, opTime: { ts: Timestamp(0, 0), t: -1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.078-0400 c20012| 2018-03-16T17:28:18.068-0400 D REPL_HB [conn9] Received heartbeat request from , { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: -2, hbv: 1, from: "", fromId: -1, term: 0, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235697, 23), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.079-0400 c20012| 2018-03-16T17:28:18.068-0400 D REPL_HB [conn8] Processed heartbeat from and generated response, { ok: 1.0, electionTime: new Date(6533657559532830722), config: { _id: "stale_mongos_updates_and_removes-configRS", version: 2, configsvr: true, protocolVersion: 1, members: [ { _id: 0, host: "ubuntu:20012", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 1, host: "ubuntu:20013", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 2, host: "ubuntu:20014", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, catchUpTimeoutMillis: -1, catchUpTakeoverDelayMillis: 30000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 }, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f') } }, state: 1, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235697, 23), t: 1 }, opTime: { ts: Timestamp(1521235698, 1), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.080-0400 c20012| 2018-03-16T17:28:18.068-0400 D REPL_HB [conn9] Processed heartbeat from and generated response, { ok: 1.0, electionTime: new Date(6533657559532830722), config: { _id: "stale_mongos_updates_and_removes-configRS", version: 2, configsvr: true, protocolVersion: 1, members: [ { _id: 0, host: "ubuntu:20012", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 1, host: "ubuntu:20013", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 2, host: "ubuntu:20014", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, catchUpTimeoutMillis: -1, catchUpTakeoverDelayMillis: 30000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 }, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f') } }, state: 1, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235697, 23), t: 1 }, opTime: { ts: Timestamp(1521235698, 1), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.081-0400 c20012| 2018-03-16T17:28:18.068-0400 D REPL_HB [replexec-2] Received response to heartbeat (requestId: 8) from ubuntu:20014, { ok: 1.0, state: 0, v: -2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", durableOpTime: { ts: Timestamp(0, 0), t: -1 }, opTime: { ts: Timestamp(0, 0), t: -1 }, $replData: { term: -1, lastOpCommitted: { ts: Timestamp(0, 0), t: -1 }, lastOpVisible: { ts: Timestamp(0, 0), t: -1 }, configVersion: 1, replicaSetId: ObjectId('000000000000000000000000'), primaryIndex: -1, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235698, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(0, 0) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.081-0400 c20012| 2018-03-16T17:28:18.068-0400 I REPL [replexec-2] Member ubuntu:20014 is now in state STARTUP [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.081-0400 c20012| 2018-03-16T17:28:18.068-0400 D REPL_HB [replexec-2] Scheduling heartbeat to ubuntu:20014 at 2018-03-16T21:28:20.068Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.083-0400 c20013| 2018-03-16T17:28:18.069-0400 D REPL_HB [replexec-0] Received response to heartbeat (requestId: 1) from ubuntu:20012, { ok: 1.0, electionTime: new Date(6533657559532830722), config: { _id: "stale_mongos_updates_and_removes-configRS", version: 2, configsvr: true, protocolVersion: 1, members: [ { _id: 0, host: "ubuntu:20012", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 1, host: "ubuntu:20013", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 2, host: "ubuntu:20014", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, catchUpTimeoutMillis: -1, catchUpTakeoverDelayMillis: 30000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 }, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f') } }, state: 1, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235697, 23), t: 1 }, opTime: { ts: Timestamp(1521235698, 1), t: 1 }, operationTime: Timestamp(1521235698, 1), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235697, 23), t: 1 }, lastOpVisible: { ts: Timestamp(0, 0), t: -1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235698, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235697, 23) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.083-0400 c20013| 2018-03-16T17:28:18.069-0400 D REPL_HB [replexec-0] Scheduling heartbeat to ubuntu:20012 at 2018-03-16T21:28:20.069Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.084-0400 c20014| 2018-03-16T17:28:18.069-0400 D REPL_HB [replexec-0] Received response to heartbeat (requestId: 1) from ubuntu:20012, { ok: 1.0, electionTime: new Date(6533657559532830722), config: { _id: "stale_mongos_updates_and_removes-configRS", version: 2, configsvr: true, protocolVersion: 1, members: [ { _id: 0, host: "ubuntu:20012", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 1, host: "ubuntu:20013", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 2, host: "ubuntu:20014", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, catchUpTimeoutMillis: -1, catchUpTakeoverDelayMillis: 30000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 }, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f') } }, state: 1, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235697, 23), t: 1 }, opTime: { ts: Timestamp(1521235698, 1), t: 1 }, operationTime: Timestamp(1521235698, 1), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235697, 23), t: 1 }, lastOpVisible: { ts: Timestamp(0, 0), t: -1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235698, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235697, 23) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.084-0400 c20013| 2018-03-16T17:28:18.069-0400 D REPL_HB [replexec-0] Received new config via heartbeat with version 2 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.084-0400 c20014| 2018-03-16T17:28:18.069-0400 D REPL_HB [replexec-0] Scheduling heartbeat to ubuntu:20012 at 2018-03-16T21:28:20.069Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.085-0400 c20014| 2018-03-16T17:28:18.069-0400 D REPL_HB [replexec-0] Received new config via heartbeat with version 2 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.085-0400 c20012| 2018-03-16T17:28:18.069-0400 I NETWORK [listener] connection accepted from 127.0.0.1:60620 #10 (4 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.085-0400 c20012| 2018-03-16T17:28:18.069-0400 I NETWORK [listener] connection accepted from 127.0.0.1:60622 #11 (5 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.085-0400 c20012| 2018-03-16T17:28:18.069-0400 I NETWORK [conn10] end connection 127.0.0.1:60620 (4 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.085-0400 c20012| 2018-03-16T17:28:18.069-0400 I NETWORK [conn11] end connection 127.0.0.1:60622 (3 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.085-0400 c20013| 2018-03-16T17:28:18.070-0400 I NETWORK [listener] connection accepted from 127.0.0.1:46990 #5 (3 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.085-0400 c20013| 2018-03-16T17:28:18.070-0400 I NETWORK [listener] connection accepted from 127.0.0.1:46992 #7 (4 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.086-0400 c20013| 2018-03-16T17:28:18.070-0400 I NETWORK [conn5] end connection 127.0.0.1:46990 (3 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.086-0400 c20013| 2018-03-16T17:28:18.070-0400 I NETWORK [conn7] end connection 127.0.0.1:46992 (2 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.086-0400 c20014| 2018-03-16T17:28:18.070-0400 I NETWORK [listener] connection accepted from 127.0.0.1:44414 #7 (3 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.086-0400 c20014| 2018-03-16T17:28:18.070-0400 I NETWORK [listener] connection accepted from 127.0.0.1:44416 #9 (4 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.086-0400 c20014| 2018-03-16T17:28:18.070-0400 I NETWORK [conn7] end connection 127.0.0.1:44414 (3 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.086-0400 c20013| 2018-03-16T17:28:18.070-0400 D REPL_HB [replexec-0] Config with version 2 validated for reconfig; persisting to disk. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.087-0400 c20013| 2018-03-16T17:28:18.070-0400 I STORAGE [replexec-0] createCollection: local.system.replset with generated UUID: 904912d5-7884-4454-aab0-babac8d690a3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.087-0400 c20014| 2018-03-16T17:28:18.070-0400 D REPL_HB [replexec-0] Config with version 2 validated for reconfig; persisting to disk. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.087-0400 c20014| 2018-03-16T17:28:18.070-0400 I NETWORK [conn9] end connection 127.0.0.1:44416 (2 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.087-0400 c20014| 2018-03-16T17:28:18.070-0400 I STORAGE [replexec-0] createCollection: local.system.replset with generated UUID: a79b4fd6-d7db-4f7e-9964-ce39588115c8 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.097-0400 c20013| 2018-03-16T17:28:18.097-0400 D REPL_HB [replexec-0] New configuration with version 2 persisted to local storage; installing new config in memory [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.097-0400 c20013| 2018-03-16T17:28:18.097-0400 D REPL_HB [replexec-0] Cancelling all heartbeats. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.098-0400 c20013| 2018-03-16T17:28:18.097-0400 I REPL [replexec-0] New replica set config in use: { _id: "stale_mongos_updates_and_removes-configRS", version: 2, configsvr: true, protocolVersion: 1, members: [ { _id: 0, host: "ubuntu:20012", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 1, host: "ubuntu:20013", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 2, host: "ubuntu:20014", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, catchUpTimeoutMillis: -1, catchUpTakeoverDelayMillis: 30000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 }, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f') } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.098-0400 c20013| 2018-03-16T17:28:18.097-0400 I REPL [replexec-0] This node is ubuntu:20013 in the config [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.098-0400 c20013| 2018-03-16T17:28:18.097-0400 I REPL [replexec-0] transition to STARTUP2 from STARTUP [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.098-0400 c20013| 2018-03-16T17:28:18.097-0400 D REPL_HB [replexec-0] Scheduling heartbeat to ubuntu:20012 at 2018-03-16T21:28:18.097Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.099-0400 c20013| 2018-03-16T17:28:18.097-0400 D REPL_HB [replexec-0] Scheduling heartbeat to ubuntu:20014 at 2018-03-16T21:28:18.097Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.099-0400 c20013| 2018-03-16T17:28:18.097-0400 D REPL_HB [replexec-1] Sending heartbeat (requestId: 4) to ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 0 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.099-0400 c20013| 2018-03-16T17:28:18.097-0400 I REPL [replexec-0] Starting replication storage threads [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.099-0400 c20013| 2018-03-16T17:28:18.097-0400 D REPL_HB [replexec-2] Sending heartbeat (requestId: 5) to ubuntu:20014, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 0 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.099-0400 c20013| 2018-03-16T17:28:18.097-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to ubuntu:20014 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.100-0400 c20012| 2018-03-16T17:28:18.097-0400 D REPL_HB [conn9] Received heartbeat request from ubuntu:20013, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 0, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235698, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.100-0400 c20012| 2018-03-16T17:28:18.097-0400 D REPL_HB [conn9] Processed heartbeat from ubuntu:20013 and generated response, { ok: 1.0, electionTime: new Date(6533657559532830722), state: 1, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235697, 23), t: 1 }, opTime: { ts: Timestamp(1521235698, 1), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.101-0400 c20013| 2018-03-16T17:28:18.097-0400 D REPL_HB [replexec-3] Received response to heartbeat (requestId: 4) from ubuntu:20012, { ok: 1.0, electionTime: new Date(6533657559532830722), state: 1, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235697, 23), t: 1 }, opTime: { ts: Timestamp(1521235698, 1), t: 1 }, operationTime: Timestamp(1521235698, 1), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235697, 23), t: 1 }, lastOpVisible: { ts: Timestamp(0, 0), t: -1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235698, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235697, 23) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.101-0400 c20014| 2018-03-16T17:28:18.097-0400 I NETWORK [listener] connection accepted from 127.0.0.1:44418 #10 (3 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.101-0400 c20013| 2018-03-16T17:28:18.097-0400 I REPL [replexec-3] Member ubuntu:20012 is now in state PRIMARY [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.101-0400 c20013| 2018-03-16T17:28:18.097-0400 D REPL_HB [replexec-3] Scheduling heartbeat to ubuntu:20012 at 2018-03-16T21:28:18.597Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.101-0400 c20014| 2018-03-16T17:28:18.097-0400 I NETWORK [conn10] received client metadata from 127.0.0.1:44418 conn10: { driver: { name: "NetworkInterfaceASIO-Replication", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.102-0400 c20013| 2018-03-16T17:28:18.098-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Successfully connected to ubuntu:20014, took 1ms (1 connections now open to ubuntu:20014) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.102-0400 c20014| 2018-03-16T17:28:18.098-0400 D REPL_HB [conn10] Received heartbeat request from ubuntu:20013, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 0, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235698, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.102-0400 c20014| 2018-03-16T17:28:18.098-0400 D REPL_HB [conn10] Scheduling heartbeat to ubuntu:20013 at 2018-03-16T21:28:18.098Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.102-0400 c20014| 2018-03-16T17:28:18.098-0400 D REPL_HB [conn10] Processed heartbeat from ubuntu:20013 and generated response, { ok: 1.0, state: 0, v: -2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, durableOpTime: { ts: Timestamp(0, 0), t: -1 }, opTime: { ts: Timestamp(0, 0), t: -1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.103-0400 c20014| 2018-03-16T17:28:18.098-0400 D REPL_HB [replexec-1] Sending heartbeat (requestId: 4) to ubuntu:20013, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: -2, hbv: 1, from: "", fromId: -1, term: 0 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.103-0400 c20013| 2018-03-16T17:28:18.098-0400 D REPL_HB [replexec-1] Received response to heartbeat (requestId: 5) from ubuntu:20014, { ok: 1.0, state: 0, v: -2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, durableOpTime: { ts: Timestamp(0, 0), t: -1 }, opTime: { ts: Timestamp(0, 0), t: -1 }, $replData: { term: 1, lastOpCommitted: { ts: Timestamp(0, 0), t: -1 }, lastOpVisible: { ts: Timestamp(0, 0), t: -1 }, configVersion: 1, replicaSetId: ObjectId('000000000000000000000000'), primaryIndex: -1, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235698, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(0, 0) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.104-0400 c20013| 2018-03-16T17:28:18.098-0400 I REPL [replexec-1] Member ubuntu:20014 is now in state STARTUP [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.104-0400 c20013| 2018-03-16T17:28:18.098-0400 D REPL_HB [replexec-1] Scheduling heartbeat to ubuntu:20014 at 2018-03-16T21:28:18.598Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.104-0400 c20014| 2018-03-16T17:28:18.098-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to ubuntu:20013 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.104-0400 c20013| 2018-03-16T17:28:18.098-0400 I NETWORK [listener] connection accepted from 127.0.0.1:47000 #9 (3 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.104-0400 c20013| 2018-03-16T17:28:18.098-0400 I NETWORK [conn9] received client metadata from 127.0.0.1:47000 conn9: { driver: { name: "NetworkInterfaceASIO-Replication", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.104-0400 c20014| 2018-03-16T17:28:18.098-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Successfully connected to ubuntu:20013, took 0ms (1 connections now open to ubuntu:20013) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.105-0400 c20013| 2018-03-16T17:28:18.098-0400 D REPL_HB [conn9] Received heartbeat request from , { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: -2, hbv: 1, from: "", fromId: -1, term: 0, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235698, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.106-0400 c20013| 2018-03-16T17:28:18.098-0400 D REPL_HB [conn9] Processed heartbeat from and generated response, { ok: 1.0, config: { _id: "stale_mongos_updates_and_removes-configRS", version: 2, configsvr: true, protocolVersion: 1, members: [ { _id: 0, host: "ubuntu:20012", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 1, host: "ubuntu:20013", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 2, host: "ubuntu:20014", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, catchUpTimeoutMillis: -1, catchUpTakeoverDelayMillis: 30000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 }, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f') } }, state: 5, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(0, 0), t: -1 }, opTime: { ts: Timestamp(0, 0), t: -1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.107-0400 c20014| 2018-03-16T17:28:18.099-0400 D REPL_HB [replexec-1] Received response to heartbeat (requestId: 4) from ubuntu:20013, { ok: 1.0, config: { _id: "stale_mongos_updates_and_removes-configRS", version: 2, configsvr: true, protocolVersion: 1, members: [ { _id: 0, host: "ubuntu:20012", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 1, host: "ubuntu:20013", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 2, host: "ubuntu:20014", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, catchUpTimeoutMillis: -1, catchUpTakeoverDelayMillis: 30000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 }, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f') } }, state: 5, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(0, 0), t: -1 }, opTime: { ts: Timestamp(0, 0), t: -1 }, $replData: { term: 1, lastOpCommitted: { ts: Timestamp(0, 0), t: -1 }, lastOpVisible: { ts: Timestamp(0, 0), t: -1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235698, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(0, 0) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.107-0400 c20014| 2018-03-16T17:28:18.099-0400 D REPL_HB [replexec-1] Scheduling heartbeat to ubuntu:20013 at 2018-03-16T21:28:20.099Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.107-0400 c20014| 2018-03-16T17:28:18.099-0400 D REPL_HB [replexec-1] Ignoring new configuration with version 2 because already in the midst of a configuration process. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.107-0400 c20014| 2018-03-16T17:28:18.099-0400 D REPL_HB [replexec-0] New configuration with version 2 persisted to local storage; installing new config in memory [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.108-0400 c20014| 2018-03-16T17:28:18.099-0400 D REPL_HB [replexec-0] Cancelling all heartbeats. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.108-0400 c20014| 2018-03-16T17:28:18.099-0400 I REPL [replexec-0] New replica set config in use: { _id: "stale_mongos_updates_and_removes-configRS", version: 2, configsvr: true, protocolVersion: 1, members: [ { _id: 0, host: "ubuntu:20012", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 1, host: "ubuntu:20013", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 2, host: "ubuntu:20014", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, catchUpTimeoutMillis: -1, catchUpTakeoverDelayMillis: 30000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 }, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f') } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.108-0400 c20014| 2018-03-16T17:28:18.099-0400 I REPL [replexec-0] This node is ubuntu:20014 in the config [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.109-0400 c20014| 2018-03-16T17:28:18.099-0400 I REPL [replexec-0] transition to STARTUP2 from STARTUP [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.109-0400 c20014| 2018-03-16T17:28:18.099-0400 D REPL_HB [replexec-0] Scheduling heartbeat to ubuntu:20012 at 2018-03-16T21:28:18.099Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.109-0400 c20014| 2018-03-16T17:28:18.099-0400 D REPL_HB [replexec-0] Scheduling heartbeat to ubuntu:20013 at 2018-03-16T21:28:18.099Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.109-0400 c20014| 2018-03-16T17:28:18.099-0400 D REPL_HB [replexec-3] Sending heartbeat (requestId: 7) to ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 0 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.109-0400 c20014| 2018-03-16T17:28:18.099-0400 I REPL [replexec-0] Starting replication storage threads [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.109-0400 c20014| 2018-03-16T17:28:18.099-0400 D REPL_HB [replexec-4] Sending heartbeat (requestId: 8) to ubuntu:20013, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 0 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.110-0400 c20013| 2018-03-16T17:28:18.099-0400 I STORAGE [replexec-0] createCollection: local.temp_oplog_buffer with generated UUID: 5b139b5d-5eb7-419d-b973-1a48e30c85ae [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.110-0400 c20012| 2018-03-16T17:28:18.099-0400 D REPL_HB [conn8] Received heartbeat request from ubuntu:20014, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 0, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235698, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.110-0400 c20012| 2018-03-16T17:28:18.099-0400 D REPL_HB [conn8] Processed heartbeat from ubuntu:20014 and generated response, { ok: 1.0, electionTime: new Date(6533657559532830722), state: 1, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235697, 23), t: 1 }, opTime: { ts: Timestamp(1521235698, 1), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.111-0400 c20013| 2018-03-16T17:28:18.100-0400 D REPL_HB [conn9] Received heartbeat request from ubuntu:20014, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 0, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235698, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.111-0400 c20013| 2018-03-16T17:28:18.100-0400 D REPL_HB [conn9] Processed heartbeat from ubuntu:20014 and generated response, { ok: 1.0, state: 5, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(0, 0), t: -1 }, opTime: { ts: Timestamp(0, 0), t: -1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.112-0400 c20014| 2018-03-16T17:28:18.100-0400 D REPL_HB [replexec-3] Received response to heartbeat (requestId: 7) from ubuntu:20012, { ok: 1.0, electionTime: new Date(6533657559532830722), state: 1, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235697, 23), t: 1 }, opTime: { ts: Timestamp(1521235698, 1), t: 1 }, operationTime: Timestamp(1521235698, 1), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235697, 23), t: 1 }, lastOpVisible: { ts: Timestamp(0, 0), t: -1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235698, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235697, 23) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.112-0400 c20014| 2018-03-16T17:28:18.100-0400 I REPL [replexec-3] Member ubuntu:20012 is now in state PRIMARY [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.112-0400 c20014| 2018-03-16T17:28:18.100-0400 D REPL_HB [replexec-3] Scheduling heartbeat to ubuntu:20012 at 2018-03-16T21:28:18.600Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.113-0400 c20014| 2018-03-16T17:28:18.100-0400 D REPL_HB [replexec-2] Received response to heartbeat (requestId: 8) from ubuntu:20013, { ok: 1.0, state: 5, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(0, 0), t: -1 }, opTime: { ts: Timestamp(0, 0), t: -1 }, $replData: { term: 1, lastOpCommitted: { ts: Timestamp(0, 0), t: -1 }, lastOpVisible: { ts: Timestamp(0, 0), t: -1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235698, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(0, 0) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.113-0400 c20014| 2018-03-16T17:28:18.100-0400 I REPL [replexec-2] Member ubuntu:20013 is now in state STARTUP2 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.113-0400 c20014| 2018-03-16T17:28:18.100-0400 D REPL_HB [replexec-2] Scheduling heartbeat to ubuntu:20013 at 2018-03-16T21:28:18.600Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.113-0400 c20014| 2018-03-16T17:28:18.102-0400 I STORAGE [replexec-0] createCollection: local.temp_oplog_buffer with generated UUID: a5c7dacf-72f4-4531-bd1a-fb7291102cb1 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.121-0400 c20013| 2018-03-16T17:28:18.120-0400 I REPL [replication-0] Starting initial sync (attempt 1 of 1) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.121-0400 c20013| 2018-03-16T17:28:18.121-0400 I STORAGE [replication-0] Finishing collection drop for local.temp_oplog_buffer (5b139b5d-5eb7-419d-b973-1a48e30c85ae). [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.129-0400 c20013| 2018-03-16T17:28:18.128-0400 I STORAGE [replication-0] createCollection: local.temp_oplog_buffer with generated UUID: ddb36586-d2ee-45df-91ea-6e625bd06e07 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.131-0400 c20014| 2018-03-16T17:28:18.131-0400 I REPL [replication-0] Starting initial sync (attempt 1 of 1) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.132-0400 c20014| 2018-03-16T17:28:18.131-0400 I STORAGE [replication-0] Finishing collection drop for local.temp_oplog_buffer (a5c7dacf-72f4-4531-bd1a-fb7291102cb1). [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.142-0400 c20014| 2018-03-16T17:28:18.142-0400 I STORAGE [replication-0] createCollection: local.temp_oplog_buffer with generated UUID: fcca0d4f-610c-4b68-a9c9-071a25be4587 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.168-0400 c20014| 2018-03-16T17:28:18.168-0400 I REPL [replication-0] sync source candidate: ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.168-0400 c20014| 2018-03-16T17:28:18.168-0400 I REPL [replication-0] ****** [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.168-0400 c20014| 2018-03-16T17:28:18.168-0400 I REPL [replication-0] creating replication oplog of size: 40MB... [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.169-0400 c20014| 2018-03-16T17:28:18.168-0400 I STORAGE [replication-0] createCollection: local.oplog.rs with generated UUID: 418030f5-6050-48bd-b00e-590e5dcb63de [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.176-0400 c20014| 2018-03-16T17:28:18.175-0400 I STORAGE [replication-0] Starting WiredTigerRecordStoreThread local.oplog.rs [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.176-0400 c20014| 2018-03-16T17:28:18.175-0400 I STORAGE [replication-0] The size storer reports that the oplog contains 0 records totaling to 0 bytes [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.176-0400 c20014| 2018-03-16T17:28:18.176-0400 I STORAGE [replication-0] Scanning the oplog to determine where to place markers for truncation [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.236-0400 c20014| 2018-03-16T17:28:18.236-0400 I REPL [replication-0] ****** [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.236-0400 c20014| 2018-03-16T17:28:18.236-0400 I STORAGE [replication-0] dropAllDatabasesExceptLocal 1 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.236-0400 c20014| 2018-03-16T17:28:18.236-0400 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.237-0400 c20012| 2018-03-16T17:28:18.236-0400 I NETWORK [listener] connection accepted from 127.0.0.1:60636 #12 (4 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.237-0400 c20012| 2018-03-16T17:28:18.236-0400 I NETWORK [conn12] received client metadata from 127.0.0.1:60636 conn12: { driver: { name: "NetworkInterfaceASIO-RS", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.237-0400 c20014| 2018-03-16T17:28:18.237-0400 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to ubuntu:20012, took 1ms (1 connections now open to ubuntu:20012) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.238-0400 c20014| 2018-03-16T17:28:18.237-0400 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.238-0400 c20012| 2018-03-16T17:28:18.238-0400 I NETWORK [listener] connection accepted from 127.0.0.1:60638 #13 (5 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.238-0400 c20012| 2018-03-16T17:28:18.238-0400 I NETWORK [conn13] received client metadata from 127.0.0.1:60638 conn13: { driver: { name: "NetworkInterfaceASIO-RS", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.238-0400 c20014| 2018-03-16T17:28:18.238-0400 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to ubuntu:20012, took 1ms (2 connections now open to ubuntu:20012) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.239-0400 c20014| 2018-03-16T17:28:18.238-0400 I REPL [replication-1] CollectionCloner::start called, on ns:admin.system.version [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.239-0400 c20014| 2018-03-16T17:28:18.239-0400 I STORAGE [repl writer worker 13] createCollection: admin.system.version with provided UUID: daa3215e-f467-4b5e-bffb-efcfd121a869 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.279-0400 c20014| 2018-03-16T17:28:18.279-0400 I INDEX [repl writer worker 13] build index on: admin.system.version properties: { v: 2, key: { _id: 1 }, name: "_id_", ns: "admin.system.version" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.279-0400 c20014| 2018-03-16T17:28:18.279-0400 I INDEX [repl writer worker 13] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.280-0400 c20014| 2018-03-16T17:28:18.280-0400 I COMMAND [repl writer worker 0] setting featureCompatibilityVersion to 4.0 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.280-0400 c20014| 2018-03-16T17:28:18.280-0400 I NETWORK [repl writer worker 0] Skip closing connection for connection # 10 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.280-0400 c20014| 2018-03-16T17:28:18.280-0400 I NETWORK [repl writer worker 0] Skip closing connection for connection # 4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.281-0400 c20014| 2018-03-16T17:28:18.280-0400 I NETWORK [repl writer worker 0] Skip closing connection for connection # 1 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.281-0400 c20014| 2018-03-16T17:28:18.280-0400 I REPL [repl writer worker 0] CollectionCloner ns:admin.system.version finished cloning with status: OK [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.284-0400 c20014| 2018-03-16T17:28:18.284-0400 I REPL [repl writer worker 0] CollectionCloner::start called, on ns:admin.system.keys [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.285-0400 c20014| 2018-03-16T17:28:18.284-0400 I STORAGE [repl writer worker 15] createCollection: admin.system.keys with provided UUID: 32997d36-cabe-4e06-8861-b98e77fe07ee [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.301-0400 c20014| 2018-03-16T17:28:18.300-0400 I INDEX [repl writer worker 15] build index on: admin.system.keys properties: { v: 2, key: { _id: 1 }, name: "_id_", ns: "admin.system.keys" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.301-0400 c20014| 2018-03-16T17:28:18.300-0400 I INDEX [repl writer worker 15] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.302-0400 c20014| 2018-03-16T17:28:18.301-0400 I REPL [repl writer worker 2] CollectionCloner ns:admin.system.keys finished cloning with status: OK [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.306-0400 c20014| 2018-03-16T17:28:18.306-0400 I REPL [replication-0] CollectionCloner::start called, on ns:config.shards [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.307-0400 c20014| 2018-03-16T17:28:18.307-0400 I STORAGE [repl writer worker 3] createCollection: config.shards with provided UUID: c13519a3-4118-49d6-8a1d-2a757c74d1f7 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.324-0400 c20014| 2018-03-16T17:28:18.323-0400 I INDEX [repl writer worker 3] build index on: config.shards properties: { v: 2, unique: true, key: { host: 1 }, name: "host_1", ns: "config.shards" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.324-0400 c20014| 2018-03-16T17:28:18.323-0400 I INDEX [repl writer worker 3] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.333-0400 c20014| 2018-03-16T17:28:18.333-0400 I INDEX [repl writer worker 3] build index on: config.shards properties: { v: 2, key: { _id: 1 }, name: "_id_", ns: "config.shards" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.333-0400 c20014| 2018-03-16T17:28:18.333-0400 I INDEX [repl writer worker 3] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.334-0400 c20014| 2018-03-16T17:28:18.333-0400 W REPL [repl writer worker 4] _insertDocumentsCallback, but no documents to insert for ns:config.shards [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.334-0400 c20014| 2018-03-16T17:28:18.334-0400 I REPL [repl writer worker 4] CollectionCloner ns:config.shards finished cloning with status: OK [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.335-0400 c20014| 2018-03-16T17:28:18.335-0400 I REPL [repl writer worker 4] CollectionCloner::start called, on ns:config.chunks [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.336-0400 c20014| 2018-03-16T17:28:18.336-0400 I STORAGE [repl writer worker 5] createCollection: config.chunks with provided UUID: ca74780e-951c-4639-ab72-4df24247c87b [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.357-0400 c20014| 2018-03-16T17:28:18.356-0400 I INDEX [repl writer worker 5] build index on: config.chunks properties: { v: 2, unique: true, key: { ns: 1, min: 1 }, name: "ns_1_min_1", ns: "config.chunks" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.357-0400 c20014| 2018-03-16T17:28:18.356-0400 I INDEX [repl writer worker 5] building index using bulk method; build may temporarily use up to 166 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.366-0400 c20014| 2018-03-16T17:28:18.366-0400 I INDEX [repl writer worker 5] build index on: config.chunks properties: { v: 2, unique: true, key: { ns: 1, shard: 1, min: 1 }, name: "ns_1_shard_1_min_1", ns: "config.chunks" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.366-0400 c20014| 2018-03-16T17:28:18.366-0400 I INDEX [repl writer worker 5] building index using bulk method; build may temporarily use up to 166 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.395-0400 c20014| 2018-03-16T17:28:18.394-0400 I INDEX [repl writer worker 5] build index on: config.chunks properties: { v: 2, unique: true, key: { ns: 1, lastmod: 1 }, name: "ns_1_lastmod_1", ns: "config.chunks" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.395-0400 c20014| 2018-03-16T17:28:18.395-0400 I INDEX [repl writer worker 5] building index using bulk method; build may temporarily use up to 166 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.403-0400 c20014| 2018-03-16T17:28:18.403-0400 I INDEX [repl writer worker 5] build index on: config.chunks properties: { v: 2, key: { _id: 1 }, name: "_id_", ns: "config.chunks" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.404-0400 c20014| 2018-03-16T17:28:18.403-0400 I INDEX [repl writer worker 5] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.404-0400 c20014| 2018-03-16T17:28:18.404-0400 W REPL [repl writer worker 14] _insertDocumentsCallback, but no documents to insert for ns:config.chunks [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.404-0400 c20014| 2018-03-16T17:28:18.404-0400 I REPL [repl writer worker 14] CollectionCloner ns:config.chunks finished cloning with status: OK [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.407-0400 c20014| 2018-03-16T17:28:18.406-0400 I REPL [repl writer worker 14] CollectionCloner::start called, on ns:config.version [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.407-0400 c20014| 2018-03-16T17:28:18.407-0400 I STORAGE [repl writer worker 6] createCollection: config.version with provided UUID: 3b13a852-5aec-4401-9e13-c21941b9b63f [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.424-0400 c20014| 2018-03-16T17:28:18.423-0400 I INDEX [repl writer worker 6] build index on: config.version properties: { v: 2, key: { _id: 1 }, name: "_id_", ns: "config.version" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.424-0400 c20014| 2018-03-16T17:28:18.423-0400 I INDEX [repl writer worker 6] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.424-0400 c20014| 2018-03-16T17:28:18.424-0400 I REPL [repl writer worker 1] CollectionCloner ns:config.version finished cloning with status: OK [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.428-0400 c20014| 2018-03-16T17:28:18.428-0400 I REPL [repl writer worker 1] CollectionCloner::start called, on ns:config.transactions [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.429-0400 c20014| 2018-03-16T17:28:18.428-0400 I STORAGE [repl writer worker 7] createCollection: config.transactions with provided UUID: 01eb99c7-45a1-4fab-8a52-0bfdf9e027e2 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.446-0400 c20014| 2018-03-16T17:28:18.445-0400 I INDEX [repl writer worker 7] build index on: config.transactions properties: { v: 2, key: { _id: 1 }, name: "_id_", ns: "config.transactions" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.446-0400 c20014| 2018-03-16T17:28:18.445-0400 I INDEX [repl writer worker 7] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.446-0400 c20014| 2018-03-16T17:28:18.446-0400 W REPL [repl writer worker 8] _insertDocumentsCallback, but no documents to insert for ns:config.transactions [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.446-0400 c20014| 2018-03-16T17:28:18.446-0400 I REPL [repl writer worker 8] CollectionCloner ns:config.transactions finished cloning with status: OK [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.447-0400 c20014| 2018-03-16T17:28:18.447-0400 I REPL [repl writer worker 8] CollectionCloner::start called, on ns:config.locks [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.448-0400 c20014| 2018-03-16T17:28:18.448-0400 I STORAGE [repl writer worker 9] createCollection: config.locks with provided UUID: 437da60d-d424-4d25-bc6a-eb52c6ac482a [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.464-0400 c20014| 2018-03-16T17:28:18.464-0400 I INDEX [repl writer worker 9] build index on: config.locks properties: { v: 2, key: { ts: 1 }, name: "ts_1", ns: "config.locks" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.465-0400 c20014| 2018-03-16T17:28:18.464-0400 I INDEX [repl writer worker 9] building index using bulk method; build may temporarily use up to 250 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.473-0400 c20014| 2018-03-16T17:28:18.473-0400 I INDEX [repl writer worker 9] build index on: config.locks properties: { v: 2, key: { state: 1, process: 1 }, name: "state_1_process_1", ns: "config.locks" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.473-0400 c20014| 2018-03-16T17:28:18.473-0400 I INDEX [repl writer worker 9] building index using bulk method; build may temporarily use up to 250 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.481-0400 c20014| 2018-03-16T17:28:18.481-0400 I INDEX [repl writer worker 9] build index on: config.locks properties: { v: 2, key: { _id: 1 }, name: "_id_", ns: "config.locks" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.481-0400 c20014| 2018-03-16T17:28:18.481-0400 I INDEX [repl writer worker 9] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.481-0400 c20014| 2018-03-16T17:28:18.481-0400 W REPL [repl writer worker 10] _insertDocumentsCallback, but no documents to insert for ns:config.locks [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.481-0400 c20014| 2018-03-16T17:28:18.481-0400 I REPL [repl writer worker 10] CollectionCloner ns:config.locks finished cloning with status: OK [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.483-0400 c20014| 2018-03-16T17:28:18.483-0400 I REPL [repl writer worker 10] CollectionCloner::start called, on ns:config.tags [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.484-0400 c20014| 2018-03-16T17:28:18.484-0400 I STORAGE [repl writer worker 11] createCollection: config.tags with provided UUID: e213515a-d815-4811-a6df-51ef6c4f2652 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.500-0400 c20014| 2018-03-16T17:28:18.500-0400 I INDEX [repl writer worker 11] build index on: config.tags properties: { v: 2, unique: true, key: { ns: 1, min: 1 }, name: "ns_1_min_1", ns: "config.tags" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.500-0400 c20014| 2018-03-16T17:28:18.500-0400 I INDEX [repl writer worker 11] building index using bulk method; build may temporarily use up to 250 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.509-0400 c20014| 2018-03-16T17:28:18.508-0400 I INDEX [repl writer worker 11] build index on: config.tags properties: { v: 2, key: { ns: 1, tag: 1 }, name: "ns_1_tag_1", ns: "config.tags" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.509-0400 c20014| 2018-03-16T17:28:18.508-0400 I INDEX [repl writer worker 11] building index using bulk method; build may temporarily use up to 250 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.516-0400 c20014| 2018-03-16T17:28:18.515-0400 I INDEX [repl writer worker 11] build index on: config.tags properties: { v: 2, key: { _id: 1 }, name: "_id_", ns: "config.tags" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.516-0400 c20014| 2018-03-16T17:28:18.515-0400 I INDEX [repl writer worker 11] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.516-0400 c20014| 2018-03-16T17:28:18.516-0400 W REPL [repl writer worker 12] _insertDocumentsCallback, but no documents to insert for ns:config.tags [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.516-0400 c20014| 2018-03-16T17:28:18.516-0400 I REPL [repl writer worker 12] CollectionCloner ns:config.tags finished cloning with status: OK [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.518-0400 c20014| 2018-03-16T17:28:18.518-0400 I REPL [repl writer worker 12] CollectionCloner::start called, on ns:config.migrations [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.519-0400 c20014| 2018-03-16T17:28:18.519-0400 I STORAGE [repl writer worker 13] createCollection: config.migrations with provided UUID: 7121c15b-943b-4d4b-b5dd-50f357a13814 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.535-0400 c20014| 2018-03-16T17:28:18.534-0400 I INDEX [repl writer worker 13] build index on: config.migrations properties: { v: 2, unique: true, key: { ns: 1, min: 1 }, name: "ns_1_min_1", ns: "config.migrations" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.535-0400 c20014| 2018-03-16T17:28:18.534-0400 I INDEX [repl writer worker 13] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.543-0400 c20014| 2018-03-16T17:28:18.543-0400 I INDEX [repl writer worker 13] build index on: config.migrations properties: { v: 2, key: { _id: 1 }, name: "_id_", ns: "config.migrations" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.543-0400 c20014| 2018-03-16T17:28:18.543-0400 I INDEX [repl writer worker 13] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.544-0400 c20014| 2018-03-16T17:28:18.543-0400 W REPL [repl writer worker 0] _insertDocumentsCallback, but no documents to insert for ns:config.migrations [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.544-0400 c20014| 2018-03-16T17:28:18.543-0400 I REPL [repl writer worker 0] CollectionCloner ns:config.migrations finished cloning with status: OK [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.545-0400 c20014| 2018-03-16T17:28:18.545-0400 I REPL [repl writer worker 0] CollectionCloner::start called, on ns:config.lockpings [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.546-0400 c20014| 2018-03-16T17:28:18.545-0400 I STORAGE [repl writer worker 15] createCollection: config.lockpings with provided UUID: 9c0b3109-1569-46d5-8030-c76bd8db19ee [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.562-0400 c20014| 2018-03-16T17:28:18.562-0400 I INDEX [repl writer worker 15] build index on: config.lockpings properties: { v: 2, key: { ping: 1 }, name: "ping_1", ns: "config.lockpings" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.562-0400 c20014| 2018-03-16T17:28:18.562-0400 I INDEX [repl writer worker 15] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.570-0400 c20014| 2018-03-16T17:28:18.570-0400 I INDEX [repl writer worker 15] build index on: config.lockpings properties: { v: 2, key: { _id: 1 }, name: "_id_", ns: "config.lockpings" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.570-0400 c20014| 2018-03-16T17:28:18.570-0400 I INDEX [repl writer worker 15] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.571-0400 c20014| 2018-03-16T17:28:18.571-0400 W REPL [repl writer worker 2] _insertDocumentsCallback, but no documents to insert for ns:config.lockpings [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.571-0400 c20014| 2018-03-16T17:28:18.571-0400 I REPL [repl writer worker 2] CollectionCloner ns:config.lockpings finished cloning with status: OK [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.573-0400 c20014| 2018-03-16T17:28:18.572-0400 I REPL [repl writer worker 2] Finished cloning data: OK. Beginning oplog replay. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.573-0400 c20014| 2018-03-16T17:28:18.573-0400 I REPL [replication-1] No need to apply operations. (currently at { : Timestamp(1521235698, 1) }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.573-0400 c20014| 2018-03-16T17:28:18.573-0400 I ASIO [NetworkInterfaceASIO-RS-0] Ending connection to host ubuntu:20012 due to bad connection status; 1 connections to that host remain open [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.574-0400 c20014| 2018-03-16T17:28:18.573-0400 I REPL [replication-1] Finished fetching oplog during initial sync: CallbackCanceled: error in fetcher batch callback: oplog fetcher is shutting down. Last fetched optime and hash: { ts: Timestamp(0, 0), t: -1 }[0] [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.574-0400 c20014| 2018-03-16T17:28:18.573-0400 I REPL [replication-1] Initial sync attempt finishing up. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.576-0400 c20014| 2018-03-16T17:28:18.573-0400 I REPL [replication-1] Initial Sync Attempt Statistics: { failedInitialSyncAttempts: 0, maxFailedInitialSyncAttempts: 1, initialSyncStart: new Date(1521235698131), initialSyncAttempts: [], fetchedMissingDocs: 0, appliedOps: 0, initialSyncOplogStart: Timestamp(1521235698, 1), initialSyncOplogEnd: Timestamp(1521235698, 1), databases: { databasesCloned: 2, admin: { collections: 2, clonedCollections: 2, start: new Date(1521235698238), end: new Date(1521235698305), elapsedMillis: 67, admin.system.version: { documentsToCopy: 1, documentsCopied: 1, indexes: 1, fetchedBatches: 1, start: new Date(1521235698238), end: new Date(1521235698284), elapsedMillis: 46 }, admin.system.keys: { documentsToCopy: 2, documentsCopied: 2, indexes: 1, fetchedBatches: 1, start: new Date(1521235698284), end: new Date(1521235698305), elapsedMillis: 21 } }, config: { collections: 8, clonedCollections: 8, start: new Date(1521235698305), end: new Date(1521235698573), elapsedMillis: 268, config.shards: { documentsToCopy: 0, documentsCopied: 0, indexes: 2, fetchedBatches: 0, start: new Date(1521235698306), end: new Date(1521235698335), elapsedMillis: 29 }, config.chunks: { documentsToCopy: 0, documentsCopied: 0, indexes: 4, fetchedBatches: 0, start: new Date(1521235698335), end: new Date(1521235698406), elapsedMillis: 71 }, config.version: { documentsToCopy: 1, documentsCopied: 1, indexes: 1, fetchedBatches: 1, start: new Date(1521235698406), end: new Date(1521235698428), elapsedMillis: 22 }, config.transactions: { documentsToCopy: 0, documentsCopied: 0, indexes: 1, fetchedBatches: 0, start: new Date(1521235698428), end: new Date(1521235698447), elapsedMillis: 19 }, config.locks: { documentsToCopy: 0, documentsCopied: 0, indexes: 3, fetchedBatches: 0, start: new Date(1521235698447), end: new Date(1521235698483), elapsedMillis: 36 }, config.tags: { documentsToCopy: 0, documentsCopied: 0, indexes: 3, fetchedBatches: 0, start: new Date(1521235698483), end: new Date(1521235698518), elapsedMillis: 35 }, config.migrations: { documentsToCopy: 0, documentsCopied: 0, indexes: 2, fetchedBatches: 0, start: new Date(1521235698518), end: new Date(1521235698545), elapsedMillis: 27 }, config.lockpings: { documentsToCopy: 0, documentsCopied: 0, indexes: 2, fetchedBatches: 0, start: new Date(1521235698545), end: new Date(1521235698573), elapsedMillis: 28 } } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.576-0400 c20014| 2018-03-16T17:28:18.573-0400 I STORAGE [replication-0] Finishing collection drop for local.temp_oplog_buffer (fcca0d4f-610c-4b68-a9c9-071a25be4587). [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.578-0400 c20014| 2018-03-16T17:28:18.578-0400 I REPL [replication-0] initial sync done; took 0s. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.578-0400 c20014| 2018-03-16T17:28:18.578-0400 I REPL [replication-0] transition to RECOVERING from STARTUP2 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.578-0400 c20014| 2018-03-16T17:28:18.578-0400 I REPL [replication-0] Starting replication fetcher thread [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.578-0400 c20014| 2018-03-16T17:28:18.578-0400 I REPL [replication-0] Starting replication applier thread [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.578-0400 c20014| 2018-03-16T17:28:18.578-0400 I REPL [rsBackgroundSync] could not find member to sync from [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.579-0400 c20014| 2018-03-16T17:28:18.578-0400 D REPL_HB [rsBackgroundSync] Cancelling all heartbeats. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.579-0400 c20014| 2018-03-16T17:28:18.578-0400 D REPL_HB [rsBackgroundSync] Scheduling heartbeat to ubuntu:20012 at 2018-03-16T21:28:18.578Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.579-0400 c20014| 2018-03-16T17:28:18.578-0400 D REPL_HB [rsBackgroundSync] Scheduling heartbeat to ubuntu:20013 at 2018-03-16T21:28:18.578Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.579-0400 c20014| 2018-03-16T17:28:18.578-0400 D REPL_HB [replexec-0] Sending heartbeat (requestId: 113) to ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.579-0400 c20014| 2018-03-16T17:28:18.578-0400 D REPL_HB [replexec-2] Sending heartbeat (requestId: 114) to ubuntu:20013, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.580-0400 c20012| 2018-03-16T17:28:18.578-0400 D REPL_HB [conn8] Received heartbeat request from ubuntu:20014, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235698, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.580-0400 c20013| 2018-03-16T17:28:18.579-0400 D REPL_HB [conn9] Received heartbeat request from ubuntu:20014, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235698, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.580-0400 c20014| 2018-03-16T17:28:18.579-0400 I REPL [replication-0] Starting replication reporter thread [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.581-0400 c20012| 2018-03-16T17:28:18.579-0400 D REPL_HB [conn8] Processed heartbeat from ubuntu:20014 and generated response, { ok: 1.0, electionTime: new Date(6533657559532830722), state: 1, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235698, 1), t: 1 }, opTime: { ts: Timestamp(1521235698, 1), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.581-0400 c20013| 2018-03-16T17:28:18.579-0400 D REPL_HB [conn9] Processed heartbeat from ubuntu:20014 and generated response, { ok: 1.0, state: 5, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(0, 0), t: -1 }, opTime: { ts: Timestamp(0, 0), t: -1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.582-0400 c20014| 2018-03-16T17:28:18.579-0400 D REPL_HB [replexec-3] Received response to heartbeat (requestId: 113) from ubuntu:20012, { ok: 1.0, electionTime: new Date(6533657559532830722), state: 1, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235698, 1), t: 1 }, opTime: { ts: Timestamp(1521235698, 1), t: 1 }, operationTime: Timestamp(1521235698, 1), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235697, 23), t: 1 }, lastOpVisible: { ts: Timestamp(0, 0), t: -1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235698, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235697, 23) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.582-0400 c20014| 2018-03-16T17:28:18.579-0400 D REPL_HB [replexec-3] Scheduling heartbeat to ubuntu:20012 at 2018-03-16T21:28:19.079Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.582-0400 c20014| 2018-03-16T17:28:18.579-0400 D REPL_HB [replexec-4] Received response to heartbeat (requestId: 114) from ubuntu:20013, { ok: 1.0, state: 5, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(0, 0), t: -1 }, opTime: { ts: Timestamp(0, 0), t: -1 }, $replData: { term: 1, lastOpCommitted: { ts: Timestamp(0, 0), t: -1 }, lastOpVisible: { ts: Timestamp(0, 0), t: -1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235698, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(0, 0) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.583-0400 c20014| 2018-03-16T17:28:18.579-0400 D REPL_HB [replexec-4] Scheduling heartbeat to ubuntu:20013 at 2018-03-16T21:28:19.079Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.583-0400 c20014| 2018-03-16T17:28:18.579-0400 I REPL [rsSync] transition to SECONDARY from RECOVERING [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.598-0400 c20013| 2018-03-16T17:28:18.598-0400 D REPL_HB [replexec-2] Sending heartbeat (requestId: 9) to ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.598-0400 c20012| 2018-03-16T17:28:18.598-0400 D REPL_HB [conn9] Received heartbeat request from ubuntu:20013, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235698, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.599-0400 c20012| 2018-03-16T17:28:18.598-0400 D REPL_HB [conn9] Processed heartbeat from ubuntu:20013 and generated response, { ok: 1.0, electionTime: new Date(6533657559532830722), state: 1, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235698, 1), t: 1 }, opTime: { ts: Timestamp(1521235698, 1), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.599-0400 c20013| 2018-03-16T17:28:18.598-0400 D REPL_HB [replexec-1] Received response to heartbeat (requestId: 9) from ubuntu:20012, { ok: 1.0, electionTime: new Date(6533657559532830722), state: 1, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235698, 1), t: 1 }, opTime: { ts: Timestamp(1521235698, 1), t: 1 }, operationTime: Timestamp(1521235698, 1), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235697, 23), t: 1 }, lastOpVisible: { ts: Timestamp(0, 0), t: -1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235698, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235697, 23) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.600-0400 c20013| 2018-03-16T17:28:18.598-0400 D REPL_HB [replexec-1] Scheduling heartbeat to ubuntu:20012 at 2018-03-16T21:28:19.098Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.600-0400 c20013| 2018-03-16T17:28:18.598-0400 D REPL_HB [replexec-3] Sending heartbeat (requestId: 11) to ubuntu:20014, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.600-0400 c20014| 2018-03-16T17:28:18.598-0400 D REPL_HB [conn10] Received heartbeat request from ubuntu:20013, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235698, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.601-0400 c20014| 2018-03-16T17:28:18.598-0400 D REPL_HB [conn10] Processed heartbeat from ubuntu:20013 and generated response, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235698, 1), t: 1 }, opTime: { ts: Timestamp(1521235698, 1), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.601-0400 c20013| 2018-03-16T17:28:18.598-0400 D REPL_HB [replexec-0] Received response to heartbeat (requestId: 11) from ubuntu:20014, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235698, 1), t: 1 }, opTime: { ts: Timestamp(1521235698, 1), t: 1 }, operationTime: Timestamp(1521235698, 1), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235697, 23), t: 1 }, lastOpVisible: { ts: Timestamp(0, 0), t: -1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235698, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235697, 23) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.601-0400 c20013| 2018-03-16T17:28:18.599-0400 I REPL [replexec-0] Member ubuntu:20014 is now in state SECONDARY [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:18.602-0400 c20013| 2018-03-16T17:28:18.599-0400 D REPL_HB [replexec-0] Scheduling heartbeat to ubuntu:20014 at 2018-03-16T21:28:19.099Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.079-0400 c20014| 2018-03-16T17:28:19.079-0400 D REPL_HB [replexec-1] Sending heartbeat (requestId: 117) to ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.080-0400 c20014| 2018-03-16T17:28:19.079-0400 D REPL_HB [replexec-0] Sending heartbeat (requestId: 118) to ubuntu:20013, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.080-0400 c20012| 2018-03-16T17:28:19.079-0400 D REPL_HB [conn8] Received heartbeat request from ubuntu:20014, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235698, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.080-0400 c20013| 2018-03-16T17:28:19.079-0400 D REPL_HB [conn9] Received heartbeat request from ubuntu:20014, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235698, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.081-0400 c20013| 2018-03-16T17:28:19.079-0400 D REPL_HB [conn9] Processed heartbeat from ubuntu:20014 and generated response, { ok: 1.0, state: 5, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(0, 0), t: -1 }, opTime: { ts: Timestamp(0, 0), t: -1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.081-0400 c20012| 2018-03-16T17:28:19.079-0400 D REPL_HB [conn8] Processed heartbeat from ubuntu:20014 and generated response, { ok: 1.0, electionTime: new Date(6533657559532830722), state: 1, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235698, 1), t: 1 }, opTime: { ts: Timestamp(1521235698, 1), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.082-0400 c20014| 2018-03-16T17:28:19.079-0400 D REPL_HB [replexec-2] Received response to heartbeat (requestId: 117) from ubuntu:20012, { ok: 1.0, electionTime: new Date(6533657559532830722), state: 1, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235698, 1), t: 1 }, opTime: { ts: Timestamp(1521235698, 1), t: 1 }, operationTime: Timestamp(1521235698, 1), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235697, 23), t: 1 }, lastOpVisible: { ts: Timestamp(0, 0), t: -1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235698, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235697, 23) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.082-0400 c20014| 2018-03-16T17:28:19.079-0400 D REPL_HB [replexec-2] Scheduling heartbeat to ubuntu:20012 at 2018-03-16T21:28:19.579Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.083-0400 c20014| 2018-03-16T17:28:19.080-0400 D REPL_HB [replexec-3] Received response to heartbeat (requestId: 118) from ubuntu:20013, { ok: 1.0, state: 5, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(0, 0), t: -1 }, opTime: { ts: Timestamp(0, 0), t: -1 }, $replData: { term: 1, lastOpCommitted: { ts: Timestamp(0, 0), t: -1 }, lastOpVisible: { ts: Timestamp(0, 0), t: -1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235698, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(0, 0) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.083-0400 c20014| 2018-03-16T17:28:19.080-0400 D REPL_HB [replexec-3] Scheduling heartbeat to ubuntu:20013 at 2018-03-16T21:28:19.580Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.099-0400 c20013| 2018-03-16T17:28:19.098-0400 D REPL_HB [replexec-2] Sending heartbeat (requestId: 13) to ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.099-0400 c20012| 2018-03-16T17:28:19.098-0400 D REPL_HB [conn9] Received heartbeat request from ubuntu:20013, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235698, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.099-0400 c20012| 2018-03-16T17:28:19.098-0400 D REPL_HB [conn9] Processed heartbeat from ubuntu:20013 and generated response, { ok: 1.0, electionTime: new Date(6533657559532830722), state: 1, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235698, 1), t: 1 }, opTime: { ts: Timestamp(1521235698, 1), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.100-0400 c20013| 2018-03-16T17:28:19.099-0400 D REPL_HB [replexec-1] Received response to heartbeat (requestId: 13) from ubuntu:20012, { ok: 1.0, electionTime: new Date(6533657559532830722), state: 1, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235698, 1), t: 1 }, opTime: { ts: Timestamp(1521235698, 1), t: 1 }, operationTime: Timestamp(1521235698, 1), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235697, 23), t: 1 }, lastOpVisible: { ts: Timestamp(0, 0), t: -1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235698, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235697, 23) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.100-0400 c20013| 2018-03-16T17:28:19.099-0400 D REPL_HB [replexec-1] Scheduling heartbeat to ubuntu:20012 at 2018-03-16T21:28:19.599Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.100-0400 c20013| 2018-03-16T17:28:19.099-0400 D REPL_HB [replexec-3] Sending heartbeat (requestId: 15) to ubuntu:20014, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.101-0400 c20014| 2018-03-16T17:28:19.099-0400 D REPL_HB [conn10] Received heartbeat request from ubuntu:20013, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235698, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.101-0400 c20014| 2018-03-16T17:28:19.099-0400 D REPL_HB [conn10] Processed heartbeat from ubuntu:20013 and generated response, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235698, 1), t: 1 }, opTime: { ts: Timestamp(1521235698, 1), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.102-0400 c20013| 2018-03-16T17:28:19.099-0400 D REPL_HB [replexec-0] Received response to heartbeat (requestId: 15) from ubuntu:20014, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235698, 1), t: 1 }, opTime: { ts: Timestamp(1521235698, 1), t: 1 }, operationTime: Timestamp(1521235698, 1), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235697, 23), t: 1 }, lastOpVisible: { ts: Timestamp(0, 0), t: -1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235698, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235697, 23) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.102-0400 c20013| 2018-03-16T17:28:19.099-0400 D REPL_HB [replexec-0] Scheduling heartbeat to ubuntu:20014 at 2018-03-16T21:28:19.599Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.158-0400 c20013| 2018-03-16T17:28:19.158-0400 I REPL [replication-0] sync source candidate: ubuntu:20014 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.159-0400 c20013| 2018-03-16T17:28:19.158-0400 I REPL [replication-0] ****** [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.159-0400 c20013| 2018-03-16T17:28:19.158-0400 I REPL [replication-0] creating replication oplog of size: 40MB... [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.159-0400 c20013| 2018-03-16T17:28:19.158-0400 I STORAGE [replication-0] createCollection: local.oplog.rs with generated UUID: c831192f-e162-4001-b941-514e7edf6468 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.168-0400 c20013| 2018-03-16T17:28:19.168-0400 I STORAGE [replication-0] Starting WiredTigerRecordStoreThread local.oplog.rs [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.168-0400 c20013| 2018-03-16T17:28:19.168-0400 I STORAGE [replication-0] The size storer reports that the oplog contains 0 records totaling to 0 bytes [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.168-0400 c20013| 2018-03-16T17:28:19.168-0400 I STORAGE [replication-0] Scanning the oplog to determine where to place markers for truncation [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.226-0400 c20013| 2018-03-16T17:28:19.226-0400 I REPL [replication-0] ****** [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.226-0400 c20013| 2018-03-16T17:28:19.226-0400 I STORAGE [replication-0] dropAllDatabasesExceptLocal 1 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.226-0400 c20013| 2018-03-16T17:28:19.226-0400 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to ubuntu:20014 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.227-0400 c20014| 2018-03-16T17:28:19.226-0400 I NETWORK [listener] connection accepted from 127.0.0.1:44426 #11 (4 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.227-0400 c20014| 2018-03-16T17:28:19.226-0400 I NETWORK [conn11] received client metadata from 127.0.0.1:44426 conn11: { driver: { name: "NetworkInterfaceASIO-RS", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.227-0400 c20013| 2018-03-16T17:28:19.227-0400 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to ubuntu:20014, took 1ms (1 connections now open to ubuntu:20014) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.228-0400 c20013| 2018-03-16T17:28:19.228-0400 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to ubuntu:20014 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.228-0400 c20014| 2018-03-16T17:28:19.228-0400 I NETWORK [listener] connection accepted from 127.0.0.1:44428 #12 (5 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.228-0400 c20014| 2018-03-16T17:28:19.228-0400 I NETWORK [conn12] received client metadata from 127.0.0.1:44428 conn12: { driver: { name: "NetworkInterfaceASIO-RS", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.229-0400 c20013| 2018-03-16T17:28:19.228-0400 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to ubuntu:20014, took 0ms (2 connections now open to ubuntu:20014) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.229-0400 c20013| 2018-03-16T17:28:19.228-0400 I REPL [replication-1] CollectionCloner::start called, on ns:admin.system.version [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.229-0400 c20013| 2018-03-16T17:28:19.229-0400 I STORAGE [repl writer worker 0] createCollection: admin.system.version with provided UUID: daa3215e-f467-4b5e-bffb-efcfd121a869 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.262-0400 c20013| 2018-03-16T17:28:19.262-0400 I INDEX [repl writer worker 0] build index on: admin.system.version properties: { v: 2, key: { _id: 1 }, name: "_id_", ns: "admin.system.version" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.263-0400 c20013| 2018-03-16T17:28:19.262-0400 I INDEX [repl writer worker 0] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.263-0400 c20013| 2018-03-16T17:28:19.263-0400 I COMMAND [repl writer worker 1] setting featureCompatibilityVersion to 4.0 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.264-0400 c20013| 2018-03-16T17:28:19.263-0400 I NETWORK [repl writer worker 1] Skip closing connection for connection # 9 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.264-0400 c20013| 2018-03-16T17:28:19.263-0400 I NETWORK [repl writer worker 1] Skip closing connection for connection # 3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.264-0400 c20013| 2018-03-16T17:28:19.263-0400 I NETWORK [repl writer worker 1] Skip closing connection for connection # 1 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.264-0400 c20013| 2018-03-16T17:28:19.263-0400 I REPL [repl writer worker 1] CollectionCloner ns:admin.system.version finished cloning with status: OK [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.267-0400 c20013| 2018-03-16T17:28:19.267-0400 I REPL [repl writer worker 1] CollectionCloner::start called, on ns:admin.system.keys [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.268-0400 c20013| 2018-03-16T17:28:19.268-0400 I STORAGE [repl writer worker 2] createCollection: admin.system.keys with provided UUID: 32997d36-cabe-4e06-8861-b98e77fe07ee [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.284-0400 c20013| 2018-03-16T17:28:19.284-0400 I INDEX [repl writer worker 2] build index on: admin.system.keys properties: { v: 2, key: { _id: 1 }, name: "_id_", ns: "admin.system.keys" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.284-0400 c20013| 2018-03-16T17:28:19.284-0400 I INDEX [repl writer worker 2] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.285-0400 c20013| 2018-03-16T17:28:19.285-0400 I REPL [repl writer worker 15] CollectionCloner ns:admin.system.keys finished cloning with status: OK [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.289-0400 c20013| 2018-03-16T17:28:19.289-0400 I REPL [replication-0] CollectionCloner::start called, on ns:config.shards [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.290-0400 c20013| 2018-03-16T17:28:19.290-0400 I STORAGE [repl writer worker 3] createCollection: config.shards with provided UUID: c13519a3-4118-49d6-8a1d-2a757c74d1f7 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.307-0400 c20013| 2018-03-16T17:28:19.306-0400 I INDEX [repl writer worker 3] build index on: config.shards properties: { v: 2, unique: true, key: { host: 1 }, name: "host_1", ns: "config.shards" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.307-0400 c20013| 2018-03-16T17:28:19.306-0400 I INDEX [repl writer worker 3] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.316-0400 c20013| 2018-03-16T17:28:19.315-0400 I INDEX [repl writer worker 3] build index on: config.shards properties: { v: 2, key: { _id: 1 }, name: "_id_", ns: "config.shards" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.316-0400 c20013| 2018-03-16T17:28:19.315-0400 I INDEX [repl writer worker 3] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.316-0400 c20013| 2018-03-16T17:28:19.316-0400 W REPL [repl writer worker 4] _insertDocumentsCallback, but no documents to insert for ns:config.shards [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.316-0400 c20013| 2018-03-16T17:28:19.316-0400 I REPL [repl writer worker 4] CollectionCloner ns:config.shards finished cloning with status: OK [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.318-0400 c20013| 2018-03-16T17:28:19.318-0400 I REPL [repl writer worker 4] CollectionCloner::start called, on ns:config.chunks [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.319-0400 c20013| 2018-03-16T17:28:19.318-0400 I STORAGE [repl writer worker 6] createCollection: config.chunks with provided UUID: ca74780e-951c-4639-ab72-4df24247c87b [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.335-0400 c20013| 2018-03-16T17:28:19.335-0400 I INDEX [repl writer worker 6] build index on: config.chunks properties: { v: 2, unique: true, key: { ns: 1, min: 1 }, name: "ns_1_min_1", ns: "config.chunks" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.335-0400 c20013| 2018-03-16T17:28:19.335-0400 I INDEX [repl writer worker 6] building index using bulk method; build may temporarily use up to 166 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.344-0400 c20013| 2018-03-16T17:28:19.344-0400 I INDEX [repl writer worker 6] build index on: config.chunks properties: { v: 2, unique: true, key: { ns: 1, shard: 1, min: 1 }, name: "ns_1_shard_1_min_1", ns: "config.chunks" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.344-0400 c20013| 2018-03-16T17:28:19.344-0400 I INDEX [repl writer worker 6] building index using bulk method; build may temporarily use up to 166 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.351-0400 c20013| 2018-03-16T17:28:19.351-0400 I INDEX [repl writer worker 6] build index on: config.chunks properties: { v: 2, unique: true, key: { ns: 1, lastmod: 1 }, name: "ns_1_lastmod_1", ns: "config.chunks" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.351-0400 c20013| 2018-03-16T17:28:19.351-0400 I INDEX [repl writer worker 6] building index using bulk method; build may temporarily use up to 166 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.360-0400 c20013| 2018-03-16T17:28:19.360-0400 I INDEX [repl writer worker 6] build index on: config.chunks properties: { v: 2, key: { _id: 1 }, name: "_id_", ns: "config.chunks" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.360-0400 c20013| 2018-03-16T17:28:19.360-0400 I INDEX [repl writer worker 6] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.361-0400 c20013| 2018-03-16T17:28:19.361-0400 W REPL [repl writer worker 5] _insertDocumentsCallback, but no documents to insert for ns:config.chunks [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.361-0400 c20013| 2018-03-16T17:28:19.361-0400 I REPL [repl writer worker 5] CollectionCloner ns:config.chunks finished cloning with status: OK [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.364-0400 c20013| 2018-03-16T17:28:19.364-0400 I REPL [repl writer worker 5] CollectionCloner::start called, on ns:config.version [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.365-0400 c20013| 2018-03-16T17:28:19.365-0400 I STORAGE [repl writer worker 7] createCollection: config.version with provided UUID: 3b13a852-5aec-4401-9e13-c21941b9b63f [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.381-0400 c20013| 2018-03-16T17:28:19.381-0400 I INDEX [repl writer worker 7] build index on: config.version properties: { v: 2, key: { _id: 1 }, name: "_id_", ns: "config.version" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.382-0400 c20013| 2018-03-16T17:28:19.381-0400 I INDEX [repl writer worker 7] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.382-0400 c20013| 2018-03-16T17:28:19.382-0400 I REPL [repl writer worker 8] CollectionCloner ns:config.version finished cloning with status: OK [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.386-0400 c20013| 2018-03-16T17:28:19.385-0400 I REPL [repl writer worker 8] CollectionCloner::start called, on ns:config.transactions [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.386-0400 c20013| 2018-03-16T17:28:19.386-0400 I STORAGE [repl writer worker 9] createCollection: config.transactions with provided UUID: 01eb99c7-45a1-4fab-8a52-0bfdf9e027e2 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.403-0400 c20013| 2018-03-16T17:28:19.402-0400 I INDEX [repl writer worker 9] build index on: config.transactions properties: { v: 2, key: { _id: 1 }, name: "_id_", ns: "config.transactions" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.403-0400 c20013| 2018-03-16T17:28:19.402-0400 I INDEX [repl writer worker 9] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.403-0400 c20013| 2018-03-16T17:28:19.403-0400 W REPL [repl writer worker 10] _insertDocumentsCallback, but no documents to insert for ns:config.transactions [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.404-0400 c20013| 2018-03-16T17:28:19.403-0400 I REPL [repl writer worker 10] CollectionCloner ns:config.transactions finished cloning with status: OK [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.404-0400 c20013| 2018-03-16T17:28:19.404-0400 I REPL [repl writer worker 10] CollectionCloner::start called, on ns:config.locks [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.405-0400 c20013| 2018-03-16T17:28:19.405-0400 I STORAGE [repl writer worker 11] createCollection: config.locks with provided UUID: 437da60d-d424-4d25-bc6a-eb52c6ac482a [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.421-0400 c20013| 2018-03-16T17:28:19.421-0400 I INDEX [repl writer worker 11] build index on: config.locks properties: { v: 2, key: { ts: 1 }, name: "ts_1", ns: "config.locks" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.421-0400 c20013| 2018-03-16T17:28:19.421-0400 I INDEX [repl writer worker 11] building index using bulk method; build may temporarily use up to 250 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.430-0400 c20013| 2018-03-16T17:28:19.430-0400 I INDEX [repl writer worker 11] build index on: config.locks properties: { v: 2, key: { state: 1, process: 1 }, name: "state_1_process_1", ns: "config.locks" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.430-0400 c20013| 2018-03-16T17:28:19.430-0400 I INDEX [repl writer worker 11] building index using bulk method; build may temporarily use up to 250 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.437-0400 c20013| 2018-03-16T17:28:19.437-0400 I INDEX [repl writer worker 11] build index on: config.locks properties: { v: 2, key: { _id: 1 }, name: "_id_", ns: "config.locks" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.438-0400 c20013| 2018-03-16T17:28:19.437-0400 I INDEX [repl writer worker 11] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.438-0400 c20013| 2018-03-16T17:28:19.437-0400 W REPL [repl writer worker 12] _insertDocumentsCallback, but no documents to insert for ns:config.locks [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.438-0400 c20013| 2018-03-16T17:28:19.438-0400 I REPL [repl writer worker 12] CollectionCloner ns:config.locks finished cloning with status: OK [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.440-0400 c20013| 2018-03-16T17:28:19.440-0400 I REPL [repl writer worker 12] CollectionCloner::start called, on ns:config.tags [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.440-0400 c20013| 2018-03-16T17:28:19.440-0400 I STORAGE [repl writer worker 13] createCollection: config.tags with provided UUID: e213515a-d815-4811-a6df-51ef6c4f2652 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.456-0400 c20013| 2018-03-16T17:28:19.456-0400 I INDEX [repl writer worker 13] build index on: config.tags properties: { v: 2, unique: true, key: { ns: 1, min: 1 }, name: "ns_1_min_1", ns: "config.tags" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.457-0400 c20013| 2018-03-16T17:28:19.456-0400 I INDEX [repl writer worker 13] building index using bulk method; build may temporarily use up to 250 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.465-0400 c20013| 2018-03-16T17:28:19.464-0400 I INDEX [repl writer worker 13] build index on: config.tags properties: { v: 2, key: { ns: 1, tag: 1 }, name: "ns_1_tag_1", ns: "config.tags" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.465-0400 c20013| 2018-03-16T17:28:19.465-0400 I INDEX [repl writer worker 13] building index using bulk method; build may temporarily use up to 250 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.472-0400 c20013| 2018-03-16T17:28:19.472-0400 I INDEX [repl writer worker 13] build index on: config.tags properties: { v: 2, key: { _id: 1 }, name: "_id_", ns: "config.tags" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.472-0400 c20013| 2018-03-16T17:28:19.472-0400 I INDEX [repl writer worker 13] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.472-0400 c20013| 2018-03-16T17:28:19.472-0400 W REPL [repl writer worker 14] _insertDocumentsCallback, but no documents to insert for ns:config.tags [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.472-0400 c20013| 2018-03-16T17:28:19.472-0400 I REPL [repl writer worker 14] CollectionCloner ns:config.tags finished cloning with status: OK [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.474-0400 c20013| 2018-03-16T17:28:19.474-0400 I REPL [repl writer worker 14] CollectionCloner::start called, on ns:config.migrations [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.475-0400 c20013| 2018-03-16T17:28:19.475-0400 I STORAGE [repl writer worker 0] createCollection: config.migrations with provided UUID: 7121c15b-943b-4d4b-b5dd-50f357a13814 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.494-0400 c20013| 2018-03-16T17:28:19.494-0400 I INDEX [repl writer worker 0] build index on: config.migrations properties: { v: 2, unique: true, key: { ns: 1, min: 1 }, name: "ns_1_min_1", ns: "config.migrations" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.495-0400 c20013| 2018-03-16T17:28:19.494-0400 I INDEX [repl writer worker 0] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.502-0400 c20013| 2018-03-16T17:28:19.502-0400 I INDEX [repl writer worker 0] build index on: config.migrations properties: { v: 2, key: { _id: 1 }, name: "_id_", ns: "config.migrations" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.502-0400 c20013| 2018-03-16T17:28:19.502-0400 I INDEX [repl writer worker 0] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.503-0400 c20013| 2018-03-16T17:28:19.502-0400 W REPL [repl writer worker 1] _insertDocumentsCallback, but no documents to insert for ns:config.migrations [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.503-0400 c20013| 2018-03-16T17:28:19.502-0400 I REPL [repl writer worker 1] CollectionCloner ns:config.migrations finished cloning with status: OK [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.504-0400 c20013| 2018-03-16T17:28:19.504-0400 I REPL [repl writer worker 1] CollectionCloner::start called, on ns:config.lockpings [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.505-0400 c20013| 2018-03-16T17:28:19.505-0400 I STORAGE [repl writer worker 2] createCollection: config.lockpings with provided UUID: 9c0b3109-1569-46d5-8030-c76bd8db19ee [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.520-0400 c20013| 2018-03-16T17:28:19.519-0400 I INDEX [repl writer worker 2] build index on: config.lockpings properties: { v: 2, key: { ping: 1 }, name: "ping_1", ns: "config.lockpings" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.520-0400 c20013| 2018-03-16T17:28:19.519-0400 I INDEX [repl writer worker 2] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.528-0400 c20013| 2018-03-16T17:28:19.528-0400 I INDEX [repl writer worker 2] build index on: config.lockpings properties: { v: 2, key: { _id: 1 }, name: "_id_", ns: "config.lockpings" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.528-0400 c20013| 2018-03-16T17:28:19.528-0400 I INDEX [repl writer worker 2] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.529-0400 c20013| 2018-03-16T17:28:19.528-0400 W REPL [repl writer worker 15] _insertDocumentsCallback, but no documents to insert for ns:config.lockpings [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.529-0400 c20013| 2018-03-16T17:28:19.528-0400 I REPL [repl writer worker 15] CollectionCloner ns:config.lockpings finished cloning with status: OK [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.530-0400 c20013| 2018-03-16T17:28:19.530-0400 I REPL [repl writer worker 15] Finished cloning data: OK. Beginning oplog replay. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.530-0400 c20013| 2018-03-16T17:28:19.530-0400 I REPL [replication-0] No need to apply operations. (currently at { : Timestamp(1521235698, 1) }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.531-0400 c20013| 2018-03-16T17:28:19.530-0400 I ASIO [NetworkInterfaceASIO-RS-0] Ending connection to host ubuntu:20014 due to bad connection status; 1 connections to that host remain open [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.531-0400 c20013| 2018-03-16T17:28:19.530-0400 I REPL [replication-0] Finished fetching oplog during initial sync: CallbackCanceled: error in fetcher batch callback: oplog fetcher is shutting down. Last fetched optime and hash: { ts: Timestamp(0, 0), t: -1 }[0] [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.531-0400 c20013| 2018-03-16T17:28:19.531-0400 I REPL [replication-0] Initial sync attempt finishing up. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.533-0400 c20013| 2018-03-16T17:28:19.531-0400 I REPL [replication-0] Initial Sync Attempt Statistics: { failedInitialSyncAttempts: 0, maxFailedInitialSyncAttempts: 1, initialSyncStart: new Date(1521235698120), initialSyncAttempts: [], fetchedMissingDocs: 0, appliedOps: 0, initialSyncOplogStart: Timestamp(1521235698, 1), initialSyncOplogEnd: Timestamp(1521235698, 1), databases: { databasesCloned: 2, admin: { collections: 2, clonedCollections: 2, start: new Date(1521235699228), end: new Date(1521235699289), elapsedMillis: 61, admin.system.version: { documentsToCopy: 1, documentsCopied: 1, indexes: 1, fetchedBatches: 1, start: new Date(1521235699228), end: new Date(1521235699267), elapsedMillis: 39 }, admin.system.keys: { documentsToCopy: 2, documentsCopied: 2, indexes: 1, fetchedBatches: 1, start: new Date(1521235699267), end: new Date(1521235699289), elapsedMillis: 22 } }, config: { collections: 8, clonedCollections: 8, start: new Date(1521235699289), end: new Date(1521235699530), elapsedMillis: 241, config.shards: { documentsToCopy: 0, documentsCopied: 0, indexes: 2, fetchedBatches: 0, start: new Date(1521235699289), end: new Date(1521235699318), elapsedMillis: 29 }, config.chunks: { documentsToCopy: 0, documentsCopied: 0, indexes: 4, fetchedBatches: 0, start: new Date(1521235699318), end: new Date(1521235699364), elapsedMillis: 46 }, config.version: { documentsToCopy: 1, documentsCopied: 1, indexes: 1, fetchedBatches: 1, start: new Date(1521235699364), end: new Date(1521235699386), elapsedMillis: 22 }, config.transactions: { documentsToCopy: 0, documentsCopied: 0, indexes: 1, fetchedBatches: 0, start: new Date(1521235699385), end: new Date(1521235699404), elapsedMillis: 19 }, config.locks: { documentsToCopy: 0, documentsCopied: 0, indexes: 3, fetchedBatches: 0, start: new Date(1521235699404), end: new Date(1521235699440), elapsedMillis: 36 }, config.tags: { documentsToCopy: 0, documentsCopied: 0, indexes: 3, fetchedBatches: 0, start: new Date(1521235699440), end: new Date(1521235699474), elapsedMillis: 34 }, config.migrations: { documentsToCopy: 0, documentsCopied: 0, indexes: 2, fetchedBatches: 0, start: new Date(1521235699474), end: new Date(1521235699504), elapsedMillis: 30 }, config.lockpings: { documentsToCopy: 0, documentsCopied: 0, indexes: 2, fetchedBatches: 0, start: new Date(1521235699504), end: new Date(1521235699530), elapsedMillis: 26 } } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.533-0400 c20013| 2018-03-16T17:28:19.531-0400 I STORAGE [replication-1] Finishing collection drop for local.temp_oplog_buffer (ddb36586-d2ee-45df-91ea-6e625bd06e07). [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.536-0400 c20013| 2018-03-16T17:28:19.535-0400 I REPL [replication-1] initial sync done; took 1s. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.536-0400 c20013| 2018-03-16T17:28:19.536-0400 I REPL [replication-1] transition to RECOVERING from STARTUP2 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.536-0400 c20013| 2018-03-16T17:28:19.536-0400 I REPL [replication-1] Starting replication fetcher thread [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.536-0400 c20013| 2018-03-16T17:28:19.536-0400 I REPL [replication-1] Starting replication applier thread [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.536-0400 c20013| 2018-03-16T17:28:19.536-0400 I REPL [rsBackgroundSync] could not find member to sync from [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.536-0400 c20013| 2018-03-16T17:28:19.536-0400 D REPL_HB [rsBackgroundSync] Cancelling all heartbeats. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.537-0400 c20013| 2018-03-16T17:28:19.536-0400 D REPL_HB [rsBackgroundSync] Scheduling heartbeat to ubuntu:20012 at 2018-03-16T21:28:19.536Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.537-0400 c20013| 2018-03-16T17:28:19.536-0400 D REPL_HB [rsBackgroundSync] Scheduling heartbeat to ubuntu:20014 at 2018-03-16T21:28:19.536Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.537-0400 c20013| 2018-03-16T17:28:19.536-0400 D REPL_HB [replexec-3] Sending heartbeat (requestId: 119) to ubuntu:20014, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.537-0400 c20013| 2018-03-16T17:28:19.536-0400 D REPL_HB [replexec-0] Sending heartbeat (requestId: 120) to ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.538-0400 c20014| 2018-03-16T17:28:19.536-0400 D REPL_HB [conn10] Received heartbeat request from ubuntu:20013, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235698, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.538-0400 c20014| 2018-03-16T17:28:19.536-0400 D REPL_HB [conn10] Processed heartbeat from ubuntu:20013 and generated response, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235698, 1), t: 1 }, opTime: { ts: Timestamp(1521235698, 1), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.538-0400 c20012| 2018-03-16T17:28:19.536-0400 D REPL_HB [conn9] Received heartbeat request from ubuntu:20013, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235698, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.539-0400 c20012| 2018-03-16T17:28:19.536-0400 D REPL_HB [conn9] Processed heartbeat from ubuntu:20013 and generated response, { ok: 1.0, electionTime: new Date(6533657559532830722), state: 1, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235698, 1), t: 1 }, opTime: { ts: Timestamp(1521235698, 1), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.539-0400 c20013| 2018-03-16T17:28:19.536-0400 I REPL [replication-1] Starting replication reporter thread [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.539-0400 c20013| 2018-03-16T17:28:19.536-0400 D REPL_HB [replexec-3] Received response to heartbeat (requestId: 119) from ubuntu:20014, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235698, 1), t: 1 }, opTime: { ts: Timestamp(1521235698, 1), t: 1 }, operationTime: Timestamp(1521235698, 1), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235697, 23), t: 1 }, lastOpVisible: { ts: Timestamp(0, 0), t: -1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235698, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235697, 23) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.540-0400 c20013| 2018-03-16T17:28:19.536-0400 D REPL_HB [replexec-3] Scheduling heartbeat to ubuntu:20014 at 2018-03-16T21:28:20.036Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.540-0400 c20013| 2018-03-16T17:28:19.536-0400 D REPL_HB [replexec-1] Received response to heartbeat (requestId: 120) from ubuntu:20012, { ok: 1.0, electionTime: new Date(6533657559532830722), state: 1, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235698, 1), t: 1 }, opTime: { ts: Timestamp(1521235698, 1), t: 1 }, operationTime: Timestamp(1521235698, 1), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235697, 23), t: 1 }, lastOpVisible: { ts: Timestamp(0, 0), t: -1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235698, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235697, 23) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.540-0400 c20013| 2018-03-16T17:28:19.536-0400 D REPL_HB [replexec-1] Scheduling heartbeat to ubuntu:20012 at 2018-03-16T21:28:20.036Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.541-0400 c20013| 2018-03-16T17:28:19.536-0400 I REPL [rsSync] transition to SECONDARY from RECOVERING [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.580-0400 c20014| 2018-03-16T17:28:19.580-0400 D REPL_HB [replexec-1] Sending heartbeat (requestId: 121) to ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.580-0400 c20014| 2018-03-16T17:28:19.580-0400 D REPL_HB [replexec-1] Sending heartbeat (requestId: 122) to ubuntu:20013, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.581-0400 c20012| 2018-03-16T17:28:19.580-0400 D REPL_HB [conn8] Received heartbeat request from ubuntu:20014, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235698, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.581-0400 c20013| 2018-03-16T17:28:19.580-0400 D REPL_HB [conn9] Received heartbeat request from ubuntu:20014, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235698, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.581-0400 c20013| 2018-03-16T17:28:19.580-0400 D REPL_HB [conn9] Processed heartbeat from ubuntu:20014 and generated response, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235698, 1), t: 1 }, opTime: { ts: Timestamp(1521235698, 1), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.582-0400 c20012| 2018-03-16T17:28:19.580-0400 D REPL_HB [conn8] Processed heartbeat from ubuntu:20014 and generated response, { ok: 1.0, electionTime: new Date(6533657559532830722), state: 1, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235698, 1), t: 1 }, opTime: { ts: Timestamp(1521235698, 1), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.582-0400 c20014| 2018-03-16T17:28:19.580-0400 D REPL_HB [replexec-4] Received response to heartbeat (requestId: 121) from ubuntu:20012, { ok: 1.0, electionTime: new Date(6533657559532830722), state: 1, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235698, 1), t: 1 }, opTime: { ts: Timestamp(1521235698, 1), t: 1 }, operationTime: Timestamp(1521235698, 1), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235697, 23), t: 1 }, lastOpVisible: { ts: Timestamp(0, 0), t: -1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235698, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235697, 23) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.582-0400 c20014| 2018-03-16T17:28:19.580-0400 D REPL_HB [replexec-4] Scheduling heartbeat to ubuntu:20012 at 2018-03-16T21:28:20.080Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.583-0400 c20014| 2018-03-16T17:28:19.580-0400 D REPL_HB [replexec-2] Received response to heartbeat (requestId: 122) from ubuntu:20013, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235698, 1), t: 1 }, opTime: { ts: Timestamp(1521235698, 1), t: 1 }, operationTime: Timestamp(1521235698, 1), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235697, 23), t: 1 }, lastOpVisible: { ts: Timestamp(0, 0), t: -1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235698, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235697, 23) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.583-0400 c20014| 2018-03-16T17:28:19.580-0400 I REPL [replexec-2] Member ubuntu:20013 is now in state SECONDARY [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.583-0400 c20014| 2018-03-16T17:28:19.580-0400 D REPL_HB [replexec-2] Scheduling heartbeat to ubuntu:20013 at 2018-03-16T21:28:20.080Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.691-0400 Set shouldWaitForKeys from RS options: false [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.693-0400 c20012| 2018-03-16T17:28:19.693-0400 I COMMAND [conn1] setting featureCompatibilityVersion to downgrading to 3.6 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.693-0400 c20012| 2018-03-16T17:28:19.693-0400 I NETWORK [conn1] Skip closing connection for connection # 13 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.693-0400 c20012| 2018-03-16T17:28:19.693-0400 I NETWORK [conn1] Skip closing connection for connection # 12 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.694-0400 c20012| 2018-03-16T17:28:19.693-0400 I NETWORK [conn1] Skip closing connection for connection # 9 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.694-0400 c20012| 2018-03-16T17:28:19.693-0400 I NETWORK [conn1] Skip closing connection for connection # 8 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.694-0400 c20012| 2018-03-16T17:28:19.693-0400 I NETWORK [conn1] Skip closing connection for connection # 1 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:19.696-0400 c20012| 2018-03-16T17:28:19.696-0400 I NETWORK [conn13] end connection 127.0.0.1:60638 (4 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.037-0400 c20013| 2018-03-16T17:28:20.036-0400 D REPL_HB [replexec-0] Sending heartbeat (requestId: 123) to ubuntu:20014, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.037-0400 c20013| 2018-03-16T17:28:20.037-0400 D REPL_HB [replexec-2] Sending heartbeat (requestId: 124) to ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.037-0400 c20012| 2018-03-16T17:28:20.037-0400 D REPL_HB [conn9] Received heartbeat request from ubuntu:20013, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235698, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.038-0400 c20014| 2018-03-16T17:28:20.037-0400 D REPL_HB [conn10] Received heartbeat request from ubuntu:20013, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235698, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.038-0400 c20012| 2018-03-16T17:28:20.037-0400 D REPL_HB [conn9] Processed heartbeat from ubuntu:20013 and generated response, { ok: 1.0, electionTime: new Date(6533657559532830722), state: 1, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235699, 1), t: 1 }, opTime: { ts: Timestamp(1521235699, 1), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.038-0400 c20014| 2018-03-16T17:28:20.037-0400 D REPL_HB [conn10] Processed heartbeat from ubuntu:20013 and generated response, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235698, 1), t: 1 }, opTime: { ts: Timestamp(1521235698, 1), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.039-0400 c20013| 2018-03-16T17:28:20.037-0400 D REPL_HB [replexec-1] Received response to heartbeat (requestId: 123) from ubuntu:20014, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235698, 1), t: 1 }, opTime: { ts: Timestamp(1521235698, 1), t: 1 }, operationTime: Timestamp(1521235698, 1), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235697, 23), t: 1 }, lastOpVisible: { ts: Timestamp(0, 0), t: -1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235698, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235697, 23) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.039-0400 c20013| 2018-03-16T17:28:20.037-0400 D REPL_HB [replexec-1] Scheduling heartbeat to ubuntu:20014 at 2018-03-16T21:28:20.537Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.040-0400 c20013| 2018-03-16T17:28:20.037-0400 D REPL_HB [replexec-3] Received response to heartbeat (requestId: 124) from ubuntu:20012, { ok: 1.0, electionTime: new Date(6533657559532830722), state: 1, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235699, 1), t: 1 }, opTime: { ts: Timestamp(1521235699, 1), t: 1 }, operationTime: Timestamp(1521235699, 1), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235697, 23), t: 1 }, lastOpVisible: { ts: Timestamp(0, 0), t: -1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235699, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235697, 23) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.040-0400 c20013| 2018-03-16T17:28:20.037-0400 D REPL_HB [replexec-3] Scheduling heartbeat to ubuntu:20012 at 2018-03-16T21:28:20.537Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.069-0400 c20012| 2018-03-16T17:28:20.068-0400 D REPL_HB [replexec-1] Sending heartbeat (requestId: 12) to ubuntu:20013, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20012", fromId: 0, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.069-0400 c20013| 2018-03-16T17:28:20.068-0400 D REPL_HB [conn3] Received heartbeat request from ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20012", fromId: 0, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235699, 1), signature: { hash: BinData(0, 22D7D58402B8C70353AB3B7F6B243E958858A14B), keyId: 6533657568122765331 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.069-0400 c20013| 2018-03-16T17:28:20.068-0400 D REPL_HB [conn3] Processed heartbeat from ubuntu:20012 and generated response, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235698, 1), t: 1 }, opTime: { ts: Timestamp(1521235698, 1), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.070-0400 c20012| 2018-03-16T17:28:20.069-0400 D REPL_HB [replexec-0] Sending heartbeat (requestId: 13) to ubuntu:20014, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20012", fromId: 0, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.070-0400 c20012| 2018-03-16T17:28:20.069-0400 D REPL_HB [replexec-2] Received response to heartbeat (requestId: 12) from ubuntu:20013, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235698, 1), t: 1 }, opTime: { ts: Timestamp(1521235698, 1), t: 1 }, operationTime: Timestamp(1521235698, 1), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235697, 23), t: 1 }, lastOpVisible: { ts: Timestamp(0, 0), t: -1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235699, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235697, 23) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.070-0400 c20012| 2018-03-16T17:28:20.069-0400 I REPL [replexec-2] Member ubuntu:20013 is now in state SECONDARY [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.071-0400 c20012| 2018-03-16T17:28:20.069-0400 D REPL_HB [replexec-2] Scheduling heartbeat to ubuntu:20013 at 2018-03-16T21:28:22.069Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.071-0400 c20014| 2018-03-16T17:28:20.069-0400 D REPL_HB [conn4] Received heartbeat request from ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20012", fromId: 0, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235699, 1), signature: { hash: BinData(0, 22D7D58402B8C70353AB3B7F6B243E958858A14B), keyId: 6533657568122765331 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.071-0400 c20014| 2018-03-16T17:28:20.069-0400 D REPL_HB [conn4] Processed heartbeat from ubuntu:20012 and generated response, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235698, 1), t: 1 }, opTime: { ts: Timestamp(1521235698, 1), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.072-0400 c20012| 2018-03-16T17:28:20.069-0400 D REPL_HB [replexec-1] Received response to heartbeat (requestId: 13) from ubuntu:20014, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235698, 1), t: 1 }, opTime: { ts: Timestamp(1521235698, 1), t: 1 }, operationTime: Timestamp(1521235698, 1), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235697, 23), t: 1 }, lastOpVisible: { ts: Timestamp(0, 0), t: -1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235699, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235697, 23) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.072-0400 c20012| 2018-03-16T17:28:20.069-0400 I REPL [replexec-1] Member ubuntu:20014 is now in state SECONDARY [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.072-0400 c20012| 2018-03-16T17:28:20.069-0400 D REPL_HB [replexec-1] Scheduling heartbeat to ubuntu:20014 at 2018-03-16T21:28:22.069Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.080-0400 c20014| 2018-03-16T17:28:20.080-0400 D REPL_HB [replexec-0] Sending heartbeat (requestId: 125) to ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.081-0400 c20014| 2018-03-16T17:28:20.080-0400 D REPL_HB [replexec-1] Sending heartbeat (requestId: 126) to ubuntu:20013, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.081-0400 c20012| 2018-03-16T17:28:20.080-0400 D REPL_HB [conn8] Received heartbeat request from ubuntu:20014, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235699, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.081-0400 c20012| 2018-03-16T17:28:20.080-0400 D REPL_HB [conn8] Processed heartbeat from ubuntu:20014 and generated response, { ok: 1.0, electionTime: new Date(6533657559532830722), state: 1, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235699, 1), t: 1 }, opTime: { ts: Timestamp(1521235699, 1), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.082-0400 c20013| 2018-03-16T17:28:20.080-0400 D REPL_HB [conn9] Received heartbeat request from ubuntu:20014, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235699, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.082-0400 c20013| 2018-03-16T17:28:20.080-0400 D REPL_HB [conn9] Processed heartbeat from ubuntu:20014 and generated response, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235698, 1), t: 1 }, opTime: { ts: Timestamp(1521235698, 1), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.083-0400 c20014| 2018-03-16T17:28:20.080-0400 D REPL_HB [replexec-3] Received response to heartbeat (requestId: 125) from ubuntu:20012, { ok: 1.0, electionTime: new Date(6533657559532830722), state: 1, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235699, 1), t: 1 }, opTime: { ts: Timestamp(1521235699, 1), t: 1 }, operationTime: Timestamp(1521235699, 1), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235698, 1), t: 1 }, lastOpVisible: { ts: Timestamp(1521235698, 1), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235699, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235698, 1) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.083-0400 c20014| 2018-03-16T17:28:20.081-0400 D REPL_HB [replexec-3] Scheduling heartbeat to ubuntu:20012 at 2018-03-16T21:28:20.580Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.084-0400 c20014| 2018-03-16T17:28:20.081-0400 D REPL_HB [replexec-4] Received response to heartbeat (requestId: 126) from ubuntu:20013, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235698, 1), t: 1 }, opTime: { ts: Timestamp(1521235698, 1), t: 1 }, operationTime: Timestamp(1521235698, 1), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235697, 23), t: 1 }, lastOpVisible: { ts: Timestamp(0, 0), t: -1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235699, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235697, 23) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.084-0400 c20014| 2018-03-16T17:28:20.081-0400 D REPL_HB [replexec-4] Scheduling heartbeat to ubuntu:20013 at 2018-03-16T21:28:20.581Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.536-0400 c20013| 2018-03-16T17:28:20.536-0400 I REPL [rsBackgroundSync] sync source candidate: ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.537-0400 c20013| 2018-03-16T17:28:20.536-0400 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.537-0400 c20012| 2018-03-16T17:28:20.537-0400 I NETWORK [listener] connection accepted from 127.0.0.1:60644 #14 (5 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.537-0400 c20012| 2018-03-16T17:28:20.537-0400 I NETWORK [conn14] received client metadata from 127.0.0.1:60644 conn14: { driver: { name: "NetworkInterfaceASIO-RS", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.538-0400 c20013| 2018-03-16T17:28:20.537-0400 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to ubuntu:20012, took 1ms (1 connections now open to ubuntu:20012) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.538-0400 c20013| 2018-03-16T17:28:20.537-0400 D REPL_HB [replexec-2] Sending heartbeat (requestId: 129) to ubuntu:20014, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.538-0400 c20013| 2018-03-16T17:28:20.537-0400 D REPL_HB [replexec-1] Sending heartbeat (requestId: 130) to ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.538-0400 c20014| 2018-03-16T17:28:20.537-0400 D REPL_HB [conn10] Received heartbeat request from ubuntu:20013, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235699, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.539-0400 c20014| 2018-03-16T17:28:20.537-0400 D REPL_HB [conn10] Processed heartbeat from ubuntu:20013 and generated response, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235698, 1), t: 1 }, opTime: { ts: Timestamp(1521235698, 1), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.539-0400 c20012| 2018-03-16T17:28:20.537-0400 D REPL_HB [conn9] Received heartbeat request from ubuntu:20013, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235699, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.539-0400 c20012| 2018-03-16T17:28:20.537-0400 D REPL_HB [conn9] Processed heartbeat from ubuntu:20013 and generated response, { ok: 1.0, electionTime: new Date(6533657559532830722), state: 1, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235699, 1), t: 1 }, opTime: { ts: Timestamp(1521235699, 1), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.540-0400 c20013| 2018-03-16T17:28:20.537-0400 D REPL_HB [replexec-3] Received response to heartbeat (requestId: 129) from ubuntu:20014, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235698, 1), t: 1 }, opTime: { ts: Timestamp(1521235698, 1), t: 1 }, operationTime: Timestamp(1521235698, 1), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235697, 23), t: 1 }, lastOpVisible: { ts: Timestamp(0, 0), t: -1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235699, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235697, 23) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.540-0400 c20013| 2018-03-16T17:28:20.537-0400 D REPL_HB [replexec-3] Scheduling heartbeat to ubuntu:20014 at 2018-03-16T21:28:22.537Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.541-0400 c20013| 2018-03-16T17:28:20.537-0400 D REPL_HB [replexec-0] Received response to heartbeat (requestId: 130) from ubuntu:20012, { ok: 1.0, electionTime: new Date(6533657559532830722), state: 1, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235699, 1), t: 1 }, opTime: { ts: Timestamp(1521235699, 1), t: 1 }, operationTime: Timestamp(1521235699, 1), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235698, 1), t: 1 }, lastOpVisible: { ts: Timestamp(1521235698, 1), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235699, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235698, 1) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.541-0400 c20013| 2018-03-16T17:28:20.538-0400 D REPL_HB [replexec-0] Scheduling heartbeat to ubuntu:20012 at 2018-03-16T21:28:22.537Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.542-0400 c20013| 2018-03-16T17:28:20.538-0400 I REPL [rsBackgroundSync] scheduling fetcher to read remote oplog on ubuntu:20012 starting at filter: { ts: { $gte: Timestamp(1521235698, 1) } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.542-0400 c20013| 2018-03-16T17:28:20.538-0400 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.542-0400 c20012| 2018-03-16T17:28:20.538-0400 I NETWORK [listener] connection accepted from 127.0.0.1:60646 #15 (6 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.542-0400 c20012| 2018-03-16T17:28:20.538-0400 I NETWORK [conn15] received client metadata from 127.0.0.1:60646 conn15: { driver: { name: "NetworkInterfaceASIO-RS", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.542-0400 c20013| 2018-03-16T17:28:20.538-0400 I STORAGE [replication-0] Triggering the first stable checkpoint. Initial Data: Timestamp(1521235698, 1) PrevStable: Timestamp(0, 0) CurrStable: Timestamp(1521235698, 1) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.543-0400 c20013| 2018-03-16T17:28:20.538-0400 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to ubuntu:20012, took 0ms (2 connections now open to ubuntu:20012) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.543-0400 c20013| 2018-03-16T17:28:20.539-0400 I COMMAND [repl writer worker 0] setting featureCompatibilityVersion to downgrading to 3.6 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.543-0400 c20013| 2018-03-16T17:28:20.539-0400 I NETWORK [repl writer worker 0] Skip closing connection for connection # 9 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.543-0400 c20013| 2018-03-16T17:28:20.539-0400 I NETWORK [repl writer worker 0] Skip closing connection for connection # 3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.543-0400 c20013| 2018-03-16T17:28:20.539-0400 I NETWORK [repl writer worker 0] Skip closing connection for connection # 1 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.544-0400 c20012| 2018-03-16T17:28:20.543-0400 I COMMAND [conn1] command admin.$cmd appName: "MongoDB Shell" command: update { update: "system.version", writeConcern: { w: "majority" }, $db: "admin" } numYields:0 reslen:338 locks:{ Global: { acquireCount: { r: 2, w: 2 } }, Database: { acquireCount: { w: 1, W: 1 } }, Collection: { acquireCount: { w: 1 } }, Mutex: { acquireCount: { W: 1 } }, oplog: { acquireCount: { w: 1 } } } protocol:op_msg 849ms [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.544-0400 c20012| 2018-03-16T17:28:20.543-0400 I COMMAND [conn1] setting featureCompatibilityVersion to 3.6 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.547-0400 c20013| 2018-03-16T17:28:20.547-0400 I COMMAND [repl writer worker 15] setting featureCompatibilityVersion to 3.6 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.556-0400 c20012| 2018-03-16T17:28:20.556-0400 I COMMAND [conn1] command admin.$cmd appName: "MongoDB Shell" command: setFeatureCompatibilityVersion { setFeatureCompatibilityVersion: "3.6", $clusterTime: { clusterTime: Timestamp(1521235698, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, $readPreference: { mode: "secondaryPreferred" }, $db: "admin" } numYields:0 reslen:256 locks:{ Global: { acquireCount: { r: 7, w: 4, R: 1 } }, Database: { acquireCount: { r: 1, w: 2, W: 2 } }, Collection: { acquireCount: { r: 1, w: 2 } }, Mutex: { acquireCount: { r: 1, W: 1 } }, oplog: { acquireCount: { w: 2 } } } protocol:op_msg 863ms [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.562-0400 ReplSetTest awaitReplication: starting: optime for primary, ubuntu:20012, is { "ts" : Timestamp(1521235700, 1), "t" : NumberLong(1) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.563-0400 ReplSetTest awaitReplication: checking secondaries against latest primary optime { "ts" : Timestamp(1521235700, 1), "t" : NumberLong(1) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.564-0400 ReplSetTest awaitReplication: checking secondary #1: ubuntu:20013 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.565-0400 ReplSetTest awaitReplication: secondary #1, ubuntu:20013, is synced [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.567-0400 ReplSetTest awaitReplication: checking secondary #2: ubuntu:20014 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.567-0400 ReplSetTest awaitReplication: optime for secondary #2, ubuntu:20014, is { "ts" : Timestamp(1521235698, 1), "t" : NumberLong(1) } but latest is { "ts" : Timestamp(1521235700, 1), "t" : NumberLong(1) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.568-0400 ReplSetTest awaitReplication: secondary #2, ubuntu:20014, is NOT synced [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.579-0400 c20014| 2018-03-16T17:28:20.579-0400 I REPL [rsBackgroundSync] sync source candidate: ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.580-0400 c20014| 2018-03-16T17:28:20.579-0400 I REPL [rsBackgroundSync] scheduling fetcher to read remote oplog on ubuntu:20012 starting at filter: { ts: { $gte: Timestamp(1521235698, 1) } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.580-0400 c20014| 2018-03-16T17:28:20.580-0400 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.580-0400 c20014| 2018-03-16T17:28:20.580-0400 D REPL_HB [replexec-0] Sending heartbeat (requestId: 136) to ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.580-0400 c20012| 2018-03-16T17:28:20.580-0400 D REPL_HB [conn8] Received heartbeat request from ubuntu:20014, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235700, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.581-0400 c20012| 2018-03-16T17:28:20.580-0400 D REPL_HB [conn8] Processed heartbeat from ubuntu:20014 and generated response, { ok: 1.0, electionTime: new Date(6533657559532830722), state: 1, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235700, 1), t: 1 }, opTime: { ts: Timestamp(1521235700, 1), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.581-0400 c20012| 2018-03-16T17:28:20.580-0400 I NETWORK [listener] connection accepted from 127.0.0.1:60648 #16 (7 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.582-0400 c20014| 2018-03-16T17:28:20.580-0400 D REPL_HB [replexec-1] Received response to heartbeat (requestId: 136) from ubuntu:20012, { ok: 1.0, electionTime: new Date(6533657559532830722), state: 1, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235700, 1), t: 1 }, opTime: { ts: Timestamp(1521235700, 1), t: 1 }, operationTime: Timestamp(1521235700, 1), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235700, 1), t: 1 }, lastOpVisible: { ts: Timestamp(1521235700, 1), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235700, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235700, 1) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.582-0400 c20014| 2018-03-16T17:28:20.580-0400 D REPL_HB [replexec-1] Scheduling heartbeat to ubuntu:20012 at 2018-03-16T21:28:22.580Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.582-0400 c20012| 2018-03-16T17:28:20.580-0400 I NETWORK [conn16] received client metadata from 127.0.0.1:60648 conn16: { driver: { name: "NetworkInterfaceASIO-RS", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.582-0400 c20014| 2018-03-16T17:28:20.580-0400 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to ubuntu:20012, took 0ms (2 connections now open to ubuntu:20012) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.583-0400 c20014| 2018-03-16T17:28:20.580-0400 I STORAGE [replication-0] Triggering the first stable checkpoint. Initial Data: Timestamp(1521235698, 1) PrevStable: Timestamp(0, 0) CurrStable: Timestamp(1521235698, 1) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.583-0400 c20014| 2018-03-16T17:28:20.580-0400 I NETWORK [conn12] end connection 127.0.0.1:44428 (4 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.583-0400 c20014| 2018-03-16T17:28:20.581-0400 D REPL_HB [replexec-4] Sending heartbeat (requestId: 140) to ubuntu:20013, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.583-0400 c20013| 2018-03-16T17:28:20.581-0400 D REPL_HB [conn9] Received heartbeat request from ubuntu:20014, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235700, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.584-0400 c20013| 2018-03-16T17:28:20.581-0400 D REPL_HB [conn9] Processed heartbeat from ubuntu:20014 and generated response, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", syncingTo: "ubuntu:20012", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235700, 1), t: 1 }, opTime: { ts: Timestamp(1521235700, 1), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.584-0400 c20014| 2018-03-16T17:28:20.581-0400 I COMMAND [repl writer worker 1] setting featureCompatibilityVersion to downgrading to 3.6 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.584-0400 c20014| 2018-03-16T17:28:20.581-0400 I NETWORK [repl writer worker 1] Skip closing connection for connection # 11 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.584-0400 c20014| 2018-03-16T17:28:20.581-0400 I NETWORK [repl writer worker 1] Skip closing connection for connection # 10 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.584-0400 c20014| 2018-03-16T17:28:20.581-0400 I NETWORK [repl writer worker 1] Skip closing connection for connection # 4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.584-0400 c20014| 2018-03-16T17:28:20.581-0400 I NETWORK [repl writer worker 1] Skip closing connection for connection # 1 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.585-0400 c20014| 2018-03-16T17:28:20.581-0400 I COMMAND [repl writer worker 1] setting featureCompatibilityVersion to 3.6 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.585-0400 c20014| 2018-03-16T17:28:20.581-0400 D REPL_HB [replexec-0] Received response to heartbeat (requestId: 140) from ubuntu:20013, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", syncingTo: "ubuntu:20012", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235700, 1), t: 1 }, opTime: { ts: Timestamp(1521235700, 1), t: 1 }, operationTime: Timestamp(1521235700, 1), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235700, 1), t: 1 }, lastOpVisible: { ts: Timestamp(1521235700, 1), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235700, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235700, 1) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.585-0400 c20014| 2018-03-16T17:28:20.581-0400 D REPL_HB [replexec-0] Scheduling heartbeat to ubuntu:20013 at 2018-03-16T21:28:22.581Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.768-0400 ReplSetTest awaitReplication: checking secondaries against latest primary optime { "ts" : Timestamp(1521235700, 1), "t" : NumberLong(1) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.769-0400 ReplSetTest awaitReplication: checking secondary #1: ubuntu:20013 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.770-0400 ReplSetTest awaitReplication: secondary #1, ubuntu:20013, is synced [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.771-0400 ReplSetTest awaitReplication: checking secondary #2: ubuntu:20014 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.772-0400 ReplSetTest awaitReplication: secondary #2, ubuntu:20014, is synced [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.772-0400 ReplSetTest awaitReplication: finished: all 2 secondaries synced at optime { "ts" : Timestamp(1521235700, 1), "t" : NumberLong(1) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.772-0400 "Config servers: stale_mongos_updates_and_removes-configRS/ubuntu:20012,ubuntu:20013,ubuntu:20014" [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.772-0400 2018-03-16T17:28:20.772-0400 I NETWORK [js] Starting new replica set monitor for stale_mongos_updates_and_removes-configRS/ubuntu:20012,ubuntu:20013,ubuntu:20014 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.772-0400 c20014| 2018-03-16T17:28:20.772-0400 I NETWORK [listener] connection accepted from 127.0.0.1:44438 #13 (5 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.773-0400 c20012| 2018-03-16T17:28:20.772-0400 I NETWORK [listener] connection accepted from 127.0.0.1:60650 #17 (8 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.773-0400 c20014| 2018-03-16T17:28:20.772-0400 I NETWORK [conn13] received client metadata from 127.0.0.1:44438 conn13: { driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.773-0400 c20012| 2018-03-16T17:28:20.772-0400 I NETWORK [conn17] received client metadata from 127.0.0.1:60650 conn17: { driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.773-0400 2018-03-16T17:28:20.772-0400 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to ubuntu:20014 (1 connections now open to ubuntu:20014 with a 5 second timeout) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.773-0400 2018-03-16T17:28:20.773-0400 I NETWORK [js] Successfully connected to ubuntu:20012 (1 connections now open to ubuntu:20012 with a 5 second timeout) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.773-0400 ShardingTest stale_mongos_updates_and_removes : [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.774-0400 { [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.774-0400 "config" : "stale_mongos_updates_and_removes-configRS/ubuntu:20012,ubuntu:20013,ubuntu:20014", [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.774-0400 "shards" : [ [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.774-0400 connection to stale_mongos_updates_and_removes-rs0/ubuntu:20010, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.774-0400 connection to stale_mongos_updates_and_removes-rs1/ubuntu:20011 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.774-0400 ] [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.774-0400 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.774-0400 c20013| 2018-03-16T17:28:20.773-0400 I NETWORK [listener] connection accepted from 127.0.0.1:47020 #10 (4 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.775-0400 c20013| 2018-03-16T17:28:20.773-0400 I NETWORK [conn10] received client metadata from 127.0.0.1:47020 conn10: { driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.775-0400 2018-03-16T17:28:20.773-0400 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to ubuntu:20013 (1 connections now open to ubuntu:20013 with a 5 second timeout) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.775-0400 2018-03-16T17:28:20.775-0400 I - [js] shell: started program (sh24552): /home/janna/server/mongo/mongos-3.6 -vv --port 20015 --configdb stale_mongos_updates_and_removes-configRS/ubuntu:20012,ubuntu:20013,ubuntu:20014 --bind_ip 0.0.0.0 --setParameter enableTestCommands=1 --setParameter disableLogicalSessionCacheRefresh=true --setParameter logComponentVerbosity={"tracking":0} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.791-0400 s20015| 2018-03-16T17:28:20.791-0400 I CONTROL [main] [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.791-0400 s20015| 2018-03-16T17:28:20.791-0400 I CONTROL [main] ** WARNING: Access control is not enabled for the database. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.791-0400 s20015| 2018-03-16T17:28:20.791-0400 I CONTROL [main] ** Read and write access to data and configuration is unrestricted. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.791-0400 s20015| 2018-03-16T17:28:20.791-0400 I CONTROL [main] [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.792-0400 s20015| 2018-03-16T17:28:20.791-0400 I SHARDING [mongosMain] mongos version v3.6.3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.792-0400 s20015| 2018-03-16T17:28:20.791-0400 I CONTROL [mongosMain] git version: 9586e557d54ef70f9ca4b43c26892cd55257e1a5 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.792-0400 s20015| 2018-03-16T17:28:20.791-0400 I CONTROL [mongosMain] OpenSSL version: OpenSSL 1.0.2g 1 Mar 2016 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.792-0400 s20015| 2018-03-16T17:28:20.791-0400 I CONTROL [mongosMain] allocator: tcmalloc [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.792-0400 s20015| 2018-03-16T17:28:20.791-0400 I CONTROL [mongosMain] modules: none [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.792-0400 s20015| 2018-03-16T17:28:20.791-0400 I CONTROL [mongosMain] build environment: [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.792-0400 s20015| 2018-03-16T17:28:20.791-0400 I CONTROL [mongosMain] distmod: ubuntu1604 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.793-0400 s20015| 2018-03-16T17:28:20.791-0400 I CONTROL [mongosMain] distarch: x86_64 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.793-0400 s20015| 2018-03-16T17:28:20.791-0400 I CONTROL [mongosMain] target_arch: x86_64 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.793-0400 s20015| 2018-03-16T17:28:20.791-0400 I CONTROL [mongosMain] db version v3.6.3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.793-0400 s20015| 2018-03-16T17:28:20.791-0400 I CONTROL [mongosMain] git version: 9586e557d54ef70f9ca4b43c26892cd55257e1a5 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.793-0400 s20015| 2018-03-16T17:28:20.791-0400 I CONTROL [mongosMain] OpenSSL version: OpenSSL 1.0.2g 1 Mar 2016 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.793-0400 s20015| 2018-03-16T17:28:20.791-0400 I CONTROL [mongosMain] allocator: tcmalloc [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.793-0400 s20015| 2018-03-16T17:28:20.791-0400 I CONTROL [mongosMain] modules: none [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.793-0400 s20015| 2018-03-16T17:28:20.791-0400 I CONTROL [mongosMain] build environment: [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.794-0400 s20015| 2018-03-16T17:28:20.791-0400 I CONTROL [mongosMain] distmod: ubuntu1604 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.794-0400 s20015| 2018-03-16T17:28:20.791-0400 I CONTROL [mongosMain] distarch: x86_64 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.794-0400 s20015| 2018-03-16T17:28:20.791-0400 I CONTROL [mongosMain] target_arch: x86_64 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.794-0400 s20015| 2018-03-16T17:28:20.791-0400 I CONTROL [mongosMain] options: { net: { bindIp: "0.0.0.0", port: 20015 }, setParameter: { disableLogicalSessionCacheRefresh: "true", enableTestCommands: "1", logComponentVerbosity: "{"tracking":0}" }, sharding: { configDB: "stale_mongos_updates_and_removes-configRS/ubuntu:20012,ubuntu:20013,ubuntu:20014" }, systemLog: { verbosity: 2 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.794-0400 s20015| 2018-03-16T17:28:20.791-0400 D NETWORK [mongosMain] fd limit hard:1048576 soft:1024 max conn: 819 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.795-0400 s20015| 2018-03-16T17:28:20.792-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-0-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.795-0400 s20015| 2018-03-16T17:28:20.792-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-1-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.795-0400 s20015| 2018-03-16T17:28:20.792-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-2-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.795-0400 s20015| 2018-03-16T17:28:20.792-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-3-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.795-0400 s20015| 2018-03-16T17:28:20.792-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-4-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.795-0400 s20015| 2018-03-16T17:28:20.792-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-5-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.796-0400 s20015| 2018-03-16T17:28:20.792-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-6-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.796-0400 s20015| 2018-03-16T17:28:20.792-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.796-0400 s20015| 2018-03-16T17:28:20.792-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-8-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.796-0400 s20015| 2018-03-16T17:28:20.792-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-7-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.796-0400 s20015| 2018-03-16T17:28:20.792-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-9-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.796-0400 s20015| 2018-03-16T17:28:20.792-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-10-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.797-0400 s20015| 2018-03-16T17:28:20.792-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-11-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.797-0400 s20015| 2018-03-16T17:28:20.792-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-13-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.797-0400 s20015| 2018-03-16T17:28:20.792-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-15-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.797-0400 s20015| 2018-03-16T17:28:20.792-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-14-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.797-0400 s20015| 2018-03-16T17:28:20.792-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-12-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.797-0400 s20015| 2018-03-16T17:28:20.792-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-16-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.798-0400 s20015| 2018-03-16T17:28:20.792-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-19-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.798-0400 s20015| 2018-03-16T17:28:20.792-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-17-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.798-0400 s20015| 2018-03-16T17:28:20.792-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-22-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.798-0400 s20015| 2018-03-16T17:28:20.792-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-21-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.798-0400 s20015| 2018-03-16T17:28:20.792-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-20-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.799-0400 s20015| 2018-03-16T17:28:20.792-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-18-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.799-0400 s20015| 2018-03-16T17:28:20.792-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-23-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.799-0400 s20015| 2018-03-16T17:28:20.792-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-26-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.799-0400 s20015| 2018-03-16T17:28:20.792-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-28-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.799-0400 s20015| 2018-03-16T17:28:20.792-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-29-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.799-0400 s20015| 2018-03-16T17:28:20.792-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-27-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.800-0400 s20015| 2018-03-16T17:28:20.792-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-30-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.800-0400 s20015| 2018-03-16T17:28:20.792-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-31-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.800-0400 s20015| 2018-03-16T17:28:20.792-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-32-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.800-0400 s20015| 2018-03-16T17:28:20.793-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-35-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.800-0400 s20015| 2018-03-16T17:28:20.793-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-34-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.800-0400 s20015| 2018-03-16T17:28:20.793-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-36-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.801-0400 s20015| 2018-03-16T17:28:20.793-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-33-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.801-0400 s20015| 2018-03-16T17:28:20.793-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-24-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.801-0400 s20015| 2018-03-16T17:28:20.793-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-39-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.801-0400 s20015| 2018-03-16T17:28:20.793-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-37-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.801-0400 s20015| 2018-03-16T17:28:20.793-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-38-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.801-0400 s20015| 2018-03-16T17:28:20.793-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-25-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.802-0400 s20015| 2018-03-16T17:28:20.793-0400 D NETWORK [mongosMain] Starting up task executor for monitoring replica sets in response to request to monitor set: stale_mongos_updates_and_removes-configRS/ubuntu:20012,ubuntu:20013,ubuntu:20014 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.802-0400 s20015| 2018-03-16T17:28:20.793-0400 I NETWORK [mongosMain] Starting new replica set monitor for stale_mongos_updates_and_removes-configRS/ubuntu:20012,ubuntu:20013,ubuntu:20014 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.802-0400 s20015| 2018-03-16T17:28:20.793-0400 D ASIO [ReplicaSetMonitor-TaskExecutor-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.802-0400 s20015| 2018-03-16T17:28:20.793-0400 D NETWORK [mongosMain] Started targeter for stale_mongos_updates_and_removes-configRS/ubuntu:20012,ubuntu:20013,ubuntu:20014 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.802-0400 s20015| 2018-03-16T17:28:20.793-0400 D NETWORK [ReplicaSetMonitor-TaskExecutor-0] Starting new refresh of replica set stale_mongos_updates_and_removes-configRS [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.803-0400 s20015| 2018-03-16T17:28:20.793-0400 D SHARDING [mongosMain] Starting up task executor for periodic reloading of ShardRegistry [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.803-0400 s20015| 2018-03-16T17:28:20.793-0400 D NETWORK [ReplicaSetMonitor-TaskExecutor-0] creating new connection to:ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.803-0400 s20015| 2018-03-16T17:28:20.793-0400 D ASIO [ShardRegistryUpdater-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.803-0400 s20015| 2018-03-16T17:28:20.793-0400 D SHARDING [ShardRegistryUpdater-0] Reloading shardRegistry [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.803-0400 s20015| 2018-03-16T17:28:20.793-0400 I SHARDING [thread1] creating distributed lock ping thread for process ubuntu:20015:1521235700:-7388722962896244705 (sleeping for 30000ms) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.803-0400 s20015| 2018-03-16T17:28:20.793-0400 D NETWORK [mongosMain] creating new connection to:ubuntu:20014 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.804-0400 s20015| 2018-03-16T17:28:20.793-0400 D NETWORK [ReplicaSetMonitor-TaskExecutor-0] connected to server ubuntu:20012 (127.0.1.1) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.804-0400 s20015| 2018-03-16T17:28:20.793-0400 D NETWORK [shard registry reload] creating new connection to:ubuntu:20013 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.804-0400 c20012| 2018-03-16T17:28:20.793-0400 I NETWORK [listener] connection accepted from 127.0.0.1:60658 #18 (9 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.804-0400 s20015| 2018-03-16T17:28:20.793-0400 D NETWORK [mongosMain] connected to server ubuntu:20014 (127.0.1.1) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.804-0400 s20015| 2018-03-16T17:28:20.793-0400 D NETWORK [shard registry reload] connected to server ubuntu:20013 (127.0.1.1) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.804-0400 c20014| 2018-03-16T17:28:20.793-0400 I NETWORK [listener] connection accepted from 127.0.0.1:44446 #14 (6 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.805-0400 c20013| 2018-03-16T17:28:20.793-0400 I NETWORK [listener] connection accepted from 127.0.0.1:47028 #11 (5 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.805-0400 c20012| 2018-03-16T17:28:20.793-0400 I NETWORK [conn18] received client metadata from 127.0.0.1:60658 conn18: { driver: { name: "MongoDB Internal Client", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.805-0400 c20014| 2018-03-16T17:28:20.793-0400 I NETWORK [conn14] received client metadata from 127.0.0.1:44446 conn14: { driver: { name: "MongoDB Internal Client", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.805-0400 c20013| 2018-03-16T17:28:20.793-0400 I NETWORK [conn11] received client metadata from 127.0.0.1:47028 conn11: { driver: { name: "MongoDB Internal Client", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.806-0400 s20015| 2018-03-16T17:28:20.793-0400 D NETWORK [ReplicaSetMonitor-TaskExecutor-0] connected connection! [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.806-0400 s20015| 2018-03-16T17:28:20.793-0400 D NETWORK [mongosMain] connected connection! [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.806-0400 s20015| 2018-03-16T17:28:20.794-0400 D NETWORK [shard registry reload] connected connection! [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.806-0400 s20015| 2018-03-16T17:28:20.794-0400 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to ubuntu:20012 (1 connections now open to ubuntu:20012 with a 5 second timeout) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.806-0400 s20015| 2018-03-16T17:28:20.794-0400 I NETWORK [shard registry reload] Successfully connected to ubuntu:20013 (1 connections now open to ubuntu:20013 with a 5 second timeout) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.806-0400 s20015| 2018-03-16T17:28:20.794-0400 I NETWORK [mongosMain] Successfully connected to ubuntu:20014 (1 connections now open to ubuntu:20014 with a 5 second timeout) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.807-0400 s20015| 2018-03-16T17:28:20.794-0400 D ASIO [replSetDistLockPinger] startCommand: RemoteCommand 1 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:50.794-0400 cmd:{ findAndModify: "lockpings", query: { _id: "ubuntu:20015:1521235700:-7388722962896244705" }, update: { $set: { ping: new Date(1521235700793) } }, upsert: true, writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.807-0400 s20015| 2018-03-16T17:28:20.794-0400 D ASIO [shard registry reload] startCommand: RemoteCommand 2 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:50.794-0400 cmd:{ find: "shards", readConcern: { level: "majority", afterOpTime: { ts: Timestamp(0, 0), t: -1 } }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.807-0400 s20015| 2018-03-16T17:28:20.794-0400 D ASIO [monitoring keys for HMAC] startCommand: RemoteCommand 3 -- target:ubuntu:20013 db:admin expDate:2018-03-16T17:28:50.794-0400 cmd:{ find: "system.keys", filter: { purpose: "HMAC", expiresAt: { $gt: Timestamp(0, 0) } }, sort: { expiresAt: 1 }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp(0, 0), t: -1 } }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.807-0400 s20015| 2018-03-16T17:28:20.794-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Connecting to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.808-0400 s20015| 2018-03-16T17:28:20.794-0400 D NETWORK [ReplicaSetMonitor-TaskExecutor-0] Refreshing replica set stale_mongos_updates_and_removes-configRS took 1 msec [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.808-0400 s20015| 2018-03-16T17:28:20.794-0400 D ASIO [mongosMain] startCommand: RemoteCommand 7 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:50.794-0400 cmd:{ find: "version", readConcern: { level: "majority", afterOpTime: { ts: Timestamp(0, 0), t: -1 } }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.808-0400 s20015| 2018-03-16T17:28:20.794-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Connecting to ubuntu:20013 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.808-0400 s20015| 2018-03-16T17:28:20.794-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Connecting to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.808-0400 c20012| 2018-03-16T17:28:20.794-0400 I NETWORK [listener] connection accepted from 127.0.0.1:60664 #19 (10 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.809-0400 s20015| 2018-03-16T17:28:20.794-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 4 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.809-0400 c20013| 2018-03-16T17:28:20.794-0400 I NETWORK [listener] connection accepted from 127.0.0.1:47032 #12 (6 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.809-0400 c20012| 2018-03-16T17:28:20.794-0400 I NETWORK [listener] connection accepted from 127.0.0.1:60668 #20 (11 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.809-0400 s20015| 2018-03-16T17:28:20.794-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 5 on host ubuntu:20013 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.809-0400 c20012| 2018-03-16T17:28:20.795-0400 I NETWORK [conn19] received client metadata from 127.0.0.1:60664 conn19: { driver: { name: "NetworkInterfaceASIO-ShardRegistry", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.809-0400 s20015| 2018-03-16T17:28:20.795-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 6 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.810-0400 c20013| 2018-03-16T17:28:20.795-0400 I NETWORK [conn12] received client metadata from 127.0.0.1:47032 conn12: { driver: { name: "NetworkInterfaceASIO-ShardRegistry", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.810-0400 s20015| 2018-03-16T17:28:20.795-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to ubuntu:20012, took 1ms (2 connections now open to ubuntu:20012) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.810-0400 s20015| 2018-03-16T17:28:20.795-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 4 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.810-0400 c20012| 2018-03-16T17:28:20.795-0400 I NETWORK [conn20] received client metadata from 127.0.0.1:60668 conn20: { driver: { name: "NetworkInterfaceASIO-ShardRegistry", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.810-0400 s20015| 2018-03-16T17:28:20.795-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Connecting to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.811-0400 s20015| 2018-03-16T17:28:20.795-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 2 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.811-0400 s20015| 2018-03-16T17:28:20.795-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to ubuntu:20013, took 1ms (1 connections now open to ubuntu:20013) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.811-0400 s20015| 2018-03-16T17:28:20.795-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 5 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.811-0400 c20012| 2018-03-16T17:28:20.795-0400 I NETWORK [listener] connection accepted from 127.0.0.1:60670 #21 (12 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.811-0400 s20015| 2018-03-16T17:28:20.795-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 3 on host ubuntu:20013 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.811-0400 s20015| 2018-03-16T17:28:20.795-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to ubuntu:20012, took 1ms (3 connections now open to ubuntu:20012) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.812-0400 s20015| 2018-03-16T17:28:20.795-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 6 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.812-0400 s20015| 2018-03-16T17:28:20.795-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 8 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.812-0400 s20015| 2018-03-16T17:28:20.795-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 1 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.812-0400 c20012| 2018-03-16T17:28:20.795-0400 I NETWORK [conn21] received client metadata from 127.0.0.1:60670 conn21: { driver: { name: "NetworkInterfaceASIO-ShardRegistry", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.813-0400 s20015| 2018-03-16T17:28:20.795-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 2 finished with response: { cursor: { firstBatch: [], id: 0, ns: "config.shards" }, ok: 1.0, operationTime: Timestamp(1521235700, 1), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235700, 1), t: 1 }, lastOpVisible: { ts: Timestamp(1521235700, 1), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235700, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235700, 1) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.813-0400 s20015| 2018-03-16T17:28:20.795-0400 D SHARDING [shard registry reload] found 0 shards listed on config server(s) with lastVisibleOpTime: { ts: Timestamp(1521235700, 1), t: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.813-0400 s20015| 2018-03-16T17:28:20.795-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 7 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.814-0400 s20015| 2018-03-16T17:28:20.795-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 3 finished with response: { cursor: { firstBatch: [ { _id: 6533657568122765331, purpose: "HMAC", key: BinData(0, 5B47B1F30E0CF2413E50FA7BFA35505CB7C97E21), expiresAt: Timestamp(1529011697, 0) }, { _id: 6533657568122765332, purpose: "HMAC", key: BinData(0, DDCFFA4B6DC56EC9F68A43CE5A9FF843430AD67C), expiresAt: Timestamp(1536787697, 0) } ], id: 0, ns: "admin.system.keys" }, ok: 1.0, operationTime: Timestamp(1521235700, 1), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235700, 1), t: 1 }, lastOpVisible: { ts: Timestamp(1521235700, 1), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235700, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235700, 1) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.814-0400 s20015| 2018-03-16T17:28:20.795-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to ubuntu:20012, took 0ms (3 connections now open to ubuntu:20012) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.815-0400 s20015| 2018-03-16T17:28:20.795-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 8 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.815-0400 s20015| 2018-03-16T17:28:20.796-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 7 finished with response: { cursor: { firstBatch: [ { _id: 1, minCompatibleVersion: 5, currentVersion: 6, clusterId: ObjectId('5aac36f188c1f30185b7ac89') } ], id: 0, ns: "config.version" }, ok: 1.0, operationTime: Timestamp(1521235700, 1), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235700, 1), t: 1 }, lastOpVisible: { ts: Timestamp(1521235700, 1), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235700, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235700, 1) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.815-0400 s20015| 2018-03-16T17:28:20.796-0400 I SHARDING [mongosMain] Not waiting for signing keys, not supported by the config shard stale_mongos_updates_and_removes-configRS [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.816-0400 s20015| 2018-03-16T17:28:20.796-0400 D ASIO [mongosMain] startCommand: RemoteCommand 12 -- target:ubuntu:20013 db:config expDate:2018-03-16T17:28:50.796-0400 cmd:{ find: "settings", filter: { _id: "balancer" }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235700, 1), t: 1 } }, limit: 1, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.816-0400 s20015| 2018-03-16T17:28:20.796-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 12 on host ubuntu:20013 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.817-0400 s20015| 2018-03-16T17:28:20.796-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 12 finished with response: { cursor: { id: 0, ns: "config.settings", firstBatch: [] }, ok: 1.0, operationTime: Timestamp(1521235700, 1), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235700, 1), t: 1 }, lastOpVisible: { ts: Timestamp(1521235700, 1), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235700, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235700, 1) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.817-0400 s20015| 2018-03-16T17:28:20.796-0400 D ASIO [mongosMain] startCommand: RemoteCommand 14 -- target:ubuntu:20013 db:config expDate:2018-03-16T17:28:50.796-0400 cmd:{ find: "settings", filter: { _id: "chunksize" }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235700, 1), t: 1 } }, limit: 1, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.817-0400 s20015| 2018-03-16T17:28:20.796-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 14 on host ubuntu:20013 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.818-0400 s20015| 2018-03-16T17:28:20.796-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 14 finished with response: { cursor: { id: 0, ns: "config.settings", firstBatch: [] }, ok: 1.0, operationTime: Timestamp(1521235700, 1), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235700, 1), t: 1 }, lastOpVisible: { ts: Timestamp(1521235700, 1), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235700, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235700, 1) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.818-0400 s20015| 2018-03-16T17:28:20.796-0400 D ASIO [mongosMain] startCommand: RemoteCommand 16 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:50.796-0400 cmd:{ find: "settings", filter: { _id: "autosplit" }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235700, 1), t: 1 } }, limit: 1, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.818-0400 s20015| 2018-03-16T17:28:20.797-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 16 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.819-0400 s20015| 2018-03-16T17:28:20.797-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 16 finished with response: { cursor: { id: 0, ns: "config.settings", firstBatch: [] }, ok: 1.0, operationTime: Timestamp(1521235700, 1), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235700, 1), t: 1 }, lastOpVisible: { ts: Timestamp(1521235700, 1), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235700, 2), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235700, 1) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.819-0400 s20015| 2018-03-16T17:28:20.797-0400 W FTDC [mongosMain] FTDC is disabled because neither '--logpath' nor set parameter 'diagnosticDataCollectionDirectoryPath' are specified. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.819-0400 s20015| 2018-03-16T17:28:20.797-0400 I FTDC [mongosMain] Initializing full-time diagnostic data capture with directory '' [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.819-0400 s20015| 2018-03-16T17:28:20.797-0400 D ASIO [mongosMain] startCommand: RemoteCommand 18 -- target:ubuntu:20012 db:admin expDate:2018-03-16T17:28:50.797-0400 cmd:{ _getUserCacheGeneration: 1, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.820-0400 s20015| 2018-03-16T17:28:20.797-0400 D COMMAND [ClusterCursorCleanupJob] BackgroundJob starting: ClusterCursorCleanupJob [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.820-0400 s20015| 2018-03-16T17:28:20.797-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 18 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.820-0400 s20015| 2018-03-16T17:28:20.797-0400 D ASIO [Uptime reporter] startCommand: RemoteCommand 19 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:50.797-0400 cmd:{ update: "mongos", bypassDocumentValidation: false, ordered: true, updates: [ { q: { _id: "ubuntu:20015" }, u: { $set: { _id: "ubuntu:20015", ping: new Date(1521235700797), up: 0, waiting: true, mongoVersion: "3.6.3" } }, multi: false, upsert: true } ], writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.820-0400 s20015| 2018-03-16T17:28:20.797-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 19 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.821-0400 s20015| 2018-03-16T17:28:20.797-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 18 finished with response: { cacheGeneration: ObjectId('5aac36f488c1f30185b7acfe'), ok: 1.0, operationTime: Timestamp(1521235700, 2), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235700, 1), t: 1 }, lastOpVisible: { ts: Timestamp(1521235700, 1), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235700, 2), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235700, 1) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.821-0400 c20012| 2018-03-16T17:28:20.797-0400 I STORAGE [conn21] createCollection: config.mongos with generated UUID: a356f24a-7ad5-4f85-ab74-7c1b9fd98da1 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.821-0400 s20015| 2018-03-16T17:28:20.797-0400 D COMMAND [UserCacheInvalidatorThread] BackgroundJob starting: UserCacheInvalidatorThread [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.821-0400 s20015| 2018-03-16T17:28:20.798-0400 D COMMAND [PeriodicTaskRunner] BackgroundJob starting: PeriodicTaskRunner [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.822-0400 s20015| 2018-03-16T17:28:20.798-0400 I NETWORK [mongosMain] waiting for connections on port 20015 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.822-0400 s20015| 2018-03-16T17:28:20.809-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 1 finished with response: { lastErrorObject: { n: 1, updatedExisting: false, upserted: "ubuntu:20015:1521235700:-7388722962896244705" }, value: null, ok: 1.0, operationTime: Timestamp(1521235700, 2), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235700, 2), t: 1 }, lastOpVisible: { ts: Timestamp(1521235700, 2), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235700, 2), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235700, 2), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235700, 2) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.822-0400 s20015| 2018-03-16T17:28:20.809-0400 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: LockStateChangeFailed: findAndModify query predicate didn't match any lock document [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.823-0400 c20013| 2018-03-16T17:28:20.821-0400 I STORAGE [repl writer worker 4] createCollection: config.mongos with provided UUID: a356f24a-7ad5-4f85-ab74-7c1b9fd98da1 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.823-0400 c20014| 2018-03-16T17:28:20.821-0400 I STORAGE [repl writer worker 3] createCollection: config.mongos with provided UUID: a356f24a-7ad5-4f85-ab74-7c1b9fd98da1 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.856-0400 s20015| 2018-03-16T17:28:20.855-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 19 finished with response: { n: 1, nModified: 0, upserted: [ { index: 0, _id: "ubuntu:20015" } ], opTime: { ts: Timestamp(1521235700, 4), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235700, 4), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235700, 4), t: 1 }, lastOpVisible: { ts: Timestamp(1521235700, 4), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235700, 4), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235700, 4), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235700, 4) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.856-0400 s20015| 2018-03-16T17:28:20.855-0400 D ASIO [Uptime reporter] startCommand: RemoteCommand 23 -- target:ubuntu:20013 db:config expDate:2018-03-16T17:28:50.855-0400 cmd:{ find: "settings", filter: { _id: "balancer" }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235700, 4), t: 1 } }, limit: 1, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.856-0400 s20015| 2018-03-16T17:28:20.855-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 23 on host ubuntu:20013 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.857-0400 s20015| 2018-03-16T17:28:20.855-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 23 finished with response: { cursor: { id: 0, ns: "config.settings", firstBatch: [] }, ok: 1.0, operationTime: Timestamp(1521235700, 4), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235700, 4), t: 1 }, lastOpVisible: { ts: Timestamp(1521235700, 4), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235700, 4), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235700, 4) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.857-0400 s20015| 2018-03-16T17:28:20.855-0400 D ASIO [Uptime reporter] startCommand: RemoteCommand 25 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:50.855-0400 cmd:{ find: "settings", filter: { _id: "chunksize" }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235700, 4), t: 1 } }, limit: 1, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.857-0400 s20015| 2018-03-16T17:28:20.855-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 25 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.858-0400 s20015| 2018-03-16T17:28:20.856-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 25 finished with response: { cursor: { id: 0, ns: "config.settings", firstBatch: [] }, ok: 1.0, operationTime: Timestamp(1521235700, 4), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235700, 4), t: 1 }, lastOpVisible: { ts: Timestamp(1521235700, 4), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235700, 4), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235700, 4), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235700, 4) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.858-0400 s20015| 2018-03-16T17:28:20.856-0400 D ASIO [Uptime reporter] startCommand: RemoteCommand 27 -- target:ubuntu:20013 db:config expDate:2018-03-16T17:28:50.856-0400 cmd:{ find: "settings", filter: { _id: "autosplit" }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235700, 4), t: 1 } }, limit: 1, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.858-0400 s20015| 2018-03-16T17:28:20.856-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 27 on host ubuntu:20013 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.859-0400 s20015| 2018-03-16T17:28:20.856-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 27 finished with response: { cursor: { id: 0, ns: "config.settings", firstBatch: [] }, ok: 1.0, operationTime: Timestamp(1521235700, 4), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235700, 4), t: 1 }, lastOpVisible: { ts: Timestamp(1521235700, 4), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235700, 4), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235700, 4) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.976-0400 s20015| 2018-03-16T17:28:20.975-0400 I NETWORK [listener] connection accepted from 127.0.0.1:47840 #1 (1 connection now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.976-0400 s20015| 2018-03-16T17:28:20.976-0400 I NETWORK [conn1] received client metadata from 127.0.0.1:47840 conn: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.978-0400 2018-03-16T17:28:20.978-0400 I - [js] shell: started program (sh24617): /home/janna/server/mongo/mongos-3.6 -vv --port 20016 --configdb stale_mongos_updates_and_removes-configRS/ubuntu:20012,ubuntu:20013,ubuntu:20014 --bind_ip 0.0.0.0 --setParameter enableTestCommands=1 --setParameter disableLogicalSessionCacheRefresh=true --setParameter logComponentVerbosity={"tracking":0} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.994-0400 s20016| 2018-03-16T17:28:20.993-0400 I CONTROL [main] [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.994-0400 s20016| 2018-03-16T17:28:20.993-0400 I CONTROL [main] ** WARNING: Access control is not enabled for the database. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.994-0400 s20016| 2018-03-16T17:28:20.993-0400 I CONTROL [main] ** Read and write access to data and configuration is unrestricted. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.994-0400 s20016| 2018-03-16T17:28:20.993-0400 I CONTROL [main] [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.994-0400 s20016| 2018-03-16T17:28:20.994-0400 I SHARDING [mongosMain] mongos version v3.6.3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.994-0400 s20016| 2018-03-16T17:28:20.994-0400 I CONTROL [mongosMain] git version: 9586e557d54ef70f9ca4b43c26892cd55257e1a5 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.995-0400 s20016| 2018-03-16T17:28:20.994-0400 I CONTROL [mongosMain] OpenSSL version: OpenSSL 1.0.2g 1 Mar 2016 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.995-0400 s20016| 2018-03-16T17:28:20.994-0400 I CONTROL [mongosMain] allocator: tcmalloc [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.995-0400 s20016| 2018-03-16T17:28:20.994-0400 I CONTROL [mongosMain] modules: none [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.995-0400 s20016| 2018-03-16T17:28:20.994-0400 I CONTROL [mongosMain] build environment: [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.995-0400 s20016| 2018-03-16T17:28:20.994-0400 I CONTROL [mongosMain] distmod: ubuntu1604 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.995-0400 s20016| 2018-03-16T17:28:20.994-0400 I CONTROL [mongosMain] distarch: x86_64 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.995-0400 s20016| 2018-03-16T17:28:20.994-0400 I CONTROL [mongosMain] target_arch: x86_64 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.995-0400 s20016| 2018-03-16T17:28:20.994-0400 I CONTROL [mongosMain] db version v3.6.3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.996-0400 s20016| 2018-03-16T17:28:20.994-0400 I CONTROL [mongosMain] git version: 9586e557d54ef70f9ca4b43c26892cd55257e1a5 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.996-0400 s20016| 2018-03-16T17:28:20.994-0400 I CONTROL [mongosMain] OpenSSL version: OpenSSL 1.0.2g 1 Mar 2016 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.996-0400 s20016| 2018-03-16T17:28:20.994-0400 I CONTROL [mongosMain] allocator: tcmalloc [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.996-0400 s20016| 2018-03-16T17:28:20.994-0400 I CONTROL [mongosMain] modules: none [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.996-0400 s20016| 2018-03-16T17:28:20.994-0400 I CONTROL [mongosMain] build environment: [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.996-0400 s20016| 2018-03-16T17:28:20.994-0400 I CONTROL [mongosMain] distmod: ubuntu1604 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.996-0400 s20016| 2018-03-16T17:28:20.994-0400 I CONTROL [mongosMain] distarch: x86_64 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.997-0400 s20016| 2018-03-16T17:28:20.994-0400 I CONTROL [mongosMain] target_arch: x86_64 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.997-0400 s20016| 2018-03-16T17:28:20.994-0400 I CONTROL [mongosMain] options: { net: { bindIp: "0.0.0.0", port: 20016 }, setParameter: { disableLogicalSessionCacheRefresh: "true", enableTestCommands: "1", logComponentVerbosity: "{"tracking":0}" }, sharding: { configDB: "stale_mongos_updates_and_removes-configRS/ubuntu:20012,ubuntu:20013,ubuntu:20014" }, systemLog: { verbosity: 2 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.997-0400 s20016| 2018-03-16T17:28:20.994-0400 D NETWORK [mongosMain] fd limit hard:1048576 soft:1024 max conn: 819 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.997-0400 s20016| 2018-03-16T17:28:20.994-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.997-0400 s20016| 2018-03-16T17:28:20.994-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-1-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.998-0400 s20016| 2018-03-16T17:28:20.994-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-0-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.998-0400 s20016| 2018-03-16T17:28:20.994-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-2-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.998-0400 s20016| 2018-03-16T17:28:20.994-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-3-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.998-0400 s20016| 2018-03-16T17:28:20.994-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-5-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.998-0400 s20016| 2018-03-16T17:28:20.994-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-6-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.998-0400 s20016| 2018-03-16T17:28:20.994-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-4-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.999-0400 s20016| 2018-03-16T17:28:20.994-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-7-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.999-0400 s20016| 2018-03-16T17:28:20.994-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-9-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.999-0400 s20016| 2018-03-16T17:28:20.995-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-10-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.999-0400 s20016| 2018-03-16T17:28:20.995-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-11-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.999-0400 s20016| 2018-03-16T17:28:20.994-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-8-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:20.999-0400 s20016| 2018-03-16T17:28:20.995-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-12-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.000-0400 s20016| 2018-03-16T17:28:20.995-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-13-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.000-0400 s20016| 2018-03-16T17:28:20.995-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-17-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.000-0400 s20016| 2018-03-16T17:28:20.995-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-15-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.000-0400 s20016| 2018-03-16T17:28:20.995-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-18-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.000-0400 s20016| 2018-03-16T17:28:20.995-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-16-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.000-0400 s20016| 2018-03-16T17:28:20.995-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-14-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.001-0400 s20016| 2018-03-16T17:28:20.995-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-19-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.001-0400 s20016| 2018-03-16T17:28:20.995-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-22-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.001-0400 s20016| 2018-03-16T17:28:20.995-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-20-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.001-0400 s20016| 2018-03-16T17:28:20.995-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-21-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.001-0400 s20016| 2018-03-16T17:28:20.995-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-26-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.001-0400 s20016| 2018-03-16T17:28:20.995-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-27-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.002-0400 s20016| 2018-03-16T17:28:20.995-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-24-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.002-0400 s20016| 2018-03-16T17:28:20.995-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-23-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.002-0400 s20016| 2018-03-16T17:28:20.995-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-25-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.002-0400 s20016| 2018-03-16T17:28:20.995-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-28-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.002-0400 s20016| 2018-03-16T17:28:20.995-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-31-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.003-0400 s20016| 2018-03-16T17:28:20.995-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-30-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.003-0400 s20016| 2018-03-16T17:28:20.995-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-32-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.003-0400 s20016| 2018-03-16T17:28:20.995-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-29-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.003-0400 s20016| 2018-03-16T17:28:20.995-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-33-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.003-0400 s20016| 2018-03-16T17:28:20.995-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-34-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.003-0400 s20016| 2018-03-16T17:28:20.995-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-35-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.004-0400 s20016| 2018-03-16T17:28:20.995-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-36-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.004-0400 s20016| 2018-03-16T17:28:20.995-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-38-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.004-0400 s20016| 2018-03-16T17:28:20.995-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-39-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.004-0400 s20016| 2018-03-16T17:28:20.995-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-37-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.005-0400 s20016| 2018-03-16T17:28:20.995-0400 D NETWORK [mongosMain] Starting up task executor for monitoring replica sets in response to request to monitor set: stale_mongos_updates_and_removes-configRS/ubuntu:20012,ubuntu:20013,ubuntu:20014 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.005-0400 s20016| 2018-03-16T17:28:20.995-0400 I NETWORK [mongosMain] Starting new replica set monitor for stale_mongos_updates_and_removes-configRS/ubuntu:20012,ubuntu:20013,ubuntu:20014 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.005-0400 s20016| 2018-03-16T17:28:20.996-0400 D NETWORK [mongosMain] Started targeter for stale_mongos_updates_and_removes-configRS/ubuntu:20012,ubuntu:20013,ubuntu:20014 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.005-0400 s20016| 2018-03-16T17:28:20.996-0400 D SHARDING [mongosMain] Starting up task executor for periodic reloading of ShardRegistry [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.005-0400 s20016| 2018-03-16T17:28:20.996-0400 D ASIO [ShardRegistryUpdater-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.005-0400 s20016| 2018-03-16T17:28:20.996-0400 I SHARDING [thread1] creating distributed lock ping thread for process ubuntu:20016:1521235700:-4633782234644071543 (sleeping for 30000ms) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.006-0400 s20016| 2018-03-16T17:28:20.996-0400 D NETWORK [mongosMain] Starting new refresh of replica set stale_mongos_updates_and_removes-configRS [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.006-0400 s20016| 2018-03-16T17:28:20.996-0400 D NETWORK [mongosMain] creating new connection to:ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.006-0400 s20016| 2018-03-16T17:28:20.996-0400 D SHARDING [ShardRegistryUpdater-0] Reloading shardRegistry [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.006-0400 s20016| 2018-03-16T17:28:20.996-0400 D ASIO [ReplicaSetMonitor-TaskExecutor-0] The NetworkInterfaceASIO worker thread is spinning up [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.006-0400 s20016| 2018-03-16T17:28:20.996-0400 D NETWORK [monitoring keys for HMAC] creating new connection to:ubuntu:20014 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.006-0400 s20016| 2018-03-16T17:28:20.996-0400 D NETWORK [replSetDistLockPinger] creating new connection to:ubuntu:20013 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.007-0400 s20016| 2018-03-16T17:28:20.996-0400 D NETWORK [mongosMain] connected to server ubuntu:20012 (127.0.1.1) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.007-0400 c20012| 2018-03-16T17:28:20.996-0400 I NETWORK [listener] connection accepted from 127.0.0.1:60676 #22 (13 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.007-0400 s20016| 2018-03-16T17:28:20.996-0400 D NETWORK [monitoring keys for HMAC] connected to server ubuntu:20014 (127.0.1.1) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.007-0400 s20016| 2018-03-16T17:28:20.996-0400 D NETWORK [replSetDistLockPinger] connected to server ubuntu:20013 (127.0.1.1) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.007-0400 c20014| 2018-03-16T17:28:20.996-0400 I NETWORK [listener] connection accepted from 127.0.0.1:44464 #15 (7 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.007-0400 c20013| 2018-03-16T17:28:20.996-0400 I NETWORK [listener] connection accepted from 127.0.0.1:47044 #13 (7 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.008-0400 c20012| 2018-03-16T17:28:20.996-0400 I NETWORK [conn22] received client metadata from 127.0.0.1:60676 conn22: { driver: { name: "MongoDB Internal Client", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.008-0400 s20016| 2018-03-16T17:28:20.996-0400 D NETWORK [mongosMain] connected connection! [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.008-0400 c20014| 2018-03-16T17:28:20.996-0400 I NETWORK [conn15] received client metadata from 127.0.0.1:44464 conn15: { driver: { name: "MongoDB Internal Client", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.008-0400 c20013| 2018-03-16T17:28:20.996-0400 I NETWORK [conn13] received client metadata from 127.0.0.1:47044 conn13: { driver: { name: "MongoDB Internal Client", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.008-0400 s20016| 2018-03-16T17:28:20.996-0400 D NETWORK [monitoring keys for HMAC] connected connection! [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.009-0400 s20016| 2018-03-16T17:28:20.996-0400 D NETWORK [replSetDistLockPinger] connected connection! [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.009-0400 s20016| 2018-03-16T17:28:20.996-0400 I NETWORK [mongosMain] Successfully connected to ubuntu:20012 (1 connections now open to ubuntu:20012 with a 5 second timeout) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.009-0400 s20016| 2018-03-16T17:28:20.996-0400 I NETWORK [monitoring keys for HMAC] Successfully connected to ubuntu:20014 (1 connections now open to ubuntu:20014 with a 5 second timeout) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.009-0400 s20016| 2018-03-16T17:28:20.996-0400 I NETWORK [replSetDistLockPinger] Successfully connected to ubuntu:20013 (1 connections now open to ubuntu:20013 with a 5 second timeout) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.009-0400 s20016| 2018-03-16T17:28:20.997-0400 D ASIO [mongosMain] startCommand: RemoteCommand 1 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:50.996-0400 cmd:{ find: "version", readConcern: { level: "majority", afterOpTime: { ts: Timestamp(0, 0), t: -1 } }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.010-0400 s20016| 2018-03-16T17:28:20.997-0400 D ASIO [shard registry reload] startCommand: RemoteCommand 2 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:50.997-0400 cmd:{ find: "shards", readConcern: { level: "majority", afterOpTime: { ts: Timestamp(0, 0), t: -1 } }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.010-0400 s20016| 2018-03-16T17:28:20.997-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Connecting to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.010-0400 s20016| 2018-03-16T17:28:20.997-0400 D ASIO [monitoring keys for HMAC] startCommand: RemoteCommand 4 -- target:ubuntu:20012 db:admin expDate:2018-03-16T17:28:50.997-0400 cmd:{ find: "system.keys", filter: { purpose: "HMAC", expiresAt: { $gt: Timestamp(0, 0) } }, sort: { expiresAt: 1 }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp(0, 0), t: -1 } }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.010-0400 s20016| 2018-03-16T17:28:20.997-0400 D NETWORK [ReplicaSetMonitor-TaskExecutor-0] Refreshing replica set stale_mongos_updates_and_removes-configRS took 0 msec [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.011-0400 s20016| 2018-03-16T17:28:20.997-0400 D ASIO [replSetDistLockPinger] startCommand: RemoteCommand 6 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:50.997-0400 cmd:{ findAndModify: "lockpings", query: { _id: "ubuntu:20016:1521235700:-4633782234644071543" }, update: { $set: { ping: new Date(1521235700996) } }, upsert: true, writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.011-0400 s20016| 2018-03-16T17:28:20.997-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Connecting to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.011-0400 s20016| 2018-03-16T17:28:20.997-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 3 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.011-0400 c20012| 2018-03-16T17:28:20.997-0400 I NETWORK [listener] connection accepted from 127.0.0.1:60682 #23 (14 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.011-0400 s20016| 2018-03-16T17:28:20.997-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 5 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.011-0400 c20012| 2018-03-16T17:28:20.997-0400 I NETWORK [listener] connection accepted from 127.0.0.1:60684 #24 (15 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.012-0400 c20012| 2018-03-16T17:28:20.997-0400 I NETWORK [conn23] received client metadata from 127.0.0.1:60682 conn23: { driver: { name: "NetworkInterfaceASIO-ShardRegistry", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.012-0400 c20012| 2018-03-16T17:28:20.997-0400 I NETWORK [conn24] received client metadata from 127.0.0.1:60684 conn24: { driver: { name: "NetworkInterfaceASIO-ShardRegistry", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.012-0400 s20016| 2018-03-16T17:28:20.997-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to ubuntu:20012, took 0ms (2 connections now open to ubuntu:20012) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.012-0400 s20016| 2018-03-16T17:28:20.997-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 3 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.012-0400 s20016| 2018-03-16T17:28:20.997-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Connecting to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.013-0400 s20016| 2018-03-16T17:28:20.997-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 1 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.013-0400 s20016| 2018-03-16T17:28:20.997-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to ubuntu:20012, took 0ms (3 connections now open to ubuntu:20012) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.013-0400 s20016| 2018-03-16T17:28:20.997-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 5 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.013-0400 s20016| 2018-03-16T17:28:20.997-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Connecting to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.013-0400 s20016| 2018-03-16T17:28:20.997-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 4 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.013-0400 c20012| 2018-03-16T17:28:20.998-0400 I NETWORK [listener] connection accepted from 127.0.0.1:60686 #25 (16 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.014-0400 s20016| 2018-03-16T17:28:20.998-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 7 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.014-0400 s20016| 2018-03-16T17:28:20.998-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 8 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.014-0400 c20012| 2018-03-16T17:28:20.998-0400 I NETWORK [listener] connection accepted from 127.0.0.1:60688 #26 (17 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.014-0400 c20012| 2018-03-16T17:28:20.998-0400 I NETWORK [conn25] received client metadata from 127.0.0.1:60686 conn25: { driver: { name: "NetworkInterfaceASIO-ShardRegistry", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.015-0400 s20016| 2018-03-16T17:28:20.998-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 1 finished with response: { cursor: { firstBatch: [ { _id: 1, minCompatibleVersion: 5, currentVersion: 6, clusterId: ObjectId('5aac36f188c1f30185b7ac89') } ], id: 0, ns: "config.version" }, ok: 1.0, operationTime: Timestamp(1521235700, 4), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235700, 4), t: 1 }, lastOpVisible: { ts: Timestamp(1521235700, 4), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235700, 4), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235700, 4) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.015-0400 s20016| 2018-03-16T17:28:20.998-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 2 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.015-0400 c20012| 2018-03-16T17:28:20.998-0400 I NETWORK [conn26] received client metadata from 127.0.0.1:60688 conn26: { driver: { name: "NetworkInterfaceASIO-ShardRegistry", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.015-0400 s20016| 2018-03-16T17:28:20.998-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to ubuntu:20012, took 1ms (4 connections now open to ubuntu:20012) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.016-0400 s20016| 2018-03-16T17:28:20.998-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 7 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.016-0400 s20016| 2018-03-16T17:28:20.998-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 6 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.017-0400 s20016| 2018-03-16T17:28:20.998-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 4 finished with response: { cursor: { firstBatch: [ { _id: 6533657568122765331, purpose: "HMAC", key: BinData(0, 5B47B1F30E0CF2413E50FA7BFA35505CB7C97E21), expiresAt: Timestamp(1529011697, 0) }, { _id: 6533657568122765332, purpose: "HMAC", key: BinData(0, DDCFFA4B6DC56EC9F68A43CE5A9FF843430AD67C), expiresAt: Timestamp(1536787697, 0) } ], id: 0, ns: "admin.system.keys" }, ok: 1.0, operationTime: Timestamp(1521235700, 4), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235700, 4), t: 1 }, lastOpVisible: { ts: Timestamp(1521235700, 4), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235700, 4), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235700, 4) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.017-0400 s20016| 2018-03-16T17:28:20.998-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to ubuntu:20012, took 1ms (4 connections now open to ubuntu:20012) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.017-0400 s20016| 2018-03-16T17:28:20.998-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 8 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.018-0400 s20016| 2018-03-16T17:28:20.998-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 2 finished with response: { cursor: { firstBatch: [], id: 0, ns: "config.shards" }, ok: 1.0, operationTime: Timestamp(1521235700, 4), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235700, 4), t: 1 }, lastOpVisible: { ts: Timestamp(1521235700, 4), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235700, 4), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235700, 4) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.018-0400 s20016| 2018-03-16T17:28:20.998-0400 D SHARDING [shard registry reload] found 0 shards listed on config server(s) with lastVisibleOpTime: { ts: Timestamp(1521235700, 4), t: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.018-0400 s20016| 2018-03-16T17:28:21.003-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 6 finished with response: { lastErrorObject: { n: 1, updatedExisting: false, upserted: "ubuntu:20016:1521235700:-4633782234644071543" }, value: null, ok: 1.0, operationTime: Timestamp(1521235700, 5), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235700, 5), t: 1 }, lastOpVisible: { ts: Timestamp(1521235700, 5), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235700, 5), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235700, 5), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235700, 5) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:21.019-0400 s20016| 2018-03-16T17:28:21.003-0400 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: LockStateChangeFailed: findAndModify query predicate didn't match any lock document [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:22.069-0400 c20012| 2018-03-16T17:28:22.069-0400 D REPL_HB [replexec-2] Sending heartbeat (requestId: 16) to ubuntu:20013, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20012", fromId: 0, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:22.070-0400 c20012| 2018-03-16T17:28:22.069-0400 D REPL_HB [replexec-0] Sending heartbeat (requestId: 17) to ubuntu:20014, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20012", fromId: 0, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:22.070-0400 c20013| 2018-03-16T17:28:22.069-0400 D REPL_HB [conn3] Received heartbeat request from ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20012", fromId: 0, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235700, 5), signature: { hash: BinData(0, 59585C9E3DE748F18A7FE2B7555A1479763E400C), keyId: 6533657568122765331 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:22.071-0400 c20013| 2018-03-16T17:28:22.069-0400 D REPL_HB [conn3] Processed heartbeat from ubuntu:20012 and generated response, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", syncingTo: "ubuntu:20012", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235700, 5), t: 1 }, opTime: { ts: Timestamp(1521235700, 5), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:22.071-0400 c20012| 2018-03-16T17:28:22.070-0400 D REPL_HB [replexec-1] Received response to heartbeat (requestId: 16) from ubuntu:20013, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", syncingTo: "ubuntu:20012", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235700, 5), t: 1 }, opTime: { ts: Timestamp(1521235700, 5), t: 1 }, operationTime: Timestamp(1521235700, 5), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235700, 5), t: 1 }, lastOpVisible: { ts: Timestamp(1521235700, 5), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235700, 5), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235700, 5) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:22.072-0400 c20014| 2018-03-16T17:28:22.070-0400 D REPL_HB [conn4] Received heartbeat request from ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20012", fromId: 0, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235700, 5), signature: { hash: BinData(0, 59585C9E3DE748F18A7FE2B7555A1479763E400C), keyId: 6533657568122765331 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:22.072-0400 c20014| 2018-03-16T17:28:22.070-0400 D REPL_HB [conn4] Processed heartbeat from ubuntu:20012 and generated response, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", syncingTo: "ubuntu:20012", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235700, 5), t: 1 }, opTime: { ts: Timestamp(1521235700, 5), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:22.072-0400 c20012| 2018-03-16T17:28:22.070-0400 D REPL_HB [replexec-1] Scheduling heartbeat to ubuntu:20013 at 2018-03-16T21:28:24.070Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:22.073-0400 c20012| 2018-03-16T17:28:22.070-0400 D REPL_HB [replexec-2] Received response to heartbeat (requestId: 17) from ubuntu:20014, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", syncingTo: "ubuntu:20012", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235700, 5), t: 1 }, opTime: { ts: Timestamp(1521235700, 5), t: 1 }, operationTime: Timestamp(1521235700, 5), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235700, 5), t: 1 }, lastOpVisible: { ts: Timestamp(1521235700, 5), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235700, 5), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235700, 5) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:22.073-0400 c20012| 2018-03-16T17:28:22.070-0400 D REPL_HB [replexec-2] Scheduling heartbeat to ubuntu:20014 at 2018-03-16T21:28:24.070Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:22.537-0400 c20013| 2018-03-16T17:28:22.537-0400 D REPL_HB [replexec-1] Sending heartbeat (requestId: 188) to ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:22.537-0400 c20012| 2018-03-16T17:28:22.537-0400 D REPL_HB [conn9] Received heartbeat request from ubuntu:20013, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235700, 5), signature: { hash: BinData(0, 59585C9E3DE748F18A7FE2B7555A1479763E400C), keyId: 6533657568122765331 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:22.538-0400 c20012| 2018-03-16T17:28:22.537-0400 D REPL_HB [conn9] Processed heartbeat from ubuntu:20013 and generated response, { ok: 1.0, electionTime: new Date(6533657559532830722), state: 1, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235700, 5), t: 1 }, opTime: { ts: Timestamp(1521235700, 5), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:22.539-0400 c20013| 2018-03-16T17:28:22.537-0400 D REPL_HB [replexec-3] Received response to heartbeat (requestId: 188) from ubuntu:20012, { ok: 1.0, electionTime: new Date(6533657559532830722), state: 1, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235700, 5), t: 1 }, opTime: { ts: Timestamp(1521235700, 5), t: 1 }, operationTime: Timestamp(1521235700, 5), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235700, 5), t: 1 }, lastOpVisible: { ts: Timestamp(1521235700, 5), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235700, 5), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235700, 5) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:22.539-0400 c20013| 2018-03-16T17:28:22.537-0400 D REPL_HB [replexec-3] Scheduling heartbeat to ubuntu:20012 at 2018-03-16T21:28:24.537Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:22.539-0400 c20013| 2018-03-16T17:28:22.538-0400 D REPL_HB [replexec-2] Sending heartbeat (requestId: 190) to ubuntu:20014, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:22.539-0400 c20014| 2018-03-16T17:28:22.538-0400 D REPL_HB [conn10] Received heartbeat request from ubuntu:20013, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235700, 5), signature: { hash: BinData(0, 59585C9E3DE748F18A7FE2B7555A1479763E400C), keyId: 6533657568122765331 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:22.540-0400 c20014| 2018-03-16T17:28:22.538-0400 D REPL_HB [conn10] Processed heartbeat from ubuntu:20013 and generated response, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", syncingTo: "ubuntu:20012", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235700, 5), t: 1 }, opTime: { ts: Timestamp(1521235700, 5), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:22.540-0400 c20013| 2018-03-16T17:28:22.538-0400 D REPL_HB [replexec-1] Received response to heartbeat (requestId: 190) from ubuntu:20014, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", syncingTo: "ubuntu:20012", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235700, 5), t: 1 }, opTime: { ts: Timestamp(1521235700, 5), t: 1 }, operationTime: Timestamp(1521235700, 5), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235700, 5), t: 1 }, lastOpVisible: { ts: Timestamp(1521235700, 5), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235700, 5), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235700, 5) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:22.541-0400 c20013| 2018-03-16T17:28:22.538-0400 D REPL_HB [replexec-1] Scheduling heartbeat to ubuntu:20014 at 2018-03-16T21:28:24.538Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:22.581-0400 c20014| 2018-03-16T17:28:22.580-0400 D REPL_HB [replexec-4] Sending heartbeat (requestId: 179) to ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:22.581-0400 c20012| 2018-03-16T17:28:22.580-0400 D REPL_HB [conn8] Received heartbeat request from ubuntu:20014, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235700, 5), signature: { hash: BinData(0, 59585C9E3DE748F18A7FE2B7555A1479763E400C), keyId: 6533657568122765331 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:22.581-0400 c20012| 2018-03-16T17:28:22.580-0400 D REPL_HB [conn8] Processed heartbeat from ubuntu:20014 and generated response, { ok: 1.0, electionTime: new Date(6533657559532830722), state: 1, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235700, 5), t: 1 }, opTime: { ts: Timestamp(1521235700, 5), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:22.582-0400 c20014| 2018-03-16T17:28:22.581-0400 D REPL_HB [replexec-0] Received response to heartbeat (requestId: 179) from ubuntu:20012, { ok: 1.0, electionTime: new Date(6533657559532830722), state: 1, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235700, 5), t: 1 }, opTime: { ts: Timestamp(1521235700, 5), t: 1 }, operationTime: Timestamp(1521235700, 5), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235700, 5), t: 1 }, lastOpVisible: { ts: Timestamp(1521235700, 5), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235700, 5), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235700, 5) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:22.582-0400 c20014| 2018-03-16T17:28:22.581-0400 D REPL_HB [replexec-0] Scheduling heartbeat to ubuntu:20012 at 2018-03-16T21:28:24.581Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:22.583-0400 c20014| 2018-03-16T17:28:22.581-0400 D REPL_HB [replexec-1] Sending heartbeat (requestId: 181) to ubuntu:20013, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:22.583-0400 c20013| 2018-03-16T17:28:22.581-0400 D REPL_HB [conn9] Received heartbeat request from ubuntu:20014, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235700, 5), signature: { hash: BinData(0, 59585C9E3DE748F18A7FE2B7555A1479763E400C), keyId: 6533657568122765331 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:22.583-0400 c20013| 2018-03-16T17:28:22.581-0400 D REPL_HB [conn9] Processed heartbeat from ubuntu:20014 and generated response, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", syncingTo: "ubuntu:20012", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235700, 5), t: 1 }, opTime: { ts: Timestamp(1521235700, 5), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:22.584-0400 c20014| 2018-03-16T17:28:22.582-0400 D REPL_HB [replexec-2] Received response to heartbeat (requestId: 181) from ubuntu:20013, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", syncingTo: "ubuntu:20012", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235700, 5), t: 1 }, opTime: { ts: Timestamp(1521235700, 5), t: 1 }, operationTime: Timestamp(1521235700, 5), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235700, 5), t: 1 }, lastOpVisible: { ts: Timestamp(1521235700, 5), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235700, 5), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235700, 5) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:22.584-0400 c20014| 2018-03-16T17:28:22.582-0400 D REPL_HB [replexec-2] Scheduling heartbeat to ubuntu:20013 at 2018-03-16T21:28:24.582Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:22.998-0400 s20016| 2018-03-16T17:28:22.998-0400 I SHARDING [mongosMain] Not waiting for signing keys, not supported by the config shard stale_mongos_updates_and_removes-configRS [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:22.999-0400 s20016| 2018-03-16T17:28:22.998-0400 D ASIO [mongosMain] startCommand: RemoteCommand 13 -- target:ubuntu:20013 db:config expDate:2018-03-16T17:28:52.998-0400 cmd:{ find: "settings", filter: { _id: "balancer" }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235700, 5), t: 1 } }, limit: 1, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:22.999-0400 s20016| 2018-03-16T17:28:22.998-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Connecting to ubuntu:20013 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:22.999-0400 s20016| 2018-03-16T17:28:22.998-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 14 on host ubuntu:20013 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:22.999-0400 c20013| 2018-03-16T17:28:22.998-0400 I NETWORK [listener] connection accepted from 127.0.0.1:47076 #14 (8 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:22.999-0400 c20013| 2018-03-16T17:28:22.999-0400 I NETWORK [conn14] received client metadata from 127.0.0.1:47076 conn14: { driver: { name: "NetworkInterfaceASIO-ShardRegistry", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.000-0400 s20016| 2018-03-16T17:28:22.999-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to ubuntu:20013, took 1ms (1 connections now open to ubuntu:20013) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.000-0400 s20016| 2018-03-16T17:28:22.999-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 14 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.000-0400 s20016| 2018-03-16T17:28:22.999-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 13 on host ubuntu:20013 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.001-0400 s20016| 2018-03-16T17:28:22.999-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 13 finished with response: { cursor: { id: 0, ns: "config.settings", firstBatch: [] }, ok: 1.0, operationTime: Timestamp(1521235700, 5), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235700, 5), t: 1 }, lastOpVisible: { ts: Timestamp(1521235700, 5), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235700, 5), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235700, 5) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.001-0400 s20016| 2018-03-16T17:28:22.999-0400 D ASIO [mongosMain] startCommand: RemoteCommand 16 -- target:ubuntu:20013 db:config expDate:2018-03-16T17:28:52.999-0400 cmd:{ find: "settings", filter: { _id: "chunksize" }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235700, 5), t: 1 } }, limit: 1, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.001-0400 s20016| 2018-03-16T17:28:22.999-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 16 on host ubuntu:20013 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.002-0400 s20016| 2018-03-16T17:28:22.999-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 16 finished with response: { cursor: { id: 0, ns: "config.settings", firstBatch: [] }, ok: 1.0, operationTime: Timestamp(1521235700, 5), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235700, 5), t: 1 }, lastOpVisible: { ts: Timestamp(1521235700, 5), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235700, 5), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235700, 5) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.002-0400 s20016| 2018-03-16T17:28:23.000-0400 D ASIO [mongosMain] startCommand: RemoteCommand 18 -- target:ubuntu:20014 db:config expDate:2018-03-16T17:28:53.000-0400 cmd:{ find: "settings", filter: { _id: "autosplit" }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235700, 5), t: 1 } }, limit: 1, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.002-0400 s20016| 2018-03-16T17:28:23.000-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Connecting to ubuntu:20014 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.002-0400 s20016| 2018-03-16T17:28:23.000-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 19 on host ubuntu:20014 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.003-0400 c20014| 2018-03-16T17:28:23.000-0400 I NETWORK [listener] connection accepted from 127.0.0.1:44498 #16 (8 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.003-0400 c20014| 2018-03-16T17:28:23.002-0400 I NETWORK [conn16] received client metadata from 127.0.0.1:44498 conn16: { driver: { name: "NetworkInterfaceASIO-ShardRegistry", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.003-0400 s20016| 2018-03-16T17:28:23.002-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to ubuntu:20014, took 2ms (1 connections now open to ubuntu:20014) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.003-0400 s20016| 2018-03-16T17:28:23.002-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 19 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.003-0400 s20016| 2018-03-16T17:28:23.002-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 18 on host ubuntu:20014 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.004-0400 s20016| 2018-03-16T17:28:23.003-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 18 finished with response: { cursor: { id: 0, ns: "config.settings", firstBatch: [] }, ok: 1.0, operationTime: Timestamp(1521235700, 5), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235700, 5), t: 1 }, lastOpVisible: { ts: Timestamp(1521235700, 5), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235700, 5), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235700, 5) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.004-0400 s20016| 2018-03-16T17:28:23.003-0400 W FTDC [mongosMain] FTDC is disabled because neither '--logpath' nor set parameter 'diagnosticDataCollectionDirectoryPath' are specified. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.004-0400 s20016| 2018-03-16T17:28:23.003-0400 I FTDC [mongosMain] Initializing full-time diagnostic data capture with directory '' [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.005-0400 s20016| 2018-03-16T17:28:23.003-0400 D ASIO [mongosMain] startCommand: RemoteCommand 21 -- target:ubuntu:20012 db:admin expDate:2018-03-16T17:28:53.003-0400 cmd:{ _getUserCacheGeneration: 1, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.005-0400 s20016| 2018-03-16T17:28:23.003-0400 D COMMAND [ClusterCursorCleanupJob] BackgroundJob starting: ClusterCursorCleanupJob [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.005-0400 s20016| 2018-03-16T17:28:23.003-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 21 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.005-0400 s20016| 2018-03-16T17:28:23.003-0400 D ASIO [Uptime reporter] startCommand: RemoteCommand 22 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:53.003-0400 cmd:{ update: "mongos", bypassDocumentValidation: false, ordered: true, updates: [ { q: { _id: "ubuntu:20016" }, u: { $set: { _id: "ubuntu:20016", ping: new Date(1521235703003), up: 0, waiting: true, mongoVersion: "3.6.3" } }, multi: false, upsert: true } ], writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.005-0400 s20016| 2018-03-16T17:28:23.003-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 22 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.006-0400 s20016| 2018-03-16T17:28:23.003-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 21 finished with response: { cacheGeneration: ObjectId('5aac36f488c1f30185b7acfe'), ok: 1.0, operationTime: Timestamp(1521235700, 5), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235700, 5), t: 1 }, lastOpVisible: { ts: Timestamp(1521235700, 5), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235700, 5), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235700, 5), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235700, 5) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.006-0400 s20016| 2018-03-16T17:28:23.003-0400 D COMMAND [UserCacheInvalidatorThread] BackgroundJob starting: UserCacheInvalidatorThread [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.006-0400 s20016| 2018-03-16T17:28:23.003-0400 D COMMAND [PeriodicTaskRunner] BackgroundJob starting: PeriodicTaskRunner [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.007-0400 s20016| 2018-03-16T17:28:23.003-0400 I NETWORK [mongosMain] waiting for connections on port 20016 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.009-0400 s20016| 2018-03-16T17:28:23.008-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 22 finished with response: { n: 1, nModified: 0, upserted: [ { index: 0, _id: "ubuntu:20016" } ], opTime: { ts: Timestamp(1521235703, 1), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235703, 1), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235703, 1), t: 1 }, lastOpVisible: { ts: Timestamp(1521235703, 1), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235703, 1), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235703, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 1) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.010-0400 s20016| 2018-03-16T17:28:23.009-0400 D ASIO [Uptime reporter] startCommand: RemoteCommand 25 -- target:ubuntu:20014 db:config expDate:2018-03-16T17:28:53.009-0400 cmd:{ find: "settings", filter: { _id: "balancer" }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235703, 1), t: 1 } }, limit: 1, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.010-0400 s20016| 2018-03-16T17:28:23.009-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 25 on host ubuntu:20014 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.010-0400 s20016| 2018-03-16T17:28:23.009-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 25 finished with response: { cursor: { id: 0, ns: "config.settings", firstBatch: [] }, ok: 1.0, operationTime: Timestamp(1521235703, 1), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235703, 1), t: 1 }, lastOpVisible: { ts: Timestamp(1521235703, 1), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235703, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 1) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.011-0400 s20016| 2018-03-16T17:28:23.009-0400 D ASIO [Uptime reporter] startCommand: RemoteCommand 27 -- target:ubuntu:20013 db:config expDate:2018-03-16T17:28:53.009-0400 cmd:{ find: "settings", filter: { _id: "chunksize" }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235703, 1), t: 1 } }, limit: 1, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.011-0400 s20016| 2018-03-16T17:28:23.009-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 27 on host ubuntu:20013 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.011-0400 s20016| 2018-03-16T17:28:23.009-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 27 finished with response: { cursor: { id: 0, ns: "config.settings", firstBatch: [] }, ok: 1.0, operationTime: Timestamp(1521235703, 1), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235703, 1), t: 1 }, lastOpVisible: { ts: Timestamp(1521235703, 1), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235703, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 1) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.012-0400 s20016| 2018-03-16T17:28:23.009-0400 D ASIO [Uptime reporter] startCommand: RemoteCommand 29 -- target:ubuntu:20014 db:config expDate:2018-03-16T17:28:53.009-0400 cmd:{ find: "settings", filter: { _id: "autosplit" }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235703, 1), t: 1 } }, limit: 1, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.012-0400 s20016| 2018-03-16T17:28:23.009-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 29 on host ubuntu:20014 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.013-0400 s20016| 2018-03-16T17:28:23.010-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 29 finished with response: { cursor: { id: 0, ns: "config.settings", firstBatch: [] }, ok: 1.0, operationTime: Timestamp(1521235703, 1), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235703, 1), t: 1 }, lastOpVisible: { ts: Timestamp(1521235703, 1), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235703, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 1) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.182-0400 s20016| 2018-03-16T17:28:23.182-0400 I NETWORK [listener] connection accepted from 127.0.0.1:50986 #1 (1 connection now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.183-0400 s20016| 2018-03-16T17:28:23.182-0400 I NETWORK [conn1] received client metadata from 127.0.0.1:50986 conn: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.184-0400 s20015| 2018-03-16T17:28:23.184-0400 D ASIO [conn1] startCommand: RemoteCommand 29 -- target:ubuntu:20012 db:admin expDate:2018-03-16T17:29:23.194-0400 cmd:{ _configsvrBalancerStop: 1, maxTimeMS: 60010 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.185-0400 s20015| 2018-03-16T17:28:23.184-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 29 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.185-0400 c20012| 2018-03-16T17:28:23.184-0400 I STORAGE [conn21] createCollection: config.settings with generated UUID: 0b574d76-e668-4186-ad9d-5e002b27f735 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.210-0400 c20014| 2018-03-16T17:28:23.210-0400 I STORAGE [repl writer worker 11] createCollection: config.settings with provided UUID: 0b574d76-e668-4186-ad9d-5e002b27f735 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.210-0400 c20013| 2018-03-16T17:28:23.210-0400 I STORAGE [repl writer worker 12] createCollection: config.settings with provided UUID: 0b574d76-e668-4186-ad9d-5e002b27f735 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.235-0400 s20015| 2018-03-16T17:28:23.234-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 29 finished with response: { ok: 1.0, operationTime: Timestamp(1521235703, 3), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235703, 3), t: 1 }, lastOpVisible: { ts: Timestamp(1521235703, 3), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235703, 3), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235703, 3), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 3) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.237-0400 s20015| 2018-03-16T17:28:23.236-0400 D ASIO [conn1] startCommand: RemoteCommand 31 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:53.236-0400 cmd:{ find: "collections", filter: { _id: /^config\./ }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235703, 3), t: 1 } }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.237-0400 s20015| 2018-03-16T17:28:23.236-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 31 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.237-0400 s20015| 2018-03-16T17:28:23.237-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 31 finished with response: { cursor: { id: 0, ns: "config.collections", firstBatch: [] }, ok: 1.0, operationTime: Timestamp(1521235703, 3), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235703, 3), t: 1 }, lastOpVisible: { ts: Timestamp(1521235703, 3), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235703, 3), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235703, 3), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 3) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.238-0400 s20015| 2018-03-16T17:28:23.237-0400 D ASIO [conn1] startCommand: RemoteCommand 33 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:53.237-0400 cmd:{ update: "settings", bypassDocumentValidation: false, ordered: true, updates: [ { q: { _id: "autosplit" }, u: { $set: { enabled: false } }, multi: false, upsert: true } ], writeConcern: { w: "majority", wtimeout: 30000.0 }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.238-0400 s20015| 2018-03-16T17:28:23.237-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 33 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.242-0400 s20015| 2018-03-16T17:28:23.241-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 33 finished with response: { n: 1, nModified: 0, upserted: [ { index: 0, _id: "autosplit" } ], opTime: { ts: Timestamp(1521235703, 4), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235703, 4), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235703, 4), t: 1 }, lastOpVisible: { ts: Timestamp(1521235703, 4), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235703, 4), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235703, 4), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 4) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.242-0400 ShardingTest stale_mongos_updates_and_removes going to add shard : stale_mongos_updates_and_removes-rs0/ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.243-0400 s20015| 2018-03-16T17:28:23.242-0400 D ASIO [conn1] startCommand: RemoteCommand 35 -- target:ubuntu:20012 db:admin cmd:{ _configsvrAddShard: "stale_mongos_updates_and_removes-rs0/ubuntu:20010", writeConcern: { w: "majority", wtimeout: 60000 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.243-0400 s20015| 2018-03-16T17:28:23.242-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 35 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.243-0400 c20012| 2018-03-16T17:28:23.243-0400 I NETWORK [conn21] Starting new replica set monitor for stale_mongos_updates_and_removes-rs0/ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.243-0400 d20010| 2018-03-16T17:28:23.243-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56372 #5 (3 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.243-0400 d20010| 2018-03-16T17:28:23.243-0400 I NETWORK [conn5] received client metadata from 127.0.0.1:56372 conn5: { driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.244-0400 c20012| 2018-03-16T17:28:23.244-0400 I NETWORK [conn21] Successfully connected to ubuntu:20010 (1 connections now open to ubuntu:20010 with a 5 second timeout) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.244-0400 c20012| 2018-03-16T17:28:23.244-0400 I ASIO [AddShard-TaskExecutor-0] Connecting to ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.244-0400 d20010| 2018-03-16T17:28:23.244-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56374 #6 (4 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.245-0400 d20010| 2018-03-16T17:28:23.244-0400 I NETWORK [conn6] received client metadata from 127.0.0.1:56374 conn6: { driver: { name: "AddShard-TaskExecutor", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.245-0400 c20012| 2018-03-16T17:28:23.245-0400 I ASIO [AddShard-TaskExecutor-0] Successfully connected to ubuntu:20010, took 1ms (1 connections now open to ubuntu:20010) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.245-0400 d20010| 2018-03-16T17:28:23.245-0400 I COMMAND [conn6] CMD: drop config.system.sessions [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.246-0400 d20010| 2018-03-16T17:28:23.246-0400 I SHARDING [conn6] initializing sharding state with: { _id: "shardIdentity", configsvrConnectionString: "stale_mongos_updates_and_removes-configRS/ubuntu:20012,ubuntu:20013,ubuntu:20014", shardName: "stale_mongos_updates_and_removes-rs0", clusterId: ObjectId('5aac36f188c1f30185b7ac89') } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.246-0400 d20010| 2018-03-16T17:28:23.246-0400 I NETWORK [conn6] Starting new replica set monitor for stale_mongos_updates_and_removes-configRS/ubuntu:20012,ubuntu:20013,ubuntu:20014 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.246-0400 d20010| 2018-03-16T17:28:23.246-0400 I SHARDING [conn6] initialized sharding components for primary node. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.247-0400 d20010| 2018-03-16T17:28:23.246-0400 I SHARDING [thread6] creating distributed lock ping thread for process ubuntu:20010:1521235703:-3756998713987756063 (sleeping for 30000ms) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.247-0400 c20013| 2018-03-16T17:28:23.246-0400 I NETWORK [listener] connection accepted from 127.0.0.1:47086 #15 (9 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.247-0400 c20012| 2018-03-16T17:28:23.246-0400 I NETWORK [listener] connection accepted from 127.0.0.1:60722 #28 (18 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.247-0400 c20013| 2018-03-16T17:28:23.246-0400 I NETWORK [conn15] received client metadata from 127.0.0.1:47086 conn15: { driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.247-0400 c20014| 2018-03-16T17:28:23.246-0400 I NETWORK [listener] connection accepted from 127.0.0.1:44510 #17 (9 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.248-0400 c20012| 2018-03-16T17:28:23.247-0400 I NETWORK [conn28] received client metadata from 127.0.0.1:60722 conn28: { driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.248-0400 c20014| 2018-03-16T17:28:23.247-0400 I NETWORK [conn17] received client metadata from 127.0.0.1:44510 conn17: { driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.248-0400 d20010| 2018-03-16T17:28:23.247-0400 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to ubuntu:20013 (1 connections now open to ubuntu:20013 with a 5 second timeout) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.248-0400 d20010| 2018-03-16T17:28:23.247-0400 I NETWORK [shard registry reload] Successfully connected to ubuntu:20012 (1 connections now open to ubuntu:20012 with a 5 second timeout) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.248-0400 d20010| 2018-03-16T17:28:23.247-0400 I NETWORK [monitoring keys for HMAC] Successfully connected to ubuntu:20014 (1 connections now open to ubuntu:20014 with a 5 second timeout) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.249-0400 c20012| 2018-03-16T17:28:23.248-0400 I SHARDING [conn21] going to insert new entry for shard into config.shards: { _id: "stale_mongos_updates_and_removes-rs0", host: "stale_mongos_updates_and_removes-rs0/ubuntu:20010", state: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.252-0400 c20012| 2018-03-16T17:28:23.252-0400 I STORAGE [conn21] createCollection: config.changelog with generated UUID: ff7c9cc1-9f5d-4353-b868-d6d08204cb84 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.272-0400 c20014| 2018-03-16T17:28:23.272-0400 I STORAGE [repl writer worker 3] createCollection: config.changelog with provided UUID: ff7c9cc1-9f5d-4353-b868-d6d08204cb84 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.272-0400 c20013| 2018-03-16T17:28:23.272-0400 I STORAGE [repl writer worker 4] createCollection: config.changelog with provided UUID: ff7c9cc1-9f5d-4353-b868-d6d08204cb84 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.278-0400 d20010| 2018-03-16T17:28:23.278-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Connecting to ubuntu:20013 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.279-0400 d20010| 2018-03-16T17:28:23.278-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Connecting to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.279-0400 d20010| 2018-03-16T17:28:23.278-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Connecting to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.279-0400 c20013| 2018-03-16T17:28:23.279-0400 I NETWORK [listener] connection accepted from 127.0.0.1:47092 #16 (10 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.279-0400 c20012| 2018-03-16T17:28:23.279-0400 I NETWORK [listener] connection accepted from 127.0.0.1:60728 #29 (19 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.279-0400 c20012| 2018-03-16T17:28:23.279-0400 I NETWORK [listener] connection accepted from 127.0.0.1:60730 #30 (20 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.280-0400 c20013| 2018-03-16T17:28:23.279-0400 I NETWORK [conn16] received client metadata from 127.0.0.1:47092 conn16: { driver: { name: "NetworkInterfaceASIO-ShardRegistry", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.280-0400 c20012| 2018-03-16T17:28:23.279-0400 I NETWORK [conn29] received client metadata from 127.0.0.1:60728 conn29: { driver: { name: "NetworkInterfaceASIO-ShardRegistry", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.280-0400 c20012| 2018-03-16T17:28:23.279-0400 I NETWORK [conn30] received client metadata from 127.0.0.1:60730 conn30: { driver: { name: "NetworkInterfaceASIO-ShardRegistry", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.280-0400 d20010| 2018-03-16T17:28:23.279-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to ubuntu:20013, took 1ms (1 connections now open to ubuntu:20013) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.281-0400 d20010| 2018-03-16T17:28:23.279-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to ubuntu:20012, took 1ms (2 connections now open to ubuntu:20012) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.281-0400 d20010| 2018-03-16T17:28:23.279-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to ubuntu:20012, took 1ms (2 connections now open to ubuntu:20012) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.298-0400 c20012| 2018-03-16T17:28:23.298-0400 I SHARDING [conn21] about to log metadata event into changelog: { _id: "ubuntu-2018-03-16T17:28:23.298-0400-5aac36f788c1f30185b7ad92", server: "ubuntu", clientAddr: "127.0.0.1:60670", time: new Date(1521235703298), what: "addShard", ns: "", details: { name: "stale_mongos_updates_and_removes-rs0", host: "stale_mongos_updates_and_removes-rs0/ubuntu:20010" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.299-0400 d20010| 2018-03-16T17:28:23.298-0400 I NETWORK [shard registry reload] Starting new replica set monitor for stale_mongos_updates_and_removes-rs0/ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.299-0400 d20010| 2018-03-16T17:28:23.299-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56388 #11 (5 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.299-0400 d20010| 2018-03-16T17:28:23.299-0400 I NETWORK [conn11] received client metadata from 127.0.0.1:56388 conn11: { driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.299-0400 d20010| 2018-03-16T17:28:23.299-0400 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: LockStateChangeFailed: findAndModify query predicate didn't match any lock document [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.300-0400 d20010| 2018-03-16T17:28:23.299-0400 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to ubuntu:20010 (1 connections now open to ubuntu:20010 with a 5 second timeout) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.303-0400 s20015| 2018-03-16T17:28:23.303-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 35 finished with response: { shardAdded: "stale_mongos_updates_and_removes-rs0", ok: 1.0, operationTime: Timestamp(1521235703, 9), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235703, 9), t: 1 }, lastOpVisible: { ts: Timestamp(1521235703, 9), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235703, 9), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235703, 9), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 9) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.304-0400 s20015| 2018-03-16T17:28:23.303-0400 D ASIO [conn1] startCommand: RemoteCommand 37 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:53.303-0400 cmd:{ find: "shards", readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235703, 9), t: 1 } }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.304-0400 s20015| 2018-03-16T17:28:23.303-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 37 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.305-0400 s20015| 2018-03-16T17:28:23.303-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 37 finished with response: { cursor: { firstBatch: [ { _id: "stale_mongos_updates_and_removes-rs0", host: "stale_mongos_updates_and_removes-rs0/ubuntu:20010", state: 1 } ], id: 0, ns: "config.shards" }, ok: 1.0, operationTime: Timestamp(1521235703, 9), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235703, 9), t: 1 }, lastOpVisible: { ts: Timestamp(1521235703, 9), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235703, 9), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235703, 9), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 9) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.305-0400 s20015| 2018-03-16T17:28:23.303-0400 D SHARDING [conn1] found 1 shards listed on config server(s) with lastVisibleOpTime: { ts: Timestamp(1521235703, 9), t: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.305-0400 s20015| 2018-03-16T17:28:23.303-0400 I NETWORK [conn1] Starting new replica set monitor for stale_mongos_updates_and_removes-rs0/ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.305-0400 s20015| 2018-03-16T17:28:23.303-0400 D NETWORK [conn1] Started targeter for stale_mongos_updates_and_removes-rs0/ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.305-0400 s20015| 2018-03-16T17:28:23.303-0400 D NETWORK [ReplicaSetMonitor-TaskExecutor-0] Starting new refresh of replica set stale_mongos_updates_and_removes-rs0 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.306-0400 s20015| 2018-03-16T17:28:23.303-0400 D NETWORK [ReplicaSetMonitor-TaskExecutor-0] creating new connection to:ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.306-0400 ShardingTest stale_mongos_updates_and_removes going to add shard : stale_mongos_updates_and_removes-rs1/ubuntu:20011 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.306-0400 s20015| 2018-03-16T17:28:23.303-0400 D NETWORK [ReplicaSetMonitor-TaskExecutor-0] connected to server ubuntu:20010 (127.0.1.1) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.306-0400 d20010| 2018-03-16T17:28:23.303-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56390 #12 (6 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.306-0400 d20010| 2018-03-16T17:28:23.304-0400 I NETWORK [conn12] received client metadata from 127.0.0.1:56390 conn12: { driver: { name: "MongoDB Internal Client", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.307-0400 s20015| 2018-03-16T17:28:23.304-0400 D ASIO [conn1] startCommand: RemoteCommand 39 -- target:ubuntu:20012 db:admin cmd:{ _configsvrAddShard: "stale_mongos_updates_and_removes-rs1/ubuntu:20011", writeConcern: { w: "majority", wtimeout: 60000 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.307-0400 s20015| 2018-03-16T17:28:23.304-0400 D NETWORK [ReplicaSetMonitor-TaskExecutor-0] connected connection! [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.307-0400 s20015| 2018-03-16T17:28:23.304-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 39 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.307-0400 s20015| 2018-03-16T17:28:23.304-0400 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to ubuntu:20010 (1 connections now open to ubuntu:20010 with a 5 second timeout) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.307-0400 s20015| 2018-03-16T17:28:23.304-0400 D NETWORK [ReplicaSetMonitor-TaskExecutor-0] Refreshing replica set stale_mongos_updates_and_removes-rs0 took 0 msec [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.307-0400 c20012| 2018-03-16T17:28:23.304-0400 I NETWORK [conn21] Starting new replica set monitor for stale_mongos_updates_and_removes-rs1/ubuntu:20011 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.308-0400 d20011| 2018-03-16T17:28:23.304-0400 I NETWORK [listener] connection accepted from 127.0.0.1:55036 #5 (3 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.308-0400 d20011| 2018-03-16T17:28:23.305-0400 I NETWORK [conn5] received client metadata from 127.0.0.1:55036 conn5: { driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.308-0400 c20012| 2018-03-16T17:28:23.305-0400 I NETWORK [conn21] Successfully connected to ubuntu:20011 (1 connections now open to ubuntu:20011 with a 5 second timeout) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.308-0400 c20012| 2018-03-16T17:28:23.305-0400 I ASIO [AddShard-TaskExecutor-0] Connecting to ubuntu:20011 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.309-0400 d20011| 2018-03-16T17:28:23.305-0400 I NETWORK [listener] connection accepted from 127.0.0.1:55038 #6 (4 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.309-0400 d20011| 2018-03-16T17:28:23.306-0400 I NETWORK [conn6] received client metadata from 127.0.0.1:55038 conn6: { driver: { name: "AddShard-TaskExecutor", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.309-0400 c20012| 2018-03-16T17:28:23.306-0400 I ASIO [AddShard-TaskExecutor-0] Successfully connected to ubuntu:20011, took 1ms (1 connections now open to ubuntu:20011) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.309-0400 d20011| 2018-03-16T17:28:23.306-0400 I COMMAND [conn6] CMD: drop config.system.sessions [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.309-0400 d20011| 2018-03-16T17:28:23.307-0400 I SHARDING [conn6] initializing sharding state with: { _id: "shardIdentity", configsvrConnectionString: "stale_mongos_updates_and_removes-configRS/ubuntu:20012,ubuntu:20013,ubuntu:20014", shardName: "stale_mongos_updates_and_removes-rs1", clusterId: ObjectId('5aac36f188c1f30185b7ac89') } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.310-0400 d20011| 2018-03-16T17:28:23.307-0400 I NETWORK [conn6] Starting new replica set monitor for stale_mongos_updates_and_removes-configRS/ubuntu:20012,ubuntu:20013,ubuntu:20014 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.310-0400 d20011| 2018-03-16T17:28:23.307-0400 I SHARDING [conn6] initialized sharding components for primary node. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.310-0400 d20011| 2018-03-16T17:28:23.307-0400 I SHARDING [thread6] creating distributed lock ping thread for process ubuntu:20011:1521235703:1064903155863675804 (sleeping for 30000ms) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.310-0400 c20013| 2018-03-16T17:28:23.307-0400 I NETWORK [listener] connection accepted from 127.0.0.1:47106 #17 (11 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.310-0400 c20012| 2018-03-16T17:28:23.307-0400 I NETWORK [listener] connection accepted from 127.0.0.1:60742 #32 (21 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.311-0400 c20013| 2018-03-16T17:28:23.308-0400 I NETWORK [conn17] received client metadata from 127.0.0.1:47106 conn17: { driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.311-0400 c20012| 2018-03-16T17:28:23.308-0400 I NETWORK [conn32] received client metadata from 127.0.0.1:60742 conn32: { driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.311-0400 d20011| 2018-03-16T17:28:23.308-0400 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to ubuntu:20013 (1 connections now open to ubuntu:20013 with a 5 second timeout) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.311-0400 d20011| 2018-03-16T17:28:23.308-0400 I NETWORK [shard registry reload] Successfully connected to ubuntu:20012 (1 connections now open to ubuntu:20012 with a 5 second timeout) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.312-0400 c20012| 2018-03-16T17:28:23.309-0400 I SHARDING [conn21] going to insert new entry for shard into config.shards: { _id: "stale_mongos_updates_and_removes-rs1", host: "stale_mongos_updates_and_removes-rs1/ubuntu:20011", state: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.314-0400 c20012| 2018-03-16T17:28:23.314-0400 I SHARDING [conn21] about to log metadata event into changelog: { _id: "ubuntu-2018-03-16T17:28:23.314-0400-5aac36f788c1f30185b7adb4", server: "ubuntu", clientAddr: "127.0.0.1:60670", time: new Date(1521235703314), what: "addShard", ns: "", details: { name: "stale_mongos_updates_and_removes-rs1", host: "stale_mongos_updates_and_removes-rs1/ubuntu:20011" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.320-0400 s20015| 2018-03-16T17:28:23.319-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 39 finished with response: { shardAdded: "stale_mongos_updates_and_removes-rs1", ok: 1.0, operationTime: Timestamp(1521235703, 12), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235703, 12), t: 1 }, lastOpVisible: { ts: Timestamp(1521235703, 12), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235703, 12), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235703, 12), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 12) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.320-0400 s20015| 2018-03-16T17:28:23.319-0400 D ASIO [conn1] startCommand: RemoteCommand 41 -- target:ubuntu:20013 db:config expDate:2018-03-16T17:28:53.319-0400 cmd:{ find: "shards", readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235703, 12), t: 1 } }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.320-0400 s20015| 2018-03-16T17:28:23.319-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 41 on host ubuntu:20013 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.321-0400 s20015| 2018-03-16T17:28:23.319-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 41 finished with response: { cursor: { firstBatch: [ { _id: "stale_mongos_updates_and_removes-rs0", host: "stale_mongos_updates_and_removes-rs0/ubuntu:20010", state: 1 }, { _id: "stale_mongos_updates_and_removes-rs1", host: "stale_mongos_updates_and_removes-rs1/ubuntu:20011", state: 1 } ], id: 0, ns: "config.shards" }, ok: 1.0, operationTime: Timestamp(1521235703, 12), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235703, 12), t: 1 }, lastOpVisible: { ts: Timestamp(1521235703, 12), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235703, 12), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 12) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.321-0400 s20015| 2018-03-16T17:28:23.319-0400 D SHARDING [conn1] found 2 shards listed on config server(s) with lastVisibleOpTime: { ts: Timestamp(1521235703, 12), t: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.321-0400 s20015| 2018-03-16T17:28:23.319-0400 D NETWORK [conn1] Started targeter for stale_mongos_updates_and_removes-rs0/ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.321-0400 s20015| 2018-03-16T17:28:23.319-0400 I NETWORK [conn1] Starting new replica set monitor for stale_mongos_updates_and_removes-rs1/ubuntu:20011 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.322-0400 s20015| 2018-03-16T17:28:23.320-0400 D NETWORK [conn1] Started targeter for stale_mongos_updates_and_removes-rs1/ubuntu:20011 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.322-0400 s20015| 2018-03-16T17:28:23.320-0400 D NETWORK [ReplicaSetMonitor-TaskExecutor-0] Starting new refresh of replica set stale_mongos_updates_and_removes-rs1 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.322-0400 s20015| 2018-03-16T17:28:23.320-0400 D NETWORK [ReplicaSetMonitor-TaskExecutor-0] creating new connection to:ubuntu:20011 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.322-0400 s20015| 2018-03-16T17:28:23.320-0400 D NETWORK [ReplicaSetMonitor-TaskExecutor-0] connected to server ubuntu:20011 (127.0.1.1) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.322-0400 Waiting for op with OpTime { "ts" : Timestamp(1521235703, 12), "t" : NumberLong(1) } to be committed on all secondaries [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.332-0400 c20012| 2018-03-16T17:28:23.332-0400 I SHARDING [conn1] distributed lock 'config' acquired for 'shardCollection', ts : 5aac36f788c1f30185b7adc9 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.334-0400 d20011| 2018-03-16T17:28:23.334-0400 I NETWORK [listener] connection accepted from 127.0.0.1:55044 #9 (5 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.335-0400 d20011| 2018-03-16T17:28:23.334-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Connecting to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.335-0400 c20014| 2018-03-16T17:28:23.334-0400 I NETWORK [listener] connection accepted from 127.0.0.1:44532 #18 (10 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.335-0400 d20011| 2018-03-16T17:28:23.334-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Connecting to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.335-0400 d20011| 2018-03-16T17:28:23.334-0400 I NETWORK [conn9] received client metadata from 127.0.0.1:55044 conn9: { driver: { name: "MongoDB Internal Client", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.335-0400 c20014| 2018-03-16T17:28:23.334-0400 I NETWORK [conn18] received client metadata from 127.0.0.1:44532 conn18: { driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.335-0400 s20015| 2018-03-16T17:28:23.334-0400 D NETWORK [ReplicaSetMonitor-TaskExecutor-0] connected connection! [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.336-0400 c20012| 2018-03-16T17:28:23.335-0400 I NETWORK [listener] connection accepted from 127.0.0.1:60748 #33 (22 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.336-0400 c20012| 2018-03-16T17:28:23.335-0400 I NETWORK [listener] connection accepted from 127.0.0.1:60750 #34 (23 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.336-0400 s20015| 2018-03-16T17:28:23.335-0400 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to ubuntu:20011 (1 connections now open to ubuntu:20011 with a 5 second timeout) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.336-0400 d20011| 2018-03-16T17:28:23.335-0400 I NETWORK [monitoring keys for HMAC] Successfully connected to ubuntu:20014 (1 connections now open to ubuntu:20014 with a 5 second timeout) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.336-0400 c20012| 2018-03-16T17:28:23.335-0400 I NETWORK [conn33] received client metadata from 127.0.0.1:60748 conn33: { driver: { name: "NetworkInterfaceASIO-ShardRegistry", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.337-0400 c20012| 2018-03-16T17:28:23.335-0400 I NETWORK [conn34] received client metadata from 127.0.0.1:60750 conn34: { driver: { name: "NetworkInterfaceASIO-ShardRegistry", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.337-0400 s20015| 2018-03-16T17:28:23.335-0400 D NETWORK [ReplicaSetMonitor-TaskExecutor-0] Refreshing replica set stale_mongos_updates_and_removes-rs1 took 15 msec [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.337-0400 d20011| 2018-03-16T17:28:23.335-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Connecting to ubuntu:20013 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.337-0400 d20011| 2018-03-16T17:28:23.335-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to ubuntu:20012, took 1ms (2 connections now open to ubuntu:20012) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.338-0400 d20011| 2018-03-16T17:28:23.335-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to ubuntu:20012, took 1ms (2 connections now open to ubuntu:20012) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.338-0400 c20013| 2018-03-16T17:28:23.335-0400 I NETWORK [listener] connection accepted from 127.0.0.1:47118 #18 (12 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.338-0400 d20011| 2018-03-16T17:28:23.335-0400 I NETWORK [shard registry reload] Starting new replica set monitor for stale_mongos_updates_and_removes-rs0/ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.338-0400 c20013| 2018-03-16T17:28:23.335-0400 I NETWORK [conn18] received client metadata from 127.0.0.1:47118 conn18: { driver: { name: "NetworkInterfaceASIO-ShardRegistry", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.338-0400 d20011| 2018-03-16T17:28:23.335-0400 I NETWORK [shard registry reload] Starting new replica set monitor for stale_mongos_updates_and_removes-rs1/ubuntu:20011 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.339-0400 d20011| 2018-03-16T17:28:23.335-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to ubuntu:20013, took 0ms (1 connections now open to ubuntu:20013) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.339-0400 d20010| 2018-03-16T17:28:23.336-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56410 #13 (7 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.339-0400 d20010| 2018-03-16T17:28:23.336-0400 I NETWORK [conn13] received client metadata from 127.0.0.1:56410 conn13: { driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.339-0400 d20011| 2018-03-16T17:28:23.336-0400 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to ubuntu:20010 (1 connections now open to ubuntu:20010 with a 5 second timeout) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.339-0400 d20011| 2018-03-16T17:28:23.336-0400 I NETWORK [listener] connection accepted from 127.0.0.1:55056 #13 (6 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.340-0400 d20011| 2018-03-16T17:28:23.336-0400 I NETWORK [conn13] received client metadata from 127.0.0.1:55056 conn13: { driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.340-0400 d20011| 2018-03-16T17:28:23.337-0400 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to ubuntu:20011 (1 connections now open to ubuntu:20011 with a 5 second timeout) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.340-0400 c20012| 2018-03-16T17:28:23.338-0400 I SHARDING [conn1] distributed lock 'config.system.sessions' acquired for 'shardCollection', ts : 5aac36f788c1f30185b7add0 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.340-0400 c20012| 2018-03-16T17:28:23.338-0400 I NETWORK [conn1] Starting new replica set monitor for stale_mongos_updates_and_removes-configRS/ubuntu:20012,ubuntu:20013,ubuntu:20014 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.340-0400 c20013| 2018-03-16T17:28:23.338-0400 I NETWORK [listener] connection accepted from 127.0.0.1:47124 #19 (13 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.340-0400 c20012| 2018-03-16T17:28:23.338-0400 I NETWORK [listener] connection accepted from 127.0.0.1:60760 #37 (24 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.341-0400 c20013| 2018-03-16T17:28:23.338-0400 I NETWORK [conn19] received client metadata from 127.0.0.1:47124 conn19: { driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.341-0400 c20012| 2018-03-16T17:28:23.338-0400 I NETWORK [conn37] received client metadata from 127.0.0.1:60760 conn37: { driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.341-0400 c20012| 2018-03-16T17:28:23.338-0400 I NETWORK [conn1] Successfully connected to ubuntu:20013 (1 connections now open to ubuntu:20013 with a 5 second timeout) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.341-0400 c20012| 2018-03-16T17:28:23.338-0400 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to ubuntu:20012 (1 connections now open to ubuntu:20012 with a 5 second timeout) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.342-0400 c20014| 2018-03-16T17:28:23.339-0400 I NETWORK [listener] connection accepted from 127.0.0.1:44548 #19 (11 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.342-0400 c20014| 2018-03-16T17:28:23.339-0400 I NETWORK [conn19] received client metadata from 127.0.0.1:44548 conn19: { driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.342-0400 c20012| 2018-03-16T17:28:23.339-0400 I NETWORK [conn1] Successfully connected to ubuntu:20014 (1 connections now open to ubuntu:20014 with a 5 second timeout) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.342-0400 c20012| 2018-03-16T17:28:23.339-0400 I NETWORK [conn1] Successfully connected to stale_mongos_updates_and_removes-configRS/ubuntu:20012,ubuntu:20013,ubuntu:20014 (1 connections now open to stale_mongos_updates_and_removes-configRS/ubuntu:20012,ubuntu:20013,ubuntu:20014 with a 0 second timeout) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.342-0400 c20012| 2018-03-16T17:28:23.339-0400 I NETWORK [listener] connection accepted from 127.0.0.1:60764 #40 (25 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.343-0400 c20012| 2018-03-16T17:28:23.339-0400 I NETWORK [conn40] received client metadata from 127.0.0.1:60764 conn40: { driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.343-0400 d20011| 2018-03-16T17:28:23.340-0400 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: LockStateChangeFailed: findAndModify query predicate didn't match any lock document [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.343-0400 c20012| 2018-03-16T17:28:23.340-0400 I NETWORK [conn1] Successfully connected to stale_mongos_updates_and_removes-rs0/ubuntu:20010 (1 connections now open to stale_mongos_updates_and_removes-rs0/ubuntu:20010 with a 0 second timeout) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.343-0400 d20010| 2018-03-16T17:28:23.340-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56422 #14 (8 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.344-0400 d20010| 2018-03-16T17:28:23.341-0400 I NETWORK [conn14] received client metadata from 127.0.0.1:56422 conn14: { driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.344-0400 c20012| 2018-03-16T17:28:23.342-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Connecting to ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.344-0400 d20010| 2018-03-16T17:28:23.342-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56424 #15 (9 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.344-0400 d20010| 2018-03-16T17:28:23.342-0400 I NETWORK [conn15] received client metadata from 127.0.0.1:56424 conn15: { driver: { name: "NetworkInterfaceASIO-ShardRegistry", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.344-0400 c20012| 2018-03-16T17:28:23.342-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to ubuntu:20010, took 0ms (1 connections now open to ubuntu:20010) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.345-0400 d20010| 2018-03-16T17:28:23.342-0400 I STORAGE [conn15] createCollection: config.system.sessions with generated UUID: b6d2c6f9-227f-4aee-b5fd-4ae6721fa95c [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.366-0400 c20012| 2018-03-16T17:28:23.366-0400 I SHARDING [conn1] CMD: shardcollection: { _configsvrShardCollection: "config.system.sessions", key: { _id: 1 }, unique: false, numInitialChunks: 0, writeConcern: { w: "majority", wtimeout: 60000 }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.395-0400 c20012| 2018-03-16T17:28:23.394-0400 I SHARDING [conn1] about to log metadata event into changelog: { _id: "ubuntu-2018-03-16T17:28:23.394-0400-5aac36f788c1f30185b7adee", server: "ubuntu", clientAddr: "127.0.0.1:41148", time: new Date(1521235703394), what: "shardCollection.start", ns: "config.system.sessions", details: { shardKey: { _id: 1 }, collection: "config.system.sessions", uuid: UUID("b6d2c6f9-227f-4aee-b5fd-4ae6721fa95c"), primary: "stale_mongos_updates_and_removes-rs0:stale_mongos_updates_and_removes-rs0/ubuntu:20010", numChunks: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.399-0400 c20012| 2018-03-16T17:28:23.399-0400 I SHARDING [conn1] ShouldAutoSplit changing from 1 to 0 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.399-0400 c20012| 2018-03-16T17:28:23.399-0400 I SHARDING [conn1] going to create 1 chunk(s) for: config.system.sessions using new epoch 5aac36f788c1f30185b7adfc [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.404-0400 c20012| 2018-03-16T17:28:23.404-0400 I STORAGE [conn1] createCollection: config.collections with generated UUID: 02b59613-8143-4ac5-bb51-05c495a9c41b [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.424-0400 c20013| 2018-03-16T17:28:23.424-0400 I STORAGE [repl writer worker 10] createCollection: config.collections with provided UUID: 02b59613-8143-4ac5-bb51-05c495a9c41b [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.425-0400 c20014| 2018-03-16T17:28:23.424-0400 I STORAGE [repl writer worker 7] createCollection: config.collections with provided UUID: 02b59613-8143-4ac5-bb51-05c495a9c41b [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.453-0400 d20010| 2018-03-16T17:28:23.453-0400 I SHARDING [conn15] XXXX in set shard version cmd obj is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.454-0400 d20010| 2018-03-16T17:28:23.453-0400 I SHARDING [conn15] { setShardVersion: "config.system.sessions", init: false, authoritative: true, configdb: "stale_mongos_updates_and_removes-configRS/ubuntu:20012,ubuntu:20013,ubuntu:20014", shard: "stale_mongos_updates_and_removes-rs0", shardHost: "stale_mongos_updates_and_removes-rs0/ubuntu:20010", version: Timestamp(1, 0), versionEpoch: ObjectId('5aac36f788c1f30185b7adfc'), noConnectionVersioning: true, $clusterTime: { clusterTime: Timestamp(1521235703, 20), signature: { hash: BinData(0, F5BAD1415D4FF72DCE994EFB8E61FE39EE018428), keyId: 6533657568122765331 } }, $client: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } }, $configServerState: { opTime: { ts: Timestamp(1521235703, 20), t: 1 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.454-0400 d20010| 2018-03-16T17:28:23.453-0400 I SHARDING [conn15] XXXX requested version is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.454-0400 d20010| 2018-03-16T17:28:23.453-0400 I SHARDING [conn15] 1|0||5aac36f788c1f30185b7adfc [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.454-0400 d20010| 2018-03-16T17:28:23.453-0400 I SHARDING [conn15] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.454-0400 d20010| 2018-03-16T17:28:23.453-0400 I SHARDING [conn15] config.system.sessions [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.454-0400 d20010| 2018-03-16T17:28:23.453-0400 I SHARDING [conn15] xxx routing info needs refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.455-0400 d20010| 2018-03-16T17:28:23.453-0400 I SHARDING [conn15] xxx calling schedule collection refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.455-0400 d20010| 2018-03-16T17:28:23.453-0400 I SHARDING [conn15] xxx is not incr [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.455-0400 d20010| 2018-03-16T17:28:23.453-0400 I SHARDING [conn15] xxx existing routing info? [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.455-0400 d20010| 2018-03-16T17:28:23.453-0400 I SHARDING [conn15] Refreshing chunks for collection config.system.sessions based on version 0|0||000000000000000000000000 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.455-0400 d20010| 2018-03-16T17:28:23.453-0400 I SHARDING [conn15] xxx getting chunks since now [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.455-0400 d20010| 2018-03-16T17:28:23.453-0400 I SHARDING [conn15] xxx getting chunks since [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.455-0400 d20010| 2018-03-16T17:28:23.453-0400 I SHARDING [conn15] xxx waiting if !refreshNot [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.455-0400 d20010| 2018-03-16T17:28:23.454-0400 I SHARDING [ShardServerCatalogCacheLoader-0] xxx primary and calling get chunks since for primary [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.456-0400 d20010| 2018-03-16T17:28:23.454-0400 I SHARDING [ShardServerCatalogCacheLoader-0] version is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.456-0400 d20010| 2018-03-16T17:28:23.454-0400 I SHARDING [ShardServerCatalogCacheLoader-0] 0|0||000000000000000000000000 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.456-0400 d20010| 2018-03-16T17:28:23.455-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] refresh in primary [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.456-0400 d20010| 2018-03-16T17:28:23.455-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] 1 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.456-0400 d20010| 2018-03-16T17:28:23.455-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx collec and changed chunks max epoch or version doesn't equal max [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.464-0400 d20010| 2018-03-16T17:28:23.464-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Cache loader remotely refreshed for collection config.system.sessions from collection version 0|0||000000000000000000000000 and found collection version 1|0||5aac36f788c1f30185b7adfc [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.464-0400 d20010| 2018-03-16T17:28:23.464-0400 I STORAGE [ShardServerCatalogCacheLoader-0] createCollection: config.cache.collections with generated UUID: cbc4b445-eb66-4b27-bbc7-a5316ee55944 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.465-0400 d20010| 2018-03-16T17:28:23.464-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Cache loader found enqueued metadata from 1|0||5aac36f788c1f30185b7adfc to 1|0||5aac36f788c1f30185b7adfc and no persisted metadata, GTE cache version 0|0||000000000000000000000000 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.465-0400 d20010| 2018-03-16T17:28:23.464-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx is okay [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.465-0400 d20010| 2018-03-16T17:28:23.464-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx finsihing refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.465-0400 d20010| 2018-03-16T17:28:23.464-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh callback [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.465-0400 d20010| 2018-03-16T17:28:23.464-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh completed [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.465-0400 d20010| 2018-03-16T17:28:23.464-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Refresh for collection config.system.sessions took 10 ms and found version 1|0||5aac36f788c1f30185b7adfc [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.466-0400 d20010| 2018-03-16T17:28:23.464-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx marking as status ok [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.466-0400 d20010| 2018-03-16T17:28:23.464-0400 I SHARDING [conn15] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.466-0400 d20010| 2018-03-16T17:28:23.464-0400 I SHARDING [conn15] config.system.sessions [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.479-0400 d20010| 2018-03-16T17:28:23.479-0400 I SHARDING [conn15] Marking collection config.system.sessions as sharded with collection version: 1|0||5aac36f788c1f30185b7adfc, shard version: 1|0||5aac36f788c1f30185b7adfc [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.479-0400 c20012| 2018-03-16T17:28:23.479-0400 I SHARDING [conn1] about to log metadata event into changelog: { _id: "ubuntu-2018-03-16T17:28:23.479-0400-5aac36f788c1f30185b7ae17", server: "ubuntu", clientAddr: "127.0.0.1:41148", time: new Date(1521235703479), what: "shardCollection.end", ns: "config.system.sessions", details: { version: "1|0||5aac36f788c1f30185b7adfc" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.479-0400 d20010| 2018-03-16T17:28:23.479-0400 I STORAGE [ShardServerCatalogCacheLoader-0] createCollection: config.cache.chunks.config.system.sessions with generated UUID: ec42243a-38d2-48bc-8ac0-d53d9f7c367a [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.493-0400 c20012| 2018-03-16T17:28:23.493-0400 I SHARDING [conn1] distributed lock with ts: 5aac36f788c1f30185b7add0' unlocked. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.501-0400 c20012| 2018-03-16T17:28:23.501-0400 I SHARDING [conn1] distributed lock with ts: 5aac36f788c1f30185b7adc9' unlocked. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.501-0400 c20012| 2018-03-16T17:28:23.501-0400 I COMMAND [conn1] command admin.$cmd appName: "MongoDB Shell" command: _configsvrShardCollection { _configsvrShardCollection: "config.system.sessions", key: { _id: 1 }, unique: false, numInitialChunks: 0, writeConcern: { w: "majority", wtimeout: 60000 }, $db: "admin" } numYields:0 reslen:339 locks:{ Global: { acquireCount: { r: 33, w: 19 } }, Database: { acquireCount: { r: 7, w: 18, W: 1 } }, Collection: { acquireCount: { r: 7, w: 9 } }, Metadata: { acquireCount: { W: 2 } }, oplog: { acquireCount: { w: 9 } } } protocol:op_msg 173ms [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.501-0400 c20012| 2018-03-16T17:28:23.501-0400 I SHARDING [conn1] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.502-0400 c20012| 2018-03-16T17:28:23.501-0400 I SHARDING [conn1] config.system.sessions [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.502-0400 c20012| 2018-03-16T17:28:23.501-0400 I SHARDING [conn1] xxx routing info needs refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.502-0400 c20012| 2018-03-16T17:28:23.501-0400 I SHARDING [conn1] xxx calling schedule collection refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.502-0400 c20012| 2018-03-16T17:28:23.501-0400 I SHARDING [conn1] xxx is not incr [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.502-0400 c20012| 2018-03-16T17:28:23.501-0400 I SHARDING [conn1] xxx existing routing info? [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.502-0400 c20012| 2018-03-16T17:28:23.501-0400 I SHARDING [conn1] Refreshing chunks for collection config.system.sessions based on version 0|0||000000000000000000000000 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.502-0400 c20012| 2018-03-16T17:28:23.501-0400 I SHARDING [conn1] xxx getting chunks since now [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.503-0400 c20012| 2018-03-16T17:28:23.501-0400 I SHARDING [conn1] xxx waiting if !refreshNot [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.503-0400 c20012| 2018-03-16T17:28:23.501-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh callback [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.503-0400 c20012| 2018-03-16T17:28:23.501-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh completed [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.503-0400 c20012| 2018-03-16T17:28:23.501-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Refresh for collection config.system.sessions took 0 ms and found version 1|0||5aac36f788c1f30185b7adfc [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.503-0400 c20012| 2018-03-16T17:28:23.501-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx marking as status ok [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.503-0400 c20012| 2018-03-16T17:28:23.501-0400 I SHARDING [conn1] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.503-0400 c20012| 2018-03-16T17:28:23.501-0400 I SHARDING [conn1] config.system.sessions [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.503-0400 c20012| 2018-03-16T17:28:23.502-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-0-0] Connecting to ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.504-0400 c20012| 2018-03-16T17:28:23.502-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-0-0] Connecting to ubuntu:20011 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.504-0400 d20010| 2018-03-16T17:28:23.502-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56426 #16 (10 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.504-0400 d20011| 2018-03-16T17:28:23.502-0400 I NETWORK [listener] connection accepted from 127.0.0.1:55072 #14 (7 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.504-0400 d20010| 2018-03-16T17:28:23.502-0400 I NETWORK [conn16] received client metadata from 127.0.0.1:56426 conn16: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-0", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.504-0400 d20011| 2018-03-16T17:28:23.502-0400 I NETWORK [conn14] received client metadata from 127.0.0.1:55072 conn14: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-0", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.505-0400 c20012| 2018-03-16T17:28:23.502-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-0-0] Successfully connected to ubuntu:20010, took 0ms (1 connections now open to ubuntu:20010) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.505-0400 c20012| 2018-03-16T17:28:23.502-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-0-0] Successfully connected to ubuntu:20011, took 0ms (1 connections now open to ubuntu:20011) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.507-0400 c20012| 2018-03-16T17:28:23.507-0400 I SHARDING [conn34] distributed lock 'config' acquired for 'createCollection', ts : 5aac36f788c1f30185b7ae36 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.511-0400 d20010| 2018-03-16T17:28:23.510-0400 I INDEX [ShardServerCatalogCacheLoader-0] build index on: config.cache.chunks.config.system.sessions properties: { v: 2, key: { lastmod: 1 }, name: "lastmod_1", ns: "config.cache.chunks.config.system.sessions" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.511-0400 d20010| 2018-03-16T17:28:23.510-0400 I INDEX [ShardServerCatalogCacheLoader-0] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.511-0400 d20010| 2018-03-16T17:28:23.511-0400 I INDEX [ShardServerCatalogCacheLoader-0] build index done. scanned 0 total records. 0 secs [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.512-0400 c20012| 2018-03-16T17:28:23.512-0400 I SHARDING [conn34] distributed lock 'config.system.sessions' acquired for 'createCollection', ts : 5aac36f788c1f30185b7ae3e [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.512-0400 c20012| 2018-03-16T17:28:23.512-0400 I STORAGE [conn34] createCollection: config.system.sessions with generated UUID: 4a45df99-aec3-46b9-8afb-98b114b919cd [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.524-0400 d20010| 2018-03-16T17:28:23.524-0400 I INDEX [conn16] build index on: config.system.sessions properties: { v: 2, key: { lastUse: 1 }, name: "lsidTTLIndex", ns: "config.system.sessions", expireAfterSeconds: 1800 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.524-0400 d20010| 2018-03-16T17:28:23.524-0400 I INDEX [conn16] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.526-0400 d20010| 2018-03-16T17:28:23.526-0400 I INDEX [conn16] build index done. scanned 0 total records. 0 secs [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.534-0400 c20014| 2018-03-16T17:28:23.533-0400 I STORAGE [repl writer worker 5] createCollection: config.system.sessions with provided UUID: 4a45df99-aec3-46b9-8afb-98b114b919cd [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.534-0400 c20013| 2018-03-16T17:28:23.533-0400 I STORAGE [repl writer worker 7] createCollection: config.system.sessions with provided UUID: 4a45df99-aec3-46b9-8afb-98b114b919cd [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.563-0400 c20012| 2018-03-16T17:28:23.562-0400 I SHARDING [conn34] distributed lock with ts: 5aac36f788c1f30185b7ae3e' unlocked. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.569-0400 c20012| 2018-03-16T17:28:23.569-0400 I SHARDING [conn34] distributed lock with ts: 5aac36f788c1f30185b7ae36' unlocked. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.570-0400 c20012| 2018-03-16T17:28:23.569-0400 I COMMAND [conn1] command admin.$cmd appName: "MongoDB Shell" command: refreshLogicalSessionCacheNow { refreshLogicalSessionCacheNow: 1.0, $clusterTime: { clusterTime: Timestamp(1521235703, 12), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, $readPreference: { mode: "secondaryPreferred" }, $db: "admin" } numYields:0 reslen:256 locks:{ Global: { acquireCount: { r: 129, w: 19 } }, Database: { acquireCount: { r: 55, w: 18, W: 1 } }, Collection: { acquireCount: { r: 53, w: 9 } }, Metadata: { acquireCount: { W: 2 } }, oplog: { acquireCount: { r: 2, w: 9 } } } protocol:op_msg 242ms [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.570-0400 s20015| 2018-03-16T17:28:23.570-0400 D ASIO [conn1] startCommand: RemoteCommand 43 -- target:ubuntu:20012 db:admin cmd:{ _configsvrEnableSharding: "test", writeConcern: { w: "majority", wtimeout: 60000 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.570-0400 s20015| 2018-03-16T17:28:23.570-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 43 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.574-0400 c20012| 2018-03-16T17:28:23.574-0400 I SHARDING [conn21] distributed lock 'test' acquired for 'enableSharding', ts : 5aac36f788c1f30185b7ae66 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.575-0400 c20012| 2018-03-16T17:28:23.575-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Connecting to ubuntu:20011 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.575-0400 d20011| 2018-03-16T17:28:23.575-0400 I NETWORK [listener] connection accepted from 127.0.0.1:55074 #15 (8 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.576-0400 d20011| 2018-03-16T17:28:23.575-0400 I NETWORK [conn15] received client metadata from 127.0.0.1:55074 conn15: { driver: { name: "NetworkInterfaceASIO-ShardRegistry", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.576-0400 c20012| 2018-03-16T17:28:23.575-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to ubuntu:20011, took 0ms (1 connections now open to ubuntu:20011) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.576-0400 c20012| 2018-03-16T17:28:23.576-0400 I SHARDING [conn21] Registering new database { _id: "test", primary: "stale_mongos_updates_and_removes-rs1", partitioned: false } in sharding catalog [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.576-0400 c20012| 2018-03-16T17:28:23.576-0400 I STORAGE [conn21] createCollection: config.databases with generated UUID: 27dc98b0-d36f-49f8-87f0-41529ddec262 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.595-0400 c20013| 2018-03-16T17:28:23.595-0400 I STORAGE [repl writer worker 0] createCollection: config.databases with provided UUID: 27dc98b0-d36f-49f8-87f0-41529ddec262 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.595-0400 c20014| 2018-03-16T17:28:23.595-0400 I STORAGE [repl writer worker 13] createCollection: config.databases with provided UUID: 27dc98b0-d36f-49f8-87f0-41529ddec262 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.626-0400 c20012| 2018-03-16T17:28:23.626-0400 I SHARDING [conn21] Enabling sharding for database [test] in config db [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.636-0400 c20012| 2018-03-16T17:28:23.636-0400 I SHARDING [conn21] distributed lock with ts: 5aac36f788c1f30185b7ae66' unlocked. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.636-0400 s20015| 2018-03-16T17:28:23.636-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 43 finished with response: { ok: 1.0, operationTime: Timestamp(1521235703, 36), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235703, 36), t: 1 }, lastOpVisible: { ts: Timestamp(1521235703, 36), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235703, 36), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235703, 36), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 36) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.637-0400 s20015| 2018-03-16T17:28:23.636-0400 D ASIO [conn1] startCommand: RemoteCommand 45 -- target:ubuntu:20012 db:admin cmd:{ _configsvrShardCollection: "test.foo", key: { x: 1.0 }, unique: false, numInitialChunks: 0, writeConcern: { w: "majority", wtimeout: 60000 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.637-0400 s20015| 2018-03-16T17:28:23.637-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 45 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.641-0400 c20012| 2018-03-16T17:28:23.641-0400 I SHARDING [conn21] distributed lock 'test' acquired for 'shardCollection', ts : 5aac36f788c1f30185b7ae95 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.645-0400 c20012| 2018-03-16T17:28:23.645-0400 I SHARDING [conn21] distributed lock 'test.foo' acquired for 'shardCollection', ts : 5aac36f788c1f30185b7ae9b [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.645-0400 c20012| 2018-03-16T17:28:23.645-0400 I NETWORK [conn21] Successfully connected to stale_mongos_updates_and_removes-rs1/ubuntu:20011 (1 connections now open to stale_mongos_updates_and_removes-rs1/ubuntu:20011 with a 0 second timeout) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.646-0400 d20011| 2018-03-16T17:28:23.645-0400 I NETWORK [listener] connection accepted from 127.0.0.1:55076 #16 (9 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.646-0400 d20011| 2018-03-16T17:28:23.645-0400 I NETWORK [conn16] received client metadata from 127.0.0.1:55076 conn16: { driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.647-0400 d20011| 2018-03-16T17:28:23.646-0400 I STORAGE [conn15] createCollection: test.foo with generated UUID: 9c8b21ea-e8a4-43a9-a011-2b25a28a947b [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.669-0400 d20011| 2018-03-16T17:28:23.669-0400 I INDEX [conn15] build index on: test.foo properties: { v: 2, key: { x: 1.0 }, name: "x_1", ns: "test.foo" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.669-0400 d20011| 2018-03-16T17:28:23.669-0400 I INDEX [conn15] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.670-0400 d20011| 2018-03-16T17:28:23.669-0400 I INDEX [conn15] build index done. scanned 0 total records. 0 secs [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.673-0400 c20012| 2018-03-16T17:28:23.673-0400 I SHARDING [conn21] CMD: shardcollection: { _configsvrShardCollection: "test.foo", key: { x: 1.0 }, unique: false, numInitialChunks: 0, writeConcern: { w: "majority", wtimeout: 60000 }, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235703, 36), signature: { hash: BinData(0, F5BAD1415D4FF72DCE994EFB8E61FE39EE018428), keyId: 6533657568122765331 } }, $client: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" }, mongos: { host: "ubuntu:20015", client: "127.0.0.1:47840", version: "3.6.3" } }, $configServerState: { opTime: { ts: Timestamp(1521235703, 36), t: 1 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.674-0400 c20012| 2018-03-16T17:28:23.673-0400 I SHARDING [conn21] about to log metadata event into changelog: { _id: "ubuntu-2018-03-16T17:28:23.673-0400-5aac36f788c1f30185b7aead", server: "ubuntu", clientAddr: "127.0.0.1:60670", time: new Date(1521235703673), what: "shardCollection.start", ns: "test.foo", details: { shardKey: { x: 1.0 }, collection: "test.foo", uuid: UUID("9c8b21ea-e8a4-43a9-a011-2b25a28a947b"), primary: "stale_mongos_updates_and_removes-rs1:stale_mongos_updates_and_removes-rs1/ubuntu:20011", numChunks: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.678-0400 c20012| 2018-03-16T17:28:23.678-0400 I SHARDING [conn21] going to create 1 chunk(s) for: test.foo using new epoch 5aac36f788c1f30185b7aeba [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.687-0400 d20011| 2018-03-16T17:28:23.686-0400 I SHARDING [conn15] XXXX in set shard version cmd obj is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.687-0400 d20011| 2018-03-16T17:28:23.686-0400 I SHARDING [conn15] { setShardVersion: "test.foo", init: false, authoritative: true, configdb: "stale_mongos_updates_and_removes-configRS/ubuntu:20012,ubuntu:20013,ubuntu:20014", shard: "stale_mongos_updates_and_removes-rs1", shardHost: "stale_mongos_updates_and_removes-rs1/ubuntu:20011", version: Timestamp(1, 0), versionEpoch: ObjectId('5aac36f788c1f30185b7aeba'), noConnectionVersioning: true, $clusterTime: { clusterTime: Timestamp(1521235703, 43), signature: { hash: BinData(0, F5BAD1415D4FF72DCE994EFB8E61FE39EE018428), keyId: 6533657568122765331 } }, $client: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" }, mongos: { host: "ubuntu:20015", client: "127.0.0.1:47840", version: "3.6.3" } }, $configServerState: { opTime: { ts: Timestamp(1521235703, 43), t: 1 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.688-0400 d20011| 2018-03-16T17:28:23.686-0400 I SHARDING [conn15] XXXX requested version is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.688-0400 d20011| 2018-03-16T17:28:23.686-0400 I SHARDING [conn15] 1|0||5aac36f788c1f30185b7aeba [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.688-0400 d20011| 2018-03-16T17:28:23.687-0400 I SHARDING [conn15] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.688-0400 d20011| 2018-03-16T17:28:23.687-0400 I SHARDING [conn15] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.688-0400 d20011| 2018-03-16T17:28:23.687-0400 I SHARDING [conn15] xxx routing info needs refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.688-0400 d20011| 2018-03-16T17:28:23.687-0400 I SHARDING [conn15] xxx calling schedule collection refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.688-0400 d20011| 2018-03-16T17:28:23.687-0400 I SHARDING [conn15] xxx is not incr [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.688-0400 d20011| 2018-03-16T17:28:23.687-0400 I SHARDING [conn15] xxx existing routing info? [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.689-0400 d20011| 2018-03-16T17:28:23.687-0400 I SHARDING [conn15] Refreshing chunks for collection test.foo based on version 0|0||000000000000000000000000 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.689-0400 d20011| 2018-03-16T17:28:23.687-0400 I SHARDING [conn15] xxx getting chunks since now [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.689-0400 d20011| 2018-03-16T17:28:23.687-0400 I SHARDING [conn15] xxx getting chunks since [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.689-0400 d20011| 2018-03-16T17:28:23.687-0400 I SHARDING [conn15] xxx waiting if !refreshNot [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.689-0400 d20011| 2018-03-16T17:28:23.687-0400 I SHARDING [ShardServerCatalogCacheLoader-0] xxx primary and calling get chunks since for primary [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.689-0400 d20011| 2018-03-16T17:28:23.687-0400 I SHARDING [ShardServerCatalogCacheLoader-0] version is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.689-0400 d20011| 2018-03-16T17:28:23.687-0400 I SHARDING [ShardServerCatalogCacheLoader-0] 0|0||000000000000000000000000 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.689-0400 d20011| 2018-03-16T17:28:23.688-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] refresh in primary [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.690-0400 d20011| 2018-03-16T17:28:23.688-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] 1 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.690-0400 d20011| 2018-03-16T17:28:23.688-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx collec and changed chunks max epoch or version doesn't equal max [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.764-0400 d20011| 2018-03-16T17:28:23.763-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Cache loader remotely refreshed for collection test.foo from collection version 0|0||000000000000000000000000 and found collection version 1|0||5aac36f788c1f30185b7aeba [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.764-0400 d20011| 2018-03-16T17:28:23.763-0400 I STORAGE [ShardServerCatalogCacheLoader-0] createCollection: config.cache.collections with generated UUID: e88e1930-aefd-4535-81c7-a61a0f8ad273 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.764-0400 d20011| 2018-03-16T17:28:23.763-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Cache loader found enqueued metadata from 1|0||5aac36f788c1f30185b7aeba to 1|0||5aac36f788c1f30185b7aeba and no persisted metadata, GTE cache version 0|0||000000000000000000000000 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.764-0400 d20011| 2018-03-16T17:28:23.763-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx is okay [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.764-0400 d20011| 2018-03-16T17:28:23.763-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx finsihing refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.765-0400 d20011| 2018-03-16T17:28:23.763-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh callback [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.765-0400 d20011| 2018-03-16T17:28:23.764-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh completed [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.765-0400 d20011| 2018-03-16T17:28:23.764-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Refresh for collection test.foo took 76 ms and found version 1|0||5aac36f788c1f30185b7aeba [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.765-0400 d20011| 2018-03-16T17:28:23.764-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx marking as status ok [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.765-0400 d20011| 2018-03-16T17:28:23.764-0400 I SHARDING [conn15] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.765-0400 d20011| 2018-03-16T17:28:23.764-0400 I SHARDING [conn15] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.766-0400 d20011| 2018-03-16T17:28:23.764-0400 I SHARDING [conn15] Marking collection test.foo as sharded with collection version: 1|0||5aac36f788c1f30185b7aeba, shard version: 1|0||5aac36f788c1f30185b7aeba [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.766-0400 c20012| 2018-03-16T17:28:23.764-0400 I SHARDING [conn21] about to log metadata event into changelog: { _id: "ubuntu-2018-03-16T17:28:23.764-0400-5aac36f788c1f30185b7aed2", server: "ubuntu", clientAddr: "127.0.0.1:60670", time: new Date(1521235703764), what: "shardCollection.end", ns: "test.foo", details: { version: "1|0||5aac36f788c1f30185b7aeba" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.777-0400 c20012| 2018-03-16T17:28:23.777-0400 I SHARDING [conn21] distributed lock with ts: 5aac36f788c1f30185b7ae9b' unlocked. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.781-0400 c20012| 2018-03-16T17:28:23.781-0400 I SHARDING [conn21] distributed lock with ts: 5aac36f788c1f30185b7ae95' unlocked. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.782-0400 c20012| 2018-03-16T17:28:23.781-0400 I COMMAND [conn21] command admin.$cmd appName: "MongoDB Shell" command: _configsvrShardCollection { _configsvrShardCollection: "test.foo", key: { x: 1.0 }, unique: false, numInitialChunks: 0, writeConcern: { w: "majority", wtimeout: 60000 }, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235703, 36), signature: { hash: BinData(0, F5BAD1415D4FF72DCE994EFB8E61FE39EE018428), keyId: 6533657568122765331 } }, $client: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" }, mongos: { host: "ubuntu:20015", client: "127.0.0.1:47840", version: "3.6.3" } }, $configServerState: { opTime: { ts: Timestamp(1521235703, 36), t: 1 } }, $db: "admin" } numYields:0 reslen:531 locks:{ Global: { acquireCount: { r: 30, w: 16 } }, Database: { acquireCount: { r: 7, w: 16 } }, Collection: { acquireCount: { r: 7, w: 8 } }, Metadata: { acquireCount: { W: 2 } }, oplog: { acquireCount: { w: 8 } } } protocol:op_msg 144ms [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.783-0400 s20015| 2018-03-16T17:28:23.781-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 45 finished with response: { collectionsharded: "test.foo", collectionUUID: UUID("9c8b21ea-e8a4-43a9-a011-2b25a28a947b"), ok: 1.0, operationTime: Timestamp(1521235703, 47), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235703, 47), t: 1 }, lastOpVisible: { ts: Timestamp(1521235703, 47), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235703, 47), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235703, 47), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 47) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.783-0400 s20015| 2018-03-16T17:28:23.782-0400 D ASIO [conn1] startCommand: RemoteCommand 47 -- target:ubuntu:20012 db:admin cmd:{ _configsvrMovePrimary: "test", to: "stale_mongos_updates_and_removes-rs0", writeConcern: { w: "majority", wtimeout: 60000 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.783-0400 s20015| 2018-03-16T17:28:23.782-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 47 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.784-0400 d20011| 2018-03-16T17:28:23.783-0400 I STORAGE [ShardServerCatalogCacheLoader-0] createCollection: config.cache.chunks.test.foo with generated UUID: 16ddefd7-5b03-4b97-a3e6-343031531217 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.787-0400 c20012| 2018-03-16T17:28:23.786-0400 I SHARDING [conn21] distributed lock 'test' acquired for 'movePrimary', ts : 5aac36f788c1f30185b7aeef [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.787-0400 c20012| 2018-03-16T17:28:23.787-0400 I SHARDING [conn21] Moving test primary from: stale_mongos_updates_and_removes-rs1:stale_mongos_updates_and_removes-rs1/ubuntu:20011 to: stale_mongos_updates_and_removes-rs0:stale_mongos_updates_and_removes-rs0/ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.787-0400 c20012| 2018-03-16T17:28:23.787-0400 I SHARDING [conn21] about to log metadata event into changelog: { _id: "ubuntu-2018-03-16T17:28:23.787-0400-5aac36f788c1f30185b7aefa", server: "ubuntu", clientAddr: "127.0.0.1:60670", time: new Date(1521235703787), what: "movePrimary.start", ns: "test", details: { database: "test", from: "stale_mongos_updates_and_removes-rs1:stale_mongos_updates_and_removes-rs1/ubuntu:20011", to: "stale_mongos_updates_and_removes-rs0:stale_mongos_updates_and_removes-rs0/ubuntu:20010", shardedCollections: [ "test.foo" ] } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.817-0400 d20011| 2018-03-16T17:28:23.817-0400 I NETWORK [listener] connection accepted from 127.0.0.1:55078 #17 (10 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.817-0400 d20011| 2018-03-16T17:28:23.817-0400 I NETWORK [conn17] end connection 127.0.0.1:55078 (9 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.818-0400 d20010| 2018-03-16T17:28:23.817-0400 I NETWORK [conn14] Starting new replica set monitor for stale_mongos_updates_and_removes-rs1/ubuntu:20011 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.818-0400 d20011| 2018-03-16T17:28:23.817-0400 I NETWORK [listener] connection accepted from 127.0.0.1:55080 #18 (10 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.818-0400 d20011| 2018-03-16T17:28:23.817-0400 I NETWORK [conn18] received client metadata from 127.0.0.1:55080 conn18: { driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.818-0400 d20010| 2018-03-16T17:28:23.818-0400 I NETWORK [conn14] Successfully connected to ubuntu:20011 (1 connections now open to ubuntu:20011 with a 5 second timeout) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.818-0400 d20011| 2018-03-16T17:28:23.818-0400 I NETWORK [listener] connection accepted from 127.0.0.1:55082 #19 (11 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.819-0400 d20011| 2018-03-16T17:28:23.818-0400 I NETWORK [conn19] received client metadata from 127.0.0.1:55082 conn19: { driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.819-0400 d20011| 2018-03-16T17:28:23.818-0400 I NETWORK [conn19] end connection 127.0.0.1:55082 (10 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.823-0400 c20012| 2018-03-16T17:28:23.823-0400 I SHARDING [conn21] about to log metadata event into changelog: { _id: "ubuntu-2018-03-16T17:28:23.823-0400-5aac36f788c1f30185b7af0c", server: "ubuntu", clientAddr: "127.0.0.1:60670", time: new Date(1521235703823), what: "movePrimary", ns: "test", details: { database: "test", from: "stale_mongos_updates_and_removes-rs1/ubuntu:20011", to: "stale_mongos_updates_and_removes-rs0:stale_mongos_updates_and_removes-rs0/ubuntu:20010", shardedCollections: [ "test.foo" ] } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.833-0400 c20012| 2018-03-16T17:28:23.833-0400 I SHARDING [conn21] distributed lock with ts: 5aac36f788c1f30185b7aeef' unlocked. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.834-0400 s20015| 2018-03-16T17:28:23.833-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 47 finished with response: { primary: "stale_mongos_updates_and_removes-rs0:stale_mongos_updates_and_removes-rs0/ubuntu:20010", ok: 1.0, operationTime: Timestamp(1521235703, 52), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235703, 52), t: 1 }, lastOpVisible: { ts: Timestamp(1521235703, 52), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235703, 52), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235703, 52), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 52) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.834-0400 d20011| 2018-03-16T17:28:23.834-0400 I INDEX [ShardServerCatalogCacheLoader-0] build index on: config.cache.chunks.test.foo properties: { v: 2, key: { lastmod: 1 }, name: "lastmod_1", ns: "config.cache.chunks.test.foo" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.834-0400 d20011| 2018-03-16T17:28:23.834-0400 I INDEX [ShardServerCatalogCacheLoader-0] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.835-0400 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.835-0400 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.835-0400 ---- [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.835-0400 xxx top of remove [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.835-0400 ---- [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.835-0400 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.835-0400 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.835-0400 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.835-0400 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.835-0400 ---- [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.835-0400 xxx top single [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.835-0400 ---- [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.835-0400 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.835-0400 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.836-0400 s20016| 2018-03-16T17:28:23.835-0400 D SHARDING [conn1] trying to acquire new distributed lock for test-movePrimary ( lock timeout : 900000 ms, ping interval : 30000 ms, process : ubuntu:20016:1521235700:-4633782234644071543 ) with lockSessionID: 5aac36f7ebcdfa1d2b617d82, why: dropCollection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.836-0400 s20016| 2018-03-16T17:28:23.835-0400 D ASIO [conn1] startCommand: RemoteCommand 31 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:53.835-0400 cmd:{ findAndModify: "locks", query: { _id: "test-movePrimary", state: 0 }, update: { $set: { ts: ObjectId('5aac36f7ebcdfa1d2b617d82'), state: 2, who: "ubuntu:20016:1521235700:-4633782234644071543:conn1", process: "ubuntu:20016:1521235700:-4633782234644071543", when: new Date(1521235703835), why: "dropCollection" } }, upsert: true, new: true, writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.836-0400 s20016| 2018-03-16T17:28:23.835-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 31 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.836-0400 d20011| 2018-03-16T17:28:23.835-0400 I INDEX [ShardServerCatalogCacheLoader-0] build index done. scanned 0 total records. 0 secs [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.839-0400 s20016| 2018-03-16T17:28:23.838-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 31 finished with response: { lastErrorObject: { n: 1, updatedExisting: false, upserted: "test-movePrimary" }, value: { _id: "test-movePrimary", state: 2, process: "ubuntu:20016:1521235700:-4633782234644071543", ts: ObjectId('5aac36f7ebcdfa1d2b617d82'), when: new Date(1521235703835), who: "ubuntu:20016:1521235700:-4633782234644071543:conn1", why: "dropCollection" }, ok: 1.0, operationTime: Timestamp(1521235703, 53), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235703, 53), t: 1 }, lastOpVisible: { ts: Timestamp(1521235703, 53), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235703, 53), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235703, 53), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 53) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.839-0400 s20016| 2018-03-16T17:28:23.838-0400 I SHARDING [conn1] distributed lock 'test-movePrimary' acquired for 'dropCollection', ts : 5aac36f7ebcdfa1d2b617d82 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.840-0400 s20016| 2018-03-16T17:28:23.838-0400 D SHARDING [conn1] trying to acquire new distributed lock for test ( lock timeout : 900000 ms, ping interval : 30000 ms, process : ubuntu:20016:1521235700:-4633782234644071543 ) with lockSessionID: 5aac36f7ebcdfa1d2b617d83, why: dropCollection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.840-0400 s20016| 2018-03-16T17:28:23.838-0400 D ASIO [conn1] startCommand: RemoteCommand 33 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:53.838-0400 cmd:{ findAndModify: "locks", query: { _id: "test", state: 0 }, update: { $set: { ts: ObjectId('5aac36f7ebcdfa1d2b617d83'), state: 2, who: "ubuntu:20016:1521235700:-4633782234644071543:conn1", process: "ubuntu:20016:1521235700:-4633782234644071543", when: new Date(1521235703838), why: "dropCollection" } }, upsert: true, new: true, writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.840-0400 s20016| 2018-03-16T17:28:23.838-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 33 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.843-0400 s20016| 2018-03-16T17:28:23.842-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 33 finished with response: { lastErrorObject: { n: 1, updatedExisting: true }, value: { _id: "test", state: 2, process: "ubuntu:20016:1521235700:-4633782234644071543", ts: ObjectId('5aac36f7ebcdfa1d2b617d83'), when: new Date(1521235703838), who: "ubuntu:20016:1521235700:-4633782234644071543:conn1", why: "dropCollection" }, ok: 1.0, operationTime: Timestamp(1521235703, 54), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235703, 54), t: 1 }, lastOpVisible: { ts: Timestamp(1521235703, 54), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235703, 54), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235703, 54), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 54) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.844-0400 s20016| 2018-03-16T17:28:23.842-0400 I SHARDING [conn1] distributed lock 'test' acquired for 'dropCollection', ts : 5aac36f7ebcdfa1d2b617d83 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.844-0400 s20016| 2018-03-16T17:28:23.842-0400 D SHARDING [conn1] trying to acquire new distributed lock for test.foo ( lock timeout : 900000 ms, ping interval : 30000 ms, process : ubuntu:20016:1521235700:-4633782234644071543 ) with lockSessionID: 5aac36f7ebcdfa1d2b617d84, why: dropCollection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.844-0400 s20016| 2018-03-16T17:28:23.843-0400 D ASIO [conn1] startCommand: RemoteCommand 35 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:53.843-0400 cmd:{ findAndModify: "locks", query: { _id: "test.foo", state: 0 }, update: { $set: { ts: ObjectId('5aac36f7ebcdfa1d2b617d84'), state: 2, who: "ubuntu:20016:1521235700:-4633782234644071543:conn1", process: "ubuntu:20016:1521235700:-4633782234644071543", when: new Date(1521235703843), why: "dropCollection" } }, upsert: true, new: true, writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.844-0400 s20016| 2018-03-16T17:28:23.843-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 35 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.849-0400 s20016| 2018-03-16T17:28:23.848-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 35 finished with response: { lastErrorObject: { n: 1, updatedExisting: true }, value: { _id: "test.foo", state: 2, process: "ubuntu:20016:1521235700:-4633782234644071543", ts: ObjectId('5aac36f7ebcdfa1d2b617d84'), when: new Date(1521235703843), who: "ubuntu:20016:1521235700:-4633782234644071543:conn1", why: "dropCollection" }, ok: 1.0, operationTime: Timestamp(1521235703, 55), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235703, 55), t: 1 }, lastOpVisible: { ts: Timestamp(1521235703, 55), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235703, 55), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235703, 55), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 55) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.849-0400 s20016| 2018-03-16T17:28:23.848-0400 I SHARDING [conn1] distributed lock 'test.foo' acquired for 'dropCollection', ts : 5aac36f7ebcdfa1d2b617d84 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.850-0400 s20016| 2018-03-16T17:28:23.848-0400 D ASIO [conn1] startCommand: RemoteCommand 37 -- target:ubuntu:20013 db:config expDate:2018-03-16T17:28:53.848-0400 cmd:{ find: "databases", filter: { _id: "test" }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235703, 55), t: 1 } }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.850-0400 s20016| 2018-03-16T17:28:23.848-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 37 on host ubuntu:20013 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.851-0400 s20016| 2018-03-16T17:28:23.849-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 37 finished with response: { cursor: { firstBatch: [ { _id: "test", primary: "stale_mongos_updates_and_removes-rs0", partitioned: true } ], id: 0, ns: "config.databases" }, ok: 1.0, operationTime: Timestamp(1521235703, 55), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235703, 55), t: 1 }, lastOpVisible: { ts: Timestamp(1521235703, 55), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235703, 55), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 55) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.851-0400 s20016| 2018-03-16T17:28:23.849-0400 D ASIO [conn1] startCommand: RemoteCommand 39 -- target:ubuntu:20014 db:config expDate:2018-03-16T17:28:53.849-0400 cmd:{ find: "collections", filter: { _id: /^test\./ }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235703, 55), t: 1 } }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.851-0400 s20016| 2018-03-16T17:28:23.849-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 39 on host ubuntu:20014 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.852-0400 s20016| 2018-03-16T17:28:23.849-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 39 finished with response: { cursor: { firstBatch: [ { _id: "test.foo", lastmodEpoch: ObjectId('5aac36f788c1f30185b7aeba'), lastmod: new Date(4294967296), dropped: false, key: { x: 1.0 }, unique: false, uuid: UUID("9c8b21ea-e8a4-43a9-a011-2b25a28a947b") } ], id: 0, ns: "config.collections" }, ok: 1.0, operationTime: Timestamp(1521235703, 55), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235703, 55), t: 1 }, lastOpVisible: { ts: Timestamp(1521235703, 55), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235703, 55), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 55) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.852-0400 s20016| 2018-03-16T17:28:23.849-0400 I SHARDING [conn1] Refreshing chunks for collection test.foo based on version 0|0||000000000000000000000000 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.852-0400 s20016| 2018-03-16T17:28:23.849-0400 D EXECUTOR [ConfigServerCatalogCacheLoader-0] starting thread in pool ConfigServerCatalogCacheLoader [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.853-0400 s20016| 2018-03-16T17:28:23.849-0400 D ASIO [ConfigServerCatalogCacheLoader-0] startCommand: RemoteCommand 41 -- target:ubuntu:20013 db:config expDate:2018-03-16T17:28:53.849-0400 cmd:{ find: "collections", filter: { _id: "test.foo" }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235703, 55), t: 1 } }, limit: 1, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.853-0400 s20016| 2018-03-16T17:28:23.850-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 41 on host ubuntu:20013 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.853-0400 s20016| 2018-03-16T17:28:23.850-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 41 finished with response: { cursor: { firstBatch: [ { _id: "test.foo", lastmodEpoch: ObjectId('5aac36f788c1f30185b7aeba'), lastmod: new Date(4294967296), dropped: false, key: { x: 1.0 }, unique: false, uuid: UUID("9c8b21ea-e8a4-43a9-a011-2b25a28a947b") } ], id: 0, ns: "config.collections" }, ok: 1.0, operationTime: Timestamp(1521235703, 55), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235703, 55), t: 1 }, lastOpVisible: { ts: Timestamp(1521235703, 55), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235703, 55), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 55) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.854-0400 s20016| 2018-03-16T17:28:23.850-0400 D ASIO [ConfigServerCatalogCacheLoader-0] startCommand: RemoteCommand 43 -- target:ubuntu:20013 db:config expDate:2018-03-16T17:28:53.850-0400 cmd:{ find: "chunks", filter: { ns: "test.foo", lastmod: { $gte: Timestamp(0, 0) } }, sort: { lastmod: 1 }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235703, 55), t: 1 } }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.854-0400 s20016| 2018-03-16T17:28:23.850-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 43 on host ubuntu:20013 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.855-0400 s20016| 2018-03-16T17:28:23.850-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 43 finished with response: { cursor: { firstBatch: [ { _id: "test.foo-x_MinKey", ns: "test.foo", min: { x: MinKey }, max: { x: MaxKey }, shard: "stale_mongos_updates_and_removes-rs1", lastmod: Timestamp(1, 0), lastmodEpoch: ObjectId('5aac36f788c1f30185b7aeba'), history: [ { validAfter: Timestamp(1521235703, 41), shard: "stale_mongos_updates_and_removes-rs1" } ] } ], id: 0, ns: "config.chunks" }, ok: 1.0, operationTime: Timestamp(1521235703, 55), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235703, 55), t: 1 }, lastOpVisible: { ts: Timestamp(1521235703, 55), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235703, 55), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 55) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.855-0400 s20016| 2018-03-16T17:28:23.850-0400 D ASIO [ConfigServerCatalogCacheLoader-0] startCommand: RemoteCommand 45 -- target:ubuntu:20014 db:config expDate:2018-03-16T17:28:53.850-0400 cmd:{ find: "shards", readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235703, 55), t: 1 } }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.855-0400 s20016| 2018-03-16T17:28:23.850-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 45 on host ubuntu:20014 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.856-0400 s20016| 2018-03-16T17:28:23.851-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 45 finished with response: { cursor: { firstBatch: [ { _id: "stale_mongos_updates_and_removes-rs0", host: "stale_mongos_updates_and_removes-rs0/ubuntu:20010", state: 1 }, { _id: "stale_mongos_updates_and_removes-rs1", host: "stale_mongos_updates_and_removes-rs1/ubuntu:20011", state: 1 } ], id: 0, ns: "config.shards" }, ok: 1.0, operationTime: Timestamp(1521235703, 55), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235703, 55), t: 1 }, lastOpVisible: { ts: Timestamp(1521235703, 55), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235703, 55), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 55) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.856-0400 s20016| 2018-03-16T17:28:23.851-0400 D SHARDING [ConfigServerCatalogCacheLoader-0] found 2 shards listed on config server(s) with lastVisibleOpTime: { ts: Timestamp(1521235703, 55), t: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.856-0400 s20016| 2018-03-16T17:28:23.851-0400 I NETWORK [ConfigServerCatalogCacheLoader-0] Starting new replica set monitor for stale_mongos_updates_and_removes-rs0/ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.857-0400 s20016| 2018-03-16T17:28:23.851-0400 D NETWORK [ConfigServerCatalogCacheLoader-0] Started targeter for stale_mongos_updates_and_removes-rs0/ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.857-0400 s20016| 2018-03-16T17:28:23.851-0400 I NETWORK [ConfigServerCatalogCacheLoader-0] Starting new replica set monitor for stale_mongos_updates_and_removes-rs1/ubuntu:20011 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.857-0400 s20016| 2018-03-16T17:28:23.851-0400 D NETWORK [ReplicaSetMonitor-TaskExecutor-0] Starting new refresh of replica set stale_mongos_updates_and_removes-rs0 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.857-0400 s20016| 2018-03-16T17:28:23.851-0400 D NETWORK [ConfigServerCatalogCacheLoader-0] Started targeter for stale_mongos_updates_and_removes-rs1/ubuntu:20011 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.857-0400 s20016| 2018-03-16T17:28:23.851-0400 D NETWORK [ReplicaSetMonitor-TaskExecutor-0] creating new connection to:ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.857-0400 s20016| 2018-03-16T17:28:23.851-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Refresh for collection test.foo took 1 ms and found version 1|0||5aac36f788c1f30185b7aeba [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.858-0400 s20016| 2018-03-16T17:28:23.851-0400 D ASIO [conn1] startCommand: RemoteCommand 47 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:53.851-0400 cmd:{ create: "changelog", capped: true, size: 10485760, writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.858-0400 s20016| 2018-03-16T17:28:23.851-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 47 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.858-0400 s20016| 2018-03-16T17:28:23.851-0400 D NETWORK [ReplicaSetMonitor-TaskExecutor-0] connected to server ubuntu:20010 (127.0.1.1) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.858-0400 d20010| 2018-03-16T17:28:23.851-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56440 #20 (11 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.858-0400 d20010| 2018-03-16T17:28:23.851-0400 I NETWORK [conn20] received client metadata from 127.0.0.1:56440 conn20: { driver: { name: "MongoDB Internal Client", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.859-0400 s20016| 2018-03-16T17:28:23.851-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 47 finished with response: { ok: 0.0, errmsg: "a collection 'config.changelog' already exists", code: 48, codeName: "NamespaceExists", operationTime: Timestamp(1521235703, 55), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235703, 55), t: 1 }, lastOpVisible: { ts: Timestamp(1521235703, 55), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235703, 55), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235703, 55), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 55) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.859-0400 s20016| 2018-03-16T17:28:23.851-0400 I SHARDING [conn1] about to log metadata event into changelog: { _id: "ubuntu-2018-03-16T17:28:23.851-0400-5aac36f7ebcdfa1d2b617d85", server: "ubuntu", clientAddr: "127.0.0.1:50986", time: new Date(1521235703851), what: "dropCollection.start", ns: "test.foo", details: {} } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.860-0400 s20016| 2018-03-16T17:28:23.851-0400 D ASIO [conn1] startCommand: RemoteCommand 49 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:53.851-0400 cmd:{ insert: "changelog", bypassDocumentValidation: false, ordered: true, documents: [ { _id: "ubuntu-2018-03-16T17:28:23.851-0400-5aac36f7ebcdfa1d2b617d85", server: "ubuntu", clientAddr: "127.0.0.1:50986", time: new Date(1521235703851), what: "dropCollection.start", ns: "test.foo", details: {} } ], writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.860-0400 s20016| 2018-03-16T17:28:23.851-0400 D NETWORK [ReplicaSetMonitor-TaskExecutor-0] connected connection! [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.860-0400 s20016| 2018-03-16T17:28:23.851-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 49 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.860-0400 s20016| 2018-03-16T17:28:23.851-0400 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to ubuntu:20010 (1 connections now open to ubuntu:20010 with a 5 second timeout) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.860-0400 s20016| 2018-03-16T17:28:23.852-0400 D NETWORK [ReplicaSetMonitor-TaskExecutor-0] Refreshing replica set stale_mongos_updates_and_removes-rs0 took 0 msec [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.861-0400 s20016| 2018-03-16T17:28:23.852-0400 D NETWORK [ReplicaSetMonitor-TaskExecutor-0] Starting new refresh of replica set stale_mongos_updates_and_removes-rs1 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.861-0400 s20016| 2018-03-16T17:28:23.852-0400 D NETWORK [ReplicaSetMonitor-TaskExecutor-0] creating new connection to:ubuntu:20011 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.861-0400 s20016| 2018-03-16T17:28:23.852-0400 D NETWORK [ReplicaSetMonitor-TaskExecutor-0] connected to server ubuntu:20011 (127.0.1.1) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.861-0400 d20011| 2018-03-16T17:28:23.852-0400 I NETWORK [listener] connection accepted from 127.0.0.1:55086 #20 (11 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.861-0400 d20011| 2018-03-16T17:28:23.852-0400 I NETWORK [conn20] received client metadata from 127.0.0.1:55086 conn20: { driver: { name: "MongoDB Internal Client", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.861-0400 s20016| 2018-03-16T17:28:23.852-0400 D NETWORK [ReplicaSetMonitor-TaskExecutor-0] connected connection! [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.862-0400 s20016| 2018-03-16T17:28:23.852-0400 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to ubuntu:20011 (1 connections now open to ubuntu:20011 with a 5 second timeout) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.862-0400 s20016| 2018-03-16T17:28:23.852-0400 D NETWORK [ReplicaSetMonitor-TaskExecutor-0] Refreshing replica set stale_mongos_updates_and_removes-rs1 took 0 msec [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.862-0400 s20016| 2018-03-16T17:28:23.856-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 49 finished with response: { n: 1, opTime: { ts: Timestamp(1521235703, 56), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235703, 56), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235703, 56), t: 1 }, lastOpVisible: { ts: Timestamp(1521235703, 56), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235703, 56), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235703, 56), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 56) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.863-0400 s20016| 2018-03-16T17:28:23.857-0400 D ASIO [conn1] startCommand: RemoteCommand 51 -- target:ubuntu:20014 db:config expDate:2018-03-16T17:28:53.857-0400 cmd:{ find: "shards", readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235703, 56), t: 1 } }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.863-0400 s20016| 2018-03-16T17:28:23.857-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 51 on host ubuntu:20014 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.864-0400 s20016| 2018-03-16T17:28:23.857-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 51 finished with response: { cursor: { firstBatch: [ { _id: "stale_mongos_updates_and_removes-rs0", host: "stale_mongos_updates_and_removes-rs0/ubuntu:20010", state: 1 }, { _id: "stale_mongos_updates_and_removes-rs1", host: "stale_mongos_updates_and_removes-rs1/ubuntu:20011", state: 1 } ], id: 0, ns: "config.shards" }, ok: 1.0, operationTime: Timestamp(1521235703, 56), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235703, 56), t: 1 }, lastOpVisible: { ts: Timestamp(1521235703, 56), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235703, 56), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 56) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.864-0400 s20016| 2018-03-16T17:28:23.857-0400 D SHARDING [conn1] dropCollection test.foo started [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.864-0400 s20016| 2018-03-16T17:28:23.857-0400 D ASIO [conn1] startCommand: RemoteCommand 53 -- target:ubuntu:20010 db:test cmd:{ drop: "foo" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.864-0400 s20016| 2018-03-16T17:28:23.857-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Connecting to ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.864-0400 s20016| 2018-03-16T17:28:23.857-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 54 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.864-0400 d20010| 2018-03-16T17:28:23.857-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56444 #21 (12 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.865-0400 d20010| 2018-03-16T17:28:23.857-0400 I NETWORK [conn21] received client metadata from 127.0.0.1:56444 conn21: { driver: { name: "NetworkInterfaceASIO-ShardRegistry", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.865-0400 s20016| 2018-03-16T17:28:23.857-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to ubuntu:20010, took 0ms (1 connections now open to ubuntu:20010) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.865-0400 s20016| 2018-03-16T17:28:23.857-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 54 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.865-0400 s20016| 2018-03-16T17:28:23.857-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 53 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.865-0400 d20010| 2018-03-16T17:28:23.857-0400 I COMMAND [conn21] CMD: drop test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.866-0400 s20016| 2018-03-16T17:28:23.858-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 53 finished with response: { ok: 0.0, errmsg: "ns not found", code: 26, codeName: "NamespaceNotFound", operationTime: Timestamp(1521235703, 30), $gleStats: { lastOpTime: { ts: Timestamp(1521235703, 30), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235703, 56), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 30), $configServerState: { opTime: { ts: Timestamp(1521235703, 56), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.866-0400 s20016| 2018-03-16T17:28:23.858-0400 D ASIO [conn1] startCommand: RemoteCommand 56 -- target:ubuntu:20011 db:test cmd:{ drop: "foo" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.866-0400 s20016| 2018-03-16T17:28:23.858-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Connecting to ubuntu:20011 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.866-0400 s20016| 2018-03-16T17:28:23.858-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 57 on host ubuntu:20011 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.866-0400 d20011| 2018-03-16T17:28:23.858-0400 I NETWORK [listener] connection accepted from 127.0.0.1:55090 #21 (12 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.867-0400 d20011| 2018-03-16T17:28:23.858-0400 I NETWORK [conn21] received client metadata from 127.0.0.1:55090 conn21: { driver: { name: "NetworkInterfaceASIO-ShardRegistry", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.867-0400 s20016| 2018-03-16T17:28:23.858-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to ubuntu:20011, took 0ms (1 connections now open to ubuntu:20011) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.867-0400 s20016| 2018-03-16T17:28:23.858-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 57 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.867-0400 s20016| 2018-03-16T17:28:23.858-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 56 on host ubuntu:20011 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.867-0400 d20011| 2018-03-16T17:28:23.858-0400 I COMMAND [conn21] CMD: drop test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.868-0400 d20011| 2018-03-16T17:28:23.858-0400 I STORAGE [conn21] dropCollection: test.foo (9c8b21ea-e8a4-43a9-a011-2b25a28a947b) - renaming to drop-pending collection: test.system.drop.1521235703i57t1.foo with drop optime { ts: Timestamp(1521235703, 57), t: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.868-0400 s20016| 2018-03-16T17:28:23.862-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 56 finished with response: { ns: "test.foo", nIndexesWas: 2, ok: 1.0, operationTime: Timestamp(1521235703, 57), $gleStats: { lastOpTime: { ts: Timestamp(1521235703, 57), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235703, 57), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 44), $configServerState: { opTime: { ts: Timestamp(1521235703, 56), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.868-0400 s20016| 2018-03-16T17:28:23.862-0400 D SHARDING [conn1] dropCollection test.foo shard data deleted [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.869-0400 s20016| 2018-03-16T17:28:23.862-0400 D ASIO [conn1] startCommand: RemoteCommand 59 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:53.862-0400 cmd:{ delete: "chunks", bypassDocumentValidation: false, ordered: true, deletes: [ { q: { ns: "test.foo" }, limit: 0 } ], writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.869-0400 s20016| 2018-03-16T17:28:23.862-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 59 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.869-0400 s20016| 2018-03-16T17:28:23.866-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 59 finished with response: { n: 1, opTime: { ts: Timestamp(1521235703, 58), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235703, 58), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235703, 58), t: 1 }, lastOpVisible: { ts: Timestamp(1521235703, 58), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235703, 58), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235703, 58), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 58) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.870-0400 s20016| 2018-03-16T17:28:23.866-0400 D SHARDING [conn1] dropCollection test.foo chunk data deleted [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.870-0400 s20016| 2018-03-16T17:28:23.866-0400 D ASIO [conn1] startCommand: RemoteCommand 61 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:53.866-0400 cmd:{ update: "collections", bypassDocumentValidation: false, ordered: true, updates: [ { q: { _id: "test.foo" }, u: { _id: "test.foo", lastmodEpoch: ObjectId('000000000000000000000000'), lastmod: new Date(1521235703866), dropped: true }, multi: false, upsert: false } ], writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.870-0400 s20016| 2018-03-16T17:28:23.866-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 61 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.871-0400 s20016| 2018-03-16T17:28:23.870-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 61 finished with response: { n: 1, nModified: 1, opTime: { ts: Timestamp(1521235703, 59), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235703, 59), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235703, 59), t: 1 }, lastOpVisible: { ts: Timestamp(1521235703, 59), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235703, 59), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235703, 59), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 59) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.871-0400 s20016| 2018-03-16T17:28:23.870-0400 D SHARDING [conn1] dropCollection test.foo collection marked as dropped [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.871-0400 s20016| 2018-03-16T17:28:23.870-0400 D ASIO [conn1] startCommand: RemoteCommand 63 -- target:ubuntu:20010 db:admin cmd:{ setShardVersion: "test.foo", init: false, authoritative: true, configdb: "stale_mongos_updates_and_removes-configRS/ubuntu:20012,ubuntu:20013,ubuntu:20014", shard: "stale_mongos_updates_and_removes-rs0", shardHost: "stale_mongos_updates_and_removes-rs0/ubuntu:20010", version: Timestamp(0, 0), versionEpoch: ObjectId('000000000000000000000000'), noConnectionVersioning: true } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.872-0400 s20016| 2018-03-16T17:28:23.870-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 63 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.872-0400 d20010| 2018-03-16T17:28:23.870-0400 I SHARDING [conn21] XXXX in set shard version cmd obj is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.872-0400 d20010| 2018-03-16T17:28:23.870-0400 I SHARDING [conn21] { setShardVersion: "test.foo", init: false, authoritative: true, configdb: "stale_mongos_updates_and_removes-configRS/ubuntu:20012,ubuntu:20013,ubuntu:20014", shard: "stale_mongos_updates_and_removes-rs0", shardHost: "stale_mongos_updates_and_removes-rs0/ubuntu:20010", version: Timestamp(0, 0), versionEpoch: ObjectId('000000000000000000000000'), noConnectionVersioning: true, $clusterTime: { clusterTime: Timestamp(1521235703, 59), signature: { hash: BinData(0, F5BAD1415D4FF72DCE994EFB8E61FE39EE018428), keyId: 6533657568122765331 } }, $client: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" }, mongos: { host: "ubuntu:20016", client: "127.0.0.1:50986", version: "3.6.3" } }, $configServerState: { opTime: { ts: Timestamp(1521235703, 59), t: 1 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.873-0400 d20010| 2018-03-16T17:28:23.870-0400 I SHARDING [conn21] XXXX requested version is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.873-0400 d20010| 2018-03-16T17:28:23.870-0400 I SHARDING [conn21] 0|0||000000000000000000000000 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.873-0400 s20016| 2018-03-16T17:28:23.870-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 63 finished with response: { oldVersion: Timestamp(0, 0), oldVersionEpoch: ObjectId('000000000000000000000000'), ok: 1.0, operationTime: Timestamp(1521235703, 30), $gleStats: { lastOpTime: { ts: Timestamp(1521235703, 30), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235703, 59), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 30), $configServerState: { opTime: { ts: Timestamp(1521235703, 59), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.873-0400 s20016| 2018-03-16T17:28:23.871-0400 D ASIO [conn1] startCommand: RemoteCommand 65 -- target:ubuntu:20010 db:admin cmd:{ unsetSharding: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.874-0400 s20016| 2018-03-16T17:28:23.871-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 65 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.874-0400 s20016| 2018-03-16T17:28:23.871-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 65 finished with response: { ok: 1.0, operationTime: Timestamp(1521235703, 30), $gleStats: { lastOpTime: { ts: Timestamp(1521235703, 30), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235703, 59), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 30), $configServerState: { opTime: { ts: Timestamp(1521235703, 59), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.874-0400 s20016| 2018-03-16T17:28:23.871-0400 D ASIO [conn1] startCommand: RemoteCommand 67 -- target:ubuntu:20011 db:admin cmd:{ setShardVersion: "test.foo", init: false, authoritative: true, configdb: "stale_mongos_updates_and_removes-configRS/ubuntu:20012,ubuntu:20013,ubuntu:20014", shard: "stale_mongos_updates_and_removes-rs1", shardHost: "stale_mongos_updates_and_removes-rs1/ubuntu:20011", version: Timestamp(0, 0), versionEpoch: ObjectId('000000000000000000000000'), noConnectionVersioning: true } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.875-0400 s20016| 2018-03-16T17:28:23.871-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 67 on host ubuntu:20011 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.875-0400 d20011| 2018-03-16T17:28:23.871-0400 I SHARDING [conn21] XXXX in set shard version cmd obj is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.876-0400 d20011| 2018-03-16T17:28:23.871-0400 I SHARDING [conn21] { setShardVersion: "test.foo", init: false, authoritative: true, configdb: "stale_mongos_updates_and_removes-configRS/ubuntu:20012,ubuntu:20013,ubuntu:20014", shard: "stale_mongos_updates_and_removes-rs1", shardHost: "stale_mongos_updates_and_removes-rs1/ubuntu:20011", version: Timestamp(0, 0), versionEpoch: ObjectId('000000000000000000000000'), noConnectionVersioning: true, $clusterTime: { clusterTime: Timestamp(1521235703, 59), signature: { hash: BinData(0, F5BAD1415D4FF72DCE994EFB8E61FE39EE018428), keyId: 6533657568122765331 } }, $client: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" }, mongos: { host: "ubuntu:20016", client: "127.0.0.1:50986", version: "3.6.3" } }, $configServerState: { opTime: { ts: Timestamp(1521235703, 59), t: 1 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.876-0400 d20011| 2018-03-16T17:28:23.871-0400 I SHARDING [conn21] XXXX requested version is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.876-0400 d20011| 2018-03-16T17:28:23.871-0400 I SHARDING [conn21] 0|0||000000000000000000000000 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.876-0400 d20011| 2018-03-16T17:28:23.871-0400 I SHARDING [conn21] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.876-0400 d20011| 2018-03-16T17:28:23.871-0400 I SHARDING [conn21] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.876-0400 d20011| 2018-03-16T17:28:23.871-0400 I SHARDING [conn21] xxx routing info needs refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.876-0400 d20011| 2018-03-16T17:28:23.871-0400 I SHARDING [conn21] xxx calling schedule collection refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.876-0400 d20011| 2018-03-16T17:28:23.871-0400 I SHARDING [conn21] xxx is incremental [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.876-0400 d20011| 2018-03-16T17:28:23.871-0400 I SHARDING [conn21] xxx existing routing info? [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.877-0400 d20011| 2018-03-16T17:28:23.871-0400 I SHARDING [conn21] 1|0||5aac36f788c1f30185b7aeba [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.877-0400 d20011| 2018-03-16T17:28:23.871-0400 I SHARDING [conn21] Refreshing chunks for collection test.foo based on version 1|0||5aac36f788c1f30185b7aeba [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.877-0400 d20011| 2018-03-16T17:28:23.871-0400 I SHARDING [conn21] xxx getting chunks since now [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.877-0400 d20011| 2018-03-16T17:28:23.871-0400 I SHARDING [conn21] xxx getting chunks since [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.877-0400 d20011| 2018-03-16T17:28:23.871-0400 I SHARDING [conn21] xxx waiting if !refreshNot [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.877-0400 d20011| 2018-03-16T17:28:23.871-0400 I SHARDING [ShardServerCatalogCacheLoader-0] xxx primary and calling get chunks since for primary [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.877-0400 d20011| 2018-03-16T17:28:23.871-0400 I SHARDING [ShardServerCatalogCacheLoader-0] version is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.878-0400 d20011| 2018-03-16T17:28:23.871-0400 I SHARDING [ShardServerCatalogCacheLoader-0] 1|0||5aac36f788c1f30185b7aeba [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.878-0400 d20011| 2018-03-16T17:28:23.872-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] XXXX namespace not found [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.883-0400 d20011| 2018-03-16T17:28:23.883-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Cache loader remotely refreshed for collection test.foo from version 1|0||5aac36f788c1f30185b7aeba and no metadata was found. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.883-0400 d20011| 2018-03-16T17:28:23.883-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx finsihing refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.883-0400 d20011| 2018-03-16T17:28:23.883-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh callback [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.884-0400 d20011| 2018-03-16T17:28:23.883-0400 I REPL [replication-0] Completing collection drop for test.system.drop.1521235703i57t1.foo with drop optime { ts: Timestamp(1521235703, 57), t: 1 } (notification optime: { ts: Timestamp(1521235703, 60), t: 1 }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.884-0400 d20011| 2018-03-16T17:28:23.883-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh completed [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.884-0400 d20011| 2018-03-16T17:28:23.883-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Refresh for collection test.foo took 11 ms and found the collection is not sharded [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.884-0400 d20011| 2018-03-16T17:28:23.883-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx marking as status ok [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.884-0400 d20011| 2018-03-16T17:28:23.883-0400 I COMMAND [ShardServerCatalogCacheLoader-0] CMD: drop config.cache.chunks.test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.884-0400 d20011| 2018-03-16T17:28:23.883-0400 I STORAGE [replication-0] Finishing collection drop for test.system.drop.1521235703i57t1.foo (9c8b21ea-e8a4-43a9-a011-2b25a28a947b). [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.885-0400 d20011| 2018-03-16T17:28:23.883-0400 I STORAGE [ShardServerCatalogCacheLoader-0] dropCollection: config.cache.chunks.test.foo (16ddefd7-5b03-4b97-a3e6-343031531217) - renaming to drop-pending collection: config.system.drop.1521235703i62t1.cache.chunks.test.foo with drop optime { ts: Timestamp(1521235703, 62), t: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.889-0400 d20011| 2018-03-16T17:28:23.889-0400 I SHARDING [conn21] Marking collection test.foo with collection version: 1|0||5aac36f788c1f30185b7aeba, shard version: 1|0||5aac36f788c1f30185b7aeba as unsharded [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.890-0400 s20016| 2018-03-16T17:28:23.889-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 67 finished with response: { oldVersion: Timestamp(0, 0), oldVersionEpoch: ObjectId('000000000000000000000000'), ok: 1.0, operationTime: Timestamp(1521235703, 62), $gleStats: { lastOpTime: { ts: Timestamp(1521235703, 57), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235703, 62), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 60), $configServerState: { opTime: { ts: Timestamp(1521235703, 59), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.890-0400 s20016| 2018-03-16T17:28:23.889-0400 D ASIO [conn1] startCommand: RemoteCommand 69 -- target:ubuntu:20011 db:admin cmd:{ unsetSharding: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.890-0400 s20016| 2018-03-16T17:28:23.889-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 69 on host ubuntu:20011 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.891-0400 s20016| 2018-03-16T17:28:23.889-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 69 finished with response: { ok: 1.0, operationTime: Timestamp(1521235703, 62), $gleStats: { lastOpTime: { ts: Timestamp(1521235703, 57), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235703, 62), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 60), $configServerState: { opTime: { ts: Timestamp(1521235703, 59), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.891-0400 s20016| 2018-03-16T17:28:23.889-0400 D SHARDING [conn1] dropCollection test.foo completed [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.891-0400 s20016| 2018-03-16T17:28:23.889-0400 I SHARDING [conn1] about to log metadata event into changelog: { _id: "ubuntu-2018-03-16T17:28:23.889-0400-5aac36f7ebcdfa1d2b617d86", server: "ubuntu", clientAddr: "127.0.0.1:50986", time: new Date(1521235703889), what: "dropCollection", ns: "test.foo", details: {} } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.892-0400 s20016| 2018-03-16T17:28:23.890-0400 D ASIO [conn1] startCommand: RemoteCommand 71 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:53.890-0400 cmd:{ insert: "changelog", bypassDocumentValidation: false, ordered: true, documents: [ { _id: "ubuntu-2018-03-16T17:28:23.889-0400-5aac36f7ebcdfa1d2b617d86", server: "ubuntu", clientAddr: "127.0.0.1:50986", time: new Date(1521235703889), what: "dropCollection", ns: "test.foo", details: {} } ], writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.892-0400 s20016| 2018-03-16T17:28:23.890-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 71 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.894-0400 s20016| 2018-03-16T17:28:23.893-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 71 finished with response: { n: 1, opTime: { ts: Timestamp(1521235703, 63), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235703, 63), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235703, 63), t: 1 }, lastOpVisible: { ts: Timestamp(1521235703, 63), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235703, 63), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235703, 63), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 63) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.894-0400 s20016| 2018-03-16T17:28:23.893-0400 D ASIO [conn1] startCommand: RemoteCommand 73 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:53.893-0400 cmd:{ findAndModify: "locks", query: { ts: ObjectId('5aac36f7ebcdfa1d2b617d84') }, update: { $set: { state: 0 } }, writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.894-0400 s20016| 2018-03-16T17:28:23.893-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 73 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.900-0400 s20016| 2018-03-16T17:28:23.899-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 73 finished with response: { lastErrorObject: { n: 1, updatedExisting: true }, value: { _id: "test.foo", state: 2, process: "ubuntu:20016:1521235700:-4633782234644071543", ts: ObjectId('5aac36f7ebcdfa1d2b617d84'), when: new Date(1521235703843), who: "ubuntu:20016:1521235700:-4633782234644071543:conn1", why: "dropCollection" }, ok: 1.0, operationTime: Timestamp(1521235703, 64), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235703, 64), t: 1 }, lastOpVisible: { ts: Timestamp(1521235703, 64), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235703, 64), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235703, 64), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 64) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.900-0400 s20016| 2018-03-16T17:28:23.899-0400 I SHARDING [conn1] distributed lock with ts: 5aac36f7ebcdfa1d2b617d84' unlocked. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.900-0400 s20016| 2018-03-16T17:28:23.899-0400 D ASIO [conn1] startCommand: RemoteCommand 75 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:53.899-0400 cmd:{ findAndModify: "locks", query: { ts: ObjectId('5aac36f7ebcdfa1d2b617d83') }, update: { $set: { state: 0 } }, writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.900-0400 s20016| 2018-03-16T17:28:23.899-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 75 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.904-0400 s20016| 2018-03-16T17:28:23.903-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 75 finished with response: { lastErrorObject: { n: 1, updatedExisting: true }, value: { _id: "test", state: 2, process: "ubuntu:20016:1521235700:-4633782234644071543", ts: ObjectId('5aac36f7ebcdfa1d2b617d83'), when: new Date(1521235703838), who: "ubuntu:20016:1521235700:-4633782234644071543:conn1", why: "dropCollection" }, ok: 1.0, operationTime: Timestamp(1521235703, 65), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235703, 65), t: 1 }, lastOpVisible: { ts: Timestamp(1521235703, 65), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235703, 65), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235703, 65), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 65) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.904-0400 s20016| 2018-03-16T17:28:23.903-0400 I SHARDING [conn1] distributed lock with ts: 5aac36f7ebcdfa1d2b617d83' unlocked. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.904-0400 s20016| 2018-03-16T17:28:23.903-0400 D ASIO [conn1] startCommand: RemoteCommand 77 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:53.903-0400 cmd:{ findAndModify: "locks", query: { ts: ObjectId('5aac36f7ebcdfa1d2b617d82') }, update: { $set: { state: 0 } }, writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.904-0400 s20016| 2018-03-16T17:28:23.903-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 77 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.907-0400 s20016| 2018-03-16T17:28:23.906-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 77 finished with response: { lastErrorObject: { n: 1, updatedExisting: true }, value: { _id: "test-movePrimary", state: 2, process: "ubuntu:20016:1521235700:-4633782234644071543", ts: ObjectId('5aac36f7ebcdfa1d2b617d82'), when: new Date(1521235703835), who: "ubuntu:20016:1521235700:-4633782234644071543:conn1", why: "dropCollection" }, ok: 1.0, operationTime: Timestamp(1521235703, 66), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235703, 66), t: 1 }, lastOpVisible: { ts: Timestamp(1521235703, 66), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235703, 66), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235703, 66), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 66) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.907-0400 s20016| 2018-03-16T17:28:23.906-0400 I SHARDING [conn1] distributed lock with ts: 5aac36f7ebcdfa1d2b617d82' unlocked. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.908-0400 s20016| 2018-03-16T17:28:23.907-0400 D ASIO [conn1] startCommand: RemoteCommand 79 -- target:ubuntu:20012 db:admin cmd:{ _configsvrShardCollection: "test.foo", key: { x: 1.0 }, unique: false, numInitialChunks: 0, writeConcern: { w: "majority", wtimeout: 60000 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.908-0400 s20016| 2018-03-16T17:28:23.907-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 79 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.921-0400 c20012| 2018-03-16T17:28:23.921-0400 I SHARDING [conn23] distributed lock 'test' acquired for 'shardCollection', ts : 5aac36f788c1f30185b7af7d [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.927-0400 c20012| 2018-03-16T17:28:23.926-0400 I SHARDING [conn23] distributed lock 'test.foo' acquired for 'shardCollection', ts : 5aac36f788c1f30185b7af86 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.927-0400 d20010| 2018-03-16T17:28:23.927-0400 I STORAGE [conn15] createCollection: test.foo with generated UUID: a26b230c-c773-414a-b498-fd1cdf23fe9f [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.951-0400 d20010| 2018-03-16T17:28:23.951-0400 I INDEX [conn15] build index on: test.foo properties: { v: 2, key: { x: 1.0 }, name: "x_1", ns: "test.foo" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.951-0400 d20010| 2018-03-16T17:28:23.951-0400 I INDEX [conn15] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.952-0400 d20010| 2018-03-16T17:28:23.952-0400 I INDEX [conn15] build index done. scanned 0 total records. 0 secs [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.955-0400 c20012| 2018-03-16T17:28:23.955-0400 I SHARDING [conn23] CMD: shardcollection: { _configsvrShardCollection: "test.foo", key: { x: 1.0 }, unique: false, numInitialChunks: 0, writeConcern: { w: "majority", wtimeout: 60000 }, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235703, 66), signature: { hash: BinData(0, F5BAD1415D4FF72DCE994EFB8E61FE39EE018428), keyId: 6533657568122765331 } }, $client: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" }, mongos: { host: "ubuntu:20016", client: "127.0.0.1:50986", version: "3.6.3" } }, $configServerState: { opTime: { ts: Timestamp(1521235703, 66), t: 1 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.956-0400 c20012| 2018-03-16T17:28:23.955-0400 I SHARDING [conn23] about to log metadata event into changelog: { _id: "ubuntu-2018-03-16T17:28:23.955-0400-5aac36f788c1f30185b7af95", server: "ubuntu", clientAddr: "127.0.0.1:60682", time: new Date(1521235703955), what: "shardCollection.start", ns: "test.foo", details: { shardKey: { x: 1.0 }, collection: "test.foo", uuid: UUID("a26b230c-c773-414a-b498-fd1cdf23fe9f"), primary: "stale_mongos_updates_and_removes-rs0:stale_mongos_updates_and_removes-rs0/ubuntu:20010", numChunks: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.959-0400 c20012| 2018-03-16T17:28:23.959-0400 I SHARDING [conn23] going to create 1 chunk(s) for: test.foo using new epoch 5aac36f788c1f30185b7afa3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.971-0400 d20010| 2018-03-16T17:28:23.971-0400 I SHARDING [conn15] XXXX in set shard version cmd obj is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.972-0400 d20010| 2018-03-16T17:28:23.971-0400 I SHARDING [conn15] { setShardVersion: "test.foo", init: false, authoritative: true, configdb: "stale_mongos_updates_and_removes-configRS/ubuntu:20012,ubuntu:20013,ubuntu:20014", shard: "stale_mongos_updates_and_removes-rs0", shardHost: "stale_mongos_updates_and_removes-rs0/ubuntu:20010", version: Timestamp(1, 0), versionEpoch: ObjectId('5aac36f788c1f30185b7afa3'), noConnectionVersioning: true, $clusterTime: { clusterTime: Timestamp(1521235703, 73), signature: { hash: BinData(0, F5BAD1415D4FF72DCE994EFB8E61FE39EE018428), keyId: 6533657568122765331 } }, $client: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" }, mongos: { host: "ubuntu:20016", client: "127.0.0.1:50986", version: "3.6.3" } }, $configServerState: { opTime: { ts: Timestamp(1521235703, 73), t: 1 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.972-0400 d20010| 2018-03-16T17:28:23.971-0400 I SHARDING [conn15] XXXX requested version is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.972-0400 d20010| 2018-03-16T17:28:23.971-0400 I SHARDING [conn15] 1|0||5aac36f788c1f30185b7afa3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.972-0400 d20010| 2018-03-16T17:28:23.971-0400 I SHARDING [conn15] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.972-0400 d20010| 2018-03-16T17:28:23.971-0400 I SHARDING [conn15] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.972-0400 d20010| 2018-03-16T17:28:23.971-0400 I SHARDING [conn15] xxx routing info needs refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.972-0400 d20010| 2018-03-16T17:28:23.971-0400 I SHARDING [conn15] xxx calling schedule collection refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.973-0400 d20010| 2018-03-16T17:28:23.972-0400 I SHARDING [conn15] xxx is not incr [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.973-0400 d20010| 2018-03-16T17:28:23.972-0400 I SHARDING [conn15] xxx existing routing info? [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.973-0400 d20010| 2018-03-16T17:28:23.972-0400 I SHARDING [conn15] Refreshing chunks for collection test.foo based on version 0|0||000000000000000000000000 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.973-0400 d20010| 2018-03-16T17:28:23.972-0400 I SHARDING [conn15] xxx getting chunks since now [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.973-0400 d20010| 2018-03-16T17:28:23.972-0400 I SHARDING [conn15] xxx getting chunks since [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.973-0400 d20010| 2018-03-16T17:28:23.972-0400 I SHARDING [conn15] xxx waiting if !refreshNot [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.973-0400 d20010| 2018-03-16T17:28:23.972-0400 I SHARDING [ShardServerCatalogCacheLoader-0] xxx primary and calling get chunks since for primary [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.974-0400 d20010| 2018-03-16T17:28:23.972-0400 I SHARDING [ShardServerCatalogCacheLoader-0] version is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.974-0400 d20010| 2018-03-16T17:28:23.972-0400 I SHARDING [ShardServerCatalogCacheLoader-0] 0|0||000000000000000000000000 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.974-0400 d20010| 2018-03-16T17:28:23.972-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] refresh in primary [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.974-0400 d20010| 2018-03-16T17:28:23.972-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] 1 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.974-0400 d20010| 2018-03-16T17:28:23.972-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx collec and changed chunks max epoch or version doesn't equal max [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.982-0400 d20011| 2018-03-16T17:28:23.982-0400 I REPL [replication-0] Completing collection drop for config.system.drop.1521235703i62t1.cache.chunks.test.foo with drop optime { ts: Timestamp(1521235703, 62), t: 1 } (notification optime: { ts: Timestamp(1521235703, 62), t: 1 }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.982-0400 d20011| 2018-03-16T17:28:23.982-0400 I STORAGE [replication-0] Finishing collection drop for config.system.drop.1521235703i62t1.cache.chunks.test.foo (16ddefd7-5b03-4b97-a3e6-343031531217). [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.983-0400 d20010| 2018-03-16T17:28:23.983-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Cache loader remotely refreshed for collection test.foo from collection version 0|0||000000000000000000000000 and found collection version 1|0||5aac36f788c1f30185b7afa3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.983-0400 d20010| 2018-03-16T17:28:23.983-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Cache loader found enqueued metadata from 1|0||5aac36f788c1f30185b7afa3 to 1|0||5aac36f788c1f30185b7afa3 and no persisted metadata, GTE cache version 0|0||000000000000000000000000 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.983-0400 d20010| 2018-03-16T17:28:23.983-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx is okay [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.984-0400 d20010| 2018-03-16T17:28:23.983-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx finsihing refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.984-0400 d20010| 2018-03-16T17:28:23.983-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh callback [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.984-0400 d20010| 2018-03-16T17:28:23.983-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh completed [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.984-0400 d20010| 2018-03-16T17:28:23.983-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Refresh for collection test.foo took 11 ms and found version 1|0||5aac36f788c1f30185b7afa3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.984-0400 d20010| 2018-03-16T17:28:23.983-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx marking as status ok [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.984-0400 d20010| 2018-03-16T17:28:23.983-0400 I SHARDING [conn15] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.984-0400 d20010| 2018-03-16T17:28:23.983-0400 I SHARDING [conn15] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.985-0400 d20010| 2018-03-16T17:28:23.983-0400 I SHARDING [conn15] Marking collection test.foo as sharded with collection version: 1|0||5aac36f788c1f30185b7afa3, shard version: 1|0||5aac36f788c1f30185b7afa3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.985-0400 d20010| 2018-03-16T17:28:23.983-0400 I STORAGE [ShardServerCatalogCacheLoader-0] createCollection: config.cache.chunks.test.foo with generated UUID: 4ca2771c-4d81-478a-b7c8-92920f3c4431 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.985-0400 c20012| 2018-03-16T17:28:23.983-0400 I SHARDING [conn23] about to log metadata event into changelog: { _id: "ubuntu-2018-03-16T17:28:23.983-0400-5aac36f788c1f30185b7afb6", server: "ubuntu", clientAddr: "127.0.0.1:60682", time: new Date(1521235703983), what: "shardCollection.end", ns: "test.foo", details: { version: "1|0||5aac36f788c1f30185b7afa3" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:23.998-0400 c20012| 2018-03-16T17:28:23.998-0400 I SHARDING [conn23] distributed lock with ts: 5aac36f788c1f30185b7af86' unlocked. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.011-0400 c20012| 2018-03-16T17:28:24.011-0400 I SHARDING [conn23] distributed lock with ts: 5aac36f788c1f30185b7af7d' unlocked. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.012-0400 c20012| 2018-03-16T17:28:24.011-0400 I COMMAND [conn23] command admin.$cmd appName: "MongoDB Shell" command: _configsvrShardCollection { _configsvrShardCollection: "test.foo", key: { x: 1.0 }, unique: false, numInitialChunks: 0, writeConcern: { w: "majority", wtimeout: 60000 }, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235703, 66), signature: { hash: BinData(0, F5BAD1415D4FF72DCE994EFB8E61FE39EE018428), keyId: 6533657568122765331 } }, $client: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" }, mongos: { host: "ubuntu:20016", client: "127.0.0.1:50986", version: "3.6.3" } }, $configServerState: { opTime: { ts: Timestamp(1521235703, 66), t: 1 } }, $db: "admin" } numYields:0 reslen:531 locks:{ Global: { acquireCount: { r: 30, w: 16 } }, Database: { acquireCount: { r: 7, w: 16 } }, Collection: { acquireCount: { r: 7, w: 8 } }, Metadata: { acquireCount: { W: 2 } }, oplog: { acquireCount: { w: 8 } } } protocol:op_msg 103ms [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.012-0400 s20016| 2018-03-16T17:28:24.011-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 79 finished with response: { collectionsharded: "test.foo", collectionUUID: UUID("a26b230c-c773-414a-b498-fd1cdf23fe9f"), ok: 1.0, operationTime: Timestamp(1521235703, 79), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235703, 79), t: 1 }, lastOpVisible: { ts: Timestamp(1521235703, 79), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235703, 79), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235703, 79), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 79) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.013-0400 s20016| 2018-03-16T17:28:24.012-0400 I SHARDING [conn1] Refreshing chunks for collection test.foo based on version 1|0||5aac36f788c1f30185b7aeba [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.013-0400 s20016| 2018-03-16T17:28:24.012-0400 D ASIO [ConfigServerCatalogCacheLoader-0] startCommand: RemoteCommand 81 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:54.012-0400 cmd:{ find: "collections", filter: { _id: "test.foo" }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235703, 79), t: 1 } }, limit: 1, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.013-0400 s20016| 2018-03-16T17:28:24.012-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 81 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.014-0400 s20016| 2018-03-16T17:28:24.012-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 81 finished with response: { cursor: { firstBatch: [ { _id: "test.foo", lastmodEpoch: ObjectId('5aac36f788c1f30185b7afa3'), lastmod: new Date(4294967296), dropped: false, key: { x: 1.0 }, unique: false, uuid: UUID("a26b230c-c773-414a-b498-fd1cdf23fe9f") } ], id: 0, ns: "config.collections" }, ok: 1.0, operationTime: Timestamp(1521235703, 79), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235703, 79), t: 1 }, lastOpVisible: { ts: Timestamp(1521235703, 79), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235703, 79), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235703, 79), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 79) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.014-0400 s20016| 2018-03-16T17:28:24.012-0400 D ASIO [ConfigServerCatalogCacheLoader-0] startCommand: RemoteCommand 83 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:54.012-0400 cmd:{ find: "chunks", filter: { ns: "test.foo", lastmod: { $gte: Timestamp(0, 0) } }, sort: { lastmod: 1 }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235703, 79), t: 1 } }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.015-0400 s20016| 2018-03-16T17:28:24.012-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 83 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.015-0400 s20016| 2018-03-16T17:28:24.013-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 83 finished with response: { cursor: { firstBatch: [ { _id: "test.foo-x_MinKey", ns: "test.foo", min: { x: MinKey }, max: { x: MaxKey }, shard: "stale_mongos_updates_and_removes-rs0", lastmod: Timestamp(1, 0), lastmodEpoch: ObjectId('5aac36f788c1f30185b7afa3'), history: [ { validAfter: Timestamp(1521235703, 71), shard: "stale_mongos_updates_and_removes-rs0" } ] } ], id: 0, ns: "config.chunks" }, ok: 1.0, operationTime: Timestamp(1521235703, 79), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235703, 79), t: 1 }, lastOpVisible: { ts: Timestamp(1521235703, 79), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235703, 79), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235703, 79), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 79) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.016-0400 s20016| 2018-03-16T17:28:24.013-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Refresh for collection test.foo took 0 ms and found version 1|0||5aac36f788c1f30185b7afa3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.016-0400 s20016| 2018-03-16T17:28:24.013-0400 D ASIO [conn1] startCommand: RemoteCommand 85 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36f81085bf286ecf4d0e'), x: 0.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f788c1f30185b7afa3') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.016-0400 s20016| 2018-03-16T17:28:24.013-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-0-0] Connecting to ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.016-0400 s20016| 2018-03-16T17:28:24.013-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-0-0] Starting asynchronous command 86 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.016-0400 d20010| 2018-03-16T17:28:24.013-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56448 #22 (13 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.017-0400 d20010| 2018-03-16T17:28:24.013-0400 I NETWORK [conn22] received client metadata from 127.0.0.1:56448 conn22: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-0", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.017-0400 s20016| 2018-03-16T17:28:24.013-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-0-0] Successfully connected to ubuntu:20010, took 0ms (1 connections now open to ubuntu:20010) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.017-0400 s20016| 2018-03-16T17:28:24.013-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-0-0] Request 86 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.017-0400 s20016| 2018-03-16T17:28:24.013-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-0-0] Starting asynchronous command 85 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.020-0400 d20010| 2018-03-16T17:28:24.020-0400 I INDEX [ShardServerCatalogCacheLoader-0] build index on: config.cache.chunks.test.foo properties: { v: 2, key: { lastmod: 1 }, name: "lastmod_1", ns: "config.cache.chunks.test.foo" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.020-0400 d20010| 2018-03-16T17:28:24.020-0400 I INDEX [ShardServerCatalogCacheLoader-0] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.021-0400 s20016| 2018-03-16T17:28:24.020-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-0-0] Request 85 finished with response: { n: 1, opTime: { ts: Timestamp(1521235704, 2), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235704, 2), $gleStats: { lastOpTime: { ts: Timestamp(1521235704, 2), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235704, 2), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 74), $configServerState: { opTime: { ts: Timestamp(1521235703, 79), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.021-0400 d20010| 2018-03-16T17:28:24.020-0400 I INDEX [ShardServerCatalogCacheLoader-0] build index done. scanned 0 total records. 0 secs [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.021-0400 s20016| 2018-03-16T17:28:24.021-0400 D ASIO [conn1] startCommand: RemoteCommand 88 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36f81085bf286ecf4d0f'), x: 0.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f788c1f30185b7afa3') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.021-0400 s20016| 2018-03-16T17:28:24.021-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-1-0] Connecting to ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.021-0400 s20016| 2018-03-16T17:28:24.021-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-1-0] Starting asynchronous command 89 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.022-0400 d20010| 2018-03-16T17:28:24.021-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56450 #23 (14 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.022-0400 d20010| 2018-03-16T17:28:24.021-0400 I NETWORK [conn23] received client metadata from 127.0.0.1:56450 conn23: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-1", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.022-0400 s20016| 2018-03-16T17:28:24.021-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-1-0] Successfully connected to ubuntu:20010, took 0ms (1 connections now open to ubuntu:20010) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.022-0400 s20016| 2018-03-16T17:28:24.021-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-1-0] Request 89 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.022-0400 s20016| 2018-03-16T17:28:24.021-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-1-0] Starting asynchronous command 88 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.023-0400 s20016| 2018-03-16T17:28:24.021-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-1-0] Request 88 finished with response: { n: 1, opTime: { ts: Timestamp(1521235704, 6), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235704, 6), $gleStats: { lastOpTime: { ts: Timestamp(1521235704, 6), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235704, 6), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 74), $configServerState: { opTime: { ts: Timestamp(1521235703, 79), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.023-0400 s20016| 2018-03-16T17:28:24.022-0400 D ASIO [conn1] startCommand: RemoteCommand 91 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36f81085bf286ecf4d10'), x: 1.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f788c1f30185b7afa3') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.023-0400 s20016| 2018-03-16T17:28:24.022-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-2-0] Connecting to ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.024-0400 s20016| 2018-03-16T17:28:24.022-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-2-0] Starting asynchronous command 92 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.024-0400 d20010| 2018-03-16T17:28:24.022-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56452 #24 (15 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.024-0400 d20010| 2018-03-16T17:28:24.023-0400 I NETWORK [conn24] received client metadata from 127.0.0.1:56452 conn24: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-2", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.024-0400 s20016| 2018-03-16T17:28:24.023-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-2-0] Successfully connected to ubuntu:20010, took 1ms (1 connections now open to ubuntu:20010) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.024-0400 s20016| 2018-03-16T17:28:24.023-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-2-0] Request 92 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.024-0400 s20016| 2018-03-16T17:28:24.023-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-2-0] Starting asynchronous command 91 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.025-0400 s20016| 2018-03-16T17:28:24.023-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-2-0] Request 91 finished with response: { n: 1, opTime: { ts: Timestamp(1521235704, 7), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235704, 7), $gleStats: { lastOpTime: { ts: Timestamp(1521235704, 7), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235704, 7), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 74), $configServerState: { opTime: { ts: Timestamp(1521235703, 79), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.025-0400 s20016| 2018-03-16T17:28:24.024-0400 D ASIO [conn1] startCommand: RemoteCommand 94 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36f81085bf286ecf4d11'), x: 1.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f788c1f30185b7afa3') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.025-0400 s20016| 2018-03-16T17:28:24.024-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-3-0] Connecting to ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.042-0400 s20016| 2018-03-16T17:28:24.042-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-3-0] Starting asynchronous command 95 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.043-0400 d20010| 2018-03-16T17:28:24.042-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56454 #25 (16 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.043-0400 d20010| 2018-03-16T17:28:24.042-0400 I NETWORK [conn25] received client metadata from 127.0.0.1:56454 conn25: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-3", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.043-0400 s20016| 2018-03-16T17:28:24.043-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-3-0] Successfully connected to ubuntu:20010, took 19ms (1 connections now open to ubuntu:20010) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.043-0400 s20016| 2018-03-16T17:28:24.043-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-3-0] Request 95 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.043-0400 s20016| 2018-03-16T17:28:24.043-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-3-0] Starting asynchronous command 94 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.044-0400 s20016| 2018-03-16T17:28:24.043-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-3-0] Request 94 finished with response: { n: 1, opTime: { ts: Timestamp(1521235704, 8), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235704, 8), $gleStats: { lastOpTime: { ts: Timestamp(1521235704, 8), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235704, 8), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 74), $configServerState: { opTime: { ts: Timestamp(1521235703, 79), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.044-0400 s20016| 2018-03-16T17:28:24.044-0400 D ASIO [conn1] startCommand: RemoteCommand 97 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36f81085bf286ecf4d12'), x: 2.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f788c1f30185b7afa3') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.044-0400 s20016| 2018-03-16T17:28:24.044-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-4-0] Connecting to ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.045-0400 s20016| 2018-03-16T17:28:24.044-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-4-0] Starting asynchronous command 98 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.045-0400 d20010| 2018-03-16T17:28:24.044-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56456 #26 (17 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.045-0400 d20010| 2018-03-16T17:28:24.044-0400 I NETWORK [conn26] received client metadata from 127.0.0.1:56456 conn26: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-4", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.045-0400 s20016| 2018-03-16T17:28:24.044-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-4-0] Successfully connected to ubuntu:20010, took 0ms (1 connections now open to ubuntu:20010) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.045-0400 s20016| 2018-03-16T17:28:24.044-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-4-0] Request 98 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.046-0400 s20016| 2018-03-16T17:28:24.044-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-4-0] Starting asynchronous command 97 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.046-0400 s20016| 2018-03-16T17:28:24.044-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-4-0] Request 97 finished with response: { n: 1, opTime: { ts: Timestamp(1521235704, 9), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235704, 9), $gleStats: { lastOpTime: { ts: Timestamp(1521235704, 9), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235704, 9), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 7), $configServerState: { opTime: { ts: Timestamp(1521235703, 79), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.046-0400 s20016| 2018-03-16T17:28:24.045-0400 D ASIO [conn1] startCommand: RemoteCommand 100 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36f81085bf286ecf4d13'), x: 2.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f788c1f30185b7afa3') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.047-0400 s20016| 2018-03-16T17:28:24.045-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-5-0] Connecting to ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.047-0400 s20016| 2018-03-16T17:28:24.046-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-5-0] Starting asynchronous command 101 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.047-0400 d20010| 2018-03-16T17:28:24.046-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56458 #27 (18 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.047-0400 d20010| 2018-03-16T17:28:24.046-0400 I NETWORK [conn27] received client metadata from 127.0.0.1:56458 conn27: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-5", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.047-0400 s20016| 2018-03-16T17:28:24.046-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-5-0] Successfully connected to ubuntu:20010, took 1ms (1 connections now open to ubuntu:20010) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.048-0400 s20016| 2018-03-16T17:28:24.046-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-5-0] Request 101 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.048-0400 s20016| 2018-03-16T17:28:24.046-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-5-0] Starting asynchronous command 100 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.048-0400 s20016| 2018-03-16T17:28:24.046-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-5-0] Request 100 finished with response: { n: 1, opTime: { ts: Timestamp(1521235704, 10), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235704, 10), $gleStats: { lastOpTime: { ts: Timestamp(1521235704, 10), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235704, 10), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 7), $configServerState: { opTime: { ts: Timestamp(1521235703, 79), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.049-0400 s20016| 2018-03-16T17:28:24.047-0400 D ASIO [conn1] startCommand: RemoteCommand 103 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36f81085bf286ecf4d14'), x: 3.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f788c1f30185b7afa3') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.049-0400 s20016| 2018-03-16T17:28:24.047-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-6-0] Connecting to ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.049-0400 s20016| 2018-03-16T17:28:24.047-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-6-0] Starting asynchronous command 104 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.049-0400 d20010| 2018-03-16T17:28:24.047-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56460 #28 (19 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.049-0400 d20010| 2018-03-16T17:28:24.047-0400 I NETWORK [conn28] received client metadata from 127.0.0.1:56460 conn28: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-6", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.049-0400 s20016| 2018-03-16T17:28:24.048-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-6-0] Successfully connected to ubuntu:20010, took 1ms (1 connections now open to ubuntu:20010) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.050-0400 s20016| 2018-03-16T17:28:24.048-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-6-0] Request 104 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.050-0400 s20016| 2018-03-16T17:28:24.048-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-6-0] Starting asynchronous command 103 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.050-0400 s20016| 2018-03-16T17:28:24.048-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-6-0] Request 103 finished with response: { n: 1, opTime: { ts: Timestamp(1521235704, 11), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235704, 11), $gleStats: { lastOpTime: { ts: Timestamp(1521235704, 11), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235704, 11), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 7), $configServerState: { opTime: { ts: Timestamp(1521235703, 79), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.051-0400 s20016| 2018-03-16T17:28:24.049-0400 D ASIO [conn1] startCommand: RemoteCommand 106 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36f81085bf286ecf4d15'), x: 3.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f788c1f30185b7afa3') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.051-0400 s20016| 2018-03-16T17:28:24.049-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-7-0] Connecting to ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.051-0400 s20016| 2018-03-16T17:28:24.049-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-7-0] Starting asynchronous command 107 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.051-0400 d20010| 2018-03-16T17:28:24.049-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56462 #29 (20 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.051-0400 d20010| 2018-03-16T17:28:24.049-0400 I NETWORK [conn29] received client metadata from 127.0.0.1:56462 conn29: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-7", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.052-0400 s20016| 2018-03-16T17:28:24.049-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-7-0] Successfully connected to ubuntu:20010, took 0ms (1 connections now open to ubuntu:20010) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.052-0400 s20016| 2018-03-16T17:28:24.049-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-7-0] Request 107 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.052-0400 s20016| 2018-03-16T17:28:24.049-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-7-0] Starting asynchronous command 106 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.052-0400 s20016| 2018-03-16T17:28:24.049-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-7-0] Request 106 finished with response: { n: 1, opTime: { ts: Timestamp(1521235704, 12), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235704, 12), $gleStats: { lastOpTime: { ts: Timestamp(1521235704, 12), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235704, 12), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 7), $configServerState: { opTime: { ts: Timestamp(1521235703, 79), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.053-0400 s20016| 2018-03-16T17:28:24.050-0400 D ASIO [conn1] startCommand: RemoteCommand 109 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36f81085bf286ecf4d16'), x: 4.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f788c1f30185b7afa3') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.053-0400 s20016| 2018-03-16T17:28:24.050-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-8-0] Connecting to ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.053-0400 s20016| 2018-03-16T17:28:24.051-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-8-0] Starting asynchronous command 110 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.053-0400 d20010| 2018-03-16T17:28:24.051-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56464 #30 (21 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.053-0400 d20010| 2018-03-16T17:28:24.051-0400 I NETWORK [conn30] received client metadata from 127.0.0.1:56464 conn30: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-8", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.054-0400 s20016| 2018-03-16T17:28:24.051-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-8-0] Successfully connected to ubuntu:20010, took 1ms (1 connections now open to ubuntu:20010) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.054-0400 s20016| 2018-03-16T17:28:24.051-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-8-0] Request 110 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.054-0400 s20016| 2018-03-16T17:28:24.051-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-8-0] Starting asynchronous command 109 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.054-0400 s20016| 2018-03-16T17:28:24.051-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-8-0] Request 109 finished with response: { n: 1, opTime: { ts: Timestamp(1521235704, 13), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235704, 13), $gleStats: { lastOpTime: { ts: Timestamp(1521235704, 13), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235704, 13), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 7), $configServerState: { opTime: { ts: Timestamp(1521235703, 79), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.055-0400 s20016| 2018-03-16T17:28:24.052-0400 D ASIO [conn1] startCommand: RemoteCommand 112 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36f81085bf286ecf4d17'), x: 4.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f788c1f30185b7afa3') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.055-0400 s20016| 2018-03-16T17:28:24.052-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-9-0] Connecting to ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.055-0400 s20016| 2018-03-16T17:28:24.052-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-9-0] Starting asynchronous command 113 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.055-0400 d20010| 2018-03-16T17:28:24.052-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56466 #31 (22 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.055-0400 d20010| 2018-03-16T17:28:24.052-0400 I NETWORK [conn31] received client metadata from 127.0.0.1:56466 conn31: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-9", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.056-0400 s20016| 2018-03-16T17:28:24.052-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-9-0] Successfully connected to ubuntu:20010, took 0ms (1 connections now open to ubuntu:20010) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.056-0400 s20016| 2018-03-16T17:28:24.052-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-9-0] Request 113 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.056-0400 s20016| 2018-03-16T17:28:24.052-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-9-0] Starting asynchronous command 112 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.057-0400 s20016| 2018-03-16T17:28:24.053-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-9-0] Request 112 finished with response: { n: 1, opTime: { ts: Timestamp(1521235704, 14), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235704, 14), $gleStats: { lastOpTime: { ts: Timestamp(1521235704, 14), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235704, 14), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 7), $configServerState: { opTime: { ts: Timestamp(1521235703, 79), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.057-0400 s20016| 2018-03-16T17:28:24.053-0400 D ASIO [conn1] startCommand: RemoteCommand 115 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36f81085bf286ecf4d18'), x: 5.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f788c1f30185b7afa3') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.057-0400 s20016| 2018-03-16T17:28:24.054-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-10-0] Connecting to ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.057-0400 s20016| 2018-03-16T17:28:24.054-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-10-0] Starting asynchronous command 116 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.057-0400 d20010| 2018-03-16T17:28:24.054-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56468 #32 (23 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.058-0400 d20010| 2018-03-16T17:28:24.054-0400 I NETWORK [conn32] received client metadata from 127.0.0.1:56468 conn32: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-10", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.058-0400 s20016| 2018-03-16T17:28:24.054-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-10-0] Successfully connected to ubuntu:20010, took 1ms (1 connections now open to ubuntu:20010) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.058-0400 s20016| 2018-03-16T17:28:24.054-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-10-0] Request 116 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.058-0400 s20016| 2018-03-16T17:28:24.054-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-10-0] Starting asynchronous command 115 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.059-0400 s20016| 2018-03-16T17:28:24.054-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-10-0] Request 115 finished with response: { n: 1, opTime: { ts: Timestamp(1521235704, 15), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235704, 15), $gleStats: { lastOpTime: { ts: Timestamp(1521235704, 15), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235704, 15), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 7), $configServerState: { opTime: { ts: Timestamp(1521235703, 79), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.059-0400 s20016| 2018-03-16T17:28:24.055-0400 D ASIO [conn1] startCommand: RemoteCommand 118 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36f81085bf286ecf4d19'), x: 5.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f788c1f30185b7afa3') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.059-0400 s20016| 2018-03-16T17:28:24.055-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-11-0] Connecting to ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.059-0400 s20016| 2018-03-16T17:28:24.055-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-11-0] Starting asynchronous command 119 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.059-0400 d20010| 2018-03-16T17:28:24.055-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56470 #33 (24 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.060-0400 d20010| 2018-03-16T17:28:24.056-0400 I NETWORK [conn33] received client metadata from 127.0.0.1:56470 conn33: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-11", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.060-0400 s20016| 2018-03-16T17:28:24.056-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-11-0] Successfully connected to ubuntu:20010, took 1ms (1 connections now open to ubuntu:20010) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.060-0400 s20016| 2018-03-16T17:28:24.056-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-11-0] Request 119 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.060-0400 s20016| 2018-03-16T17:28:24.056-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-11-0] Starting asynchronous command 118 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.061-0400 s20016| 2018-03-16T17:28:24.056-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-11-0] Request 118 finished with response: { n: 1, opTime: { ts: Timestamp(1521235704, 16), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235704, 16), $gleStats: { lastOpTime: { ts: Timestamp(1521235704, 16), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235704, 16), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 7), $configServerState: { opTime: { ts: Timestamp(1521235703, 79), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.061-0400 s20016| 2018-03-16T17:28:24.057-0400 D ASIO [conn1] startCommand: RemoteCommand 121 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36f81085bf286ecf4d1a'), x: 6.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f788c1f30185b7afa3') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.061-0400 s20016| 2018-03-16T17:28:24.057-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-12-0] Connecting to ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.061-0400 s20016| 2018-03-16T17:28:24.057-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-12-0] Starting asynchronous command 122 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.061-0400 d20010| 2018-03-16T17:28:24.057-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56472 #34 (25 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.062-0400 d20010| 2018-03-16T17:28:24.057-0400 I NETWORK [conn34] received client metadata from 127.0.0.1:56472 conn34: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-12", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.062-0400 s20016| 2018-03-16T17:28:24.057-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-12-0] Successfully connected to ubuntu:20010, took 0ms (1 connections now open to ubuntu:20010) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.062-0400 s20016| 2018-03-16T17:28:24.057-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-12-0] Request 122 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.062-0400 s20016| 2018-03-16T17:28:24.058-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-12-0] Starting asynchronous command 121 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.063-0400 s20016| 2018-03-16T17:28:24.058-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-12-0] Request 121 finished with response: { n: 1, opTime: { ts: Timestamp(1521235704, 17), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235704, 17), $gleStats: { lastOpTime: { ts: Timestamp(1521235704, 17), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235704, 17), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 7), $configServerState: { opTime: { ts: Timestamp(1521235703, 79), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.063-0400 s20016| 2018-03-16T17:28:24.059-0400 D ASIO [conn1] startCommand: RemoteCommand 124 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36f81085bf286ecf4d1b'), x: 6.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f788c1f30185b7afa3') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.063-0400 s20016| 2018-03-16T17:28:24.059-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-13-0] Connecting to ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.063-0400 s20016| 2018-03-16T17:28:24.059-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-13-0] Starting asynchronous command 125 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.063-0400 d20010| 2018-03-16T17:28:24.059-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56474 #35 (26 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.064-0400 d20010| 2018-03-16T17:28:24.059-0400 I NETWORK [conn35] received client metadata from 127.0.0.1:56474 conn35: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-13", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.064-0400 s20016| 2018-03-16T17:28:24.059-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-13-0] Successfully connected to ubuntu:20010, took 0ms (1 connections now open to ubuntu:20010) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.064-0400 s20016| 2018-03-16T17:28:24.059-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-13-0] Request 125 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.064-0400 s20016| 2018-03-16T17:28:24.059-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-13-0] Starting asynchronous command 124 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.065-0400 s20016| 2018-03-16T17:28:24.059-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-13-0] Request 124 finished with response: { n: 1, opTime: { ts: Timestamp(1521235704, 18), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235704, 18), $gleStats: { lastOpTime: { ts: Timestamp(1521235704, 18), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235704, 18), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 7), $configServerState: { opTime: { ts: Timestamp(1521235703, 79), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.065-0400 s20016| 2018-03-16T17:28:24.060-0400 D ASIO [conn1] startCommand: RemoteCommand 127 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36f81085bf286ecf4d1c'), x: 7.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f788c1f30185b7afa3') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.065-0400 s20016| 2018-03-16T17:28:24.060-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-14-0] Connecting to ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.065-0400 s20016| 2018-03-16T17:28:24.061-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-14-0] Starting asynchronous command 128 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.065-0400 d20010| 2018-03-16T17:28:24.061-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56476 #36 (27 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.066-0400 d20010| 2018-03-16T17:28:24.061-0400 I NETWORK [conn36] received client metadata from 127.0.0.1:56476 conn36: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-14", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.066-0400 s20016| 2018-03-16T17:28:24.061-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-14-0] Successfully connected to ubuntu:20010, took 1ms (1 connections now open to ubuntu:20010) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.066-0400 s20016| 2018-03-16T17:28:24.061-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-14-0] Request 128 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.066-0400 s20016| 2018-03-16T17:28:24.061-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-14-0] Starting asynchronous command 127 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.067-0400 s20016| 2018-03-16T17:28:24.061-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-14-0] Request 127 finished with response: { n: 1, opTime: { ts: Timestamp(1521235704, 19), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235704, 19), $gleStats: { lastOpTime: { ts: Timestamp(1521235704, 19), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235704, 19), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 7), $configServerState: { opTime: { ts: Timestamp(1521235703, 79), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.067-0400 s20016| 2018-03-16T17:28:24.062-0400 D ASIO [conn1] startCommand: RemoteCommand 130 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36f81085bf286ecf4d1d'), x: 7.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f788c1f30185b7afa3') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.067-0400 s20016| 2018-03-16T17:28:24.062-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-15-0] Connecting to ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.068-0400 s20016| 2018-03-16T17:28:24.062-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-15-0] Starting asynchronous command 131 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.068-0400 d20010| 2018-03-16T17:28:24.062-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56478 #37 (28 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.068-0400 d20010| 2018-03-16T17:28:24.062-0400 I NETWORK [conn37] received client metadata from 127.0.0.1:56478 conn37: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-15", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.068-0400 s20016| 2018-03-16T17:28:24.063-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-15-0] Successfully connected to ubuntu:20010, took 0ms (1 connections now open to ubuntu:20010) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.068-0400 s20016| 2018-03-16T17:28:24.063-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-15-0] Request 131 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.069-0400 s20016| 2018-03-16T17:28:24.063-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-15-0] Starting asynchronous command 130 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.069-0400 s20016| 2018-03-16T17:28:24.063-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-15-0] Request 130 finished with response: { n: 1, opTime: { ts: Timestamp(1521235704, 20), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235704, 20), $gleStats: { lastOpTime: { ts: Timestamp(1521235704, 20), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235704, 20), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 7), $configServerState: { opTime: { ts: Timestamp(1521235703, 79), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.069-0400 s20016| 2018-03-16T17:28:24.064-0400 D ASIO [conn1] startCommand: RemoteCommand 133 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36f81085bf286ecf4d1e'), x: 8.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f788c1f30185b7afa3') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.070-0400 s20016| 2018-03-16T17:28:24.064-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-16-0] Connecting to ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.070-0400 s20016| 2018-03-16T17:28:24.064-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-16-0] Starting asynchronous command 134 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.070-0400 d20010| 2018-03-16T17:28:24.064-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56480 #38 (29 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.070-0400 d20010| 2018-03-16T17:28:24.064-0400 I NETWORK [conn38] received client metadata from 127.0.0.1:56480 conn38: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-16", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.070-0400 s20016| 2018-03-16T17:28:24.064-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-16-0] Successfully connected to ubuntu:20010, took 0ms (1 connections now open to ubuntu:20010) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.070-0400 s20016| 2018-03-16T17:28:24.064-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-16-0] Request 134 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.071-0400 s20016| 2018-03-16T17:28:24.064-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-16-0] Starting asynchronous command 133 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.071-0400 s20016| 2018-03-16T17:28:24.064-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-16-0] Request 133 finished with response: { n: 1, opTime: { ts: Timestamp(1521235704, 21), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235704, 21), $gleStats: { lastOpTime: { ts: Timestamp(1521235704, 21), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235704, 21), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 7), $configServerState: { opTime: { ts: Timestamp(1521235703, 79), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.072-0400 s20016| 2018-03-16T17:28:24.065-0400 D ASIO [conn1] startCommand: RemoteCommand 136 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36f81085bf286ecf4d1f'), x: 8.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f788c1f30185b7afa3') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.072-0400 s20016| 2018-03-16T17:28:24.065-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-17-0] Connecting to ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.072-0400 s20016| 2018-03-16T17:28:24.066-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-17-0] Starting asynchronous command 137 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.072-0400 d20010| 2018-03-16T17:28:24.066-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56482 #39 (30 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.072-0400 d20010| 2018-03-16T17:28:24.066-0400 I NETWORK [conn39] received client metadata from 127.0.0.1:56482 conn39: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-17", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.072-0400 s20016| 2018-03-16T17:28:24.066-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-17-0] Successfully connected to ubuntu:20010, took 1ms (1 connections now open to ubuntu:20010) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.073-0400 s20016| 2018-03-16T17:28:24.066-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-17-0] Request 137 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.073-0400 s20016| 2018-03-16T17:28:24.066-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-17-0] Starting asynchronous command 136 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.073-0400 s20016| 2018-03-16T17:28:24.066-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-17-0] Request 136 finished with response: { n: 1, opTime: { ts: Timestamp(1521235704, 22), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235704, 22), $gleStats: { lastOpTime: { ts: Timestamp(1521235704, 22), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235704, 22), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 7), $configServerState: { opTime: { ts: Timestamp(1521235703, 79), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.074-0400 s20016| 2018-03-16T17:28:24.067-0400 D ASIO [conn1] startCommand: RemoteCommand 139 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36f81085bf286ecf4d20'), x: 9.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f788c1f30185b7afa3') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.074-0400 s20016| 2018-03-16T17:28:24.067-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-18-0] Connecting to ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.074-0400 s20016| 2018-03-16T17:28:24.067-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-18-0] Starting asynchronous command 140 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.074-0400 d20010| 2018-03-16T17:28:24.067-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56484 #40 (31 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.074-0400 d20010| 2018-03-16T17:28:24.068-0400 I NETWORK [conn40] received client metadata from 127.0.0.1:56484 conn40: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-18", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.075-0400 s20016| 2018-03-16T17:28:24.068-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-18-0] Successfully connected to ubuntu:20010, took 1ms (1 connections now open to ubuntu:20010) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.075-0400 s20016| 2018-03-16T17:28:24.068-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-18-0] Request 140 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.075-0400 s20016| 2018-03-16T17:28:24.068-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-18-0] Starting asynchronous command 139 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.075-0400 s20016| 2018-03-16T17:28:24.068-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-18-0] Request 139 finished with response: { n: 1, opTime: { ts: Timestamp(1521235704, 23), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235704, 23), $gleStats: { lastOpTime: { ts: Timestamp(1521235704, 23), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235704, 23), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 7), $configServerState: { opTime: { ts: Timestamp(1521235703, 79), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.076-0400 s20016| 2018-03-16T17:28:24.069-0400 D ASIO [conn1] startCommand: RemoteCommand 142 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36f81085bf286ecf4d21'), x: 9.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f788c1f30185b7afa3') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.076-0400 s20016| 2018-03-16T17:28:24.069-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-19-0] Connecting to ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.076-0400 c20012| 2018-03-16T17:28:24.070-0400 D REPL_HB [replexec-2] Sending heartbeat (requestId: 72) to ubuntu:20013, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20012", fromId: 0, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.076-0400 c20012| 2018-03-16T17:28:24.070-0400 D REPL_HB [replexec-0] Sending heartbeat (requestId: 73) to ubuntu:20014, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20012", fromId: 0, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.077-0400 c20013| 2018-03-16T17:28:24.070-0400 D REPL_HB [conn3] Received heartbeat request from ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20012", fromId: 0, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235703, 79), signature: { hash: BinData(0, F5BAD1415D4FF72DCE994EFB8E61FE39EE018428), keyId: 6533657568122765331 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.077-0400 c20013| 2018-03-16T17:28:24.070-0400 D REPL_HB [conn3] Processed heartbeat from ubuntu:20012 and generated response, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", syncingTo: "ubuntu:20012", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235703, 79), t: 1 }, opTime: { ts: Timestamp(1521235703, 79), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.077-0400 c20014| 2018-03-16T17:28:24.070-0400 D REPL_HB [conn4] Received heartbeat request from ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20012", fromId: 0, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235703, 79), signature: { hash: BinData(0, F5BAD1415D4FF72DCE994EFB8E61FE39EE018428), keyId: 6533657568122765331 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.078-0400 c20012| 2018-03-16T17:28:24.070-0400 D REPL_HB [replexec-1] Received response to heartbeat (requestId: 72) from ubuntu:20013, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", syncingTo: "ubuntu:20012", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235703, 79), t: 1 }, opTime: { ts: Timestamp(1521235703, 79), t: 1 }, operationTime: Timestamp(1521235703, 79), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235703, 79), t: 1 }, lastOpVisible: { ts: Timestamp(1521235703, 79), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235703, 79), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 79) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.079-0400 c20014| 2018-03-16T17:28:24.070-0400 D REPL_HB [conn4] Processed heartbeat from ubuntu:20012 and generated response, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", syncingTo: "ubuntu:20012", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235703, 79), t: 1 }, opTime: { ts: Timestamp(1521235703, 79), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.079-0400 c20012| 2018-03-16T17:28:24.070-0400 D REPL_HB [replexec-1] Scheduling heartbeat to ubuntu:20013 at 2018-03-16T21:28:26.070Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.079-0400 c20012| 2018-03-16T17:28:24.070-0400 D REPL_HB [replexec-2] Received response to heartbeat (requestId: 73) from ubuntu:20014, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", syncingTo: "ubuntu:20012", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235703, 79), t: 1 }, opTime: { ts: Timestamp(1521235703, 79), t: 1 }, operationTime: Timestamp(1521235703, 79), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235703, 79), t: 1 }, lastOpVisible: { ts: Timestamp(1521235703, 79), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235703, 79), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 79) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.080-0400 c20012| 2018-03-16T17:28:24.070-0400 D REPL_HB [replexec-2] Scheduling heartbeat to ubuntu:20014 at 2018-03-16T21:28:26.070Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.082-0400 s20016| 2018-03-16T17:28:24.082-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-19-0] Starting asynchronous command 143 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.083-0400 d20010| 2018-03-16T17:28:24.082-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56486 #41 (32 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.083-0400 d20010| 2018-03-16T17:28:24.082-0400 I NETWORK [conn41] received client metadata from 127.0.0.1:56486 conn41: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-19", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.083-0400 s20016| 2018-03-16T17:28:24.083-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-19-0] Successfully connected to ubuntu:20010, took 14ms (1 connections now open to ubuntu:20010) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.083-0400 s20016| 2018-03-16T17:28:24.083-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-19-0] Request 143 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.083-0400 s20016| 2018-03-16T17:28:24.083-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-19-0] Starting asynchronous command 142 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.084-0400 s20016| 2018-03-16T17:28:24.083-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-19-0] Request 142 finished with response: { n: 1, opTime: { ts: Timestamp(1521235704, 24), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235704, 24), $gleStats: { lastOpTime: { ts: Timestamp(1521235704, 24), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235704, 24), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 7), $configServerState: { opTime: { ts: Timestamp(1521235703, 79), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.085-0400 Waiting for op with OpTime { "ts" : Timestamp(1521235703, 79), "t" : NumberLong(1) } to be committed on all secondaries [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.089-0400 s20016| 2018-03-16T17:28:24.089-0400 D ASIO [conn1] startCommand: RemoteCommand 145 -- target:ubuntu:20013 db:config expDate:2018-03-16T17:28:54.089-0400 cmd:{ find: "collections", filter: { _id: /^config\./ }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235703, 79), t: 1 } }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.089-0400 s20016| 2018-03-16T17:28:24.089-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 145 on host ubuntu:20013 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.090-0400 s20016| 2018-03-16T17:28:24.089-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 145 finished with response: { cursor: { firstBatch: [ { _id: "config.system.sessions", lastmodEpoch: ObjectId('5aac36f788c1f30185b7adfc'), lastmod: new Date(4294967296), dropped: false, key: { _id: 1 }, unique: false, uuid: UUID("b6d2c6f9-227f-4aee-b5fd-4ae6721fa95c") } ], id: 0, ns: "config.collections" }, ok: 1.0, operationTime: Timestamp(1521235703, 79), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235703, 79), t: 1 }, lastOpVisible: { ts: Timestamp(1521235703, 79), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235704, 24), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 79) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.090-0400 s20016| 2018-03-16T17:28:24.089-0400 D ASIO [conn1] startCommand: RemoteCommand 147 -- target:ubuntu:20012 db:config cmd:{ find: "chunks", filter: { min: { x: MinKey }, max: { x: MaxKey } }, limit: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.090-0400 s20016| 2018-03-16T17:28:24.089-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-20-0] Connecting to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.090-0400 s20016| 2018-03-16T17:28:24.089-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-20-0] Starting asynchronous command 148 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.091-0400 c20012| 2018-03-16T17:28:24.089-0400 I NETWORK [listener] connection accepted from 127.0.0.1:60832 #43 (26 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.091-0400 c20012| 2018-03-16T17:28:24.090-0400 I NETWORK [conn43] received client metadata from 127.0.0.1:60832 conn43: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-20", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.091-0400 s20016| 2018-03-16T17:28:24.090-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-20-0] Successfully connected to ubuntu:20012, took 1ms (1 connections now open to ubuntu:20012) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.091-0400 s20016| 2018-03-16T17:28:24.090-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-20-0] Request 148 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.091-0400 s20016| 2018-03-16T17:28:24.090-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-20-0] Starting asynchronous command 147 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.092-0400 s20016| 2018-03-16T17:28:24.090-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-20-0] Request 147 finished with response: { cursor: { firstBatch: [ { _id: "test.foo-x_MinKey", ns: "test.foo", min: { x: MinKey }, max: { x: MaxKey }, shard: "stale_mongos_updates_and_removes-rs0", lastmod: Timestamp(1, 0), lastmodEpoch: ObjectId('5aac36f788c1f30185b7afa3'), history: [ { validAfter: Timestamp(1521235703, 71), shard: "stale_mongos_updates_and_removes-rs0" } ] } ], id: 0, ns: "config.chunks" }, ok: 1.0, operationTime: Timestamp(1521235703, 79), $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235704, 24), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 79) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.092-0400 s20015| 2018-03-16T17:28:24.091-0400 D ASIO [conn1] startCommand: RemoteCommand 49 -- target:ubuntu:20013 db:config expDate:2018-03-16T17:28:54.091-0400 cmd:{ find: "databases", filter: { _id: "test" }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235703, 52), t: 1 } }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.092-0400 s20015| 2018-03-16T17:28:24.091-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 49 on host ubuntu:20013 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.093-0400 s20015| 2018-03-16T17:28:24.091-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 49 finished with response: { cursor: { firstBatch: [ { _id: "test", primary: "stale_mongos_updates_and_removes-rs0", partitioned: true } ], id: 0, ns: "config.databases" }, ok: 1.0, operationTime: Timestamp(1521235703, 79), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235703, 79), t: 1 }, lastOpVisible: { ts: Timestamp(1521235703, 79), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235704, 24), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 79) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.093-0400 s20015| 2018-03-16T17:28:24.091-0400 D ASIO [conn1] startCommand: RemoteCommand 51 -- target:ubuntu:20014 db:config expDate:2018-03-16T17:28:54.091-0400 cmd:{ find: "collections", filter: { _id: /^test\./ }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235703, 79), t: 1 } }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.094-0400 s20015| 2018-03-16T17:28:24.091-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Connecting to ubuntu:20014 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.094-0400 s20015| 2018-03-16T17:28:24.091-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 52 on host ubuntu:20014 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.094-0400 c20014| 2018-03-16T17:28:24.091-0400 I NETWORK [listener] connection accepted from 127.0.0.1:44620 #20 (12 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.094-0400 c20014| 2018-03-16T17:28:24.092-0400 I NETWORK [conn20] received client metadata from 127.0.0.1:44620 conn20: { driver: { name: "NetworkInterfaceASIO-ShardRegistry", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.094-0400 s20015| 2018-03-16T17:28:24.092-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to ubuntu:20014, took 1ms (1 connections now open to ubuntu:20014) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.094-0400 s20015| 2018-03-16T17:28:24.092-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 52 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.095-0400 s20015| 2018-03-16T17:28:24.092-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 51 on host ubuntu:20014 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.095-0400 s20015| 2018-03-16T17:28:24.092-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 51 finished with response: { cursor: { firstBatch: [ { _id: "test.foo", lastmodEpoch: ObjectId('5aac36f788c1f30185b7afa3'), lastmod: new Date(4294967296), dropped: false, key: { x: 1.0 }, unique: false, uuid: UUID("a26b230c-c773-414a-b498-fd1cdf23fe9f") } ], id: 0, ns: "config.collections" }, ok: 1.0, operationTime: Timestamp(1521235703, 79), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235703, 79), t: 1 }, lastOpVisible: { ts: Timestamp(1521235703, 79), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235704, 24), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 79) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.096-0400 s20015| 2018-03-16T17:28:24.092-0400 I SHARDING [conn1] Refreshing chunks for collection test.foo based on version 0|0||000000000000000000000000 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.096-0400 s20015| 2018-03-16T17:28:24.092-0400 D EXECUTOR [ConfigServerCatalogCacheLoader-0] starting thread in pool ConfigServerCatalogCacheLoader [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.096-0400 s20015| 2018-03-16T17:28:24.092-0400 D ASIO [ConfigServerCatalogCacheLoader-0] startCommand: RemoteCommand 54 -- target:ubuntu:20014 db:config expDate:2018-03-16T17:28:54.092-0400 cmd:{ find: "collections", filter: { _id: "test.foo" }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235703, 79), t: 1 } }, limit: 1, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.096-0400 s20015| 2018-03-16T17:28:24.092-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 54 on host ubuntu:20014 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.097-0400 s20015| 2018-03-16T17:28:24.092-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 54 finished with response: { cursor: { firstBatch: [ { _id: "test.foo", lastmodEpoch: ObjectId('5aac36f788c1f30185b7afa3'), lastmod: new Date(4294967296), dropped: false, key: { x: 1.0 }, unique: false, uuid: UUID("a26b230c-c773-414a-b498-fd1cdf23fe9f") } ], id: 0, ns: "config.collections" }, ok: 1.0, operationTime: Timestamp(1521235703, 79), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235703, 79), t: 1 }, lastOpVisible: { ts: Timestamp(1521235703, 79), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235704, 24), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 79) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.097-0400 s20015| 2018-03-16T17:28:24.092-0400 D ASIO [ConfigServerCatalogCacheLoader-0] startCommand: RemoteCommand 56 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:54.092-0400 cmd:{ find: "chunks", filter: { ns: "test.foo", lastmod: { $gte: Timestamp(0, 0) } }, sort: { lastmod: 1 }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235703, 79), t: 1 } }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.097-0400 s20015| 2018-03-16T17:28:24.092-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 56 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.098-0400 s20015| 2018-03-16T17:28:24.093-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 56 finished with response: { cursor: { firstBatch: [ { _id: "test.foo-x_MinKey", ns: "test.foo", min: { x: MinKey }, max: { x: MaxKey }, shard: "stale_mongos_updates_and_removes-rs0", lastmod: Timestamp(1, 0), lastmodEpoch: ObjectId('5aac36f788c1f30185b7afa3'), history: [ { validAfter: Timestamp(1521235703, 71), shard: "stale_mongos_updates_and_removes-rs0" } ] } ], id: 0, ns: "config.chunks" }, ok: 1.0, operationTime: Timestamp(1521235703, 79), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235703, 79), t: 1 }, lastOpVisible: { ts: Timestamp(1521235703, 79), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235703, 52), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235704, 24), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235703, 79) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.098-0400 s20015| 2018-03-16T17:28:24.093-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Refresh for collection test.foo took 0 ms and found version 1|0||5aac36f788c1f30185b7afa3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.099-0400 s20015| 2018-03-16T17:28:24.093-0400 D ASIO [conn1] startCommand: RemoteCommand 58 -- target:ubuntu:20012 db:admin cmd:{ _configsvrMoveChunk: 1, _id: "test.foo-x_MinKey", ns: "test.foo", min: { x: MinKey }, max: { x: MaxKey }, shard: "stale_mongos_updates_and_removes-rs0", lastmod: Timestamp(1, 0), lastmodEpoch: ObjectId('5aac36f788c1f30185b7afa3'), toShard: "stale_mongos_updates_and_removes-rs1", maxChunkSizeBytes: 67108864, secondaryThrottle: {}, waitForDelete: true, writeConcern: { w: "majority", wtimeout: 15000 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.099-0400 s20015| 2018-03-16T17:28:24.093-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 58 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.099-0400 c20012| 2018-03-16T17:28:24.095-0400 I SHARDING [conn21] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.099-0400 c20012| 2018-03-16T17:28:24.095-0400 I SHARDING [conn21] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.099-0400 c20012| 2018-03-16T17:28:24.095-0400 I SHARDING [conn21] xxx routing info needs refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.099-0400 c20012| 2018-03-16T17:28:24.095-0400 I SHARDING [conn21] xxx calling schedule collection refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.100-0400 c20012| 2018-03-16T17:28:24.095-0400 I SHARDING [conn21] xxx is not incr [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.100-0400 c20012| 2018-03-16T17:28:24.095-0400 I SHARDING [conn21] xxx existing routing info? [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.100-0400 c20012| 2018-03-16T17:28:24.095-0400 I SHARDING [conn21] Refreshing chunks for collection test.foo based on version 0|0||000000000000000000000000 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.100-0400 c20012| 2018-03-16T17:28:24.095-0400 I SHARDING [conn21] xxx getting chunks since now [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.100-0400 c20012| 2018-03-16T17:28:24.095-0400 I SHARDING [conn21] xxx waiting if !refreshNot [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.100-0400 c20012| 2018-03-16T17:28:24.096-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh callback [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.100-0400 c20012| 2018-03-16T17:28:24.096-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh completed [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.101-0400 c20012| 2018-03-16T17:28:24.096-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Refresh for collection test.foo took 0 ms and found version 1|0||5aac36f788c1f30185b7afa3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.101-0400 c20012| 2018-03-16T17:28:24.096-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx marking as status ok [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.101-0400 c20012| 2018-03-16T17:28:24.096-0400 I SHARDING [conn21] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.101-0400 c20012| 2018-03-16T17:28:24.096-0400 I SHARDING [conn21] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.106-0400 c20012| 2018-03-16T17:28:24.106-0400 I SHARDING [conn21] distributed lock 'test.foo' acquired for 'Migrating chunk(s) in collection test.foo', ts : 5aac36ec88c1f30185b7ac69 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.106-0400 d20010| 2018-03-16T17:28:24.106-0400 I NETWORK [conn15] Starting new replica set monitor for stale_mongos_updates_and_removes-rs1/ubuntu:20011 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.107-0400 d20010| 2018-03-16T17:28:24.107-0400 I SHARDING [conn15] Starting chunk migration ns: test.foo, [{ x: MinKey }, { x: MaxKey }), fromShard: stale_mongos_updates_and_removes-rs0, toShard: stale_mongos_updates_and_removes-rs1 with expected collection version epoch 5aac36f788c1f30185b7afa3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.107-0400 d20010| 2018-03-16T17:28:24.107-0400 I SHARDING [conn15] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.107-0400 d20010| 2018-03-16T17:28:24.107-0400 I SHARDING [conn15] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.107-0400 d20010| 2018-03-16T17:28:24.107-0400 I SHARDING [conn15] xxx routing info needs refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.107-0400 d20010| 2018-03-16T17:28:24.107-0400 I SHARDING [conn15] xxx calling schedule collection refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.107-0400 d20010| 2018-03-16T17:28:24.107-0400 I SHARDING [conn15] xxx is incremental [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.107-0400 d20010| 2018-03-16T17:28:24.107-0400 I SHARDING [conn15] xxx existing routing info? [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.108-0400 d20010| 2018-03-16T17:28:24.107-0400 I SHARDING [conn15] 1|0||5aac36f788c1f30185b7afa3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.108-0400 d20010| 2018-03-16T17:28:24.107-0400 I SHARDING [conn15] Refreshing chunks for collection test.foo based on version 1|0||5aac36f788c1f30185b7afa3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.108-0400 d20010| 2018-03-16T17:28:24.107-0400 I SHARDING [conn15] xxx getting chunks since now [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.108-0400 d20010| 2018-03-16T17:28:24.107-0400 I SHARDING [conn15] xxx getting chunks since [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.108-0400 d20010| 2018-03-16T17:28:24.107-0400 I SHARDING [conn15] xxx waiting if !refreshNot [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.108-0400 d20010| 2018-03-16T17:28:24.107-0400 I SHARDING [ShardServerCatalogCacheLoader-0] xxx primary and calling get chunks since for primary [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.108-0400 d20010| 2018-03-16T17:28:24.107-0400 I SHARDING [ShardServerCatalogCacheLoader-0] version is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.109-0400 d20010| 2018-03-16T17:28:24.107-0400 I SHARDING [ShardServerCatalogCacheLoader-0] 1|0||5aac36f788c1f30185b7afa3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.109-0400 d20010| 2018-03-16T17:28:24.107-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Connecting to ubuntu:20014 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.109-0400 c20014| 2018-03-16T17:28:24.107-0400 I NETWORK [listener] connection accepted from 127.0.0.1:44622 #21 (13 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.109-0400 c20014| 2018-03-16T17:28:24.107-0400 I NETWORK [conn21] received client metadata from 127.0.0.1:44622 conn21: { driver: { name: "NetworkInterfaceASIO-ShardRegistry", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.109-0400 d20010| 2018-03-16T17:28:24.107-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to ubuntu:20014, took 0ms (1 connections now open to ubuntu:20014) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.109-0400 d20010| 2018-03-16T17:28:24.108-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] refresh in primary [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.110-0400 d20010| 2018-03-16T17:28:24.108-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] 1 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.110-0400 d20010| 2018-03-16T17:28:24.108-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Cache loader remotely refreshed for collection test.foo from collection version 1|0||5aac36f788c1f30185b7afa3 and found collection version 1|0||5aac36f788c1f30185b7afa3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.110-0400 d20010| 2018-03-16T17:28:24.108-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Cache loader found no enqueued metadata and persisted metadata from 1|0||5aac36f788c1f30185b7afa3 to 1|0||5aac36f788c1f30185b7afa3, GTE cache version 1|0||5aac36f788c1f30185b7afa3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.110-0400 d20010| 2018-03-16T17:28:24.108-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx is okay [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.110-0400 d20010| 2018-03-16T17:28:24.108-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx finsihing refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.110-0400 d20010| 2018-03-16T17:28:24.108-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh callback [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.111-0400 d20010| 2018-03-16T17:28:24.108-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh completed [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.111-0400 d20010| 2018-03-16T17:28:24.108-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Refresh for collection test.foo took 1 ms and found version 1|0||5aac36f788c1f30185b7afa3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.111-0400 d20010| 2018-03-16T17:28:24.108-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx marking as status ok [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.111-0400 d20010| 2018-03-16T17:28:24.108-0400 I SHARDING [conn15] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.111-0400 d20010| 2018-03-16T17:28:24.108-0400 I SHARDING [conn15] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.112-0400 d20010| 2018-03-16T17:28:24.109-0400 I SHARDING [conn15] about to log metadata event into changelog: { _id: "ubuntu-2018-03-16T17:28:24.109-0400-5aac36f8f7e5a0b40b8e34d9", server: "ubuntu", clientAddr: "127.0.0.1:56424", time: new Date(1521235704109), what: "moveChunk.start", ns: "test.foo", details: { min: { x: MinKey }, max: { x: MaxKey }, from: "stale_mongos_updates_and_removes-rs0", to: "stale_mongos_updates_and_removes-rs1" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.115-0400 d20010| 2018-03-16T17:28:24.115-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Connecting to ubuntu:20011 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.115-0400 d20011| 2018-03-16T17:28:24.115-0400 I NETWORK [listener] connection accepted from 127.0.0.1:55138 #22 (13 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.115-0400 d20011| 2018-03-16T17:28:24.115-0400 I NETWORK [conn22] received client metadata from 127.0.0.1:55138 conn22: { driver: { name: "NetworkInterfaceASIO-ShardRegistry", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.115-0400 d20010| 2018-03-16T17:28:24.115-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to ubuntu:20011, took 0ms (1 connections now open to ubuntu:20011) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.116-0400 d20011| 2018-03-16T17:28:24.115-0400 I SHARDING [conn22] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.116-0400 d20011| 2018-03-16T17:28:24.115-0400 I SHARDING [conn22] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.116-0400 d20011| 2018-03-16T17:28:24.115-0400 I SHARDING [conn22] xxx routing info needs refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.116-0400 d20011| 2018-03-16T17:28:24.115-0400 I SHARDING [conn22] xxx calling schedule collection refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.116-0400 d20011| 2018-03-16T17:28:24.115-0400 I SHARDING [conn22] xxx is not incr [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.116-0400 d20011| 2018-03-16T17:28:24.115-0400 I SHARDING [conn22] xxx existing routing info? [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.116-0400 d20011| 2018-03-16T17:28:24.115-0400 I SHARDING [conn22] Refreshing chunks for collection test.foo based on version 0|0||000000000000000000000000 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.116-0400 d20011| 2018-03-16T17:28:24.115-0400 I SHARDING [conn22] xxx getting chunks since now [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.116-0400 d20011| 2018-03-16T17:28:24.115-0400 I SHARDING [conn22] xxx getting chunks since [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.117-0400 d20011| 2018-03-16T17:28:24.115-0400 I SHARDING [conn22] xxx waiting if !refreshNot [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.117-0400 d20011| 2018-03-16T17:28:24.115-0400 I SHARDING [ShardServerCatalogCacheLoader-0] xxx primary and calling get chunks since for primary [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.117-0400 d20011| 2018-03-16T17:28:24.115-0400 I SHARDING [ShardServerCatalogCacheLoader-0] version is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.117-0400 d20011| 2018-03-16T17:28:24.115-0400 I SHARDING [ShardServerCatalogCacheLoader-0] 0|0||000000000000000000000000 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.117-0400 d20011| 2018-03-16T17:28:24.116-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] refresh in primary [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.117-0400 d20011| 2018-03-16T17:28:24.116-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] 1 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.117-0400 d20011| 2018-03-16T17:28:24.116-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx collec and changed chunks max epoch or version doesn't equal max [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.187-0400 d20011| 2018-03-16T17:28:24.186-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Cache loader remotely refreshed for collection test.foo from collection version 0|0||000000000000000000000000 and found collection version 1|0||5aac36f788c1f30185b7afa3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.187-0400 d20011| 2018-03-16T17:28:24.186-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Cache loader found enqueued metadata from 1|0||5aac36f788c1f30185b7afa3 to 1|0||5aac36f788c1f30185b7afa3 and no persisted metadata, GTE cache version 0|0||000000000000000000000000 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.187-0400 d20011| 2018-03-16T17:28:24.186-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx is okay [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.187-0400 d20011| 2018-03-16T17:28:24.186-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx finsihing refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.187-0400 d20011| 2018-03-16T17:28:24.186-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh callback [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.187-0400 d20011| 2018-03-16T17:28:24.186-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh completed [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.188-0400 d20011| 2018-03-16T17:28:24.186-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Refresh for collection test.foo took 71 ms and found version 1|0||5aac36f788c1f30185b7afa3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.188-0400 d20011| 2018-03-16T17:28:24.186-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx marking as status ok [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.188-0400 d20011| 2018-03-16T17:28:24.186-0400 I SHARDING [conn22] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.188-0400 d20011| 2018-03-16T17:28:24.186-0400 I SHARDING [conn22] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.188-0400 d20011| 2018-03-16T17:28:24.186-0400 I SHARDING [conn22] Marking collection test.foo as sharded with collection version: 1|0||5aac36f788c1f30185b7afa3, shard version: 0|0||5aac36f788c1f30185b7afa3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.188-0400 d20011| 2018-03-16T17:28:24.187-0400 I STORAGE [ShardServerCatalogCacheLoader-0] createCollection: config.cache.chunks.test.foo with generated UUID: 74f4c770-00b6-43f4-8263-89eac02e3c28 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.189-0400 d20011| 2018-03-16T17:28:24.187-0400 I SHARDING [migrateThread] Starting receiving end of migration of chunk { x: MinKey } -> { x: MaxKey } for collection test.foo from stale_mongos_updates_and_removes-rs0/ubuntu:20010 at epoch 5aac36f788c1f30185b7afa3 with session id stale_mongos_updates_and_removes-rs0_stale_mongos_updates_and_removes-rs1_5aac36f8f7e5a0b40b8e34da [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.189-0400 d20011| 2018-03-16T17:28:24.187-0400 I NETWORK [migrateThread] Successfully connected to stale_mongos_updates_and_removes-rs0/ubuntu:20010 (1 connections now open to stale_mongos_updates_and_removes-rs0/ubuntu:20010 with a 0 second timeout) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.189-0400 d20010| 2018-03-16T17:28:24.187-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56496 #42 (33 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.189-0400 d20010| 2018-03-16T17:28:24.187-0400 I NETWORK [conn42] received client metadata from 127.0.0.1:56496 conn42: { driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.189-0400 d20011| 2018-03-16T17:28:24.187-0400 I STORAGE [migrateThread] createCollection: test.foo with provided UUID: a26b230c-c773-414a-b498-fd1cdf23fe9f [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.227-0400 d20011| 2018-03-16T17:28:24.227-0400 I INDEX [ShardServerCatalogCacheLoader-0] build index on: config.cache.chunks.test.foo properties: { v: 2, key: { lastmod: 1 }, name: "lastmod_1", ns: "config.cache.chunks.test.foo" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.227-0400 d20011| 2018-03-16T17:28:24.227-0400 I INDEX [ShardServerCatalogCacheLoader-0] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.233-0400 d20011| 2018-03-16T17:28:24.233-0400 I INDEX [migrateThread] build index on: test.foo properties: { v: 2, key: { x: 1.0 }, name: "x_1", ns: "test.foo" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.233-0400 d20011| 2018-03-16T17:28:24.233-0400 I INDEX [migrateThread] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.234-0400 d20011| 2018-03-16T17:28:24.233-0400 I SHARDING [migrateThread] Scheduling deletion of any documents in test.foo range [{ x: MinKey }, { x: MaxKey }) before migrating in a chunk covering the range [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.234-0400 d20011| 2018-03-16T17:28:24.233-0400 I SHARDING [Collection Range Deleter] No documents remain to delete in test.foo range [{ x: MinKey }, { x: MaxKey }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.234-0400 d20011| 2018-03-16T17:28:24.233-0400 I SHARDING [Collection Range Deleter] Waiting for majority replication of local deletions in test.foo range [{ x: MinKey }, { x: MaxKey }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.234-0400 d20011| 2018-03-16T17:28:24.233-0400 I INDEX [ShardServerCatalogCacheLoader-0] build index done. scanned 0 total records. 0 secs [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.235-0400 d20011| 2018-03-16T17:28:24.234-0400 I SHARDING [Collection Range Deleter] Finished deleting documents in test.foo range [{ x: MinKey }, { x: MaxKey }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.235-0400 d20011| 2018-03-16T17:28:24.234-0400 I SHARDING [migrateThread] Finished deleting test.foo range [{ x: MinKey }, { x: MaxKey }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.235-0400 d20011| 2018-03-16T17:28:24.235-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Connecting to ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.235-0400 d20010| 2018-03-16T17:28:24.235-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56498 #43 (34 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.235-0400 d20010| 2018-03-16T17:28:24.235-0400 I NETWORK [conn43] received client metadata from 127.0.0.1:56498 conn43: { driver: { name: "NetworkInterfaceASIO-ShardRegistry", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.235-0400 d20011| 2018-03-16T17:28:24.235-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to ubuntu:20010, took 0ms (1 connections now open to ubuntu:20010) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.236-0400 d20011| 2018-03-16T17:28:24.236-0400 I SHARDING [migrateThread] Waiting for replication to catch up before entering critical section [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.239-0400 d20011| 2018-03-16T17:28:24.239-0400 I SHARDING [migrateThread] migrate commit succeeded flushing to secondaries for 'test.foo' { x: MinKey } -> { x: MaxKey } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.240-0400 d20010| 2018-03-16T17:28:24.239-0400 I SHARDING [conn15] moveChunk data transfer progress: { waited: true, active: true, sessionId: "stale_mongos_updates_and_removes-rs0_stale_mongos_updates_and_removes-rs1_5aac36f8f7e5a0b40b8e34da", ns: "test.foo", from: "stale_mongos_updates_and_removes-rs0/ubuntu:20010", min: { x: MinKey }, max: { x: MaxKey }, shardKeyPattern: { x: 1.0 }, state: "steady", counts: { cloned: 20, clonedBytes: 1120, catchup: 0, steady: 0 }, ok: 1.0, operationTime: Timestamp(1521235704, 57), $gleStats: { lastOpTime: { ts: Timestamp(1521235704, 31), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235704, 57), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 57), $configServerState: { opTime: { ts: Timestamp(1521235704, 27), t: 1 } } } mem used: 0 documents remaining to clone: 0 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.241-0400 d20010| 2018-03-16T17:28:24.240-0400 I SHARDING [conn15] Migration successfully entered critical section [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.250-0400 d20011| 2018-03-16T17:28:24.249-0400 I SHARDING [migrateThread] migrate commit succeeded flushing to secondaries for 'test.foo' { x: MinKey } -> { x: MaxKey } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.538-0400 c20013| 2018-03-16T17:28:24.537-0400 D REPL_HB [replexec-1] Sending heartbeat (requestId: 702) to ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.538-0400 c20012| 2018-03-16T17:28:24.537-0400 D REPL_HB [conn9] Received heartbeat request from ubuntu:20013, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235704, 27), signature: { hash: BinData(0, F44F64AF6B9C0EDEC7908BFBF94A6C2809BBED78), keyId: 6533657568122765331 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.538-0400 c20012| 2018-03-16T17:28:24.537-0400 D REPL_HB [conn9] Processed heartbeat from ubuntu:20013 and generated response, { ok: 1.0, electionTime: new Date(6533657559532830722), state: 1, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235704, 27), t: 1 }, opTime: { ts: Timestamp(1521235704, 27), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.539-0400 c20013| 2018-03-16T17:28:24.538-0400 D REPL_HB [replexec-0] Received response to heartbeat (requestId: 702) from ubuntu:20012, { ok: 1.0, electionTime: new Date(6533657559532830722), state: 1, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235704, 27), t: 1 }, opTime: { ts: Timestamp(1521235704, 27), t: 1 }, operationTime: Timestamp(1521235704, 27), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235704, 27), t: 1 }, lastOpVisible: { ts: Timestamp(1521235704, 27), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235704, 28), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 27) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.539-0400 c20013| 2018-03-16T17:28:24.538-0400 D REPL_HB [replexec-0] Scheduling heartbeat to ubuntu:20012 at 2018-03-16T21:28:26.538Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.540-0400 c20013| 2018-03-16T17:28:24.538-0400 D REPL_HB [replexec-2] Sending heartbeat (requestId: 704) to ubuntu:20014, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.540-0400 c20014| 2018-03-16T17:28:24.538-0400 D REPL_HB [conn10] Received heartbeat request from ubuntu:20013, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235704, 28), signature: { hash: BinData(0, F44F64AF6B9C0EDEC7908BFBF94A6C2809BBED78), keyId: 6533657568122765331 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.540-0400 c20014| 2018-03-16T17:28:24.538-0400 D REPL_HB [conn10] Processed heartbeat from ubuntu:20013 and generated response, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", syncingTo: "ubuntu:20012", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235704, 27), t: 1 }, opTime: { ts: Timestamp(1521235704, 27), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.541-0400 c20013| 2018-03-16T17:28:24.538-0400 D REPL_HB [replexec-1] Received response to heartbeat (requestId: 704) from ubuntu:20014, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", syncingTo: "ubuntu:20012", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235704, 27), t: 1 }, opTime: { ts: Timestamp(1521235704, 27), t: 1 }, operationTime: Timestamp(1521235704, 27), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235704, 27), t: 1 }, lastOpVisible: { ts: Timestamp(1521235704, 27), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235704, 28), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 27) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.541-0400 c20013| 2018-03-16T17:28:24.538-0400 D REPL_HB [replexec-1] Scheduling heartbeat to ubuntu:20014 at 2018-03-16T21:28:26.538Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.581-0400 c20014| 2018-03-16T17:28:24.581-0400 D REPL_HB [replexec-3] Sending heartbeat (requestId: 693) to ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.582-0400 c20012| 2018-03-16T17:28:24.581-0400 D REPL_HB [conn8] Received heartbeat request from ubuntu:20014, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235704, 28), signature: { hash: BinData(0, F44F64AF6B9C0EDEC7908BFBF94A6C2809BBED78), keyId: 6533657568122765331 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.582-0400 c20012| 2018-03-16T17:28:24.581-0400 D REPL_HB [conn8] Processed heartbeat from ubuntu:20014 and generated response, { ok: 1.0, electionTime: new Date(6533657559532830722), state: 1, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235704, 27), t: 1 }, opTime: { ts: Timestamp(1521235704, 27), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.583-0400 c20014| 2018-03-16T17:28:24.581-0400 D REPL_HB [replexec-1] Received response to heartbeat (requestId: 693) from ubuntu:20012, { ok: 1.0, electionTime: new Date(6533657559532830722), state: 1, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235704, 27), t: 1 }, opTime: { ts: Timestamp(1521235704, 27), t: 1 }, operationTime: Timestamp(1521235704, 27), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235704, 27), t: 1 }, lastOpVisible: { ts: Timestamp(1521235704, 27), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235704, 28), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 27) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.583-0400 c20014| 2018-03-16T17:28:24.581-0400 D REPL_HB [replexec-1] Scheduling heartbeat to ubuntu:20012 at 2018-03-16T21:28:26.581Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.583-0400 c20014| 2018-03-16T17:28:24.582-0400 D REPL_HB [replexec-4] Sending heartbeat (requestId: 695) to ubuntu:20013, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.584-0400 c20013| 2018-03-16T17:28:24.582-0400 D REPL_HB [conn9] Received heartbeat request from ubuntu:20014, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235704, 28), signature: { hash: BinData(0, F44F64AF6B9C0EDEC7908BFBF94A6C2809BBED78), keyId: 6533657568122765331 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.584-0400 c20013| 2018-03-16T17:28:24.582-0400 D REPL_HB [conn9] Processed heartbeat from ubuntu:20014 and generated response, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", syncingTo: "ubuntu:20012", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235704, 27), t: 1 }, opTime: { ts: Timestamp(1521235704, 27), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.585-0400 c20014| 2018-03-16T17:28:24.582-0400 D REPL_HB [replexec-0] Received response to heartbeat (requestId: 695) from ubuntu:20013, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", syncingTo: "ubuntu:20012", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235704, 27), t: 1 }, opTime: { ts: Timestamp(1521235704, 27), t: 1 }, operationTime: Timestamp(1521235704, 27), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235704, 27), t: 1 }, lastOpVisible: { ts: Timestamp(1521235704, 27), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235704, 28), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 27) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.585-0400 c20014| 2018-03-16T17:28:24.582-0400 D REPL_HB [replexec-0] Scheduling heartbeat to ubuntu:20013 at 2018-03-16T21:28:26.582Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.638-0400 d20011| 2018-03-16T17:28:24.637-0400 I SHARDING [migrateThread] about to log metadata event into changelog: { _id: "ubuntu-2018-03-16T17:28:24.637-0400-5aac36f86a0ae3d58b6402fb", server: "ubuntu", clientAddr: "", time: new Date(1521235704637), what: "moveChunk.to", ns: "test.foo", details: { min: { x: MinKey }, max: { x: MaxKey }, step 1 of 6: 46, step 2 of 6: 1, step 3 of 6: 1, step 4 of 6: 0, step 5 of 6: 13, step 6 of 6: 387, note: "success" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.644-0400 d20011| 2018-03-16T17:28:24.643-0400 I COMMAND [conn22] command admin.$cmd command: _recvChunkCommit { _recvChunkCommit: "test.foo", waitForSteadyOrDone: false, sessionId: "stale_mongos_updates_and_removes-rs0_stale_mongos_updates_and_removes-rs1_5aac36f8f7e5a0b40b8e34da", $clusterTime: { clusterTime: Timestamp(1521235704, 60), signature: { hash: BinData(0, F44F64AF6B9C0EDEC7908BFBF94A6C2809BBED78), keyId: 6533657568122765331 } }, $configServerState: { opTime: { ts: Timestamp(1521235704, 27), t: 1 } }, $db: "admin" } numYields:0 reslen:561 locks:{} protocol:op_msg 402ms [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.644-0400 d20010| 2018-03-16T17:28:24.643-0400 I SHARDING [conn15] Moving last chunk for the collection out [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.649-0400 d20010| 2018-03-16T17:28:24.649-0400 I SHARDING [conn15] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.649-0400 d20010| 2018-03-16T17:28:24.649-0400 I SHARDING [conn15] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.649-0400 d20010| 2018-03-16T17:28:24.649-0400 I SHARDING [conn15] xxx routing info needs refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.649-0400 d20010| 2018-03-16T17:28:24.649-0400 I SHARDING [conn15] xxx calling schedule collection refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.649-0400 d20010| 2018-03-16T17:28:24.649-0400 I SHARDING [conn15] xxx is incremental [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.650-0400 d20010| 2018-03-16T17:28:24.649-0400 I SHARDING [conn15] xxx existing routing info? [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.650-0400 d20010| 2018-03-16T17:28:24.649-0400 I SHARDING [conn15] 1|0||5aac36f788c1f30185b7afa3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.650-0400 d20010| 2018-03-16T17:28:24.649-0400 I SHARDING [conn15] Refreshing chunks for collection test.foo based on version 1|0||5aac36f788c1f30185b7afa3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.650-0400 d20010| 2018-03-16T17:28:24.649-0400 I SHARDING [conn15] xxx getting chunks since now [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.650-0400 d20010| 2018-03-16T17:28:24.649-0400 I SHARDING [conn15] xxx getting chunks since [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.650-0400 d20010| 2018-03-16T17:28:24.649-0400 I SHARDING [conn15] xxx waiting if !refreshNot [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.650-0400 d20010| 2018-03-16T17:28:24.649-0400 I SHARDING [ShardServerCatalogCacheLoader-0] xxx primary and calling get chunks since for primary [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.651-0400 d20010| 2018-03-16T17:28:24.649-0400 I SHARDING [ShardServerCatalogCacheLoader-0] version is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.651-0400 d20010| 2018-03-16T17:28:24.649-0400 I SHARDING [ShardServerCatalogCacheLoader-0] 1|0||5aac36f788c1f30185b7afa3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.651-0400 d20010| 2018-03-16T17:28:24.650-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] refresh in primary [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.651-0400 d20010| 2018-03-16T17:28:24.650-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] 1 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.651-0400 d20010| 2018-03-16T17:28:24.650-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx collec and changed chunks max epoch or version doesn't equal max [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.688-0400 d20010| 2018-03-16T17:28:24.688-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Cache loader remotely refreshed for collection test.foo from collection version 1|0||5aac36f788c1f30185b7afa3 and found collection version 2|0||5aac36f788c1f30185b7afa3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.689-0400 d20010| 2018-03-16T17:28:24.688-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Cache loader found enqueued metadata from 2|0||5aac36f788c1f30185b7afa3 to 2|0||5aac36f788c1f30185b7afa3 and persisted metadata from 1|0||5aac36f788c1f30185b7afa3 to 1|0||5aac36f788c1f30185b7afa3, GTE cache version 1|0||5aac36f788c1f30185b7afa3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.689-0400 d20010| 2018-03-16T17:28:24.688-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx is okay [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.689-0400 d20010| 2018-03-16T17:28:24.688-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx finsihing refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.689-0400 d20010| 2018-03-16T17:28:24.688-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh callback [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.689-0400 d20010| 2018-03-16T17:28:24.688-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh completed [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.690-0400 d20010| 2018-03-16T17:28:24.688-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Refresh for collection test.foo took 39 ms and found version 2|0||5aac36f788c1f30185b7afa3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.690-0400 d20010| 2018-03-16T17:28:24.688-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx marking as status ok [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.690-0400 d20010| 2018-03-16T17:28:24.688-0400 I SHARDING [conn15] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.690-0400 d20010| 2018-03-16T17:28:24.688-0400 I SHARDING [conn15] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.690-0400 d20010| 2018-03-16T17:28:24.688-0400 I SHARDING [conn15] Updating collection metadata for test.foo from collection version: 1|0||5aac36f788c1f30185b7afa3, shard version: 1|0||5aac36f788c1f30185b7afa3 to collection version: 2|0||5aac36f788c1f30185b7afa3, shard version: 0|0||5aac36f788c1f30185b7afa3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.690-0400 d20010| 2018-03-16T17:28:24.689-0400 I SHARDING [conn15] Migration succeeded and updated collection version to 2|0||5aac36f788c1f30185b7afa3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.691-0400 d20010| 2018-03-16T17:28:24.689-0400 I SHARDING [conn15] about to log metadata event into changelog: { _id: "ubuntu-2018-03-16T17:28:24.689-0400-5aac36f8f7e5a0b40b8e34f7", server: "ubuntu", clientAddr: "127.0.0.1:56424", time: new Date(1521235704689), what: "moveChunk.commit", ns: "test.foo", details: { min: { x: MinKey }, max: { x: MaxKey }, from: "stale_mongos_updates_and_removes-rs0", to: "stale_mongos_updates_and_removes-rs1", counts: { cloned: 20, clonedBytes: 1120, catchup: 0, steady: 0 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.695-0400 d20010| 2018-03-16T17:28:24.695-0400 I SHARDING [conn15] Scheduling immediate deletion of test.foo range [{ x: MinKey }, { x: MaxKey }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.695-0400 d20010| 2018-03-16T17:28:24.695-0400 I SHARDING [conn15] Waiting for cleanup of test.foo range [{ x: MinKey }, { x: MaxKey }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.695-0400 d20011| 2018-03-16T17:28:24.695-0400 I SHARDING [conn22] XXXX in set shard version cmd obj is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.696-0400 d20011| 2018-03-16T17:28:24.695-0400 I SHARDING [conn22] { setShardVersion: "test.foo", init: false, authoritative: false, configdb: "stale_mongos_updates_and_removes-configRS/ubuntu:20012,ubuntu:20013,ubuntu:20014", shard: "stale_mongos_updates_and_removes-rs1", shardHost: "ubuntu:20011", version: Timestamp(2, 0), versionEpoch: ObjectId('5aac36f788c1f30185b7afa3'), noConnectionVersioning: true, $clusterTime: { clusterTime: Timestamp(1521235704, 69), signature: { hash: BinData(0, F44F64AF6B9C0EDEC7908BFBF94A6C2809BBED78), keyId: 6533657568122765331 } }, $client: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" }, mongos: { host: "ubuntu:20015", client: "127.0.0.1:47840", version: "3.6.3" } }, $configServerState: { opTime: { ts: Timestamp(1521235704, 69), t: 1 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.696-0400 d20011| 2018-03-16T17:28:24.695-0400 I SHARDING [conn22] XXXX requested version is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.696-0400 d20011| 2018-03-16T17:28:24.695-0400 I SHARDING [conn22] 2|0||5aac36f788c1f30185b7afa3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.696-0400 d20010| 2018-03-16T17:28:24.695-0400 I SHARDING [Collection Range Deleter] No documents remain to delete in test.foo range [{ x: MinKey }, { x: MaxKey }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.697-0400 d20010| 2018-03-16T17:28:24.696-0400 I SHARDING [Collection Range Deleter] Waiting for majority replication of local deletions in test.foo range [{ x: MinKey }, { x: MaxKey }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.699-0400 d20010| 2018-03-16T17:28:24.699-0400 I SHARDING [Collection Range Deleter] Finished deleting documents in test.foo range [{ x: MinKey }, { x: MaxKey }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.699-0400 d20010| 2018-03-16T17:28:24.699-0400 I SHARDING [conn15] about to log metadata event into changelog: { _id: "ubuntu-2018-03-16T17:28:24.699-0400-5aac36f8f7e5a0b40b8e34f8", server: "ubuntu", clientAddr: "127.0.0.1:56424", time: new Date(1521235704699), what: "moveChunk.from", ns: "test.foo", details: { min: { x: MinKey }, max: { x: MaxKey }, step 1 of 6: 0, step 2 of 6: 1, step 3 of 6: 78, step 4 of 6: 52, step 5 of 6: 404, step 6 of 6: 55, to: "stale_mongos_updates_and_removes-rs1", from: "stale_mongos_updates_and_removes-rs0", note: "success" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.704-0400 d20010| 2018-03-16T17:28:24.703-0400 I COMMAND [conn15] command admin.$cmd appName: "MongoDB Shell" command: moveChunk { moveChunk: "test.foo", shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f788c1f30185b7afa3') ], epoch: ObjectId('5aac36f788c1f30185b7afa3'), configdb: "stale_mongos_updates_and_removes-configRS/ubuntu:20012,ubuntu:20013,ubuntu:20014", fromShard: "stale_mongos_updates_and_removes-rs0", toShard: "stale_mongos_updates_and_removes-rs1", min: { x: MinKey }, max: { x: MaxKey }, maxChunkSizeBytes: 67108864, waitForDelete: true, takeDistLock: false, $clusterTime: { clusterTime: Timestamp(1521235704, 26), signature: { hash: BinData(0, F44F64AF6B9C0EDEC7908BFBF94A6C2809BBED78), keyId: 6533657568122765331 } }, $client: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" }, mongos: { host: "ubuntu:20015", client: "127.0.0.1:47840", version: "3.6.3" } }, $configServerState: { opTime: { ts: Timestamp(1521235704, 26), t: 1 } }, $db: "admin" } numYields:0 reslen:317 locks:{ Global: { acquireCount: { r: 33, w: 15 } }, Database: { acquireCount: { r: 9, w: 13, W: 2 } }, Collection: { acquireCount: { r: 9, w: 1, W: 5 } }, oplog: { acquireCount: { w: 7 } } } protocol:op_msg 596ms [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.707-0400 c20012| 2018-03-16T17:28:24.707-0400 I SHARDING [NetworkInterfaceASIO-ShardRegistry-0] distributed lock with ts: '5aac36ec88c1f30185b7ac69' and _id: 'test.foo' unlocked. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.707-0400 c20012| 2018-03-16T17:28:24.707-0400 I SHARDING [conn21] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.708-0400 c20012| 2018-03-16T17:28:24.707-0400 I SHARDING [conn21] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.708-0400 c20012| 2018-03-16T17:28:24.707-0400 I SHARDING [conn21] xxx routing info needs refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.708-0400 c20012| 2018-03-16T17:28:24.707-0400 I SHARDING [conn21] xxx calling schedule collection refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.708-0400 c20012| 2018-03-16T17:28:24.707-0400 I SHARDING [conn21] xxx is incremental [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.708-0400 c20012| 2018-03-16T17:28:24.707-0400 I SHARDING [conn21] xxx existing routing info? [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.708-0400 c20012| 2018-03-16T17:28:24.707-0400 I SHARDING [conn21] 1|0||5aac36f788c1f30185b7afa3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.708-0400 c20012| 2018-03-16T17:28:24.707-0400 I SHARDING [conn21] Refreshing chunks for collection test.foo based on version 1|0||5aac36f788c1f30185b7afa3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.709-0400 c20012| 2018-03-16T17:28:24.707-0400 I SHARDING [conn21] xxx getting chunks since now [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.709-0400 c20012| 2018-03-16T17:28:24.707-0400 I SHARDING [conn21] xxx waiting if !refreshNot [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.709-0400 c20012| 2018-03-16T17:28:24.708-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh callback [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.709-0400 c20012| 2018-03-16T17:28:24.708-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh completed [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.709-0400 c20012| 2018-03-16T17:28:24.708-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Refresh for collection test.foo took 0 ms and found version 2|0||5aac36f788c1f30185b7afa3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.709-0400 c20012| 2018-03-16T17:28:24.708-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx marking as status ok [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.709-0400 c20012| 2018-03-16T17:28:24.708-0400 I SHARDING [conn21] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.710-0400 c20012| 2018-03-16T17:28:24.708-0400 I SHARDING [conn21] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.712-0400 c20012| 2018-03-16T17:28:24.711-0400 I COMMAND [conn21] command admin.$cmd appName: "MongoDB Shell" command: _configsvrMoveChunk { _configsvrMoveChunk: 1, _id: "test.foo-x_MinKey", ns: "test.foo", min: { x: MinKey }, max: { x: MaxKey }, shard: "stale_mongos_updates_and_removes-rs0", lastmod: Timestamp(1, 0), lastmodEpoch: ObjectId('5aac36f788c1f30185b7afa3'), toShard: "stale_mongos_updates_and_removes-rs1", maxChunkSizeBytes: 67108864, secondaryThrottle: {}, waitForDelete: true, writeConcern: { w: "majority", wtimeout: 15000 }, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235704, 24), signature: { hash: BinData(0, F44F64AF6B9C0EDEC7908BFBF94A6C2809BBED78), keyId: 6533657568122765331 } }, $client: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" }, mongos: { host: "ubuntu:20015", client: "127.0.0.1:47840", version: "3.6.3" } }, $configServerState: { opTime: { ts: Timestamp(1521235703, 79), t: 1 } }, $db: "admin" } numYields:0 reslen:462 locks:{ Global: { acquireCount: { r: 14, w: 6 } }, Database: { acquireCount: { r: 4, w: 6 } }, Collection: { acquireCount: { r: 4, w: 3 } }, oplog: { acquireCount: { w: 3 } } } protocol:op_msg 618ms [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.713-0400 s20015| 2018-03-16T17:28:24.711-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 58 finished with response: { ok: 1.0, operationTime: Timestamp(1521235704, 92), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235704, 92), t: 1 }, lastOpVisible: { ts: Timestamp(1521235704, 92), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235704, 92), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235704, 92), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 92) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.713-0400 s20016| 2018-03-16T17:28:24.712-0400 I SHARDING [conn1] Refreshing chunks for collection test.foo based on version 1|0||5aac36f788c1f30185b7afa3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.713-0400 s20016| 2018-03-16T17:28:24.712-0400 D ASIO [ConfigServerCatalogCacheLoader-0] startCommand: RemoteCommand 150 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:54.712-0400 cmd:{ find: "collections", filter: { _id: "test.foo" }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235703, 79), t: 1 } }, limit: 1, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.714-0400 s20016| 2018-03-16T17:28:24.713-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 150 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.714-0400 s20016| 2018-03-16T17:28:24.713-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 150 finished with response: { cursor: { firstBatch: [ { _id: "test.foo", lastmodEpoch: ObjectId('5aac36f788c1f30185b7afa3'), lastmod: new Date(4294967296), dropped: false, key: { x: 1.0 }, unique: false, uuid: UUID("a26b230c-c773-414a-b498-fd1cdf23fe9f") } ], id: 0, ns: "config.collections" }, ok: 1.0, operationTime: Timestamp(1521235704, 92), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235704, 92), t: 1 }, lastOpVisible: { ts: Timestamp(1521235704, 92), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235703, 79), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235704, 92), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 92) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.715-0400 s20016| 2018-03-16T17:28:24.713-0400 D ASIO [ConfigServerCatalogCacheLoader-0] startCommand: RemoteCommand 152 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:54.713-0400 cmd:{ find: "chunks", filter: { ns: "test.foo", lastmod: { $gte: Timestamp(1, 0) } }, sort: { lastmod: 1 }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235704, 92), t: 1 } }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.715-0400 s20016| 2018-03-16T17:28:24.713-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 152 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.716-0400 s20016| 2018-03-16T17:28:24.713-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 152 finished with response: { cursor: { firstBatch: [ { _id: "test.foo-x_MinKey", lastmod: Timestamp(2, 0), lastmodEpoch: ObjectId('5aac36f788c1f30185b7afa3'), ns: "test.foo", min: { x: MinKey }, max: { x: MaxKey }, shard: "stale_mongos_updates_and_removes-rs1", history: [ { validAfter: Timestamp(1521235704, 61), shard: "stale_mongos_updates_and_removes-rs1" }, { validAfter: Timestamp(1521235703, 71), shard: "stale_mongos_updates_and_removes-rs0" } ] } ], id: 0, ns: "config.chunks" }, ok: 1.0, operationTime: Timestamp(1521235704, 92), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235704, 92), t: 1 }, lastOpVisible: { ts: Timestamp(1521235704, 92), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235703, 79), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235704, 92), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 92) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.716-0400 s20016| 2018-03-16T17:28:24.713-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Refresh for collection test.foo took 0 ms and found version 2|0||5aac36f788c1f30185b7afa3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.716-0400 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.716-0400 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.716-0400 ---- [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.716-0400 xxx will fail below at do remove [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.716-0400 ---- [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.717-0400 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.717-0400 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.717-0400 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.717-0400 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.717-0400 ---- [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.717-0400 xxx top single [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.717-0400 ---- [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.717-0400 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.717-0400 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.717-0400 s20016| 2018-03-16T17:28:24.715-0400 D SHARDING [conn1] trying to acquire new distributed lock for test-movePrimary ( lock timeout : 900000 ms, ping interval : 30000 ms, process : ubuntu:20016:1521235700:-4633782234644071543 ) with lockSessionID: 5aac36f8ebcdfa1d2b617d9f, why: dropCollection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.718-0400 s20016| 2018-03-16T17:28:24.715-0400 D ASIO [conn1] startCommand: RemoteCommand 154 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:54.715-0400 cmd:{ findAndModify: "locks", query: { _id: "test-movePrimary", state: 0 }, update: { $set: { ts: ObjectId('5aac36f8ebcdfa1d2b617d9f'), state: 2, who: "ubuntu:20016:1521235700:-4633782234644071543:conn1", process: "ubuntu:20016:1521235700:-4633782234644071543", when: new Date(1521235704715), why: "dropCollection" } }, upsert: true, new: true, writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.718-0400 s20016| 2018-03-16T17:28:24.715-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 154 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.725-0400 s20016| 2018-03-16T17:28:24.724-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 154 finished with response: { lastErrorObject: { n: 1, updatedExisting: true }, value: { _id: "test-movePrimary", state: 2, process: "ubuntu:20016:1521235700:-4633782234644071543", ts: ObjectId('5aac36f8ebcdfa1d2b617d9f'), when: new Date(1521235704715), who: "ubuntu:20016:1521235700:-4633782234644071543:conn1", why: "dropCollection" }, ok: 1.0, operationTime: Timestamp(1521235704, 93), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235704, 93), t: 1 }, lastOpVisible: { ts: Timestamp(1521235704, 93), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235704, 93), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235704, 93), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 93) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.725-0400 s20016| 2018-03-16T17:28:24.724-0400 I SHARDING [conn1] distributed lock 'test-movePrimary' acquired for 'dropCollection', ts : 5aac36f8ebcdfa1d2b617d9f [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.725-0400 s20016| 2018-03-16T17:28:24.724-0400 D SHARDING [conn1] trying to acquire new distributed lock for test ( lock timeout : 900000 ms, ping interval : 30000 ms, process : ubuntu:20016:1521235700:-4633782234644071543 ) with lockSessionID: 5aac36f8ebcdfa1d2b617da0, why: dropCollection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.726-0400 s20016| 2018-03-16T17:28:24.724-0400 D ASIO [conn1] startCommand: RemoteCommand 156 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:54.724-0400 cmd:{ findAndModify: "locks", query: { _id: "test", state: 0 }, update: { $set: { ts: ObjectId('5aac36f8ebcdfa1d2b617da0'), state: 2, who: "ubuntu:20016:1521235700:-4633782234644071543:conn1", process: "ubuntu:20016:1521235700:-4633782234644071543", when: new Date(1521235704724), why: "dropCollection" } }, upsert: true, new: true, writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.726-0400 s20016| 2018-03-16T17:28:24.724-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 156 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.728-0400 s20016| 2018-03-16T17:28:24.728-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 156 finished with response: { lastErrorObject: { n: 1, updatedExisting: true }, value: { _id: "test", state: 2, process: "ubuntu:20016:1521235700:-4633782234644071543", ts: ObjectId('5aac36f8ebcdfa1d2b617da0'), when: new Date(1521235704724), who: "ubuntu:20016:1521235700:-4633782234644071543:conn1", why: "dropCollection" }, ok: 1.0, operationTime: Timestamp(1521235704, 94), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235704, 94), t: 1 }, lastOpVisible: { ts: Timestamp(1521235704, 94), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235704, 94), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235704, 94), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 94) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.729-0400 s20016| 2018-03-16T17:28:24.728-0400 I SHARDING [conn1] distributed lock 'test' acquired for 'dropCollection', ts : 5aac36f8ebcdfa1d2b617da0 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.729-0400 s20016| 2018-03-16T17:28:24.728-0400 D SHARDING [conn1] trying to acquire new distributed lock for test.foo ( lock timeout : 900000 ms, ping interval : 30000 ms, process : ubuntu:20016:1521235700:-4633782234644071543 ) with lockSessionID: 5aac36f8ebcdfa1d2b617da1, why: dropCollection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.729-0400 s20016| 2018-03-16T17:28:24.728-0400 D ASIO [conn1] startCommand: RemoteCommand 158 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:54.728-0400 cmd:{ findAndModify: "locks", query: { _id: "test.foo", state: 0 }, update: { $set: { ts: ObjectId('5aac36f8ebcdfa1d2b617da1'), state: 2, who: "ubuntu:20016:1521235700:-4633782234644071543:conn1", process: "ubuntu:20016:1521235700:-4633782234644071543", when: new Date(1521235704728), why: "dropCollection" } }, upsert: true, new: true, writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.730-0400 s20016| 2018-03-16T17:28:24.728-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 158 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.732-0400 s20016| 2018-03-16T17:28:24.732-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 158 finished with response: { lastErrorObject: { n: 1, updatedExisting: true }, value: { _id: "test.foo", state: 2, process: "ubuntu:20016:1521235700:-4633782234644071543", ts: ObjectId('5aac36f8ebcdfa1d2b617da1'), when: new Date(1521235704728), who: "ubuntu:20016:1521235700:-4633782234644071543:conn1", why: "dropCollection" }, ok: 1.0, operationTime: Timestamp(1521235704, 95), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235704, 95), t: 1 }, lastOpVisible: { ts: Timestamp(1521235704, 95), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235704, 95), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235704, 95), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 95) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.733-0400 s20016| 2018-03-16T17:28:24.732-0400 I SHARDING [conn1] distributed lock 'test.foo' acquired for 'dropCollection', ts : 5aac36f8ebcdfa1d2b617da1 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.733-0400 s20016| 2018-03-16T17:28:24.732-0400 I SHARDING [conn1] about to log metadata event into changelog: { _id: "ubuntu-2018-03-16T17:28:24.732-0400-5aac36f8ebcdfa1d2b617da2", server: "ubuntu", clientAddr: "127.0.0.1:50986", time: new Date(1521235704732), what: "dropCollection.start", ns: "test.foo", details: {} } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.733-0400 s20016| 2018-03-16T17:28:24.732-0400 D ASIO [conn1] startCommand: RemoteCommand 160 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:54.732-0400 cmd:{ insert: "changelog", bypassDocumentValidation: false, ordered: true, documents: [ { _id: "ubuntu-2018-03-16T17:28:24.732-0400-5aac36f8ebcdfa1d2b617da2", server: "ubuntu", clientAddr: "127.0.0.1:50986", time: new Date(1521235704732), what: "dropCollection.start", ns: "test.foo", details: {} } ], writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.734-0400 s20016| 2018-03-16T17:28:24.732-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 160 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.738-0400 s20016| 2018-03-16T17:28:24.737-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 160 finished with response: { n: 1, opTime: { ts: Timestamp(1521235704, 96), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235704, 96), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235704, 96), t: 1 }, lastOpVisible: { ts: Timestamp(1521235704, 96), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235704, 96), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235704, 96), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 96) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.738-0400 s20016| 2018-03-16T17:28:24.737-0400 D ASIO [conn1] startCommand: RemoteCommand 162 -- target:ubuntu:20013 db:config expDate:2018-03-16T17:28:54.737-0400 cmd:{ find: "shards", readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235704, 96), t: 1 } }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.738-0400 s20016| 2018-03-16T17:28:24.737-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 162 on host ubuntu:20013 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.739-0400 s20016| 2018-03-16T17:28:24.738-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 162 finished with response: { cursor: { firstBatch: [ { _id: "stale_mongos_updates_and_removes-rs0", host: "stale_mongos_updates_and_removes-rs0/ubuntu:20010", state: 1 }, { _id: "stale_mongos_updates_and_removes-rs1", host: "stale_mongos_updates_and_removes-rs1/ubuntu:20011", state: 1 } ], id: 0, ns: "config.shards" }, ok: 1.0, operationTime: Timestamp(1521235704, 96), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235704, 96), t: 1 }, lastOpVisible: { ts: Timestamp(1521235704, 96), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235704, 96), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 96) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.739-0400 s20016| 2018-03-16T17:28:24.738-0400 D SHARDING [conn1] dropCollection test.foo started [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.739-0400 s20016| 2018-03-16T17:28:24.738-0400 D ASIO [conn1] startCommand: RemoteCommand 164 -- target:ubuntu:20010 db:test cmd:{ drop: "foo" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.740-0400 s20016| 2018-03-16T17:28:24.738-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 164 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.740-0400 d20010| 2018-03-16T17:28:24.738-0400 I COMMAND [conn21] CMD: drop test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.740-0400 d20010| 2018-03-16T17:28:24.738-0400 I STORAGE [conn21] dropCollection: test.foo (a26b230c-c773-414a-b498-fd1cdf23fe9f) - renaming to drop-pending collection: test.system.drop.1521235704i97t1.foo with drop optime { ts: Timestamp(1521235704, 97), t: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.742-0400 s20016| 2018-03-16T17:28:24.741-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 164 finished with response: { ns: "test.foo", nIndexesWas: 2, ok: 1.0, operationTime: Timestamp(1521235704, 97), $gleStats: { lastOpTime: { ts: Timestamp(1521235704, 97), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235704, 97), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 89), $configServerState: { opTime: { ts: Timestamp(1521235704, 96), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.742-0400 s20016| 2018-03-16T17:28:24.741-0400 D ASIO [conn1] startCommand: RemoteCommand 166 -- target:ubuntu:20011 db:test cmd:{ drop: "foo" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.742-0400 s20016| 2018-03-16T17:28:24.741-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 166 on host ubuntu:20011 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.742-0400 d20011| 2018-03-16T17:28:24.742-0400 I COMMAND [conn21] CMD: drop test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.743-0400 d20011| 2018-03-16T17:28:24.742-0400 I STORAGE [conn21] dropCollection: test.foo (a26b230c-c773-414a-b498-fd1cdf23fe9f) - renaming to drop-pending collection: test.system.drop.1521235704i98t1.foo with drop optime { ts: Timestamp(1521235704, 98), t: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.744-0400 s20016| 2018-03-16T17:28:24.744-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 166 finished with response: { ns: "test.foo", nIndexesWas: 2, ok: 1.0, operationTime: Timestamp(1521235704, 98), $gleStats: { lastOpTime: { ts: Timestamp(1521235704, 98), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235704, 98), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 57), $configServerState: { opTime: { ts: Timestamp(1521235704, 96), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.745-0400 s20016| 2018-03-16T17:28:24.744-0400 D SHARDING [conn1] dropCollection test.foo shard data deleted [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.745-0400 s20016| 2018-03-16T17:28:24.744-0400 D ASIO [conn1] startCommand: RemoteCommand 168 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:54.744-0400 cmd:{ delete: "chunks", bypassDocumentValidation: false, ordered: true, deletes: [ { q: { ns: "test.foo" }, limit: 0 } ], writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.745-0400 s20016| 2018-03-16T17:28:24.744-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 168 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.749-0400 s20016| 2018-03-16T17:28:24.748-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 168 finished with response: { n: 1, opTime: { ts: Timestamp(1521235704, 99), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235704, 99), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235704, 99), t: 1 }, lastOpVisible: { ts: Timestamp(1521235704, 99), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235704, 99), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235704, 99), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 99) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.749-0400 s20016| 2018-03-16T17:28:24.748-0400 D SHARDING [conn1] dropCollection test.foo chunk data deleted [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.749-0400 s20016| 2018-03-16T17:28:24.748-0400 D ASIO [conn1] startCommand: RemoteCommand 170 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:54.748-0400 cmd:{ update: "collections", bypassDocumentValidation: false, ordered: true, updates: [ { q: { _id: "test.foo" }, u: { _id: "test.foo", lastmodEpoch: ObjectId('000000000000000000000000'), lastmod: new Date(1521235704748), dropped: true }, multi: false, upsert: false } ], writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.750-0400 s20016| 2018-03-16T17:28:24.748-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 170 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.750-0400 d20010| 2018-03-16T17:28:24.750-0400 I REPL [replication-0] Completing collection drop for test.system.drop.1521235704i97t1.foo with drop optime { ts: Timestamp(1521235704, 97), t: 1 } (notification optime: { ts: Timestamp(1521235704, 97), t: 1 }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.750-0400 d20010| 2018-03-16T17:28:24.750-0400 I STORAGE [replication-0] Finishing collection drop for test.system.drop.1521235704i97t1.foo (a26b230c-c773-414a-b498-fd1cdf23fe9f). [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.754-0400 s20016| 2018-03-16T17:28:24.753-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 170 finished with response: { n: 1, nModified: 1, opTime: { ts: Timestamp(1521235704, 100), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235704, 100), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235704, 100), t: 1 }, lastOpVisible: { ts: Timestamp(1521235704, 100), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235704, 100), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235704, 100), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 100) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.754-0400 s20016| 2018-03-16T17:28:24.754-0400 D SHARDING [conn1] dropCollection test.foo collection marked as dropped [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.755-0400 s20016| 2018-03-16T17:28:24.754-0400 D ASIO [conn1] startCommand: RemoteCommand 172 -- target:ubuntu:20010 db:admin cmd:{ setShardVersion: "test.foo", init: false, authoritative: true, configdb: "stale_mongos_updates_and_removes-configRS/ubuntu:20012,ubuntu:20013,ubuntu:20014", shard: "stale_mongos_updates_and_removes-rs0", shardHost: "stale_mongos_updates_and_removes-rs0/ubuntu:20010", version: Timestamp(0, 0), versionEpoch: ObjectId('000000000000000000000000'), noConnectionVersioning: true } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.755-0400 s20016| 2018-03-16T17:28:24.754-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 172 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.755-0400 d20010| 2018-03-16T17:28:24.754-0400 I SHARDING [conn21] XXXX in set shard version cmd obj is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.756-0400 d20010| 2018-03-16T17:28:24.754-0400 I SHARDING [conn21] { setShardVersion: "test.foo", init: false, authoritative: true, configdb: "stale_mongos_updates_and_removes-configRS/ubuntu:20012,ubuntu:20013,ubuntu:20014", shard: "stale_mongos_updates_and_removes-rs0", shardHost: "stale_mongos_updates_and_removes-rs0/ubuntu:20010", version: Timestamp(0, 0), versionEpoch: ObjectId('000000000000000000000000'), noConnectionVersioning: true, $clusterTime: { clusterTime: Timestamp(1521235704, 100), signature: { hash: BinData(0, F44F64AF6B9C0EDEC7908BFBF94A6C2809BBED78), keyId: 6533657568122765331 } }, $client: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" }, mongos: { host: "ubuntu:20016", client: "127.0.0.1:50986", version: "3.6.3" } }, $configServerState: { opTime: { ts: Timestamp(1521235704, 100), t: 1 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.756-0400 d20010| 2018-03-16T17:28:24.754-0400 I SHARDING [conn21] XXXX requested version is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.756-0400 d20010| 2018-03-16T17:28:24.754-0400 I SHARDING [conn21] 0|0||000000000000000000000000 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.756-0400 d20010| 2018-03-16T17:28:24.755-0400 I SHARDING [conn21] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.756-0400 d20010| 2018-03-16T17:28:24.756-0400 I SHARDING [conn21] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.757-0400 d20010| 2018-03-16T17:28:24.756-0400 I SHARDING [conn21] xxx routing info needs refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.757-0400 d20010| 2018-03-16T17:28:24.756-0400 I SHARDING [conn21] xxx calling schedule collection refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.757-0400 d20010| 2018-03-16T17:28:24.756-0400 I SHARDING [conn21] xxx is incremental [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.757-0400 d20010| 2018-03-16T17:28:24.756-0400 I SHARDING [conn21] xxx existing routing info? [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.757-0400 d20010| 2018-03-16T17:28:24.756-0400 I SHARDING [conn21] 2|0||5aac36f788c1f30185b7afa3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.757-0400 d20010| 2018-03-16T17:28:24.756-0400 I SHARDING [conn21] Refreshing chunks for collection test.foo based on version 2|0||5aac36f788c1f30185b7afa3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.757-0400 d20010| 2018-03-16T17:28:24.756-0400 I SHARDING [conn21] xxx getting chunks since now [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.758-0400 d20010| 2018-03-16T17:28:24.756-0400 I SHARDING [conn21] xxx getting chunks since [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.758-0400 d20010| 2018-03-16T17:28:24.756-0400 I SHARDING [conn21] xxx waiting if !refreshNot [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.758-0400 d20010| 2018-03-16T17:28:24.756-0400 I SHARDING [ShardServerCatalogCacheLoader-0] xxx primary and calling get chunks since for primary [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.758-0400 d20010| 2018-03-16T17:28:24.756-0400 I SHARDING [ShardServerCatalogCacheLoader-0] version is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.758-0400 d20010| 2018-03-16T17:28:24.756-0400 I SHARDING [ShardServerCatalogCacheLoader-0] 2|0||5aac36f788c1f30185b7afa3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.758-0400 d20010| 2018-03-16T17:28:24.756-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] XXXX namespace not found [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.788-0400 d20011| 2018-03-16T17:28:24.788-0400 I REPL [replication-0] Completing collection drop for test.system.drop.1521235704i98t1.foo with drop optime { ts: Timestamp(1521235704, 98), t: 1 } (notification optime: { ts: Timestamp(1521235704, 98), t: 1 }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.788-0400 d20011| 2018-03-16T17:28:24.788-0400 I STORAGE [replication-0] Finishing collection drop for test.system.drop.1521235704i98t1.foo (a26b230c-c773-414a-b498-fd1cdf23fe9f). [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.789-0400 d20010| 2018-03-16T17:28:24.789-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Cache loader remotely refreshed for collection test.foo from version 2|0||5aac36f788c1f30185b7afa3 and no metadata was found. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.789-0400 d20010| 2018-03-16T17:28:24.789-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx finsihing refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.790-0400 d20010| 2018-03-16T17:28:24.789-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh callback [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.790-0400 d20010| 2018-03-16T17:28:24.789-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh completed [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.790-0400 d20010| 2018-03-16T17:28:24.789-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Refresh for collection test.foo took 33 ms and found the collection is not sharded [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.790-0400 d20010| 2018-03-16T17:28:24.789-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx marking as status ok [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.790-0400 d20010| 2018-03-16T17:28:24.789-0400 I SHARDING [conn21] Marking collection test.foo with collection version: 2|0||5aac36f788c1f30185b7afa3, shard version: 0|0||5aac36f788c1f30185b7afa3 as unsharded [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.790-0400 d20010| 2018-03-16T17:28:24.789-0400 I COMMAND [ShardServerCatalogCacheLoader-0] CMD: drop config.cache.chunks.test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.791-0400 d20010| 2018-03-16T17:28:24.789-0400 I STORAGE [ShardServerCatalogCacheLoader-0] dropCollection: config.cache.chunks.test.foo (4ca2771c-4d81-478a-b7c8-92920f3c4431) - renaming to drop-pending collection: config.system.drop.1521235704i103t1.cache.chunks.test.foo with drop optime { ts: Timestamp(1521235704, 103), t: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.791-0400 s20016| 2018-03-16T17:28:24.789-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 172 finished with response: { oldVersion: Timestamp(0, 0), oldVersionEpoch: ObjectId('000000000000000000000000'), ok: 1.0, operationTime: Timestamp(1521235704, 102), $gleStats: { lastOpTime: { ts: Timestamp(1521235704, 97), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235704, 102), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 101), $configServerState: { opTime: { ts: Timestamp(1521235704, 100), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.791-0400 s20016| 2018-03-16T17:28:24.789-0400 D ASIO [conn1] startCommand: RemoteCommand 174 -- target:ubuntu:20010 db:admin cmd:{ unsetSharding: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.792-0400 s20016| 2018-03-16T17:28:24.789-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 174 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.792-0400 s20016| 2018-03-16T17:28:24.789-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 174 finished with response: { ok: 1.0, operationTime: Timestamp(1521235704, 102), $gleStats: { lastOpTime: { ts: Timestamp(1521235704, 97), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235704, 103), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 101), $configServerState: { opTime: { ts: Timestamp(1521235704, 100), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.793-0400 s20016| 2018-03-16T17:28:24.790-0400 D ASIO [conn1] startCommand: RemoteCommand 176 -- target:ubuntu:20011 db:admin cmd:{ setShardVersion: "test.foo", init: false, authoritative: true, configdb: "stale_mongos_updates_and_removes-configRS/ubuntu:20012,ubuntu:20013,ubuntu:20014", shard: "stale_mongos_updates_and_removes-rs1", shardHost: "stale_mongos_updates_and_removes-rs1/ubuntu:20011", version: Timestamp(0, 0), versionEpoch: ObjectId('000000000000000000000000'), noConnectionVersioning: true } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.793-0400 s20016| 2018-03-16T17:28:24.790-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 176 on host ubuntu:20011 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.793-0400 d20011| 2018-03-16T17:28:24.790-0400 I SHARDING [conn21] XXXX in set shard version cmd obj is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.794-0400 d20011| 2018-03-16T17:28:24.790-0400 I SHARDING [conn21] { setShardVersion: "test.foo", init: false, authoritative: true, configdb: "stale_mongos_updates_and_removes-configRS/ubuntu:20012,ubuntu:20013,ubuntu:20014", shard: "stale_mongos_updates_and_removes-rs1", shardHost: "stale_mongos_updates_and_removes-rs1/ubuntu:20011", version: Timestamp(0, 0), versionEpoch: ObjectId('000000000000000000000000'), noConnectionVersioning: true, $clusterTime: { clusterTime: Timestamp(1521235704, 103), signature: { hash: BinData(0, F44F64AF6B9C0EDEC7908BFBF94A6C2809BBED78), keyId: 6533657568122765331 } }, $client: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" }, mongos: { host: "ubuntu:20016", client: "127.0.0.1:50986", version: "3.6.3" } }, $configServerState: { opTime: { ts: Timestamp(1521235704, 100), t: 1 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.794-0400 d20011| 2018-03-16T17:28:24.790-0400 I SHARDING [conn21] XXXX requested version is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.794-0400 d20011| 2018-03-16T17:28:24.790-0400 I SHARDING [conn21] 0|0||000000000000000000000000 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.795-0400 d20011| 2018-03-16T17:28:24.795-0400 I SHARDING [conn21] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.795-0400 d20011| 2018-03-16T17:28:24.795-0400 I SHARDING [conn21] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.795-0400 d20011| 2018-03-16T17:28:24.795-0400 I SHARDING [conn21] xxx routing info needs refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.795-0400 d20011| 2018-03-16T17:28:24.795-0400 I SHARDING [conn21] xxx calling schedule collection refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.795-0400 d20011| 2018-03-16T17:28:24.795-0400 I SHARDING [conn21] xxx is incremental [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.796-0400 d20011| 2018-03-16T17:28:24.795-0400 I SHARDING [conn21] xxx existing routing info? [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.796-0400 d20011| 2018-03-16T17:28:24.795-0400 I SHARDING [conn21] 1|0||5aac36f788c1f30185b7afa3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.796-0400 d20011| 2018-03-16T17:28:24.795-0400 I SHARDING [conn21] Refreshing chunks for collection test.foo based on version 1|0||5aac36f788c1f30185b7afa3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.796-0400 d20011| 2018-03-16T17:28:24.795-0400 I SHARDING [conn21] xxx getting chunks since now [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.796-0400 d20011| 2018-03-16T17:28:24.795-0400 I SHARDING [conn21] xxx getting chunks since [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.796-0400 d20011| 2018-03-16T17:28:24.795-0400 I SHARDING [conn21] xxx waiting if !refreshNot [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.796-0400 d20011| 2018-03-16T17:28:24.795-0400 I SHARDING [ShardServerCatalogCacheLoader-0] xxx primary and calling get chunks since for primary [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.796-0400 d20011| 2018-03-16T17:28:24.795-0400 I SHARDING [ShardServerCatalogCacheLoader-0] version is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.797-0400 d20011| 2018-03-16T17:28:24.795-0400 I SHARDING [ShardServerCatalogCacheLoader-0] 1|0||5aac36f788c1f30185b7afa3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.797-0400 d20011| 2018-03-16T17:28:24.795-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] XXXX namespace not found [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.844-0400 d20011| 2018-03-16T17:28:24.844-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Cache loader remotely refreshed for collection test.foo from version 1|0||5aac36f788c1f30185b7afa3 and no metadata was found. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.844-0400 d20011| 2018-03-16T17:28:24.844-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx finsihing refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.844-0400 d20011| 2018-03-16T17:28:24.844-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh callback [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.844-0400 d20011| 2018-03-16T17:28:24.844-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh completed [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.845-0400 d20011| 2018-03-16T17:28:24.844-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Refresh for collection test.foo took 49 ms and found the collection is not sharded [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.845-0400 d20011| 2018-03-16T17:28:24.844-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx marking as status ok [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.845-0400 d20011| 2018-03-16T17:28:24.844-0400 I SHARDING [conn21] Marking collection test.foo with collection version: 1|0||5aac36f788c1f30185b7afa3, shard version: 0|0||5aac36f788c1f30185b7afa3 as unsharded [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.845-0400 d20011| 2018-03-16T17:28:24.844-0400 I COMMAND [ShardServerCatalogCacheLoader-0] CMD: drop config.cache.chunks.test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.845-0400 d20011| 2018-03-16T17:28:24.844-0400 I STORAGE [ShardServerCatalogCacheLoader-0] dropCollection: config.cache.chunks.test.foo (74f4c770-00b6-43f4-8263-89eac02e3c28) - renaming to drop-pending collection: config.system.drop.1521235704i106t1.cache.chunks.test.foo with drop optime { ts: Timestamp(1521235704, 106), t: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.846-0400 s20016| 2018-03-16T17:28:24.844-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 176 finished with response: { oldVersion: Timestamp(0, 0), oldVersionEpoch: ObjectId('000000000000000000000000'), ok: 1.0, operationTime: Timestamp(1521235704, 105), $gleStats: { lastOpTime: { ts: Timestamp(1521235704, 98), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235704, 105), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 104), $configServerState: { opTime: { ts: Timestamp(1521235704, 100), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.846-0400 s20016| 2018-03-16T17:28:24.844-0400 D ASIO [conn1] startCommand: RemoteCommand 178 -- target:ubuntu:20011 db:admin cmd:{ unsetSharding: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.846-0400 s20016| 2018-03-16T17:28:24.844-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 178 on host ubuntu:20011 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.847-0400 s20016| 2018-03-16T17:28:24.844-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 178 finished with response: { ok: 1.0, operationTime: Timestamp(1521235704, 105), $gleStats: { lastOpTime: { ts: Timestamp(1521235704, 98), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235704, 106), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 104), $configServerState: { opTime: { ts: Timestamp(1521235704, 100), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.847-0400 s20016| 2018-03-16T17:28:24.844-0400 D SHARDING [conn1] dropCollection test.foo completed [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.847-0400 s20016| 2018-03-16T17:28:24.844-0400 I SHARDING [conn1] about to log metadata event into changelog: { _id: "ubuntu-2018-03-16T17:28:24.844-0400-5aac36f8ebcdfa1d2b617da3", server: "ubuntu", clientAddr: "127.0.0.1:50986", time: new Date(1521235704844), what: "dropCollection", ns: "test.foo", details: {} } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.848-0400 s20016| 2018-03-16T17:28:24.844-0400 D ASIO [conn1] startCommand: RemoteCommand 180 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:54.844-0400 cmd:{ insert: "changelog", bypassDocumentValidation: false, ordered: true, documents: [ { _id: "ubuntu-2018-03-16T17:28:24.844-0400-5aac36f8ebcdfa1d2b617da3", server: "ubuntu", clientAddr: "127.0.0.1:50986", time: new Date(1521235704844), what: "dropCollection", ns: "test.foo", details: {} } ], writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.848-0400 s20016| 2018-03-16T17:28:24.844-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 180 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.848-0400 s20016| 2018-03-16T17:28:24.848-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 180 finished with response: { n: 1, opTime: { ts: Timestamp(1521235704, 107), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235704, 107), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235704, 107), t: 1 }, lastOpVisible: { ts: Timestamp(1521235704, 107), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235704, 107), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235704, 107), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 107) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.849-0400 s20016| 2018-03-16T17:28:24.848-0400 D ASIO [conn1] startCommand: RemoteCommand 182 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:54.848-0400 cmd:{ findAndModify: "locks", query: { ts: ObjectId('5aac36f8ebcdfa1d2b617da1') }, update: { $set: { state: 0 } }, writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.849-0400 s20016| 2018-03-16T17:28:24.848-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 182 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.855-0400 s20016| 2018-03-16T17:28:24.854-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 182 finished with response: { lastErrorObject: { n: 1, updatedExisting: true }, value: { _id: "test.foo", state: 2, process: "ubuntu:20016:1521235700:-4633782234644071543", ts: ObjectId('5aac36f8ebcdfa1d2b617da1'), when: new Date(1521235704728), who: "ubuntu:20016:1521235700:-4633782234644071543:conn1", why: "dropCollection" }, ok: 1.0, operationTime: Timestamp(1521235704, 108), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235704, 108), t: 1 }, lastOpVisible: { ts: Timestamp(1521235704, 108), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235704, 108), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235704, 108), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 108) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.855-0400 s20016| 2018-03-16T17:28:24.854-0400 I SHARDING [conn1] distributed lock with ts: 5aac36f8ebcdfa1d2b617da1' unlocked. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.855-0400 s20016| 2018-03-16T17:28:24.854-0400 D ASIO [conn1] startCommand: RemoteCommand 184 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:54.854-0400 cmd:{ findAndModify: "locks", query: { ts: ObjectId('5aac36f8ebcdfa1d2b617da0') }, update: { $set: { state: 0 } }, writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.855-0400 s20016| 2018-03-16T17:28:24.854-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 184 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.856-0400 d20010| 2018-03-16T17:28:24.856-0400 I REPL [replication-0] Completing collection drop for config.system.drop.1521235704i103t1.cache.chunks.test.foo with drop optime { ts: Timestamp(1521235704, 103), t: 1 } (notification optime: { ts: Timestamp(1521235704, 103), t: 1 }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.857-0400 d20010| 2018-03-16T17:28:24.856-0400 I STORAGE [replication-0] Finishing collection drop for config.system.drop.1521235704i103t1.cache.chunks.test.foo (4ca2771c-4d81-478a-b7c8-92920f3c4431). [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.859-0400 s20016| 2018-03-16T17:28:24.859-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 184 finished with response: { lastErrorObject: { n: 1, updatedExisting: true }, value: { _id: "test", state: 2, process: "ubuntu:20016:1521235700:-4633782234644071543", ts: ObjectId('5aac36f8ebcdfa1d2b617da0'), when: new Date(1521235704724), who: "ubuntu:20016:1521235700:-4633782234644071543:conn1", why: "dropCollection" }, ok: 1.0, operationTime: Timestamp(1521235704, 109), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235704, 109), t: 1 }, lastOpVisible: { ts: Timestamp(1521235704, 109), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235704, 109), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235704, 109), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 109) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.860-0400 s20016| 2018-03-16T17:28:24.859-0400 I SHARDING [conn1] distributed lock with ts: 5aac36f8ebcdfa1d2b617da0' unlocked. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.860-0400 s20016| 2018-03-16T17:28:24.859-0400 D ASIO [conn1] startCommand: RemoteCommand 186 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:54.859-0400 cmd:{ findAndModify: "locks", query: { ts: ObjectId('5aac36f8ebcdfa1d2b617d9f') }, update: { $set: { state: 0 } }, writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.860-0400 s20016| 2018-03-16T17:28:24.859-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 186 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.866-0400 s20016| 2018-03-16T17:28:24.865-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 186 finished with response: { lastErrorObject: { n: 1, updatedExisting: true }, value: { _id: "test-movePrimary", state: 2, process: "ubuntu:20016:1521235700:-4633782234644071543", ts: ObjectId('5aac36f8ebcdfa1d2b617d9f'), when: new Date(1521235704715), who: "ubuntu:20016:1521235700:-4633782234644071543:conn1", why: "dropCollection" }, ok: 1.0, operationTime: Timestamp(1521235704, 110), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235704, 110), t: 1 }, lastOpVisible: { ts: Timestamp(1521235704, 110), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235704, 110), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235704, 110), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 110) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.866-0400 s20016| 2018-03-16T17:28:24.865-0400 I SHARDING [conn1] distributed lock with ts: 5aac36f8ebcdfa1d2b617d9f' unlocked. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.866-0400 s20016| 2018-03-16T17:28:24.866-0400 D ASIO [conn1] startCommand: RemoteCommand 188 -- target:ubuntu:20012 db:admin cmd:{ _configsvrShardCollection: "test.foo", key: { x: 1.0 }, unique: false, numInitialChunks: 0, writeConcern: { w: "majority", wtimeout: 60000 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.867-0400 s20016| 2018-03-16T17:28:24.866-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 188 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.870-0400 c20012| 2018-03-16T17:28:24.869-0400 I SHARDING [conn23] distributed lock 'test' acquired for 'shardCollection', ts : 5aac36f888c1f30185b7b09f [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.875-0400 c20012| 2018-03-16T17:28:24.874-0400 I SHARDING [conn23] distributed lock 'test.foo' acquired for 'shardCollection', ts : 5aac36f888c1f30185b7b0a4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.875-0400 d20010| 2018-03-16T17:28:24.875-0400 I STORAGE [conn15] createCollection: test.foo with generated UUID: 650ad6da-cccc-457f-bb84-294856863b67 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.890-0400 d20011| 2018-03-16T17:28:24.890-0400 I REPL [replication-0] Completing collection drop for config.system.drop.1521235704i106t1.cache.chunks.test.foo with drop optime { ts: Timestamp(1521235704, 106), t: 1 } (notification optime: { ts: Timestamp(1521235704, 106), t: 1 }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.890-0400 d20011| 2018-03-16T17:28:24.890-0400 I STORAGE [replication-0] Finishing collection drop for config.system.drop.1521235704i106t1.cache.chunks.test.foo (74f4c770-00b6-43f4-8263-89eac02e3c28). [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.904-0400 d20010| 2018-03-16T17:28:24.903-0400 I INDEX [conn15] build index on: test.foo properties: { v: 2, key: { x: 1.0 }, name: "x_1", ns: "test.foo" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.904-0400 d20010| 2018-03-16T17:28:24.903-0400 I INDEX [conn15] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.904-0400 d20010| 2018-03-16T17:28:24.904-0400 I INDEX [conn15] build index done. scanned 0 total records. 0 secs [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.906-0400 c20012| 2018-03-16T17:28:24.906-0400 I SHARDING [conn23] CMD: shardcollection: { _configsvrShardCollection: "test.foo", key: { x: 1.0 }, unique: false, numInitialChunks: 0, writeConcern: { w: "majority", wtimeout: 60000 }, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235704, 110), signature: { hash: BinData(0, F44F64AF6B9C0EDEC7908BFBF94A6C2809BBED78), keyId: 6533657568122765331 } }, $client: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" }, mongos: { host: "ubuntu:20016", client: "127.0.0.1:50986", version: "3.6.3" } }, $configServerState: { opTime: { ts: Timestamp(1521235704, 110), t: 1 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.907-0400 c20012| 2018-03-16T17:28:24.906-0400 I SHARDING [conn23] about to log metadata event into changelog: { _id: "ubuntu-2018-03-16T17:28:24.906-0400-5aac36f888c1f30185b7b0b6", server: "ubuntu", clientAddr: "127.0.0.1:60682", time: new Date(1521235704906), what: "shardCollection.start", ns: "test.foo", details: { shardKey: { x: 1.0 }, collection: "test.foo", uuid: UUID("650ad6da-cccc-457f-bb84-294856863b67"), primary: "stale_mongos_updates_and_removes-rs0:stale_mongos_updates_and_removes-rs0/ubuntu:20010", numChunks: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.910-0400 c20012| 2018-03-16T17:28:24.910-0400 I SHARDING [conn23] going to create 1 chunk(s) for: test.foo using new epoch 5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.918-0400 d20010| 2018-03-16T17:28:24.918-0400 I SHARDING [conn15] XXXX in set shard version cmd obj is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.919-0400 d20010| 2018-03-16T17:28:24.918-0400 I SHARDING [conn15] { setShardVersion: "test.foo", init: false, authoritative: true, configdb: "stale_mongos_updates_and_removes-configRS/ubuntu:20012,ubuntu:20013,ubuntu:20014", shard: "stale_mongos_updates_and_removes-rs0", shardHost: "stale_mongos_updates_and_removes-rs0/ubuntu:20010", version: Timestamp(1, 0), versionEpoch: ObjectId('5aac36f888c1f30185b7b0c4'), noConnectionVersioning: true, $clusterTime: { clusterTime: Timestamp(1521235704, 117), signature: { hash: BinData(0, F44F64AF6B9C0EDEC7908BFBF94A6C2809BBED78), keyId: 6533657568122765331 } }, $client: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" }, mongos: { host: "ubuntu:20016", client: "127.0.0.1:50986", version: "3.6.3" } }, $configServerState: { opTime: { ts: Timestamp(1521235704, 117), t: 1 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.919-0400 d20010| 2018-03-16T17:28:24.918-0400 I SHARDING [conn15] XXXX requested version is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.919-0400 d20010| 2018-03-16T17:28:24.918-0400 I SHARDING [conn15] 1|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.919-0400 d20010| 2018-03-16T17:28:24.918-0400 I SHARDING [conn15] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.920-0400 d20010| 2018-03-16T17:28:24.918-0400 I SHARDING [conn15] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.920-0400 d20010| 2018-03-16T17:28:24.918-0400 I SHARDING [conn15] xxx routing info needs refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.920-0400 d20010| 2018-03-16T17:28:24.918-0400 I SHARDING [conn15] xxx calling schedule collection refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.920-0400 d20010| 2018-03-16T17:28:24.918-0400 I SHARDING [conn15] xxx is not incr [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.920-0400 d20010| 2018-03-16T17:28:24.918-0400 I SHARDING [conn15] xxx existing routing info? [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.920-0400 d20010| 2018-03-16T17:28:24.918-0400 I SHARDING [conn15] Refreshing chunks for collection test.foo based on version 0|0||000000000000000000000000 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.920-0400 d20010| 2018-03-16T17:28:24.918-0400 I SHARDING [conn15] xxx getting chunks since now [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.920-0400 d20010| 2018-03-16T17:28:24.918-0400 I SHARDING [conn15] xxx getting chunks since [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.921-0400 d20010| 2018-03-16T17:28:24.918-0400 I SHARDING [conn15] xxx waiting if !refreshNot [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.921-0400 d20010| 2018-03-16T17:28:24.918-0400 I SHARDING [ShardServerCatalogCacheLoader-0] xxx primary and calling get chunks since for primary [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.921-0400 d20010| 2018-03-16T17:28:24.918-0400 I SHARDING [ShardServerCatalogCacheLoader-0] version is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.921-0400 d20010| 2018-03-16T17:28:24.918-0400 I SHARDING [ShardServerCatalogCacheLoader-0] 0|0||000000000000000000000000 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.921-0400 d20010| 2018-03-16T17:28:24.919-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] refresh in primary [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.921-0400 d20010| 2018-03-16T17:28:24.919-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] 1 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.921-0400 d20010| 2018-03-16T17:28:24.919-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx collec and changed chunks max epoch or version doesn't equal max [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.993-0400 d20010| 2018-03-16T17:28:24.992-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Cache loader remotely refreshed for collection test.foo from collection version 0|0||000000000000000000000000 and found collection version 1|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.993-0400 d20010| 2018-03-16T17:28:24.992-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Cache loader found enqueued metadata from 1|0||5aac36f888c1f30185b7b0c4 to 1|0||5aac36f888c1f30185b7b0c4 and no persisted metadata, GTE cache version 0|0||000000000000000000000000 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.993-0400 d20010| 2018-03-16T17:28:24.992-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx is okay [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.993-0400 d20010| 2018-03-16T17:28:24.992-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx finsihing refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.993-0400 d20010| 2018-03-16T17:28:24.992-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh callback [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.994-0400 d20010| 2018-03-16T17:28:24.992-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh completed [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.994-0400 d20010| 2018-03-16T17:28:24.992-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Refresh for collection test.foo took 74 ms and found version 1|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.994-0400 d20010| 2018-03-16T17:28:24.992-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx marking as status ok [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.994-0400 d20010| 2018-03-16T17:28:24.993-0400 I SHARDING [conn15] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.994-0400 d20010| 2018-03-16T17:28:24.993-0400 I SHARDING [conn15] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.994-0400 d20010| 2018-03-16T17:28:24.993-0400 I SHARDING [conn15] Marking collection test.foo as sharded with collection version: 1|0||5aac36f888c1f30185b7b0c4, shard version: 1|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.995-0400 d20010| 2018-03-16T17:28:24.993-0400 I STORAGE [ShardServerCatalogCacheLoader-0] createCollection: config.cache.chunks.test.foo with generated UUID: 1659e1aa-b5d3-40dc-919d-6d4e2c140fb0 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:24.995-0400 c20012| 2018-03-16T17:28:24.993-0400 I SHARDING [conn23] about to log metadata event into changelog: { _id: "ubuntu-2018-03-16T17:28:24.993-0400-5aac36f888c1f30185b7b0d8", server: "ubuntu", clientAddr: "127.0.0.1:60682", time: new Date(1521235704993), what: "shardCollection.end", ns: "test.foo", details: { version: "1|0||5aac36f888c1f30185b7b0c4" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.003-0400 c20012| 2018-03-16T17:28:25.003-0400 I SHARDING [conn23] distributed lock with ts: 5aac36f888c1f30185b7b0a4' unlocked. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.007-0400 c20012| 2018-03-16T17:28:25.007-0400 I SHARDING [conn23] distributed lock with ts: 5aac36f888c1f30185b7b09f' unlocked. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.008-0400 c20012| 2018-03-16T17:28:25.007-0400 I COMMAND [conn23] command admin.$cmd appName: "MongoDB Shell" command: _configsvrShardCollection { _configsvrShardCollection: "test.foo", key: { x: 1.0 }, unique: false, numInitialChunks: 0, writeConcern: { w: "majority", wtimeout: 60000 }, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235704, 110), signature: { hash: BinData(0, F44F64AF6B9C0EDEC7908BFBF94A6C2809BBED78), keyId: 6533657568122765331 } }, $client: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" }, mongos: { host: "ubuntu:20016", client: "127.0.0.1:50986", version: "3.6.3" } }, $configServerState: { opTime: { ts: Timestamp(1521235704, 110), t: 1 } }, $db: "admin" } numYields:0 reslen:531 locks:{ Global: { acquireCount: { r: 30, w: 16 } }, Database: { acquireCount: { r: 7, w: 16 } }, Collection: { acquireCount: { r: 7, w: 8 } }, Metadata: { acquireCount: { W: 2 } }, oplog: { acquireCount: { w: 8 } } } protocol:op_msg 140ms [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.009-0400 s20016| 2018-03-16T17:28:25.007-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 188 finished with response: { collectionsharded: "test.foo", collectionUUID: UUID("650ad6da-cccc-457f-bb84-294856863b67"), ok: 1.0, operationTime: Timestamp(1521235705, 1), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235705, 1), t: 1 }, lastOpVisible: { ts: Timestamp(1521235705, 1), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235705, 1), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235705, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235705, 1) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.009-0400 s20016| 2018-03-16T17:28:25.008-0400 I SHARDING [conn1] Refreshing chunks for collection test.foo based on version 2|0||5aac36f788c1f30185b7afa3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.009-0400 s20016| 2018-03-16T17:28:25.008-0400 D ASIO [ConfigServerCatalogCacheLoader-0] startCommand: RemoteCommand 190 -- target:ubuntu:20014 db:config expDate:2018-03-16T17:28:55.008-0400 cmd:{ find: "collections", filter: { _id: "test.foo" }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235705, 1), t: 1 } }, limit: 1, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.009-0400 s20016| 2018-03-16T17:28:25.008-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 190 on host ubuntu:20014 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.010-0400 s20016| 2018-03-16T17:28:25.008-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 190 finished with response: { cursor: { firstBatch: [ { _id: "test.foo", lastmodEpoch: ObjectId('5aac36f888c1f30185b7b0c4'), lastmod: new Date(4294967296), dropped: false, key: { x: 1.0 }, unique: false, uuid: UUID("650ad6da-cccc-457f-bb84-294856863b67") } ], id: 0, ns: "config.collections" }, ok: 1.0, operationTime: Timestamp(1521235705, 1), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235705, 1), t: 1 }, lastOpVisible: { ts: Timestamp(1521235705, 1), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235705, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235705, 1) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.010-0400 s20016| 2018-03-16T17:28:25.008-0400 D ASIO [ConfigServerCatalogCacheLoader-0] startCommand: RemoteCommand 192 -- target:ubuntu:20014 db:config expDate:2018-03-16T17:28:55.008-0400 cmd:{ find: "chunks", filter: { ns: "test.foo", lastmod: { $gte: Timestamp(0, 0) } }, sort: { lastmod: 1 }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235705, 1), t: 1 } }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.011-0400 s20016| 2018-03-16T17:28:25.008-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 192 on host ubuntu:20014 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.011-0400 s20016| 2018-03-16T17:28:25.009-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 192 finished with response: { cursor: { firstBatch: [ { _id: "test.foo-x_MinKey", ns: "test.foo", min: { x: MinKey }, max: { x: MaxKey }, shard: "stale_mongos_updates_and_removes-rs0", lastmod: Timestamp(1, 0), lastmodEpoch: ObjectId('5aac36f888c1f30185b7b0c4'), history: [ { validAfter: Timestamp(1521235704, 115), shard: "stale_mongos_updates_and_removes-rs0" } ] } ], id: 0, ns: "config.chunks" }, ok: 1.0, operationTime: Timestamp(1521235705, 1), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235705, 1), t: 1 }, lastOpVisible: { ts: Timestamp(1521235705, 1), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235705, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235705, 1) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.012-0400 s20016| 2018-03-16T17:28:25.009-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Refresh for collection test.foo took 1 ms and found version 1|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.012-0400 s20016| 2018-03-16T17:28:25.009-0400 D ASIO [conn1] startCommand: RemoteCommand 194 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36f91085bf286ecf4d22'), x: 0.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f888c1f30185b7b0c4') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.012-0400 s20016| 2018-03-16T17:28:25.009-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-22-0] Connecting to ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.012-0400 s20016| 2018-03-16T17:28:25.009-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-22-0] Starting asynchronous command 195 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.012-0400 d20010| 2018-03-16T17:28:25.009-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56500 #44 (35 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.013-0400 d20010| 2018-03-16T17:28:25.009-0400 I NETWORK [conn44] received client metadata from 127.0.0.1:56500 conn44: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-22", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.013-0400 s20016| 2018-03-16T17:28:25.010-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-22-0] Successfully connected to ubuntu:20010, took 1ms (1 connections now open to ubuntu:20010) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.013-0400 s20016| 2018-03-16T17:28:25.010-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-22-0] Request 195 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.013-0400 s20016| 2018-03-16T17:28:25.010-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-22-0] Starting asynchronous command 194 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.014-0400 s20016| 2018-03-16T17:28:25.013-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-22-0] Request 194 finished with response: { n: 1, opTime: { ts: Timestamp(1521235705, 2), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235705, 2), $gleStats: { lastOpTime: { ts: Timestamp(1521235705, 2), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235705, 2), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 118), $configServerState: { opTime: { ts: Timestamp(1521235705, 1), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.014-0400 s20016| 2018-03-16T17:28:25.014-0400 D ASIO [conn1] startCommand: RemoteCommand 197 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36f91085bf286ecf4d23'), x: 0.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f888c1f30185b7b0c4') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.015-0400 s20016| 2018-03-16T17:28:25.014-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-23-0] Connecting to ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.015-0400 s20016| 2018-03-16T17:28:25.014-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-23-0] Starting asynchronous command 198 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.015-0400 d20010| 2018-03-16T17:28:25.014-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56502 #45 (36 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.015-0400 d20010| 2018-03-16T17:28:25.014-0400 I NETWORK [conn45] received client metadata from 127.0.0.1:56502 conn45: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-23", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.015-0400 s20016| 2018-03-16T17:28:25.015-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-23-0] Successfully connected to ubuntu:20010, took 1ms (1 connections now open to ubuntu:20010) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.015-0400 s20016| 2018-03-16T17:28:25.015-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-23-0] Request 198 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.016-0400 s20016| 2018-03-16T17:28:25.015-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-23-0] Starting asynchronous command 197 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.016-0400 s20016| 2018-03-16T17:28:25.015-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-23-0] Request 197 finished with response: { n: 1, opTime: { ts: Timestamp(1521235705, 4), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235705, 4), $gleStats: { lastOpTime: { ts: Timestamp(1521235705, 4), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235705, 4), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 118), $configServerState: { opTime: { ts: Timestamp(1521235705, 1), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.016-0400 s20016| 2018-03-16T17:28:25.016-0400 D ASIO [conn1] startCommand: RemoteCommand 200 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36f91085bf286ecf4d24'), x: 1.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f888c1f30185b7b0c4') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.017-0400 s20016| 2018-03-16T17:28:25.016-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-24-0] Connecting to ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.017-0400 s20016| 2018-03-16T17:28:25.016-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-24-0] Starting asynchronous command 201 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.017-0400 d20010| 2018-03-16T17:28:25.016-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56504 #46 (37 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.017-0400 d20010| 2018-03-16T17:28:25.016-0400 I NETWORK [conn46] received client metadata from 127.0.0.1:56504 conn46: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-24", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.017-0400 s20016| 2018-03-16T17:28:25.016-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-24-0] Successfully connected to ubuntu:20010, took 0ms (1 connections now open to ubuntu:20010) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.017-0400 s20016| 2018-03-16T17:28:25.016-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-24-0] Request 201 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.018-0400 s20016| 2018-03-16T17:28:25.016-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-24-0] Starting asynchronous command 200 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.018-0400 s20016| 2018-03-16T17:28:25.016-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-24-0] Request 200 finished with response: { n: 1, opTime: { ts: Timestamp(1521235705, 5), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235705, 5), $gleStats: { lastOpTime: { ts: Timestamp(1521235705, 5), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235705, 5), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 118), $configServerState: { opTime: { ts: Timestamp(1521235705, 1), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.018-0400 s20016| 2018-03-16T17:28:25.017-0400 D ASIO [conn1] startCommand: RemoteCommand 203 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36f91085bf286ecf4d25'), x: 1.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f888c1f30185b7b0c4') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.019-0400 s20016| 2018-03-16T17:28:25.017-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-25-0] Connecting to ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.019-0400 s20016| 2018-03-16T17:28:25.018-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-25-0] Starting asynchronous command 204 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.019-0400 d20010| 2018-03-16T17:28:25.018-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56506 #47 (38 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.019-0400 d20010| 2018-03-16T17:28:25.018-0400 I NETWORK [conn47] received client metadata from 127.0.0.1:56506 conn47: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-25", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.019-0400 s20016| 2018-03-16T17:28:25.018-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-25-0] Successfully connected to ubuntu:20010, took 1ms (1 connections now open to ubuntu:20010) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.020-0400 s20016| 2018-03-16T17:28:25.018-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-25-0] Request 204 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.020-0400 s20016| 2018-03-16T17:28:25.018-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-25-0] Starting asynchronous command 203 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.020-0400 s20016| 2018-03-16T17:28:25.018-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-25-0] Request 203 finished with response: { n: 1, opTime: { ts: Timestamp(1521235705, 6), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235705, 6), $gleStats: { lastOpTime: { ts: Timestamp(1521235705, 6), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235705, 6), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 118), $configServerState: { opTime: { ts: Timestamp(1521235705, 1), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.021-0400 s20016| 2018-03-16T17:28:25.019-0400 D ASIO [conn1] startCommand: RemoteCommand 206 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36f91085bf286ecf4d26'), x: 2.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f888c1f30185b7b0c4') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.021-0400 s20016| 2018-03-16T17:28:25.019-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-26-0] Connecting to ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.021-0400 s20016| 2018-03-16T17:28:25.019-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-26-0] Starting asynchronous command 207 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.021-0400 d20010| 2018-03-16T17:28:25.019-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56508 #48 (39 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.021-0400 d20010| 2018-03-16T17:28:25.019-0400 I NETWORK [conn48] received client metadata from 127.0.0.1:56508 conn48: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-26", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.021-0400 s20016| 2018-03-16T17:28:25.019-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-26-0] Successfully connected to ubuntu:20010, took 0ms (1 connections now open to ubuntu:20010) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.022-0400 s20016| 2018-03-16T17:28:25.019-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-26-0] Request 207 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.022-0400 s20016| 2018-03-16T17:28:25.019-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-26-0] Starting asynchronous command 206 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.022-0400 s20016| 2018-03-16T17:28:25.020-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-26-0] Request 206 finished with response: { n: 1, opTime: { ts: Timestamp(1521235705, 7), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235705, 7), $gleStats: { lastOpTime: { ts: Timestamp(1521235705, 7), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235705, 7), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 118), $configServerState: { opTime: { ts: Timestamp(1521235705, 1), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.023-0400 s20016| 2018-03-16T17:28:25.021-0400 D ASIO [conn1] startCommand: RemoteCommand 209 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36f91085bf286ecf4d27'), x: 2.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f888c1f30185b7b0c4') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.023-0400 s20016| 2018-03-16T17:28:25.021-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-27-0] Connecting to ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.023-0400 s20016| 2018-03-16T17:28:25.021-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-27-0] Starting asynchronous command 210 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.023-0400 d20010| 2018-03-16T17:28:25.021-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56510 #49 (40 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.023-0400 d20010| 2018-03-16T17:28:25.021-0400 I NETWORK [conn49] received client metadata from 127.0.0.1:56510 conn49: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-27", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.023-0400 s20016| 2018-03-16T17:28:25.021-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-27-0] Successfully connected to ubuntu:20010, took 0ms (1 connections now open to ubuntu:20010) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.024-0400 s20016| 2018-03-16T17:28:25.021-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-27-0] Request 210 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.024-0400 s20016| 2018-03-16T17:28:25.021-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-27-0] Starting asynchronous command 209 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.024-0400 s20016| 2018-03-16T17:28:25.021-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-27-0] Request 209 finished with response: { n: 1, opTime: { ts: Timestamp(1521235705, 8), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235705, 8), $gleStats: { lastOpTime: { ts: Timestamp(1521235705, 8), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235705, 8), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 118), $configServerState: { opTime: { ts: Timestamp(1521235705, 1), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.024-0400 d20010| 2018-03-16T17:28:25.021-0400 I INDEX [ShardServerCatalogCacheLoader-0] build index on: config.cache.chunks.test.foo properties: { v: 2, key: { lastmod: 1 }, name: "lastmod_1", ns: "config.cache.chunks.test.foo" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.025-0400 d20010| 2018-03-16T17:28:25.021-0400 I INDEX [ShardServerCatalogCacheLoader-0] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.025-0400 d20010| 2018-03-16T17:28:25.022-0400 I INDEX [ShardServerCatalogCacheLoader-0] build index done. scanned 0 total records. 0 secs [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.025-0400 s20016| 2018-03-16T17:28:25.022-0400 D ASIO [conn1] startCommand: RemoteCommand 212 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36f91085bf286ecf4d28'), x: 3.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f888c1f30185b7b0c4') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.025-0400 s20016| 2018-03-16T17:28:25.022-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-28-0] Connecting to ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.025-0400 s20016| 2018-03-16T17:28:25.022-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-28-0] Starting asynchronous command 213 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.026-0400 d20010| 2018-03-16T17:28:25.022-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56512 #50 (41 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.026-0400 d20010| 2018-03-16T17:28:25.023-0400 I NETWORK [conn50] received client metadata from 127.0.0.1:56512 conn50: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-28", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.026-0400 s20016| 2018-03-16T17:28:25.023-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-28-0] Successfully connected to ubuntu:20010, took 1ms (1 connections now open to ubuntu:20010) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.026-0400 s20016| 2018-03-16T17:28:25.023-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-28-0] Request 213 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.026-0400 s20016| 2018-03-16T17:28:25.023-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-28-0] Starting asynchronous command 212 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.027-0400 s20016| 2018-03-16T17:28:25.023-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-28-0] Request 212 finished with response: { n: 1, opTime: { ts: Timestamp(1521235705, 12), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235705, 12), $gleStats: { lastOpTime: { ts: Timestamp(1521235705, 12), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235705, 12), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 118), $configServerState: { opTime: { ts: Timestamp(1521235705, 1), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.027-0400 s20016| 2018-03-16T17:28:25.024-0400 D ASIO [conn1] startCommand: RemoteCommand 215 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36f91085bf286ecf4d29'), x: 3.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f888c1f30185b7b0c4') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.027-0400 s20016| 2018-03-16T17:28:25.024-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-29-0] Connecting to ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.027-0400 s20016| 2018-03-16T17:28:25.024-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-29-0] Starting asynchronous command 216 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.028-0400 d20010| 2018-03-16T17:28:25.024-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56514 #51 (42 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.028-0400 d20010| 2018-03-16T17:28:25.024-0400 I NETWORK [conn51] received client metadata from 127.0.0.1:56514 conn51: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-29", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.028-0400 s20016| 2018-03-16T17:28:25.025-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-29-0] Successfully connected to ubuntu:20010, took 1ms (1 connections now open to ubuntu:20010) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.028-0400 s20016| 2018-03-16T17:28:25.025-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-29-0] Request 216 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.028-0400 s20016| 2018-03-16T17:28:25.025-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-29-0] Starting asynchronous command 215 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.029-0400 s20016| 2018-03-16T17:28:25.025-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-29-0] Request 215 finished with response: { n: 1, opTime: { ts: Timestamp(1521235705, 13), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235705, 13), $gleStats: { lastOpTime: { ts: Timestamp(1521235705, 13), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235705, 13), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 118), $configServerState: { opTime: { ts: Timestamp(1521235705, 1), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.029-0400 s20016| 2018-03-16T17:28:25.026-0400 D ASIO [conn1] startCommand: RemoteCommand 218 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36f91085bf286ecf4d2a'), x: 4.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f888c1f30185b7b0c4') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.029-0400 s20016| 2018-03-16T17:28:25.026-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-30-0] Connecting to ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.029-0400 s20016| 2018-03-16T17:28:25.026-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-30-0] Starting asynchronous command 219 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.042-0400 d20010| 2018-03-16T17:28:25.042-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56516 #52 (43 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.043-0400 d20010| 2018-03-16T17:28:25.042-0400 I NETWORK [conn52] received client metadata from 127.0.0.1:56516 conn52: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-30", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.043-0400 s20016| 2018-03-16T17:28:25.043-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-30-0] Successfully connected to ubuntu:20010, took 17ms (1 connections now open to ubuntu:20010) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.043-0400 s20016| 2018-03-16T17:28:25.043-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-30-0] Request 219 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.043-0400 s20016| 2018-03-16T17:28:25.043-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-30-0] Starting asynchronous command 218 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.044-0400 s20016| 2018-03-16T17:28:25.043-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-30-0] Request 218 finished with response: { n: 1, opTime: { ts: Timestamp(1521235705, 14), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235705, 14), $gleStats: { lastOpTime: { ts: Timestamp(1521235705, 14), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235705, 14), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 118), $configServerState: { opTime: { ts: Timestamp(1521235705, 1), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.044-0400 s20016| 2018-03-16T17:28:25.044-0400 D ASIO [conn1] startCommand: RemoteCommand 221 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36f91085bf286ecf4d2b'), x: 4.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f888c1f30185b7b0c4') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.044-0400 s20016| 2018-03-16T17:28:25.044-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-31-0] Connecting to ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.044-0400 s20016| 2018-03-16T17:28:25.044-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-31-0] Starting asynchronous command 222 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.044-0400 d20010| 2018-03-16T17:28:25.044-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56518 #53 (44 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.045-0400 d20010| 2018-03-16T17:28:25.044-0400 I NETWORK [conn53] received client metadata from 127.0.0.1:56518 conn53: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-31", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.045-0400 s20016| 2018-03-16T17:28:25.044-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-31-0] Successfully connected to ubuntu:20010, took 0ms (1 connections now open to ubuntu:20010) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.045-0400 s20016| 2018-03-16T17:28:25.044-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-31-0] Request 222 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.045-0400 s20016| 2018-03-16T17:28:25.044-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-31-0] Starting asynchronous command 221 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.046-0400 s20016| 2018-03-16T17:28:25.045-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-31-0] Request 221 finished with response: { n: 1, opTime: { ts: Timestamp(1521235705, 15), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235705, 15), $gleStats: { lastOpTime: { ts: Timestamp(1521235705, 15), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235705, 15), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 118), $configServerState: { opTime: { ts: Timestamp(1521235705, 1), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.046-0400 s20016| 2018-03-16T17:28:25.045-0400 D ASIO [conn1] startCommand: RemoteCommand 224 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36f91085bf286ecf4d2c'), x: 5.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f888c1f30185b7b0c4') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.046-0400 s20016| 2018-03-16T17:28:25.045-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-32-0] Connecting to ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.046-0400 s20016| 2018-03-16T17:28:25.046-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-32-0] Starting asynchronous command 225 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.047-0400 d20010| 2018-03-16T17:28:25.046-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56520 #54 (45 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.047-0400 d20010| 2018-03-16T17:28:25.046-0400 I NETWORK [conn54] received client metadata from 127.0.0.1:56520 conn54: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-32", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.047-0400 s20016| 2018-03-16T17:28:25.046-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-32-0] Successfully connected to ubuntu:20010, took 1ms (1 connections now open to ubuntu:20010) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.047-0400 s20016| 2018-03-16T17:28:25.046-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-32-0] Request 225 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.047-0400 s20016| 2018-03-16T17:28:25.046-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-32-0] Starting asynchronous command 224 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.048-0400 s20016| 2018-03-16T17:28:25.046-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-32-0] Request 224 finished with response: { n: 1, opTime: { ts: Timestamp(1521235705, 16), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235705, 16), $gleStats: { lastOpTime: { ts: Timestamp(1521235705, 16), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235705, 16), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 118), $configServerState: { opTime: { ts: Timestamp(1521235705, 1), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.048-0400 s20016| 2018-03-16T17:28:25.047-0400 D ASIO [conn1] startCommand: RemoteCommand 227 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36f91085bf286ecf4d2d'), x: 5.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f888c1f30185b7b0c4') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.048-0400 s20016| 2018-03-16T17:28:25.047-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-33-0] Connecting to ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.049-0400 s20016| 2018-03-16T17:28:25.047-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-33-0] Starting asynchronous command 228 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.049-0400 d20010| 2018-03-16T17:28:25.047-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56522 #55 (46 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.049-0400 d20010| 2018-03-16T17:28:25.047-0400 I NETWORK [conn55] received client metadata from 127.0.0.1:56522 conn55: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-33", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.049-0400 s20016| 2018-03-16T17:28:25.048-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-33-0] Successfully connected to ubuntu:20010, took 1ms (1 connections now open to ubuntu:20010) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.049-0400 s20016| 2018-03-16T17:28:25.048-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-33-0] Request 228 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.049-0400 s20016| 2018-03-16T17:28:25.048-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-33-0] Starting asynchronous command 227 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.050-0400 s20016| 2018-03-16T17:28:25.048-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-33-0] Request 227 finished with response: { n: 1, opTime: { ts: Timestamp(1521235705, 17), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235705, 17), $gleStats: { lastOpTime: { ts: Timestamp(1521235705, 17), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235705, 17), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 118), $configServerState: { opTime: { ts: Timestamp(1521235705, 1), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.050-0400 s20016| 2018-03-16T17:28:25.049-0400 D ASIO [conn1] startCommand: RemoteCommand 230 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36f91085bf286ecf4d2e'), x: 6.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f888c1f30185b7b0c4') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.051-0400 s20016| 2018-03-16T17:28:25.049-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-34-0] Connecting to ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.051-0400 s20016| 2018-03-16T17:28:25.049-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-34-0] Starting asynchronous command 231 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.051-0400 d20010| 2018-03-16T17:28:25.049-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56524 #56 (47 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.051-0400 d20010| 2018-03-16T17:28:25.049-0400 I NETWORK [conn56] received client metadata from 127.0.0.1:56524 conn56: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-34", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.051-0400 s20016| 2018-03-16T17:28:25.049-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-34-0] Successfully connected to ubuntu:20010, took 0ms (1 connections now open to ubuntu:20010) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.051-0400 s20016| 2018-03-16T17:28:25.049-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-34-0] Request 231 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.052-0400 s20016| 2018-03-16T17:28:25.049-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-34-0] Starting asynchronous command 230 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.052-0400 s20016| 2018-03-16T17:28:25.050-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-34-0] Request 230 finished with response: { n: 1, opTime: { ts: Timestamp(1521235705, 18), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235705, 18), $gleStats: { lastOpTime: { ts: Timestamp(1521235705, 18), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235705, 18), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 118), $configServerState: { opTime: { ts: Timestamp(1521235705, 1), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.052-0400 s20016| 2018-03-16T17:28:25.050-0400 D ASIO [conn1] startCommand: RemoteCommand 233 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36f91085bf286ecf4d2f'), x: 6.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f888c1f30185b7b0c4') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.053-0400 s20016| 2018-03-16T17:28:25.050-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-35-0] Connecting to ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.053-0400 s20016| 2018-03-16T17:28:25.051-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-35-0] Starting asynchronous command 234 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.053-0400 d20010| 2018-03-16T17:28:25.051-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56526 #57 (48 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.053-0400 d20010| 2018-03-16T17:28:25.051-0400 I NETWORK [conn57] received client metadata from 127.0.0.1:56526 conn57: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-35", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.053-0400 s20016| 2018-03-16T17:28:25.051-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-35-0] Successfully connected to ubuntu:20010, took 1ms (1 connections now open to ubuntu:20010) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.054-0400 s20016| 2018-03-16T17:28:25.051-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-35-0] Request 234 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.054-0400 s20016| 2018-03-16T17:28:25.051-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-35-0] Starting asynchronous command 233 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.054-0400 s20016| 2018-03-16T17:28:25.051-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-35-0] Request 233 finished with response: { n: 1, opTime: { ts: Timestamp(1521235705, 19), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235705, 19), $gleStats: { lastOpTime: { ts: Timestamp(1521235705, 19), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235705, 19), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 118), $configServerState: { opTime: { ts: Timestamp(1521235705, 1), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.055-0400 s20016| 2018-03-16T17:28:25.052-0400 D ASIO [conn1] startCommand: RemoteCommand 236 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36f91085bf286ecf4d30'), x: 7.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f888c1f30185b7b0c4') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.055-0400 s20016| 2018-03-16T17:28:25.052-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-36-0] Connecting to ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.055-0400 s20016| 2018-03-16T17:28:25.052-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-36-0] Starting asynchronous command 237 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.055-0400 d20010| 2018-03-16T17:28:25.052-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56528 #58 (49 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.055-0400 d20010| 2018-03-16T17:28:25.052-0400 I NETWORK [conn58] received client metadata from 127.0.0.1:56528 conn58: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-36", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.056-0400 s20016| 2018-03-16T17:28:25.053-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-36-0] Successfully connected to ubuntu:20010, took 1ms (1 connections now open to ubuntu:20010) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.056-0400 s20016| 2018-03-16T17:28:25.053-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-36-0] Request 237 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.056-0400 s20016| 2018-03-16T17:28:25.053-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-36-0] Starting asynchronous command 236 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.056-0400 s20016| 2018-03-16T17:28:25.053-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-36-0] Request 236 finished with response: { n: 1, opTime: { ts: Timestamp(1521235705, 20), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235705, 20), $gleStats: { lastOpTime: { ts: Timestamp(1521235705, 20), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235705, 20), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 118), $configServerState: { opTime: { ts: Timestamp(1521235705, 1), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.057-0400 s20016| 2018-03-16T17:28:25.054-0400 D ASIO [conn1] startCommand: RemoteCommand 239 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36f91085bf286ecf4d31'), x: 7.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f888c1f30185b7b0c4') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.057-0400 s20016| 2018-03-16T17:28:25.054-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-37-0] Connecting to ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.057-0400 s20016| 2018-03-16T17:28:25.054-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-37-0] Starting asynchronous command 240 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.057-0400 d20010| 2018-03-16T17:28:25.054-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56530 #59 (50 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.057-0400 d20010| 2018-03-16T17:28:25.054-0400 I NETWORK [conn59] received client metadata from 127.0.0.1:56530 conn59: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-37", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.058-0400 s20016| 2018-03-16T17:28:25.054-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-37-0] Successfully connected to ubuntu:20010, took 0ms (1 connections now open to ubuntu:20010) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.058-0400 s20016| 2018-03-16T17:28:25.054-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-37-0] Request 240 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.058-0400 s20016| 2018-03-16T17:28:25.054-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-37-0] Starting asynchronous command 239 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.058-0400 s20016| 2018-03-16T17:28:25.054-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-37-0] Request 239 finished with response: { n: 1, opTime: { ts: Timestamp(1521235705, 21), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235705, 21), $gleStats: { lastOpTime: { ts: Timestamp(1521235705, 21), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235705, 21), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 118), $configServerState: { opTime: { ts: Timestamp(1521235705, 1), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.059-0400 s20016| 2018-03-16T17:28:25.055-0400 D ASIO [conn1] startCommand: RemoteCommand 242 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36f91085bf286ecf4d32'), x: 8.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f888c1f30185b7b0c4') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.059-0400 s20016| 2018-03-16T17:28:25.055-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-38-0] Connecting to ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.059-0400 s20016| 2018-03-16T17:28:25.055-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-38-0] Starting asynchronous command 243 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.059-0400 d20010| 2018-03-16T17:28:25.055-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56532 #60 (51 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.059-0400 d20010| 2018-03-16T17:28:25.056-0400 I NETWORK [conn60] received client metadata from 127.0.0.1:56532 conn60: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-38", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.060-0400 s20016| 2018-03-16T17:28:25.056-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-38-0] Successfully connected to ubuntu:20010, took 1ms (1 connections now open to ubuntu:20010) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.060-0400 s20016| 2018-03-16T17:28:25.056-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-38-0] Request 243 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.060-0400 s20016| 2018-03-16T17:28:25.056-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-38-0] Starting asynchronous command 242 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.060-0400 s20016| 2018-03-16T17:28:25.056-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-38-0] Request 242 finished with response: { n: 1, opTime: { ts: Timestamp(1521235705, 22), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235705, 22), $gleStats: { lastOpTime: { ts: Timestamp(1521235705, 22), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235705, 22), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 118), $configServerState: { opTime: { ts: Timestamp(1521235705, 1), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.061-0400 s20016| 2018-03-16T17:28:25.057-0400 D ASIO [conn1] startCommand: RemoteCommand 245 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36f91085bf286ecf4d33'), x: 8.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f888c1f30185b7b0c4') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.061-0400 s20016| 2018-03-16T17:28:25.057-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-39-0] Connecting to ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.061-0400 s20016| 2018-03-16T17:28:25.057-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-39-0] Starting asynchronous command 246 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.061-0400 d20010| 2018-03-16T17:28:25.057-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56534 #61 (52 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.061-0400 d20010| 2018-03-16T17:28:25.057-0400 I NETWORK [conn61] received client metadata from 127.0.0.1:56534 conn61: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-39", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.062-0400 s20016| 2018-03-16T17:28:25.057-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-39-0] Successfully connected to ubuntu:20010, took 0ms (1 connections now open to ubuntu:20010) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.062-0400 s20016| 2018-03-16T17:28:25.057-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-39-0] Request 246 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.062-0400 s20016| 2018-03-16T17:28:25.057-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-39-0] Starting asynchronous command 245 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.063-0400 s20016| 2018-03-16T17:28:25.058-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-39-0] Request 245 finished with response: { n: 1, opTime: { ts: Timestamp(1521235705, 23), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235705, 23), $gleStats: { lastOpTime: { ts: Timestamp(1521235705, 23), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235705, 23), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 118), $configServerState: { opTime: { ts: Timestamp(1521235705, 1), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.063-0400 s20016| 2018-03-16T17:28:25.058-0400 D ASIO [conn1] startCommand: RemoteCommand 248 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36f91085bf286ecf4d34'), x: 9.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f888c1f30185b7b0c4') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.063-0400 s20016| 2018-03-16T17:28:25.058-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-0-0] Starting asynchronous command 248 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.064-0400 s20016| 2018-03-16T17:28:25.059-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-0-0] Request 248 finished with response: { n: 1, opTime: { ts: Timestamp(1521235705, 24), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235705, 24), $gleStats: { lastOpTime: { ts: Timestamp(1521235705, 24), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235705, 24), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 118), $configServerState: { opTime: { ts: Timestamp(1521235705, 1), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.064-0400 s20016| 2018-03-16T17:28:25.059-0400 D ASIO [conn1] startCommand: RemoteCommand 250 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36f91085bf286ecf4d35'), x: 9.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f888c1f30185b7b0c4') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.064-0400 s20016| 2018-03-16T17:28:25.059-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-1-0] Starting asynchronous command 250 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.065-0400 s20016| 2018-03-16T17:28:25.060-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-1-0] Request 250 finished with response: { n: 1, opTime: { ts: Timestamp(1521235705, 25), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235705, 25), $gleStats: { lastOpTime: { ts: Timestamp(1521235705, 25), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235705, 25), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235704, 118), $configServerState: { opTime: { ts: Timestamp(1521235705, 1), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.065-0400 Waiting for op with OpTime { "ts" : Timestamp(1521235705, 1), "t" : NumberLong(1) } to be committed on all secondaries [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.066-0400 s20016| 2018-03-16T17:28:25.065-0400 D ASIO [conn1] startCommand: RemoteCommand 252 -- target:ubuntu:20012 db:config cmd:{ find: "chunks", filter: { min: { x: MinKey }, max: { x: MaxKey } }, limit: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.066-0400 s20016| 2018-03-16T17:28:25.065-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-2-0] Connecting to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.066-0400 s20016| 2018-03-16T17:28:25.066-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-2-0] Starting asynchronous command 253 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.066-0400 c20012| 2018-03-16T17:28:25.066-0400 I NETWORK [listener] connection accepted from 127.0.0.1:60880 #44 (27 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.067-0400 c20012| 2018-03-16T17:28:25.066-0400 I NETWORK [conn44] received client metadata from 127.0.0.1:60880 conn44: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-2", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.067-0400 s20016| 2018-03-16T17:28:25.066-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-2-0] Successfully connected to ubuntu:20012, took 1ms (1 connections now open to ubuntu:20012) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.067-0400 s20016| 2018-03-16T17:28:25.066-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-2-0] Request 253 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.067-0400 s20016| 2018-03-16T17:28:25.066-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-2-0] Starting asynchronous command 252 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.068-0400 s20016| 2018-03-16T17:28:25.066-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-2-0] Request 252 finished with response: { cursor: { firstBatch: [ { _id: "test.foo-x_MinKey", ns: "test.foo", min: { x: MinKey }, max: { x: MaxKey }, shard: "stale_mongos_updates_and_removes-rs0", lastmod: Timestamp(1, 0), lastmodEpoch: ObjectId('5aac36f888c1f30185b7b0c4'), history: [ { validAfter: Timestamp(1521235704, 115), shard: "stale_mongos_updates_and_removes-rs0" } ] } ], id: 0, ns: "config.chunks" }, ok: 1.0, operationTime: Timestamp(1521235705, 1), $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235705, 25), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235705, 1) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.068-0400 s20015| 2018-03-16T17:28:25.067-0400 I SHARDING [conn1] Refreshing chunks for collection test.foo based on version 1|0||5aac36f788c1f30185b7afa3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.068-0400 s20015| 2018-03-16T17:28:25.067-0400 D ASIO [ConfigServerCatalogCacheLoader-0] startCommand: RemoteCommand 60 -- target:ubuntu:20014 db:config expDate:2018-03-16T17:28:55.067-0400 cmd:{ find: "collections", filter: { _id: "test.foo" }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235704, 92), t: 1 } }, limit: 1, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.068-0400 s20015| 2018-03-16T17:28:25.067-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 60 on host ubuntu:20014 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.069-0400 s20015| 2018-03-16T17:28:25.067-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 60 finished with response: { cursor: { firstBatch: [ { _id: "test.foo", lastmodEpoch: ObjectId('5aac36f888c1f30185b7b0c4'), lastmod: new Date(4294967296), dropped: false, key: { x: 1.0 }, unique: false, uuid: UUID("650ad6da-cccc-457f-bb84-294856863b67") } ], id: 0, ns: "config.collections" }, ok: 1.0, operationTime: Timestamp(1521235705, 1), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235705, 1), t: 1 }, lastOpVisible: { ts: Timestamp(1521235705, 1), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235705, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235705, 1) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.069-0400 s20015| 2018-03-16T17:28:25.067-0400 D ASIO [ConfigServerCatalogCacheLoader-0] startCommand: RemoteCommand 62 -- target:ubuntu:20014 db:config expDate:2018-03-16T17:28:55.067-0400 cmd:{ find: "chunks", filter: { ns: "test.foo", lastmod: { $gte: Timestamp(0, 0) } }, sort: { lastmod: 1 }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235705, 1), t: 1 } }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.070-0400 s20015| 2018-03-16T17:28:25.068-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 62 on host ubuntu:20014 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.071-0400 s20015| 2018-03-16T17:28:25.068-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 62 finished with response: { cursor: { firstBatch: [ { _id: "test.foo-x_MinKey", ns: "test.foo", min: { x: MinKey }, max: { x: MaxKey }, shard: "stale_mongos_updates_and_removes-rs0", lastmod: Timestamp(1, 0), lastmodEpoch: ObjectId('5aac36f888c1f30185b7b0c4'), history: [ { validAfter: Timestamp(1521235704, 115), shard: "stale_mongos_updates_and_removes-rs0" } ] } ], id: 0, ns: "config.chunks" }, ok: 1.0, operationTime: Timestamp(1521235705, 1), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235705, 1), t: 1 }, lastOpVisible: { ts: Timestamp(1521235705, 1), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235705, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235705, 1) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.071-0400 s20015| 2018-03-16T17:28:25.068-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Refresh for collection test.foo took 0 ms and found version 1|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.071-0400 s20015| 2018-03-16T17:28:25.068-0400 D ASIO [conn1] startCommand: RemoteCommand 64 -- target:ubuntu:20012 db:admin cmd:{ _configsvrMoveChunk: 1, _id: "test.foo-x_MinKey", ns: "test.foo", min: { x: MinKey }, max: { x: MaxKey }, shard: "stale_mongos_updates_and_removes-rs0", lastmod: Timestamp(1, 0), lastmodEpoch: ObjectId('5aac36f888c1f30185b7b0c4'), toShard: "stale_mongos_updates_and_removes-rs1", maxChunkSizeBytes: 67108864, secondaryThrottle: {}, waitForDelete: true, writeConcern: { w: "majority", wtimeout: 15000 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.071-0400 s20015| 2018-03-16T17:28:25.068-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 64 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.071-0400 c20012| 2018-03-16T17:28:25.070-0400 I SHARDING [conn21] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.072-0400 c20012| 2018-03-16T17:28:25.070-0400 I SHARDING [conn21] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.072-0400 c20012| 2018-03-16T17:28:25.070-0400 I SHARDING [conn21] xxx routing info needs refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.072-0400 c20012| 2018-03-16T17:28:25.070-0400 I SHARDING [conn21] xxx calling schedule collection refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.072-0400 c20012| 2018-03-16T17:28:25.070-0400 I SHARDING [conn21] xxx is incremental [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.072-0400 c20012| 2018-03-16T17:28:25.070-0400 I SHARDING [conn21] xxx existing routing info? [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.072-0400 c20012| 2018-03-16T17:28:25.070-0400 I SHARDING [conn21] 2|0||5aac36f788c1f30185b7afa3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.072-0400 c20012| 2018-03-16T17:28:25.070-0400 I SHARDING [conn21] Refreshing chunks for collection test.foo based on version 2|0||5aac36f788c1f30185b7afa3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.072-0400 c20012| 2018-03-16T17:28:25.070-0400 I SHARDING [conn21] xxx getting chunks since now [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.073-0400 c20012| 2018-03-16T17:28:25.070-0400 I SHARDING [conn21] xxx waiting if !refreshNot [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.073-0400 c20012| 2018-03-16T17:28:25.070-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh callback [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.073-0400 c20012| 2018-03-16T17:28:25.070-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh completed [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.073-0400 c20012| 2018-03-16T17:28:25.070-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Refresh for collection test.foo took 0 ms and found version 1|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.073-0400 c20012| 2018-03-16T17:28:25.070-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx marking as status ok [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.073-0400 c20012| 2018-03-16T17:28:25.070-0400 I SHARDING [conn21] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.073-0400 c20012| 2018-03-16T17:28:25.070-0400 I SHARDING [conn21] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.081-0400 c20012| 2018-03-16T17:28:25.080-0400 I SHARDING [conn21] distributed lock 'test.foo' acquired for 'Migrating chunk(s) in collection test.foo', ts : 5aac36ec88c1f30185b7ac69 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.081-0400 d20010| 2018-03-16T17:28:25.081-0400 I SHARDING [conn15] Starting chunk migration ns: test.foo, [{ x: MinKey }, { x: MaxKey }), fromShard: stale_mongos_updates_and_removes-rs0, toShard: stale_mongos_updates_and_removes-rs1 with expected collection version epoch 5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.081-0400 d20010| 2018-03-16T17:28:25.081-0400 I SHARDING [conn15] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.082-0400 d20010| 2018-03-16T17:28:25.081-0400 I SHARDING [conn15] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.082-0400 d20010| 2018-03-16T17:28:25.081-0400 I SHARDING [conn15] xxx routing info needs refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.082-0400 d20010| 2018-03-16T17:28:25.081-0400 I SHARDING [conn15] xxx calling schedule collection refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.082-0400 d20010| 2018-03-16T17:28:25.081-0400 I SHARDING [conn15] xxx is incremental [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.082-0400 d20010| 2018-03-16T17:28:25.081-0400 I SHARDING [conn15] xxx existing routing info? [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.082-0400 d20010| 2018-03-16T17:28:25.081-0400 I SHARDING [conn15] 1|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.082-0400 d20010| 2018-03-16T17:28:25.081-0400 I SHARDING [conn15] Refreshing chunks for collection test.foo based on version 1|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.082-0400 d20010| 2018-03-16T17:28:25.081-0400 I SHARDING [conn15] xxx getting chunks since now [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.083-0400 d20010| 2018-03-16T17:28:25.081-0400 I SHARDING [conn15] xxx getting chunks since [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.083-0400 d20010| 2018-03-16T17:28:25.081-0400 I SHARDING [conn15] xxx waiting if !refreshNot [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.083-0400 d20010| 2018-03-16T17:28:25.081-0400 I SHARDING [ShardServerCatalogCacheLoader-0] xxx primary and calling get chunks since for primary [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.083-0400 d20010| 2018-03-16T17:28:25.081-0400 I SHARDING [ShardServerCatalogCacheLoader-0] version is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.083-0400 d20010| 2018-03-16T17:28:25.081-0400 I SHARDING [ShardServerCatalogCacheLoader-0] 1|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.083-0400 d20010| 2018-03-16T17:28:25.082-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] refresh in primary [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.083-0400 d20010| 2018-03-16T17:28:25.082-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] 1 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.083-0400 d20010| 2018-03-16T17:28:25.082-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Cache loader remotely refreshed for collection test.foo from collection version 1|0||5aac36f888c1f30185b7b0c4 and found collection version 1|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.084-0400 d20010| 2018-03-16T17:28:25.082-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Cache loader found no enqueued metadata and persisted metadata from 1|0||5aac36f888c1f30185b7b0c4 to 1|0||5aac36f888c1f30185b7b0c4, GTE cache version 1|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.084-0400 d20010| 2018-03-16T17:28:25.082-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx is okay [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.084-0400 d20010| 2018-03-16T17:28:25.082-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx finsihing refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.084-0400 d20010| 2018-03-16T17:28:25.082-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh callback [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.084-0400 d20010| 2018-03-16T17:28:25.082-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh completed [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.084-0400 d20010| 2018-03-16T17:28:25.082-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Refresh for collection test.foo took 1 ms and found version 1|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.085-0400 d20010| 2018-03-16T17:28:25.082-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx marking as status ok [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.085-0400 d20010| 2018-03-16T17:28:25.082-0400 I SHARDING [conn15] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.085-0400 d20010| 2018-03-16T17:28:25.082-0400 I SHARDING [conn15] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.085-0400 d20010| 2018-03-16T17:28:25.082-0400 I SHARDING [conn15] about to log metadata event into changelog: { _id: "ubuntu-2018-03-16T17:28:25.082-0400-5aac36f9f7e5a0b40b8e353d", server: "ubuntu", clientAddr: "127.0.0.1:56424", time: new Date(1521235705082), what: "moveChunk.start", ns: "test.foo", details: { min: { x: MinKey }, max: { x: MaxKey }, from: "stale_mongos_updates_and_removes-rs0", to: "stale_mongos_updates_and_removes-rs1" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.091-0400 d20011| 2018-03-16T17:28:25.090-0400 I SHARDING [conn22] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.091-0400 d20011| 2018-03-16T17:28:25.090-0400 I SHARDING [conn22] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.091-0400 d20011| 2018-03-16T17:28:25.090-0400 I SHARDING [conn22] xxx routing info needs refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.091-0400 d20011| 2018-03-16T17:28:25.090-0400 I SHARDING [conn22] xxx calling schedule collection refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.091-0400 d20011| 2018-03-16T17:28:25.090-0400 I SHARDING [conn22] xxx is not incr [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.091-0400 d20011| 2018-03-16T17:28:25.090-0400 I SHARDING [conn22] xxx existing routing info? [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.091-0400 d20011| 2018-03-16T17:28:25.090-0400 I SHARDING [conn22] Refreshing chunks for collection test.foo based on version 0|0||000000000000000000000000 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.091-0400 d20011| 2018-03-16T17:28:25.090-0400 I SHARDING [conn22] xxx getting chunks since now [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.092-0400 d20011| 2018-03-16T17:28:25.090-0400 I SHARDING [conn22] xxx getting chunks since [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.092-0400 d20011| 2018-03-16T17:28:25.090-0400 I SHARDING [conn22] xxx waiting if !refreshNot [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.092-0400 d20011| 2018-03-16T17:28:25.090-0400 I SHARDING [ShardServerCatalogCacheLoader-0] xxx primary and calling get chunks since for primary [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.092-0400 d20011| 2018-03-16T17:28:25.090-0400 I SHARDING [ShardServerCatalogCacheLoader-0] version is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.092-0400 d20011| 2018-03-16T17:28:25.090-0400 I SHARDING [ShardServerCatalogCacheLoader-0] 0|0||000000000000000000000000 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.092-0400 d20011| 2018-03-16T17:28:25.091-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Connecting to ubuntu:20014 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.092-0400 c20014| 2018-03-16T17:28:25.091-0400 I NETWORK [listener] connection accepted from 127.0.0.1:44668 #22 (14 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.093-0400 c20014| 2018-03-16T17:28:25.091-0400 I NETWORK [conn22] received client metadata from 127.0.0.1:44668 conn22: { driver: { name: "NetworkInterfaceASIO-ShardRegistry", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.093-0400 d20011| 2018-03-16T17:28:25.091-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to ubuntu:20014, took 0ms (1 connections now open to ubuntu:20014) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.093-0400 d20011| 2018-03-16T17:28:25.092-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] refresh in primary [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.093-0400 d20011| 2018-03-16T17:28:25.092-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] 1 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.093-0400 d20011| 2018-03-16T17:28:25.092-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx collec and changed chunks max epoch or version doesn't equal max [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.193-0400 d20011| 2018-03-16T17:28:25.193-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Cache loader remotely refreshed for collection test.foo from collection version 0|0||000000000000000000000000 and found collection version 1|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.193-0400 d20011| 2018-03-16T17:28:25.193-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Cache loader found enqueued metadata from 1|0||5aac36f888c1f30185b7b0c4 to 1|0||5aac36f888c1f30185b7b0c4 and no persisted metadata, GTE cache version 0|0||000000000000000000000000 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.193-0400 d20011| 2018-03-16T17:28:25.193-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx is okay [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.193-0400 d20011| 2018-03-16T17:28:25.193-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx finsihing refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.194-0400 d20011| 2018-03-16T17:28:25.193-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh callback [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.194-0400 d20011| 2018-03-16T17:28:25.193-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh completed [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.194-0400 d20011| 2018-03-16T17:28:25.193-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Refresh for collection test.foo took 102 ms and found version 1|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.194-0400 d20011| 2018-03-16T17:28:25.193-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx marking as status ok [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.194-0400 d20011| 2018-03-16T17:28:25.193-0400 I SHARDING [conn22] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.194-0400 d20011| 2018-03-16T17:28:25.193-0400 I SHARDING [conn22] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.194-0400 d20011| 2018-03-16T17:28:25.193-0400 I SHARDING [conn22] Marking collection test.foo as sharded with collection version: 1|0||5aac36f888c1f30185b7b0c4, shard version: 0|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.195-0400 d20011| 2018-03-16T17:28:25.193-0400 I STORAGE [ShardServerCatalogCacheLoader-0] createCollection: config.cache.chunks.test.foo with generated UUID: 90d2b6b6-c94f-4a34-b6ef-f2815aeaa7a7 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.195-0400 d20011| 2018-03-16T17:28:25.193-0400 I COMMAND [conn22] command admin.$cmd appName: "MongoDB Shell" command: _recvChunkStart { _recvChunkStart: "test.foo", sessionId: "stale_mongos_updates_and_removes-rs0_stale_mongos_updates_and_removes-rs1_5aac36f9f7e5a0b40b8e353e", from: "stale_mongos_updates_and_removes-rs0/ubuntu:20010", fromShardName: "stale_mongos_updates_and_removes-rs0", toShardName: "stale_mongos_updates_and_removes-rs1", min: { x: MinKey }, max: { x: MaxKey }, shardKeyPattern: { x: 1.0 }, $clusterTime: { clusterTime: Timestamp(1521235705, 29), signature: { hash: BinData(0, 6DF82EEC68E53F390B4A84C15A0D2C6D32CF7D93), keyId: 6533657568122765331 } }, $configServerState: { opTime: { ts: Timestamp(1521235705, 28), t: 1 } }, $db: "admin" } numYields:0 reslen:327 locks:{ Global: { acquireCount: { r: 3, w: 1 } }, Database: { acquireCount: { r: 1, w: 1 } }, Collection: { acquireCount: { r: 1, W: 1 } } } protocol:op_msg 102ms [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.196-0400 d20011| 2018-03-16T17:28:25.193-0400 I SHARDING [migrateThread] Starting receiving end of migration of chunk { x: MinKey } -> { x: MaxKey } for collection test.foo from stale_mongos_updates_and_removes-rs0/ubuntu:20010 at epoch 5aac36f888c1f30185b7b0c4 with session id stale_mongos_updates_and_removes-rs0_stale_mongos_updates_and_removes-rs1_5aac36f9f7e5a0b40b8e353e [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.196-0400 d20011| 2018-03-16T17:28:25.194-0400 I STORAGE [migrateThread] createCollection: test.foo with provided UUID: 650ad6da-cccc-457f-bb84-294856863b67 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.230-0400 d20011| 2018-03-16T17:28:25.229-0400 I INDEX [ShardServerCatalogCacheLoader-0] build index on: config.cache.chunks.test.foo properties: { v: 2, key: { lastmod: 1 }, name: "lastmod_1", ns: "config.cache.chunks.test.foo" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.230-0400 d20011| 2018-03-16T17:28:25.229-0400 I INDEX [ShardServerCatalogCacheLoader-0] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.242-0400 d20011| 2018-03-16T17:28:25.242-0400 I INDEX [migrateThread] build index on: test.foo properties: { v: 2, key: { x: 1.0 }, name: "x_1", ns: "test.foo" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.242-0400 d20011| 2018-03-16T17:28:25.242-0400 I INDEX [migrateThread] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.242-0400 d20011| 2018-03-16T17:28:25.242-0400 I SHARDING [migrateThread] Scheduling deletion of any documents in test.foo range [{ x: MinKey }, { x: MaxKey }) before migrating in a chunk covering the range [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.243-0400 d20011| 2018-03-16T17:28:25.242-0400 I SHARDING [Collection Range Deleter] No documents remain to delete in test.foo range [{ x: MinKey }, { x: MaxKey }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.243-0400 d20011| 2018-03-16T17:28:25.242-0400 I SHARDING [Collection Range Deleter] Waiting for majority replication of local deletions in test.foo range [{ x: MinKey }, { x: MaxKey }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.243-0400 d20011| 2018-03-16T17:28:25.242-0400 I INDEX [ShardServerCatalogCacheLoader-0] build index done. scanned 0 total records. 0 secs [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.243-0400 d20011| 2018-03-16T17:28:25.243-0400 I SHARDING [Collection Range Deleter] Finished deleting documents in test.foo range [{ x: MinKey }, { x: MaxKey }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.243-0400 d20011| 2018-03-16T17:28:25.243-0400 I SHARDING [migrateThread] Finished deleting test.foo range [{ x: MinKey }, { x: MaxKey }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.245-0400 d20011| 2018-03-16T17:28:25.245-0400 I SHARDING [migrateThread] Waiting for replication to catch up before entering critical section [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.248-0400 d20011| 2018-03-16T17:28:25.247-0400 I SHARDING [migrateThread] migrate commit succeeded flushing to secondaries for 'test.foo' { x: MinKey } -> { x: MaxKey } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.248-0400 d20010| 2018-03-16T17:28:25.247-0400 I SHARDING [conn15] moveChunk data transfer progress: { waited: true, active: true, sessionId: "stale_mongos_updates_and_removes-rs0_stale_mongos_updates_and_removes-rs1_5aac36f9f7e5a0b40b8e353e", ns: "test.foo", from: "stale_mongos_updates_and_removes-rs0/ubuntu:20010", min: { x: MinKey }, max: { x: MaxKey }, shardKeyPattern: { x: 1.0 }, state: "steady", counts: { cloned: 20, clonedBytes: 1120, catchup: 0, steady: 0 }, ok: 1.0, operationTime: Timestamp(1521235705, 58), $gleStats: { lastOpTime: { ts: Timestamp(1521235705, 32), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235705, 58), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235705, 58), $configServerState: { opTime: { ts: Timestamp(1521235705, 28), t: 1 } } } mem used: 0 documents remaining to clone: 0 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.249-0400 d20010| 2018-03-16T17:28:25.249-0400 I SHARDING [conn15] Migration successfully entered critical section [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.258-0400 d20011| 2018-03-16T17:28:25.258-0400 I SHARDING [migrateThread] migrate commit succeeded flushing to secondaries for 'test.foo' { x: MinKey } -> { x: MaxKey } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.646-0400 d20011| 2018-03-16T17:28:25.645-0400 I SHARDING [migrateThread] about to log metadata event into changelog: { _id: "ubuntu-2018-03-16T17:28:25.645-0400-5aac36f96a0ae3d58b640310", server: "ubuntu", clientAddr: "", time: new Date(1521235705645), what: "moveChunk.to", ns: "test.foo", details: { min: { x: MinKey }, max: { x: MaxKey }, step 1 of 6: 48, step 2 of 6: 1, step 3 of 6: 1, step 4 of 6: 0, step 5 of 6: 13, step 6 of 6: 387, note: "success" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.652-0400 d20011| 2018-03-16T17:28:25.652-0400 I COMMAND [conn22] command admin.$cmd appName: "MongoDB Shell" command: _recvChunkCommit { _recvChunkCommit: "test.foo", waitForSteadyOrDone: false, sessionId: "stale_mongos_updates_and_removes-rs0_stale_mongos_updates_and_removes-rs1_5aac36f9f7e5a0b40b8e353e", $clusterTime: { clusterTime: Timestamp(1521235705, 61), signature: { hash: BinData(0, 6DF82EEC68E53F390B4A84C15A0D2C6D32CF7D93), keyId: 6533657568122765331 } }, $configServerState: { opTime: { ts: Timestamp(1521235705, 28), t: 1 } }, $db: "admin" } numYields:0 reslen:561 locks:{} protocol:op_msg 402ms [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.653-0400 d20010| 2018-03-16T17:28:25.652-0400 I SHARDING [conn15] Moving last chunk for the collection out [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.658-0400 d20010| 2018-03-16T17:28:25.658-0400 I SHARDING [conn15] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.658-0400 d20010| 2018-03-16T17:28:25.658-0400 I SHARDING [conn15] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.658-0400 d20010| 2018-03-16T17:28:25.658-0400 I SHARDING [conn15] xxx routing info needs refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.658-0400 d20010| 2018-03-16T17:28:25.658-0400 I SHARDING [conn15] xxx calling schedule collection refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.658-0400 d20010| 2018-03-16T17:28:25.658-0400 I SHARDING [conn15] xxx is incremental [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.658-0400 d20010| 2018-03-16T17:28:25.658-0400 I SHARDING [conn15] xxx existing routing info? [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.659-0400 d20010| 2018-03-16T17:28:25.658-0400 I SHARDING [conn15] 1|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.659-0400 d20010| 2018-03-16T17:28:25.658-0400 I SHARDING [conn15] Refreshing chunks for collection test.foo based on version 1|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.659-0400 d20010| 2018-03-16T17:28:25.658-0400 I SHARDING [conn15] xxx getting chunks since now [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.659-0400 d20010| 2018-03-16T17:28:25.658-0400 I SHARDING [conn15] xxx getting chunks since [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.659-0400 d20010| 2018-03-16T17:28:25.658-0400 I SHARDING [conn15] xxx waiting if !refreshNot [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.659-0400 d20010| 2018-03-16T17:28:25.658-0400 I SHARDING [ShardServerCatalogCacheLoader-0] xxx primary and calling get chunks since for primary [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.659-0400 d20010| 2018-03-16T17:28:25.658-0400 I SHARDING [ShardServerCatalogCacheLoader-0] version is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.660-0400 d20010| 2018-03-16T17:28:25.658-0400 I SHARDING [ShardServerCatalogCacheLoader-0] 1|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.660-0400 d20010| 2018-03-16T17:28:25.659-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] refresh in primary [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.660-0400 d20010| 2018-03-16T17:28:25.659-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] 1 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.660-0400 d20010| 2018-03-16T17:28:25.659-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx collec and changed chunks max epoch or version doesn't equal max [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.695-0400 d20010| 2018-03-16T17:28:25.695-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Cache loader remotely refreshed for collection test.foo from collection version 1|0||5aac36f888c1f30185b7b0c4 and found collection version 2|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.696-0400 d20010| 2018-03-16T17:28:25.695-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Cache loader found enqueued metadata from 2|0||5aac36f888c1f30185b7b0c4 to 2|0||5aac36f888c1f30185b7b0c4 and persisted metadata from 1|0||5aac36f888c1f30185b7b0c4 to 1|0||5aac36f888c1f30185b7b0c4, GTE cache version 1|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.696-0400 d20010| 2018-03-16T17:28:25.695-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx is okay [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.696-0400 d20010| 2018-03-16T17:28:25.695-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx finsihing refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.696-0400 d20010| 2018-03-16T17:28:25.695-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh callback [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.696-0400 d20010| 2018-03-16T17:28:25.695-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh completed [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.697-0400 d20010| 2018-03-16T17:28:25.695-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Refresh for collection test.foo took 37 ms and found version 2|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.697-0400 d20010| 2018-03-16T17:28:25.695-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx marking as status ok [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.697-0400 d20010| 2018-03-16T17:28:25.695-0400 I SHARDING [conn15] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.697-0400 d20010| 2018-03-16T17:28:25.696-0400 I SHARDING [conn15] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.697-0400 d20010| 2018-03-16T17:28:25.696-0400 I SHARDING [conn15] Updating collection metadata for test.foo from collection version: 1|0||5aac36f888c1f30185b7b0c4, shard version: 1|0||5aac36f888c1f30185b7b0c4 to collection version: 2|0||5aac36f888c1f30185b7b0c4, shard version: 0|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.697-0400 d20010| 2018-03-16T17:28:25.696-0400 I SHARDING [conn15] Migration succeeded and updated collection version to 2|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.698-0400 d20010| 2018-03-16T17:28:25.696-0400 I SHARDING [conn15] about to log metadata event into changelog: { _id: "ubuntu-2018-03-16T17:28:25.696-0400-5aac36f9f7e5a0b40b8e3559", server: "ubuntu", clientAddr: "127.0.0.1:56424", time: new Date(1521235705696), what: "moveChunk.commit", ns: "test.foo", details: { min: { x: MinKey }, max: { x: MaxKey }, from: "stale_mongos_updates_and_removes-rs0", to: "stale_mongos_updates_and_removes-rs1", counts: { cloned: 20, clonedBytes: 1120, catchup: 0, steady: 0 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.705-0400 d20010| 2018-03-16T17:28:25.705-0400 I SHARDING [conn15] Scheduling immediate deletion of test.foo range [{ x: MinKey }, { x: MaxKey }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.706-0400 d20010| 2018-03-16T17:28:25.705-0400 I SHARDING [conn15] Waiting for cleanup of test.foo range [{ x: MinKey }, { x: MaxKey }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.706-0400 d20011| 2018-03-16T17:28:25.705-0400 I SHARDING [conn22] XXXX in set shard version cmd obj is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.706-0400 d20011| 2018-03-16T17:28:25.705-0400 I SHARDING [conn22] { setShardVersion: "test.foo", init: false, authoritative: false, configdb: "stale_mongos_updates_and_removes-configRS/ubuntu:20012,ubuntu:20013,ubuntu:20014", shard: "stale_mongos_updates_and_removes-rs1", shardHost: "ubuntu:20011", version: Timestamp(2, 0), versionEpoch: ObjectId('5aac36f888c1f30185b7b0c4'), noConnectionVersioning: true, $clusterTime: { clusterTime: Timestamp(1521235705, 70), signature: { hash: BinData(0, 6DF82EEC68E53F390B4A84C15A0D2C6D32CF7D93), keyId: 6533657568122765331 } }, $client: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" }, mongos: { host: "ubuntu:20015", client: "127.0.0.1:47840", version: "3.6.3" } }, $configServerState: { opTime: { ts: Timestamp(1521235705, 70), t: 1 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.707-0400 d20011| 2018-03-16T17:28:25.705-0400 I SHARDING [conn22] XXXX requested version is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.707-0400 d20011| 2018-03-16T17:28:25.705-0400 I SHARDING [conn22] 2|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.707-0400 d20010| 2018-03-16T17:28:25.706-0400 I SHARDING [Collection Range Deleter] No documents remain to delete in test.foo range [{ x: MinKey }, { x: MaxKey }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.707-0400 d20010| 2018-03-16T17:28:25.706-0400 I SHARDING [Collection Range Deleter] Waiting for majority replication of local deletions in test.foo range [{ x: MinKey }, { x: MaxKey }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.709-0400 d20010| 2018-03-16T17:28:25.708-0400 I SHARDING [Collection Range Deleter] Finished deleting documents in test.foo range [{ x: MinKey }, { x: MaxKey }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.709-0400 d20010| 2018-03-16T17:28:25.708-0400 I SHARDING [conn15] about to log metadata event into changelog: { _id: "ubuntu-2018-03-16T17:28:25.708-0400-5aac36f9f7e5a0b40b8e355a", server: "ubuntu", clientAddr: "127.0.0.1:56424", time: new Date(1521235705708), what: "moveChunk.from", ns: "test.foo", details: { min: { x: MinKey }, max: { x: MaxKey }, step 1 of 6: 0, step 2 of 6: 1, step 3 of 6: 110, step 4 of 6: 54, step 5 of 6: 404, step 6 of 6: 56, to: "stale_mongos_updates_and_removes-rs1", from: "stale_mongos_updates_and_removes-rs0", note: "success" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.714-0400 d20010| 2018-03-16T17:28:25.712-0400 I COMMAND [conn15] command admin.$cmd appName: "MongoDB Shell" command: moveChunk { moveChunk: "test.foo", shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f888c1f30185b7b0c4') ], epoch: ObjectId('5aac36f888c1f30185b7b0c4'), configdb: "stale_mongos_updates_and_removes-configRS/ubuntu:20012,ubuntu:20013,ubuntu:20014", fromShard: "stale_mongos_updates_and_removes-rs0", toShard: "stale_mongos_updates_and_removes-rs1", min: { x: MinKey }, max: { x: MaxKey }, maxChunkSizeBytes: 67108864, waitForDelete: true, takeDistLock: false, $clusterTime: { clusterTime: Timestamp(1521235705, 27), signature: { hash: BinData(0, 6DF82EEC68E53F390B4A84C15A0D2C6D32CF7D93), keyId: 6533657568122765331 } }, $client: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" }, mongos: { host: "ubuntu:20015", client: "127.0.0.1:47840", version: "3.6.3" } }, $configServerState: { opTime: { ts: Timestamp(1521235705, 27), t: 1 } }, $db: "admin" } numYields:0 reslen:317 locks:{ Global: { acquireCount: { r: 33, w: 15 } }, Database: { acquireCount: { r: 9, w: 13, W: 2 } }, Collection: { acquireCount: { r: 9, w: 1, W: 5 } }, oplog: { acquireCount: { w: 7 } } } protocol:op_msg 631ms [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.718-0400 c20012| 2018-03-16T17:28:25.717-0400 I SHARDING [NetworkInterfaceASIO-ShardRegistry-0] distributed lock with ts: '5aac36ec88c1f30185b7ac69' and _id: 'test.foo' unlocked. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.718-0400 c20012| 2018-03-16T17:28:25.717-0400 I SHARDING [conn21] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.718-0400 c20012| 2018-03-16T17:28:25.717-0400 I SHARDING [conn21] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.718-0400 c20012| 2018-03-16T17:28:25.718-0400 I SHARDING [conn21] xxx routing info needs refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.719-0400 c20012| 2018-03-16T17:28:25.718-0400 I SHARDING [conn21] xxx calling schedule collection refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.719-0400 c20012| 2018-03-16T17:28:25.718-0400 I SHARDING [conn21] xxx is incremental [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.719-0400 c20012| 2018-03-16T17:28:25.718-0400 I SHARDING [conn21] xxx existing routing info? [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.719-0400 c20012| 2018-03-16T17:28:25.718-0400 I SHARDING [conn21] 1|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.719-0400 c20012| 2018-03-16T17:28:25.718-0400 I SHARDING [conn21] Refreshing chunks for collection test.foo based on version 1|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.719-0400 c20012| 2018-03-16T17:28:25.718-0400 I SHARDING [conn21] xxx getting chunks since now [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.719-0400 c20012| 2018-03-16T17:28:25.718-0400 I SHARDING [conn21] xxx waiting if !refreshNot [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.720-0400 c20012| 2018-03-16T17:28:25.718-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh callback [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.720-0400 c20012| 2018-03-16T17:28:25.718-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh completed [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.720-0400 c20012| 2018-03-16T17:28:25.718-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Refresh for collection test.foo took 0 ms and found version 2|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.720-0400 c20012| 2018-03-16T17:28:25.718-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx marking as status ok [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.720-0400 c20012| 2018-03-16T17:28:25.718-0400 I SHARDING [conn21] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.720-0400 c20012| 2018-03-16T17:28:25.718-0400 I SHARDING [conn21] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.723-0400 c20012| 2018-03-16T17:28:25.722-0400 I COMMAND [conn21] command admin.$cmd appName: "MongoDB Shell" command: _configsvrMoveChunk { _configsvrMoveChunk: 1, _id: "test.foo-x_MinKey", ns: "test.foo", min: { x: MinKey }, max: { x: MaxKey }, shard: "stale_mongos_updates_and_removes-rs0", lastmod: Timestamp(1, 0), lastmodEpoch: ObjectId('5aac36f888c1f30185b7b0c4'), toShard: "stale_mongos_updates_and_removes-rs1", maxChunkSizeBytes: 67108864, secondaryThrottle: {}, waitForDelete: true, writeConcern: { w: "majority", wtimeout: 15000 }, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235705, 1), signature: { hash: BinData(0, 6DF82EEC68E53F390B4A84C15A0D2C6D32CF7D93), keyId: 6533657568122765331 } }, $client: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" }, mongos: { host: "ubuntu:20015", client: "127.0.0.1:47840", version: "3.6.3" } }, $configServerState: { opTime: { ts: Timestamp(1521235705, 1), t: 1 } }, $db: "admin" } numYields:0 reslen:462 locks:{ Global: { acquireCount: { r: 10, w: 6 } }, Database: { acquireCount: { r: 2, w: 6 } }, Collection: { acquireCount: { r: 2, w: 3 } }, oplog: { acquireCount: { w: 3 } } } protocol:op_msg 653ms [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.723-0400 s20015| 2018-03-16T17:28:25.722-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 64 finished with response: { ok: 1.0, operationTime: Timestamp(1521235705, 93), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235705, 93), t: 1 }, lastOpVisible: { ts: Timestamp(1521235705, 93), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235705, 93), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235705, 93), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235705, 93) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.724-0400 s20016| 2018-03-16T17:28:25.723-0400 D ASIO [conn1] startCommand: RemoteCommand 255 -- target:ubuntu:20010 db:test cmd:{ delete: "foo", bypassDocumentValidation: false, ordered: true, deletes: [ { q: {}, limit: 0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f888c1f30185b7b0c4') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.724-0400 s20016| 2018-03-16T17:28:25.723-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-4-0] Starting asynchronous command 255 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.724-0400 d20010| 2018-03-16T17:28:25.723-0400 I SHARDING [conn26] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.724-0400 d20010| 2018-03-16T17:28:25.723-0400 I SHARDING [conn26] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.724-0400 d20010| 2018-03-16T17:28:25.723-0400 I SHARDING [conn26] xxx routing info needs refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.724-0400 d20010| 2018-03-16T17:28:25.723-0400 I SHARDING [conn26] xxx calling schedule collection refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.724-0400 d20010| 2018-03-16T17:28:25.723-0400 I SHARDING [conn26] xxx is incremental [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.725-0400 d20010| 2018-03-16T17:28:25.723-0400 I SHARDING [conn26] xxx existing routing info? [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.725-0400 d20010| 2018-03-16T17:28:25.723-0400 I SHARDING [conn26] 2|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.725-0400 d20010| 2018-03-16T17:28:25.723-0400 I SHARDING [conn26] Refreshing chunks for collection test.foo based on version 2|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.725-0400 d20010| 2018-03-16T17:28:25.723-0400 I SHARDING [conn26] xxx getting chunks since now [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.725-0400 d20010| 2018-03-16T17:28:25.723-0400 I SHARDING [conn26] xxx getting chunks since [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.725-0400 d20010| 2018-03-16T17:28:25.723-0400 I SHARDING [conn26] xxx waiting if !refreshNot [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.725-0400 d20010| 2018-03-16T17:28:25.723-0400 I SHARDING [ShardServerCatalogCacheLoader-0] xxx primary and calling get chunks since for primary [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.725-0400 d20010| 2018-03-16T17:28:25.723-0400 I SHARDING [ShardServerCatalogCacheLoader-0] version is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.726-0400 d20010| 2018-03-16T17:28:25.723-0400 I SHARDING [ShardServerCatalogCacheLoader-0] 2|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.726-0400 d20010| 2018-03-16T17:28:25.724-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] refresh in primary [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.726-0400 d20010| 2018-03-16T17:28:25.724-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] 1 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.726-0400 d20010| 2018-03-16T17:28:25.724-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Cache loader remotely refreshed for collection test.foo from collection version 2|0||5aac36f888c1f30185b7b0c4 and found collection version 2|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.726-0400 d20010| 2018-03-16T17:28:25.725-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Cache loader found no enqueued metadata and persisted metadata from 2|0||5aac36f888c1f30185b7b0c4 to 2|0||5aac36f888c1f30185b7b0c4, GTE cache version 2|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.727-0400 d20010| 2018-03-16T17:28:25.725-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx is okay [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.727-0400 d20010| 2018-03-16T17:28:25.725-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx finsihing refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.727-0400 d20010| 2018-03-16T17:28:25.725-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh callback [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.727-0400 d20010| 2018-03-16T17:28:25.725-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh completed [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.727-0400 d20010| 2018-03-16T17:28:25.725-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Refresh for collection test.foo took 1 ms and found version 2|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.727-0400 d20010| 2018-03-16T17:28:25.725-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx marking as status ok [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.727-0400 d20010| 2018-03-16T17:28:25.725-0400 I SHARDING [conn26] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.728-0400 d20010| 2018-03-16T17:28:25.725-0400 I SHARDING [conn26] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.728-0400 s20016| 2018-03-16T17:28:25.725-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-4-0] Request 255 finished with response: { n: 0, writeErrors: [ { index: 0, code: 63, errInfo: { vWanted: Timestamp(0, 0), vWantedEpoch: ObjectId('5aac36f888c1f30185b7b0c4') }, errmsg: "shard version not ok: this shard no longer contains chunks for test.foo, the collection may have been dropped" } ], opTime: { ts: Timestamp(1521235705, 90), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235705, 90), $gleStats: { lastOpTime: { ts: Timestamp(1521235705, 90), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235705, 93), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235705, 90), $configServerState: { opTime: { ts: Timestamp(1521235705, 93), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.728-0400 s20016| 2018-03-16T17:28:25.725-0400 I SHARDING [conn1] Refreshing chunks for collection test.foo based on version 1|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.729-0400 s20016| 2018-03-16T17:28:25.725-0400 D ASIO [ConfigServerCatalogCacheLoader-0] startCommand: RemoteCommand 257 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:55.725-0400 cmd:{ find: "collections", filter: { _id: "test.foo" }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235705, 93), t: 1 } }, limit: 1, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.729-0400 s20016| 2018-03-16T17:28:25.725-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 257 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.730-0400 s20016| 2018-03-16T17:28:25.725-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 257 finished with response: { cursor: { firstBatch: [ { _id: "test.foo", lastmodEpoch: ObjectId('5aac36f888c1f30185b7b0c4'), lastmod: new Date(4294967296), dropped: false, key: { x: 1.0 }, unique: false, uuid: UUID("650ad6da-cccc-457f-bb84-294856863b67") } ], id: 0, ns: "config.collections" }, ok: 1.0, operationTime: Timestamp(1521235705, 93), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235705, 93), t: 1 }, lastOpVisible: { ts: Timestamp(1521235705, 93), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235705, 1), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235705, 93), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235705, 93) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.730-0400 s20016| 2018-03-16T17:28:25.726-0400 D ASIO [ConfigServerCatalogCacheLoader-0] startCommand: RemoteCommand 259 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:55.726-0400 cmd:{ find: "chunks", filter: { ns: "test.foo", lastmod: { $gte: Timestamp(1, 0) } }, sort: { lastmod: 1 }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235705, 93), t: 1 } }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.730-0400 s20016| 2018-03-16T17:28:25.726-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 259 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.731-0400 s20016| 2018-03-16T17:28:25.726-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 259 finished with response: { cursor: { firstBatch: [ { _id: "test.foo-x_MinKey", lastmod: Timestamp(2, 0), lastmodEpoch: ObjectId('5aac36f888c1f30185b7b0c4'), ns: "test.foo", min: { x: MinKey }, max: { x: MaxKey }, shard: "stale_mongos_updates_and_removes-rs1", history: [ { validAfter: Timestamp(1521235705, 62), shard: "stale_mongos_updates_and_removes-rs1" }, { validAfter: Timestamp(1521235704, 115), shard: "stale_mongos_updates_and_removes-rs0" } ] } ], id: 0, ns: "config.chunks" }, ok: 1.0, operationTime: Timestamp(1521235705, 93), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235705, 93), t: 1 }, lastOpVisible: { ts: Timestamp(1521235705, 93), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235705, 1), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235705, 93), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235705, 93) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.731-0400 s20016| 2018-03-16T17:28:25.726-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Refresh for collection test.foo took 1 ms and found version 2|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.732-0400 s20016| 2018-03-16T17:28:25.726-0400 D ASIO [conn1] startCommand: RemoteCommand 261 -- target:ubuntu:20011 db:test cmd:{ delete: "foo", bypassDocumentValidation: false, ordered: true, deletes: [ { q: {}, limit: 0 } ], shardVersion: [ Timestamp(2, 0), ObjectId('5aac36f888c1f30185b7b0c4') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.732-0400 s20016| 2018-03-16T17:28:25.726-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-5-0] Connecting to ubuntu:20011 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.732-0400 s20016| 2018-03-16T17:28:25.726-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-5-0] Starting asynchronous command 262 on host ubuntu:20011 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.732-0400 d20011| 2018-03-16T17:28:25.726-0400 I NETWORK [listener] connection accepted from 127.0.0.1:55184 #24 (14 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.732-0400 d20011| 2018-03-16T17:28:25.727-0400 I NETWORK [conn24] received client metadata from 127.0.0.1:55184 conn24: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-5", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.733-0400 s20016| 2018-03-16T17:28:25.727-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-5-0] Successfully connected to ubuntu:20011, took 1ms (1 connections now open to ubuntu:20011) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.733-0400 s20016| 2018-03-16T17:28:25.727-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-5-0] Request 262 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.733-0400 s20016| 2018-03-16T17:28:25.727-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-5-0] Starting asynchronous command 261 on host ubuntu:20011 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.733-0400 d20011| 2018-03-16T17:28:25.727-0400 I SHARDING [conn24] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.733-0400 d20011| 2018-03-16T17:28:25.727-0400 I SHARDING [conn24] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.733-0400 d20011| 2018-03-16T17:28:25.727-0400 I SHARDING [conn24] xxx routing info needs refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.733-0400 d20011| 2018-03-16T17:28:25.727-0400 I SHARDING [conn24] xxx calling schedule collection refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.734-0400 d20011| 2018-03-16T17:28:25.727-0400 I SHARDING [conn24] xxx is incremental [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.734-0400 d20011| 2018-03-16T17:28:25.727-0400 I SHARDING [conn24] xxx existing routing info? [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.734-0400 d20011| 2018-03-16T17:28:25.727-0400 I SHARDING [conn24] 1|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.734-0400 d20011| 2018-03-16T17:28:25.727-0400 I SHARDING [conn24] Refreshing chunks for collection test.foo based on version 1|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.734-0400 d20011| 2018-03-16T17:28:25.727-0400 I SHARDING [conn24] xxx getting chunks since now [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.734-0400 d20011| 2018-03-16T17:28:25.727-0400 I SHARDING [conn24] xxx getting chunks since [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.734-0400 d20011| 2018-03-16T17:28:25.727-0400 I SHARDING [conn24] xxx waiting if !refreshNot [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.734-0400 d20011| 2018-03-16T17:28:25.727-0400 I SHARDING [ShardServerCatalogCacheLoader-0] xxx primary and calling get chunks since for primary [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.735-0400 d20011| 2018-03-16T17:28:25.727-0400 I SHARDING [ShardServerCatalogCacheLoader-0] version is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.735-0400 d20011| 2018-03-16T17:28:25.727-0400 I SHARDING [ShardServerCatalogCacheLoader-0] 1|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.735-0400 d20011| 2018-03-16T17:28:25.728-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] refresh in primary [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.735-0400 d20011| 2018-03-16T17:28:25.728-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] 1 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.735-0400 d20011| 2018-03-16T17:28:25.728-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx collec and changed chunks max epoch or version doesn't equal max [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.796-0400 d20011| 2018-03-16T17:28:25.796-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Cache loader remotely refreshed for collection test.foo from collection version 1|0||5aac36f888c1f30185b7b0c4 and found collection version 2|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.797-0400 d20011| 2018-03-16T17:28:25.796-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Cache loader found enqueued metadata from 2|0||5aac36f888c1f30185b7b0c4 to 2|0||5aac36f888c1f30185b7b0c4 and persisted metadata from 1|0||5aac36f888c1f30185b7b0c4 to 1|0||5aac36f888c1f30185b7b0c4, GTE cache version 1|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.797-0400 d20011| 2018-03-16T17:28:25.796-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx is okay [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.797-0400 d20011| 2018-03-16T17:28:25.796-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx finsihing refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.797-0400 d20011| 2018-03-16T17:28:25.796-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh callback [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.797-0400 d20011| 2018-03-16T17:28:25.796-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh completed [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.797-0400 d20011| 2018-03-16T17:28:25.796-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Refresh for collection test.foo took 69 ms and found version 2|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.798-0400 d20011| 2018-03-16T17:28:25.796-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx marking as status ok [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.798-0400 d20011| 2018-03-16T17:28:25.796-0400 I SHARDING [conn24] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.798-0400 d20011| 2018-03-16T17:28:25.796-0400 I SHARDING [conn24] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.798-0400 d20011| 2018-03-16T17:28:25.796-0400 I SHARDING [conn24] Updating collection metadata for test.foo from collection version: 1|0||5aac36f888c1f30185b7b0c4, shard version: 0|0||5aac36f888c1f30185b7b0c4 to collection version: 2|0||5aac36f888c1f30185b7b0c4, shard version: 2|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.799-0400 s20016| 2018-03-16T17:28:25.797-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-5-0] Request 261 finished with response: { n: 0, writeErrors: [ { index: 0, code: 63, errInfo: { vWanted: Timestamp(0, 0), vWantedEpoch: ObjectId('5aac36f888c1f30185b7b0c4') }, errmsg: "shard version not ok: this shard no longer contains chunks for test.foo, the collection may have been dropped" } ], opTime: { ts: Timestamp(1521235705, 96), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235705, 96), $gleStats: { lastOpTime: { ts: Timestamp(1521235705, 96), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235705, 97), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235705, 94), $configServerState: { opTime: { ts: Timestamp(1521235705, 93), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.799-0400 s20016| 2018-03-16T17:28:25.797-0400 I SHARDING [conn1] Refreshing chunks for collection test.foo based on version 2|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.799-0400 s20016| 2018-03-16T17:28:25.797-0400 D ASIO [ConfigServerCatalogCacheLoader-0] startCommand: RemoteCommand 264 -- target:ubuntu:20013 db:config expDate:2018-03-16T17:28:55.797-0400 cmd:{ find: "collections", filter: { _id: "test.foo" }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235705, 93), t: 1 } }, limit: 1, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.800-0400 s20016| 2018-03-16T17:28:25.797-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 264 on host ubuntu:20013 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.800-0400 s20016| 2018-03-16T17:28:25.797-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 264 finished with response: { cursor: { firstBatch: [ { _id: "test.foo", lastmodEpoch: ObjectId('5aac36f888c1f30185b7b0c4'), lastmod: new Date(4294967296), dropped: false, key: { x: 1.0 }, unique: false, uuid: UUID("650ad6da-cccc-457f-bb84-294856863b67") } ], id: 0, ns: "config.collections" }, ok: 1.0, operationTime: Timestamp(1521235705, 93), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235705, 93), t: 1 }, lastOpVisible: { ts: Timestamp(1521235705, 93), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235705, 97), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235705, 93) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.801-0400 s20016| 2018-03-16T17:28:25.797-0400 D ASIO [ConfigServerCatalogCacheLoader-0] startCommand: RemoteCommand 266 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:55.797-0400 cmd:{ find: "chunks", filter: { ns: "test.foo", lastmod: { $gte: Timestamp(2, 0) } }, sort: { lastmod: 1 }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235705, 93), t: 1 } }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.801-0400 s20016| 2018-03-16T17:28:25.797-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 266 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.802-0400 s20016| 2018-03-16T17:28:25.798-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 266 finished with response: { cursor: { firstBatch: [ { _id: "test.foo-x_MinKey", lastmod: Timestamp(2, 0), lastmodEpoch: ObjectId('5aac36f888c1f30185b7b0c4'), ns: "test.foo", min: { x: MinKey }, max: { x: MaxKey }, shard: "stale_mongos_updates_and_removes-rs1", history: [ { validAfter: Timestamp(1521235705, 62), shard: "stale_mongos_updates_and_removes-rs1" }, { validAfter: Timestamp(1521235704, 115), shard: "stale_mongos_updates_and_removes-rs0" } ] } ], id: 0, ns: "config.chunks" }, ok: 1.0, operationTime: Timestamp(1521235705, 93), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235705, 93), t: 1 }, lastOpVisible: { ts: Timestamp(1521235705, 93), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235705, 1), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235705, 97), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235705, 93) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.802-0400 s20016| 2018-03-16T17:28:25.798-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Refresh for collection test.foo took 1 ms and found version 2|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.802-0400 s20016| 2018-03-16T17:28:25.798-0400 D ASIO [conn1] startCommand: RemoteCommand 268 -- target:ubuntu:20011 db:test cmd:{ delete: "foo", bypassDocumentValidation: false, ordered: true, deletes: [ { q: {}, limit: 0 } ], shardVersion: [ Timestamp(2, 0), ObjectId('5aac36f888c1f30185b7b0c4') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.803-0400 s20016| 2018-03-16T17:28:25.798-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-6-0] Connecting to ubuntu:20011 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.803-0400 s20016| 2018-03-16T17:28:25.798-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-6-0] Starting asynchronous command 269 on host ubuntu:20011 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.803-0400 d20011| 2018-03-16T17:28:25.798-0400 I NETWORK [listener] connection accepted from 127.0.0.1:55186 #25 (15 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.803-0400 d20011| 2018-03-16T17:28:25.798-0400 I NETWORK [conn25] received client metadata from 127.0.0.1:55186 conn25: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-6", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.803-0400 s20016| 2018-03-16T17:28:25.799-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-6-0] Successfully connected to ubuntu:20011, took 1ms (1 connections now open to ubuntu:20011) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.804-0400 s20016| 2018-03-16T17:28:25.799-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-6-0] Request 269 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.804-0400 s20016| 2018-03-16T17:28:25.799-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-6-0] Starting asynchronous command 268 on host ubuntu:20011 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.804-0400 s20016| 2018-03-16T17:28:25.799-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-6-0] Request 268 finished with response: { n: 20, opTime: { ts: Timestamp(1521235705, 118), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235705, 118), $gleStats: { lastOpTime: { ts: Timestamp(1521235705, 118), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235705, 118), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235705, 94), $configServerState: { opTime: { ts: Timestamp(1521235705, 93), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.805-0400 s20016| 2018-03-16T17:28:25.800-0400 D ASIO [conn1] startCommand: RemoteCommand 271 -- target:ubuntu:20011 db:test cmd:{ find: "foo", shardVersion: [ Timestamp(2, 0), ObjectId('5aac36f888c1f30185b7b0c4') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.805-0400 s20016| 2018-03-16T17:28:25.801-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-7-0] Connecting to ubuntu:20011 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.805-0400 s20016| 2018-03-16T17:28:25.801-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-7-0] Starting asynchronous command 272 on host ubuntu:20011 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.805-0400 d20011| 2018-03-16T17:28:25.801-0400 I NETWORK [listener] connection accepted from 127.0.0.1:55188 #26 (16 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.805-0400 d20011| 2018-03-16T17:28:25.801-0400 I NETWORK [conn26] received client metadata from 127.0.0.1:55188 conn26: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-7", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.806-0400 s20016| 2018-03-16T17:28:25.801-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-7-0] Successfully connected to ubuntu:20011, took 0ms (1 connections now open to ubuntu:20011) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.806-0400 s20016| 2018-03-16T17:28:25.801-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-7-0] Request 272 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.806-0400 s20016| 2018-03-16T17:28:25.801-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-7-0] Starting asynchronous command 271 on host ubuntu:20011 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.806-0400 s20016| 2018-03-16T17:28:25.801-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-7-0] Request 271 finished with response: { cursor: { firstBatch: [], id: 0, ns: "test.foo" }, ok: 1.0, operationTime: Timestamp(1521235705, 118), $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235705, 118), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235705, 94), $configServerState: { opTime: { ts: Timestamp(1521235705, 93), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.806-0400 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.807-0400 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.807-0400 ---- [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.807-0400 xxx top of remove [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.807-0400 ---- [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.807-0400 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.807-0400 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.807-0400 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.807-0400 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.807-0400 ---- [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.807-0400 xxx top multi stale [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.807-0400 ---- [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.807-0400 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.807-0400 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.808-0400 s20016| 2018-03-16T17:28:25.802-0400 D SHARDING [conn1] trying to acquire new distributed lock for test-movePrimary ( lock timeout : 900000 ms, ping interval : 30000 ms, process : ubuntu:20016:1521235700:-4633782234644071543 ) with lockSessionID: 5aac36f9ebcdfa1d2b617dbd, why: dropCollection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.808-0400 s20016| 2018-03-16T17:28:25.802-0400 D ASIO [conn1] startCommand: RemoteCommand 274 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:55.802-0400 cmd:{ findAndModify: "locks", query: { _id: "test-movePrimary", state: 0 }, update: { $set: { ts: ObjectId('5aac36f9ebcdfa1d2b617dbd'), state: 2, who: "ubuntu:20016:1521235700:-4633782234644071543:conn1", process: "ubuntu:20016:1521235700:-4633782234644071543", when: new Date(1521235705802), why: "dropCollection" } }, upsert: true, new: true, writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.808-0400 s20016| 2018-03-16T17:28:25.802-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 274 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.809-0400 s20016| 2018-03-16T17:28:25.809-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 274 finished with response: { lastErrorObject: { n: 1, updatedExisting: true }, value: { _id: "test-movePrimary", state: 2, process: "ubuntu:20016:1521235700:-4633782234644071543", ts: ObjectId('5aac36f9ebcdfa1d2b617dbd'), when: new Date(1521235705802), who: "ubuntu:20016:1521235700:-4633782234644071543:conn1", why: "dropCollection" }, ok: 1.0, operationTime: Timestamp(1521235705, 119), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235705, 119), t: 1 }, lastOpVisible: { ts: Timestamp(1521235705, 119), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235705, 119), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235705, 119), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235705, 119) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.810-0400 s20016| 2018-03-16T17:28:25.809-0400 I SHARDING [conn1] distributed lock 'test-movePrimary' acquired for 'dropCollection', ts : 5aac36f9ebcdfa1d2b617dbd [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.810-0400 s20016| 2018-03-16T17:28:25.809-0400 D SHARDING [conn1] trying to acquire new distributed lock for test ( lock timeout : 900000 ms, ping interval : 30000 ms, process : ubuntu:20016:1521235700:-4633782234644071543 ) with lockSessionID: 5aac36f9ebcdfa1d2b617dbe, why: dropCollection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.810-0400 s20016| 2018-03-16T17:28:25.809-0400 D ASIO [conn1] startCommand: RemoteCommand 276 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:55.809-0400 cmd:{ findAndModify: "locks", query: { _id: "test", state: 0 }, update: { $set: { ts: ObjectId('5aac36f9ebcdfa1d2b617dbe'), state: 2, who: "ubuntu:20016:1521235700:-4633782234644071543:conn1", process: "ubuntu:20016:1521235700:-4633782234644071543", when: new Date(1521235705809), why: "dropCollection" } }, upsert: true, new: true, writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.811-0400 s20016| 2018-03-16T17:28:25.809-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 276 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.813-0400 s20016| 2018-03-16T17:28:25.812-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 276 finished with response: { lastErrorObject: { n: 1, updatedExisting: true }, value: { _id: "test", state: 2, process: "ubuntu:20016:1521235700:-4633782234644071543", ts: ObjectId('5aac36f9ebcdfa1d2b617dbe'), when: new Date(1521235705809), who: "ubuntu:20016:1521235700:-4633782234644071543:conn1", why: "dropCollection" }, ok: 1.0, operationTime: Timestamp(1521235705, 120), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235705, 120), t: 1 }, lastOpVisible: { ts: Timestamp(1521235705, 120), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235705, 120), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235705, 120), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235705, 120) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.813-0400 s20016| 2018-03-16T17:28:25.812-0400 I SHARDING [conn1] distributed lock 'test' acquired for 'dropCollection', ts : 5aac36f9ebcdfa1d2b617dbe [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.814-0400 s20016| 2018-03-16T17:28:25.812-0400 D SHARDING [conn1] trying to acquire new distributed lock for test.foo ( lock timeout : 900000 ms, ping interval : 30000 ms, process : ubuntu:20016:1521235700:-4633782234644071543 ) with lockSessionID: 5aac36f9ebcdfa1d2b617dbf, why: dropCollection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.814-0400 s20016| 2018-03-16T17:28:25.812-0400 D ASIO [conn1] startCommand: RemoteCommand 278 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:55.812-0400 cmd:{ findAndModify: "locks", query: { _id: "test.foo", state: 0 }, update: { $set: { ts: ObjectId('5aac36f9ebcdfa1d2b617dbf'), state: 2, who: "ubuntu:20016:1521235700:-4633782234644071543:conn1", process: "ubuntu:20016:1521235700:-4633782234644071543", when: new Date(1521235705812), why: "dropCollection" } }, upsert: true, new: true, writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.814-0400 s20016| 2018-03-16T17:28:25.812-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 278 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.817-0400 s20016| 2018-03-16T17:28:25.816-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 278 finished with response: { lastErrorObject: { n: 1, updatedExisting: true }, value: { _id: "test.foo", state: 2, process: "ubuntu:20016:1521235700:-4633782234644071543", ts: ObjectId('5aac36f9ebcdfa1d2b617dbf'), when: new Date(1521235705812), who: "ubuntu:20016:1521235700:-4633782234644071543:conn1", why: "dropCollection" }, ok: 1.0, operationTime: Timestamp(1521235705, 121), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235705, 121), t: 1 }, lastOpVisible: { ts: Timestamp(1521235705, 121), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235705, 121), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235705, 121), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235705, 121) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.817-0400 s20016| 2018-03-16T17:28:25.816-0400 I SHARDING [conn1] distributed lock 'test.foo' acquired for 'dropCollection', ts : 5aac36f9ebcdfa1d2b617dbf [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.818-0400 s20016| 2018-03-16T17:28:25.817-0400 I SHARDING [conn1] about to log metadata event into changelog: { _id: "ubuntu-2018-03-16T17:28:25.816-0400-5aac36f9ebcdfa1d2b617dc0", server: "ubuntu", clientAddr: "127.0.0.1:50986", time: new Date(1521235705816), what: "dropCollection.start", ns: "test.foo", details: {} } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.818-0400 s20016| 2018-03-16T17:28:25.817-0400 D ASIO [conn1] startCommand: RemoteCommand 280 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:55.817-0400 cmd:{ insert: "changelog", bypassDocumentValidation: false, ordered: true, documents: [ { _id: "ubuntu-2018-03-16T17:28:25.816-0400-5aac36f9ebcdfa1d2b617dc0", server: "ubuntu", clientAddr: "127.0.0.1:50986", time: new Date(1521235705816), what: "dropCollection.start", ns: "test.foo", details: {} } ], writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.818-0400 s20016| 2018-03-16T17:28:25.817-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 280 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.821-0400 s20016| 2018-03-16T17:28:25.820-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 280 finished with response: { n: 1, opTime: { ts: Timestamp(1521235705, 122), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235705, 122), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235705, 122), t: 1 }, lastOpVisible: { ts: Timestamp(1521235705, 122), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235705, 122), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235705, 122), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235705, 122) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.822-0400 s20016| 2018-03-16T17:28:25.820-0400 D ASIO [conn1] startCommand: RemoteCommand 282 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:55.820-0400 cmd:{ find: "shards", readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235705, 122), t: 1 } }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.822-0400 s20016| 2018-03-16T17:28:25.821-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 282 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.822-0400 s20016| 2018-03-16T17:28:25.821-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 282 finished with response: { cursor: { firstBatch: [ { _id: "stale_mongos_updates_and_removes-rs0", host: "stale_mongos_updates_and_removes-rs0/ubuntu:20010", state: 1 }, { _id: "stale_mongos_updates_and_removes-rs1", host: "stale_mongos_updates_and_removes-rs1/ubuntu:20011", state: 1 } ], id: 0, ns: "config.shards" }, ok: 1.0, operationTime: Timestamp(1521235705, 122), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235705, 122), t: 1 }, lastOpVisible: { ts: Timestamp(1521235705, 122), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235705, 122), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235705, 122), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235705, 122) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.823-0400 s20016| 2018-03-16T17:28:25.821-0400 D SHARDING [conn1] dropCollection test.foo started [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.823-0400 s20016| 2018-03-16T17:28:25.821-0400 D ASIO [conn1] startCommand: RemoteCommand 284 -- target:ubuntu:20010 db:test cmd:{ drop: "foo" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.823-0400 s20016| 2018-03-16T17:28:25.821-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 284 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.823-0400 d20010| 2018-03-16T17:28:25.821-0400 I COMMAND [conn21] CMD: drop test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.823-0400 d20010| 2018-03-16T17:28:25.821-0400 I STORAGE [conn21] dropCollection: test.foo (650ad6da-cccc-457f-bb84-294856863b67) - renaming to drop-pending collection: test.system.drop.1521235705i123t1.foo with drop optime { ts: Timestamp(1521235705, 123), t: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.824-0400 s20016| 2018-03-16T17:28:25.824-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 284 finished with response: { ns: "test.foo", nIndexesWas: 2, ok: 1.0, operationTime: Timestamp(1521235705, 123), $gleStats: { lastOpTime: { ts: Timestamp(1521235705, 123), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235705, 123), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235705, 90), $configServerState: { opTime: { ts: Timestamp(1521235705, 122), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.824-0400 s20016| 2018-03-16T17:28:25.824-0400 D ASIO [conn1] startCommand: RemoteCommand 286 -- target:ubuntu:20011 db:test cmd:{ drop: "foo" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.825-0400 s20016| 2018-03-16T17:28:25.824-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 286 on host ubuntu:20011 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.825-0400 d20011| 2018-03-16T17:28:25.824-0400 I COMMAND [conn21] CMD: drop test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.825-0400 d20011| 2018-03-16T17:28:25.824-0400 I STORAGE [conn21] dropCollection: test.foo (650ad6da-cccc-457f-bb84-294856863b67) - renaming to drop-pending collection: test.system.drop.1521235705i124t1.foo with drop optime { ts: Timestamp(1521235705, 124), t: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.828-0400 s20016| 2018-03-16T17:28:25.828-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 286 finished with response: { ns: "test.foo", nIndexesWas: 2, ok: 1.0, operationTime: Timestamp(1521235705, 124), $gleStats: { lastOpTime: { ts: Timestamp(1521235705, 124), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235705, 124), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235705, 118), $configServerState: { opTime: { ts: Timestamp(1521235705, 122), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.828-0400 s20016| 2018-03-16T17:28:25.828-0400 D SHARDING [conn1] dropCollection test.foo shard data deleted [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.829-0400 s20016| 2018-03-16T17:28:25.828-0400 D ASIO [conn1] startCommand: RemoteCommand 288 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:55.828-0400 cmd:{ delete: "chunks", bypassDocumentValidation: false, ordered: true, deletes: [ { q: { ns: "test.foo" }, limit: 0 } ], writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.829-0400 s20016| 2018-03-16T17:28:25.828-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 288 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.833-0400 s20016| 2018-03-16T17:28:25.833-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 288 finished with response: { n: 1, opTime: { ts: Timestamp(1521235705, 125), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235705, 125), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235705, 125), t: 1 }, lastOpVisible: { ts: Timestamp(1521235705, 125), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235705, 125), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235705, 125), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235705, 125) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.834-0400 s20016| 2018-03-16T17:28:25.833-0400 D SHARDING [conn1] dropCollection test.foo chunk data deleted [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.834-0400 s20016| 2018-03-16T17:28:25.833-0400 D ASIO [conn1] startCommand: RemoteCommand 290 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:55.833-0400 cmd:{ update: "collections", bypassDocumentValidation: false, ordered: true, updates: [ { q: { _id: "test.foo" }, u: { _id: "test.foo", lastmodEpoch: ObjectId('000000000000000000000000'), lastmod: new Date(1521235705833), dropped: true }, multi: false, upsert: false } ], writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.834-0400 s20016| 2018-03-16T17:28:25.833-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 290 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.837-0400 s20016| 2018-03-16T17:28:25.837-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 290 finished with response: { n: 1, nModified: 1, opTime: { ts: Timestamp(1521235705, 126), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235705, 126), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235705, 126), t: 1 }, lastOpVisible: { ts: Timestamp(1521235705, 126), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235705, 126), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235705, 126), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235705, 126) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.837-0400 s20016| 2018-03-16T17:28:25.837-0400 D SHARDING [conn1] dropCollection test.foo collection marked as dropped [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.838-0400 s20016| 2018-03-16T17:28:25.837-0400 D ASIO [conn1] startCommand: RemoteCommand 292 -- target:ubuntu:20010 db:admin cmd:{ setShardVersion: "test.foo", init: false, authoritative: true, configdb: "stale_mongos_updates_and_removes-configRS/ubuntu:20012,ubuntu:20013,ubuntu:20014", shard: "stale_mongos_updates_and_removes-rs0", shardHost: "stale_mongos_updates_and_removes-rs0/ubuntu:20010", version: Timestamp(0, 0), versionEpoch: ObjectId('000000000000000000000000'), noConnectionVersioning: true } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.838-0400 s20016| 2018-03-16T17:28:25.837-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 292 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.838-0400 d20010| 2018-03-16T17:28:25.837-0400 I SHARDING [conn21] XXXX in set shard version cmd obj is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.839-0400 d20010| 2018-03-16T17:28:25.837-0400 I SHARDING [conn21] { setShardVersion: "test.foo", init: false, authoritative: true, configdb: "stale_mongos_updates_and_removes-configRS/ubuntu:20012,ubuntu:20013,ubuntu:20014", shard: "stale_mongos_updates_and_removes-rs0", shardHost: "stale_mongos_updates_and_removes-rs0/ubuntu:20010", version: Timestamp(0, 0), versionEpoch: ObjectId('000000000000000000000000'), noConnectionVersioning: true, $clusterTime: { clusterTime: Timestamp(1521235705, 126), signature: { hash: BinData(0, 6DF82EEC68E53F390B4A84C15A0D2C6D32CF7D93), keyId: 6533657568122765331 } }, $client: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" }, mongos: { host: "ubuntu:20016", client: "127.0.0.1:50986", version: "3.6.3" } }, $configServerState: { opTime: { ts: Timestamp(1521235705, 126), t: 1 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.839-0400 d20010| 2018-03-16T17:28:25.837-0400 I SHARDING [conn21] XXXX requested version is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.839-0400 d20010| 2018-03-16T17:28:25.837-0400 I SHARDING [conn21] 0|0||000000000000000000000000 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.839-0400 d20010| 2018-03-16T17:28:25.837-0400 I SHARDING [conn21] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.840-0400 d20010| 2018-03-16T17:28:25.837-0400 I SHARDING [conn21] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.840-0400 d20010| 2018-03-16T17:28:25.837-0400 I SHARDING [conn21] xxx routing info needs refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.840-0400 d20010| 2018-03-16T17:28:25.837-0400 I SHARDING [conn21] xxx calling schedule collection refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.840-0400 d20010| 2018-03-16T17:28:25.837-0400 I SHARDING [conn21] xxx is incremental [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.840-0400 d20010| 2018-03-16T17:28:25.837-0400 I SHARDING [conn21] xxx existing routing info? [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.840-0400 d20010| 2018-03-16T17:28:25.837-0400 I SHARDING [conn21] 2|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.840-0400 d20010| 2018-03-16T17:28:25.837-0400 I SHARDING [conn21] Refreshing chunks for collection test.foo based on version 2|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.840-0400 d20010| 2018-03-16T17:28:25.837-0400 I SHARDING [conn21] xxx getting chunks since now [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.841-0400 d20010| 2018-03-16T17:28:25.837-0400 I SHARDING [conn21] xxx getting chunks since [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.841-0400 d20010| 2018-03-16T17:28:25.837-0400 I SHARDING [conn21] xxx waiting if !refreshNot [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.841-0400 d20010| 2018-03-16T17:28:25.837-0400 I SHARDING [ShardServerCatalogCacheLoader-0] xxx primary and calling get chunks since for primary [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.841-0400 d20010| 2018-03-16T17:28:25.837-0400 I SHARDING [ShardServerCatalogCacheLoader-0] version is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.841-0400 d20010| 2018-03-16T17:28:25.837-0400 I SHARDING [ShardServerCatalogCacheLoader-0] 2|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.841-0400 d20010| 2018-03-16T17:28:25.838-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] XXXX namespace not found [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.897-0400 d20010| 2018-03-16T17:28:25.896-0400 I REPL [replication-0] Completing collection drop for test.system.drop.1521235705i123t1.foo with drop optime { ts: Timestamp(1521235705, 123), t: 1 } (notification optime: { ts: Timestamp(1521235705, 127), t: 1 }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.897-0400 d20010| 2018-03-16T17:28:25.896-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Cache loader remotely refreshed for collection test.foo from version 2|0||5aac36f888c1f30185b7b0c4 and no metadata was found. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.897-0400 d20010| 2018-03-16T17:28:25.896-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx finsihing refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.897-0400 d20010| 2018-03-16T17:28:25.896-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh callback [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.897-0400 d20010| 2018-03-16T17:28:25.896-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh completed [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.897-0400 d20010| 2018-03-16T17:28:25.896-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Refresh for collection test.foo took 59 ms and found the collection is not sharded [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.898-0400 d20010| 2018-03-16T17:28:25.896-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx marking as status ok [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.898-0400 d20010| 2018-03-16T17:28:25.896-0400 I STORAGE [replication-0] Finishing collection drop for test.system.drop.1521235705i123t1.foo (650ad6da-cccc-457f-bb84-294856863b67). [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.898-0400 d20010| 2018-03-16T17:28:25.896-0400 I COMMAND [ShardServerCatalogCacheLoader-0] CMD: drop config.cache.chunks.test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.898-0400 d20010| 2018-03-16T17:28:25.896-0400 I STORAGE [ShardServerCatalogCacheLoader-0] dropCollection: config.cache.chunks.test.foo (1659e1aa-b5d3-40dc-919d-6d4e2c140fb0) - renaming to drop-pending collection: config.system.drop.1521235705i129t1.cache.chunks.test.foo with drop optime { ts: Timestamp(1521235705, 129), t: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.898-0400 d20010| 2018-03-16T17:28:25.897-0400 I SHARDING [conn21] Marking collection test.foo with collection version: 2|0||5aac36f888c1f30185b7b0c4, shard version: 0|0||5aac36f888c1f30185b7b0c4 as unsharded [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.899-0400 s20016| 2018-03-16T17:28:25.897-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 292 finished with response: { oldVersion: Timestamp(0, 0), oldVersionEpoch: ObjectId('000000000000000000000000'), ok: 1.0, operationTime: Timestamp(1521235705, 128), $gleStats: { lastOpTime: { ts: Timestamp(1521235705, 123), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235705, 129), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235705, 127), $configServerState: { opTime: { ts: Timestamp(1521235705, 126), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.899-0400 s20016| 2018-03-16T17:28:25.897-0400 D ASIO [conn1] startCommand: RemoteCommand 294 -- target:ubuntu:20010 db:admin cmd:{ unsetSharding: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.899-0400 s20016| 2018-03-16T17:28:25.897-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 294 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.900-0400 s20016| 2018-03-16T17:28:25.897-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 294 finished with response: { ok: 1.0, operationTime: Timestamp(1521235705, 128), $gleStats: { lastOpTime: { ts: Timestamp(1521235705, 123), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235705, 129), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235705, 127), $configServerState: { opTime: { ts: Timestamp(1521235705, 126), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.900-0400 s20016| 2018-03-16T17:28:25.897-0400 D ASIO [conn1] startCommand: RemoteCommand 296 -- target:ubuntu:20011 db:admin cmd:{ setShardVersion: "test.foo", init: false, authoritative: true, configdb: "stale_mongos_updates_and_removes-configRS/ubuntu:20012,ubuntu:20013,ubuntu:20014", shard: "stale_mongos_updates_and_removes-rs1", shardHost: "stale_mongos_updates_and_removes-rs1/ubuntu:20011", version: Timestamp(0, 0), versionEpoch: ObjectId('000000000000000000000000'), noConnectionVersioning: true } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.900-0400 s20016| 2018-03-16T17:28:25.897-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 296 on host ubuntu:20011 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.901-0400 d20011| 2018-03-16T17:28:25.897-0400 I SHARDING [conn21] XXXX in set shard version cmd obj is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.901-0400 d20011| 2018-03-16T17:28:25.897-0400 I SHARDING [conn21] { setShardVersion: "test.foo", init: false, authoritative: true, configdb: "stale_mongos_updates_and_removes-configRS/ubuntu:20012,ubuntu:20013,ubuntu:20014", shard: "stale_mongos_updates_and_removes-rs1", shardHost: "stale_mongos_updates_and_removes-rs1/ubuntu:20011", version: Timestamp(0, 0), versionEpoch: ObjectId('000000000000000000000000'), noConnectionVersioning: true, $clusterTime: { clusterTime: Timestamp(1521235705, 129), signature: { hash: BinData(0, 6DF82EEC68E53F390B4A84C15A0D2C6D32CF7D93), keyId: 6533657568122765331 } }, $client: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" }, mongos: { host: "ubuntu:20016", client: "127.0.0.1:50986", version: "3.6.3" } }, $configServerState: { opTime: { ts: Timestamp(1521235705, 126), t: 1 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.901-0400 d20011| 2018-03-16T17:28:25.897-0400 I SHARDING [conn21] XXXX requested version is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.902-0400 d20011| 2018-03-16T17:28:25.897-0400 I SHARDING [conn21] 0|0||000000000000000000000000 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.902-0400 d20011| 2018-03-16T17:28:25.897-0400 I SHARDING [conn21] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.902-0400 d20011| 2018-03-16T17:28:25.897-0400 I SHARDING [conn21] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.902-0400 d20011| 2018-03-16T17:28:25.897-0400 I SHARDING [conn21] xxx routing info needs refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.902-0400 d20011| 2018-03-16T17:28:25.897-0400 I SHARDING [conn21] xxx calling schedule collection refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.902-0400 d20011| 2018-03-16T17:28:25.897-0400 I SHARDING [conn21] xxx is incremental [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.902-0400 d20011| 2018-03-16T17:28:25.897-0400 I SHARDING [conn21] xxx existing routing info? [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.902-0400 d20011| 2018-03-16T17:28:25.897-0400 I SHARDING [conn21] 2|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.903-0400 d20011| 2018-03-16T17:28:25.897-0400 I SHARDING [conn21] Refreshing chunks for collection test.foo based on version 2|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.903-0400 d20011| 2018-03-16T17:28:25.897-0400 I SHARDING [conn21] xxx getting chunks since now [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.903-0400 d20011| 2018-03-16T17:28:25.897-0400 I SHARDING [conn21] xxx getting chunks since [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.903-0400 d20011| 2018-03-16T17:28:25.897-0400 I SHARDING [conn21] xxx waiting if !refreshNot [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.903-0400 d20011| 2018-03-16T17:28:25.897-0400 I SHARDING [ShardServerCatalogCacheLoader-0] xxx primary and calling get chunks since for primary [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.903-0400 d20011| 2018-03-16T17:28:25.897-0400 I SHARDING [ShardServerCatalogCacheLoader-0] version is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.903-0400 d20011| 2018-03-16T17:28:25.897-0400 I SHARDING [ShardServerCatalogCacheLoader-0] 2|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.903-0400 d20011| 2018-03-16T17:28:25.898-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] XXXX namespace not found [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.904-0400 d20011| 2018-03-16T17:28:25.899-0400 I REPL [replication-0] Completing collection drop for test.system.drop.1521235705i124t1.foo with drop optime { ts: Timestamp(1521235705, 124), t: 1 } (notification optime: { ts: Timestamp(1521235705, 124), t: 1 }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.904-0400 d20011| 2018-03-16T17:28:25.899-0400 I STORAGE [replication-0] Finishing collection drop for test.system.drop.1521235705i124t1.foo (650ad6da-cccc-457f-bb84-294856863b67). [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.926-0400 d20010| 2018-03-16T17:28:25.925-0400 I REPL [replication-0] Completing collection drop for config.system.drop.1521235705i129t1.cache.chunks.test.foo with drop optime { ts: Timestamp(1521235705, 129), t: 1 } (notification optime: { ts: Timestamp(1521235705, 129), t: 1 }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.926-0400 d20010| 2018-03-16T17:28:25.925-0400 I STORAGE [replication-0] Finishing collection drop for config.system.drop.1521235705i129t1.cache.chunks.test.foo (1659e1aa-b5d3-40dc-919d-6d4e2c140fb0). [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.928-0400 d20011| 2018-03-16T17:28:25.927-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Cache loader remotely refreshed for collection test.foo from version 2|0||5aac36f888c1f30185b7b0c4 and no metadata was found. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.928-0400 d20011| 2018-03-16T17:28:25.927-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx finsihing refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.928-0400 d20011| 2018-03-16T17:28:25.927-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh callback [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.928-0400 d20011| 2018-03-16T17:28:25.927-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh completed [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.928-0400 d20011| 2018-03-16T17:28:25.927-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Refresh for collection test.foo took 30 ms and found the collection is not sharded [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.929-0400 d20011| 2018-03-16T17:28:25.927-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx marking as status ok [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.929-0400 d20011| 2018-03-16T17:28:25.928-0400 I SHARDING [conn21] Marking collection test.foo with collection version: 2|0||5aac36f888c1f30185b7b0c4, shard version: 2|0||5aac36f888c1f30185b7b0c4 as unsharded [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.929-0400 d20011| 2018-03-16T17:28:25.928-0400 I COMMAND [ShardServerCatalogCacheLoader-0] CMD: drop config.cache.chunks.test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.929-0400 s20016| 2018-03-16T17:28:25.928-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 296 finished with response: { oldVersion: Timestamp(0, 0), oldVersionEpoch: ObjectId('000000000000000000000000'), ok: 1.0, operationTime: Timestamp(1521235705, 130), $gleStats: { lastOpTime: { ts: Timestamp(1521235705, 124), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235705, 131), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235705, 130), $configServerState: { opTime: { ts: Timestamp(1521235705, 126), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.930-0400 d20011| 2018-03-16T17:28:25.928-0400 I STORAGE [ShardServerCatalogCacheLoader-0] dropCollection: config.cache.chunks.test.foo (90d2b6b6-c94f-4a34-b6ef-f2815aeaa7a7) - renaming to drop-pending collection: config.system.drop.1521235705i132t1.cache.chunks.test.foo with drop optime { ts: Timestamp(1521235705, 132), t: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.930-0400 s20016| 2018-03-16T17:28:25.928-0400 D ASIO [conn1] startCommand: RemoteCommand 298 -- target:ubuntu:20011 db:admin cmd:{ unsetSharding: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.930-0400 s20016| 2018-03-16T17:28:25.928-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 298 on host ubuntu:20011 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.931-0400 s20016| 2018-03-16T17:28:25.928-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 298 finished with response: { ok: 1.0, operationTime: Timestamp(1521235705, 131), $gleStats: { lastOpTime: { ts: Timestamp(1521235705, 124), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235705, 132), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235705, 130), $configServerState: { opTime: { ts: Timestamp(1521235705, 126), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.931-0400 s20016| 2018-03-16T17:28:25.928-0400 D SHARDING [conn1] dropCollection test.foo completed [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.931-0400 s20016| 2018-03-16T17:28:25.928-0400 I SHARDING [conn1] about to log metadata event into changelog: { _id: "ubuntu-2018-03-16T17:28:25.928-0400-5aac36f9ebcdfa1d2b617dc1", server: "ubuntu", clientAddr: "127.0.0.1:50986", time: new Date(1521235705928), what: "dropCollection", ns: "test.foo", details: {} } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.931-0400 s20016| 2018-03-16T17:28:25.928-0400 D ASIO [conn1] startCommand: RemoteCommand 300 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:55.928-0400 cmd:{ insert: "changelog", bypassDocumentValidation: false, ordered: true, documents: [ { _id: "ubuntu-2018-03-16T17:28:25.928-0400-5aac36f9ebcdfa1d2b617dc1", server: "ubuntu", clientAddr: "127.0.0.1:50986", time: new Date(1521235705928), what: "dropCollection", ns: "test.foo", details: {} } ], writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.932-0400 s20016| 2018-03-16T17:28:25.928-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 300 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.933-0400 s20016| 2018-03-16T17:28:25.932-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 300 finished with response: { n: 1, opTime: { ts: Timestamp(1521235705, 133), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235705, 133), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235705, 133), t: 1 }, lastOpVisible: { ts: Timestamp(1521235705, 133), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235705, 133), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235705, 133), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235705, 133) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.933-0400 s20016| 2018-03-16T17:28:25.932-0400 D ASIO [conn1] startCommand: RemoteCommand 302 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:55.932-0400 cmd:{ findAndModify: "locks", query: { ts: ObjectId('5aac36f9ebcdfa1d2b617dbf') }, update: { $set: { state: 0 } }, writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.933-0400 s20016| 2018-03-16T17:28:25.932-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 302 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.937-0400 s20016| 2018-03-16T17:28:25.936-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 302 finished with response: { lastErrorObject: { n: 1, updatedExisting: true }, value: { _id: "test.foo", state: 2, process: "ubuntu:20016:1521235700:-4633782234644071543", ts: ObjectId('5aac36f9ebcdfa1d2b617dbf'), when: new Date(1521235705812), who: "ubuntu:20016:1521235700:-4633782234644071543:conn1", why: "dropCollection" }, ok: 1.0, operationTime: Timestamp(1521235705, 134), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235705, 134), t: 1 }, lastOpVisible: { ts: Timestamp(1521235705, 134), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235705, 134), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235705, 134), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235705, 134) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.937-0400 s20016| 2018-03-16T17:28:25.936-0400 I SHARDING [conn1] distributed lock with ts: 5aac36f9ebcdfa1d2b617dbf' unlocked. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.937-0400 s20016| 2018-03-16T17:28:25.936-0400 D ASIO [conn1] startCommand: RemoteCommand 304 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:55.936-0400 cmd:{ findAndModify: "locks", query: { ts: ObjectId('5aac36f9ebcdfa1d2b617dbe') }, update: { $set: { state: 0 } }, writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.937-0400 s20016| 2018-03-16T17:28:25.936-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 304 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.944-0400 s20016| 2018-03-16T17:28:25.943-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 304 finished with response: { lastErrorObject: { n: 1, updatedExisting: true }, value: { _id: "test", state: 2, process: "ubuntu:20016:1521235700:-4633782234644071543", ts: ObjectId('5aac36f9ebcdfa1d2b617dbe'), when: new Date(1521235705809), who: "ubuntu:20016:1521235700:-4633782234644071543:conn1", why: "dropCollection" }, ok: 1.0, operationTime: Timestamp(1521235705, 135), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235705, 135), t: 1 }, lastOpVisible: { ts: Timestamp(1521235705, 135), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235705, 135), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235705, 135), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235705, 135) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.944-0400 s20016| 2018-03-16T17:28:25.943-0400 I SHARDING [conn1] distributed lock with ts: 5aac36f9ebcdfa1d2b617dbe' unlocked. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.945-0400 s20016| 2018-03-16T17:28:25.943-0400 D ASIO [conn1] startCommand: RemoteCommand 306 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:55.943-0400 cmd:{ findAndModify: "locks", query: { ts: ObjectId('5aac36f9ebcdfa1d2b617dbd') }, update: { $set: { state: 0 } }, writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.945-0400 s20016| 2018-03-16T17:28:25.943-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 306 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.948-0400 s20016| 2018-03-16T17:28:25.947-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 306 finished with response: { lastErrorObject: { n: 1, updatedExisting: true }, value: { _id: "test-movePrimary", state: 2, process: "ubuntu:20016:1521235700:-4633782234644071543", ts: ObjectId('5aac36f9ebcdfa1d2b617dbd'), when: new Date(1521235705802), who: "ubuntu:20016:1521235700:-4633782234644071543:conn1", why: "dropCollection" }, ok: 1.0, operationTime: Timestamp(1521235705, 136), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235705, 136), t: 1 }, lastOpVisible: { ts: Timestamp(1521235705, 136), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235705, 136), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235705, 136), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235705, 136) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.949-0400 s20016| 2018-03-16T17:28:25.947-0400 I SHARDING [conn1] distributed lock with ts: 5aac36f9ebcdfa1d2b617dbd' unlocked. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.949-0400 s20016| 2018-03-16T17:28:25.948-0400 D ASIO [conn1] startCommand: RemoteCommand 308 -- target:ubuntu:20012 db:admin cmd:{ _configsvrShardCollection: "test.foo", key: { x: 1.0 }, unique: false, numInitialChunks: 0, writeConcern: { w: "majority", wtimeout: 60000 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.949-0400 s20016| 2018-03-16T17:28:25.948-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 308 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.952-0400 c20012| 2018-03-16T17:28:25.952-0400 I SHARDING [conn23] distributed lock 'test' acquired for 'shardCollection', ts : 5aac36f988c1f30185b7b1b9 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.957-0400 c20012| 2018-03-16T17:28:25.957-0400 I SHARDING [conn23] distributed lock 'test.foo' acquired for 'shardCollection', ts : 5aac36f988c1f30185b7b1bf [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.958-0400 d20010| 2018-03-16T17:28:25.958-0400 I STORAGE [conn15] createCollection: test.foo with generated UUID: 53837380-bed7-429b-ba96-f32fe994b394 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.982-0400 d20010| 2018-03-16T17:28:25.981-0400 I INDEX [conn15] build index on: test.foo properties: { v: 2, key: { x: 1.0 }, name: "x_1", ns: "test.foo" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.982-0400 d20010| 2018-03-16T17:28:25.981-0400 I INDEX [conn15] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.982-0400 d20010| 2018-03-16T17:28:25.982-0400 I INDEX [conn15] build index done. scanned 0 total records. 0 secs [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.986-0400 c20012| 2018-03-16T17:28:25.985-0400 I SHARDING [conn23] CMD: shardcollection: { _configsvrShardCollection: "test.foo", key: { x: 1.0 }, unique: false, numInitialChunks: 0, writeConcern: { w: "majority", wtimeout: 60000 }, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235705, 136), signature: { hash: BinData(0, 6DF82EEC68E53F390B4A84C15A0D2C6D32CF7D93), keyId: 6533657568122765331 } }, $client: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" }, mongos: { host: "ubuntu:20016", client: "127.0.0.1:50986", version: "3.6.3" } }, $configServerState: { opTime: { ts: Timestamp(1521235705, 136), t: 1 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.986-0400 c20012| 2018-03-16T17:28:25.985-0400 I SHARDING [conn23] about to log metadata event into changelog: { _id: "ubuntu-2018-03-16T17:28:25.985-0400-5aac36f988c1f30185b7b1d1", server: "ubuntu", clientAddr: "127.0.0.1:60682", time: new Date(1521235705985), what: "shardCollection.start", ns: "test.foo", details: { shardKey: { x: 1.0 }, collection: "test.foo", uuid: UUID("53837380-bed7-429b-ba96-f32fe994b394"), primary: "stale_mongos_updates_and_removes-rs0:stale_mongos_updates_and_removes-rs0/ubuntu:20010", numChunks: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.990-0400 c20012| 2018-03-16T17:28:25.989-0400 I SHARDING [conn23] going to create 1 chunk(s) for: test.foo using new epoch 5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.999-0400 d20010| 2018-03-16T17:28:25.998-0400 I SHARDING [conn15] XXXX in set shard version cmd obj is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.999-0400 d20010| 2018-03-16T17:28:25.998-0400 I SHARDING [conn15] { setShardVersion: "test.foo", init: false, authoritative: true, configdb: "stale_mongos_updates_and_removes-configRS/ubuntu:20012,ubuntu:20013,ubuntu:20014", shard: "stale_mongos_updates_and_removes-rs0", shardHost: "stale_mongos_updates_and_removes-rs0/ubuntu:20010", version: Timestamp(1, 0), versionEpoch: ObjectId('5aac36f988c1f30185b7b1df'), noConnectionVersioning: true, $clusterTime: { clusterTime: Timestamp(1521235705, 143), signature: { hash: BinData(0, 6DF82EEC68E53F390B4A84C15A0D2C6D32CF7D93), keyId: 6533657568122765331 } }, $client: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" }, mongos: { host: "ubuntu:20016", client: "127.0.0.1:50986", version: "3.6.3" } }, $configServerState: { opTime: { ts: Timestamp(1521235705, 143), t: 1 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:25.999-0400 d20010| 2018-03-16T17:28:25.998-0400 I SHARDING [conn15] XXXX requested version is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.000-0400 d20010| 2018-03-16T17:28:25.998-0400 I SHARDING [conn15] 1|0||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.000-0400 d20010| 2018-03-16T17:28:25.998-0400 I SHARDING [conn15] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.000-0400 d20010| 2018-03-16T17:28:25.998-0400 I SHARDING [conn15] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.000-0400 d20010| 2018-03-16T17:28:25.998-0400 I SHARDING [conn15] xxx routing info needs refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.000-0400 d20010| 2018-03-16T17:28:25.998-0400 I SHARDING [conn15] xxx calling schedule collection refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.000-0400 d20010| 2018-03-16T17:28:25.998-0400 I SHARDING [conn15] xxx is not incr [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.000-0400 d20010| 2018-03-16T17:28:25.998-0400 I SHARDING [conn15] xxx existing routing info? [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.000-0400 d20010| 2018-03-16T17:28:25.998-0400 I SHARDING [conn15] Refreshing chunks for collection test.foo based on version 0|0||000000000000000000000000 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.001-0400 d20010| 2018-03-16T17:28:25.998-0400 I SHARDING [conn15] xxx getting chunks since now [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.001-0400 d20010| 2018-03-16T17:28:25.998-0400 I SHARDING [conn15] xxx getting chunks since [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.001-0400 d20010| 2018-03-16T17:28:25.998-0400 I SHARDING [conn15] xxx waiting if !refreshNot [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.001-0400 d20010| 2018-03-16T17:28:25.998-0400 I SHARDING [ShardServerCatalogCacheLoader-0] xxx primary and calling get chunks since for primary [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.001-0400 d20010| 2018-03-16T17:28:25.999-0400 I SHARDING [ShardServerCatalogCacheLoader-0] version is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.001-0400 d20010| 2018-03-16T17:28:25.999-0400 I SHARDING [ShardServerCatalogCacheLoader-0] 0|0||000000000000000000000000 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.001-0400 d20010| 2018-03-16T17:28:25.999-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] refresh in primary [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.002-0400 d20010| 2018-03-16T17:28:25.999-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] 1 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.002-0400 d20010| 2018-03-16T17:28:25.999-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx collec and changed chunks max epoch or version doesn't equal max [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.002-0400 d20011| 2018-03-16T17:28:26.001-0400 I REPL [replication-0] Completing collection drop for config.system.drop.1521235705i132t1.cache.chunks.test.foo with drop optime { ts: Timestamp(1521235705, 132), t: 1 } (notification optime: { ts: Timestamp(1521235705, 132), t: 1 }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.002-0400 d20011| 2018-03-16T17:28:26.001-0400 I STORAGE [replication-0] Finishing collection drop for config.system.drop.1521235705i132t1.cache.chunks.test.foo (90d2b6b6-c94f-4a34-b6ef-f2815aeaa7a7). [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.071-0400 c20012| 2018-03-16T17:28:26.070-0400 D REPL_HB [replexec-2] Sending heartbeat (requestId: 99) to ubuntu:20013, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20012", fromId: 0, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.071-0400 c20012| 2018-03-16T17:28:26.070-0400 D REPL_HB [replexec-1] Sending heartbeat (requestId: 100) to ubuntu:20014, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20012", fromId: 0, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.071-0400 c20013| 2018-03-16T17:28:26.070-0400 D REPL_HB [conn3] Received heartbeat request from ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20012", fromId: 0, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235705, 143), signature: { hash: BinData(0, 6DF82EEC68E53F390B4A84C15A0D2C6D32CF7D93), keyId: 6533657568122765331 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.072-0400 c20013| 2018-03-16T17:28:26.071-0400 D REPL_HB [conn3] Processed heartbeat from ubuntu:20012 and generated response, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", syncingTo: "ubuntu:20012", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235705, 143), t: 1 }, opTime: { ts: Timestamp(1521235705, 143), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.072-0400 c20014| 2018-03-16T17:28:26.071-0400 D REPL_HB [conn4] Received heartbeat request from ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20012", fromId: 0, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235705, 143), signature: { hash: BinData(0, 6DF82EEC68E53F390B4A84C15A0D2C6D32CF7D93), keyId: 6533657568122765331 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.072-0400 c20014| 2018-03-16T17:28:26.071-0400 D REPL_HB [conn4] Processed heartbeat from ubuntu:20012 and generated response, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", syncingTo: "ubuntu:20012", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235705, 143), t: 1 }, opTime: { ts: Timestamp(1521235705, 143), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.073-0400 c20012| 2018-03-16T17:28:26.071-0400 D REPL_HB [replexec-0] Received response to heartbeat (requestId: 99) from ubuntu:20013, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", syncingTo: "ubuntu:20012", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235705, 143), t: 1 }, opTime: { ts: Timestamp(1521235705, 143), t: 1 }, operationTime: Timestamp(1521235705, 143), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235705, 143), t: 1 }, lastOpVisible: { ts: Timestamp(1521235705, 143), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235705, 143), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235705, 143) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.073-0400 c20012| 2018-03-16T17:28:26.071-0400 D REPL_HB [replexec-0] Scheduling heartbeat to ubuntu:20013 at 2018-03-16T21:28:28.071Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.074-0400 c20012| 2018-03-16T17:28:26.071-0400 D REPL_HB [replexec-2] Received response to heartbeat (requestId: 100) from ubuntu:20014, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", syncingTo: "ubuntu:20012", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235705, 143), t: 1 }, opTime: { ts: Timestamp(1521235705, 143), t: 1 }, operationTime: Timestamp(1521235705, 143), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235705, 143), t: 1 }, lastOpVisible: { ts: Timestamp(1521235705, 143), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235705, 143), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235705, 143) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.074-0400 c20012| 2018-03-16T17:28:26.071-0400 D REPL_HB [replexec-2] Scheduling heartbeat to ubuntu:20014 at 2018-03-16T21:28:28.071Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.074-0400 d20010| 2018-03-16T17:28:26.073-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Cache loader remotely refreshed for collection test.foo from collection version 0|0||000000000000000000000000 and found collection version 1|0||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.075-0400 d20010| 2018-03-16T17:28:26.073-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Cache loader found enqueued metadata from 1|0||5aac36f988c1f30185b7b1df to 1|0||5aac36f988c1f30185b7b1df and no persisted metadata, GTE cache version 0|0||000000000000000000000000 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.075-0400 d20010| 2018-03-16T17:28:26.073-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx is okay [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.075-0400 d20010| 2018-03-16T17:28:26.073-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx finsihing refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.075-0400 d20010| 2018-03-16T17:28:26.073-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh callback [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.075-0400 d20010| 2018-03-16T17:28:26.073-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh completed [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.075-0400 d20010| 2018-03-16T17:28:26.073-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Refresh for collection test.foo took 74 ms and found version 1|0||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.076-0400 d20010| 2018-03-16T17:28:26.073-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx marking as status ok [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.076-0400 d20010| 2018-03-16T17:28:26.073-0400 I SHARDING [conn15] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.076-0400 d20010| 2018-03-16T17:28:26.073-0400 I SHARDING [conn15] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.076-0400 d20010| 2018-03-16T17:28:26.073-0400 I SHARDING [conn15] Marking collection test.foo as sharded with collection version: 1|0||5aac36f988c1f30185b7b1df, shard version: 1|0||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.076-0400 d20010| 2018-03-16T17:28:26.073-0400 I STORAGE [ShardServerCatalogCacheLoader-0] createCollection: config.cache.chunks.test.foo with generated UUID: 59771135-3e65-4e94-a4d8-1a81b63cddcb [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.077-0400 c20012| 2018-03-16T17:28:26.073-0400 I SHARDING [conn23] about to log metadata event into changelog: { _id: "ubuntu-2018-03-16T17:28:26.073-0400-5aac36fa88c1f30185b7b1f3", server: "ubuntu", clientAddr: "127.0.0.1:60682", time: new Date(1521235706073), what: "shardCollection.end", ns: "test.foo", details: { version: "1|0||5aac36f988c1f30185b7b1df" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.089-0400 c20012| 2018-03-16T17:28:26.089-0400 I SHARDING [conn23] distributed lock with ts: 5aac36f988c1f30185b7b1bf' unlocked. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.094-0400 c20012| 2018-03-16T17:28:26.094-0400 I SHARDING [conn23] distributed lock with ts: 5aac36f988c1f30185b7b1b9' unlocked. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.095-0400 c20012| 2018-03-16T17:28:26.094-0400 I COMMAND [conn23] command admin.$cmd appName: "MongoDB Shell" command: _configsvrShardCollection { _configsvrShardCollection: "test.foo", key: { x: 1.0 }, unique: false, numInitialChunks: 0, writeConcern: { w: "majority", wtimeout: 60000 }, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235705, 136), signature: { hash: BinData(0, 6DF82EEC68E53F390B4A84C15A0D2C6D32CF7D93), keyId: 6533657568122765331 } }, $client: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" }, mongos: { host: "ubuntu:20016", client: "127.0.0.1:50986", version: "3.6.3" } }, $configServerState: { opTime: { ts: Timestamp(1521235705, 136), t: 1 } }, $db: "admin" } numYields:0 reslen:531 locks:{ Global: { acquireCount: { r: 30, w: 16 } }, Database: { acquireCount: { r: 7, w: 16 } }, Collection: { acquireCount: { r: 7, w: 8 } }, Metadata: { acquireCount: { W: 2 } }, oplog: { acquireCount: { w: 8 } } } protocol:op_msg 146ms [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.096-0400 s20016| 2018-03-16T17:28:26.094-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 308 finished with response: { collectionsharded: "test.foo", collectionUUID: UUID("53837380-bed7-429b-ba96-f32fe994b394"), ok: 1.0, operationTime: Timestamp(1521235706, 5), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235706, 5), t: 1 }, lastOpVisible: { ts: Timestamp(1521235706, 5), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235706, 5), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235706, 5), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235706, 5) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.096-0400 s20016| 2018-03-16T17:28:26.095-0400 I SHARDING [conn1] Refreshing chunks for collection test.foo based on version 2|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.097-0400 s20016| 2018-03-16T17:28:26.095-0400 D ASIO [ConfigServerCatalogCacheLoader-0] startCommand: RemoteCommand 310 -- target:ubuntu:20014 db:config expDate:2018-03-16T17:28:56.095-0400 cmd:{ find: "collections", filter: { _id: "test.foo" }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235706, 5), t: 1 } }, limit: 1, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.097-0400 s20016| 2018-03-16T17:28:26.095-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 310 on host ubuntu:20014 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.097-0400 s20016| 2018-03-16T17:28:26.096-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 310 finished with response: { cursor: { firstBatch: [ { _id: "test.foo", lastmodEpoch: ObjectId('5aac36f988c1f30185b7b1df'), lastmod: new Date(4294967296), dropped: false, key: { x: 1.0 }, unique: false, uuid: UUID("53837380-bed7-429b-ba96-f32fe994b394") } ], id: 0, ns: "config.collections" }, ok: 1.0, operationTime: Timestamp(1521235706, 5), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235706, 5), t: 1 }, lastOpVisible: { ts: Timestamp(1521235706, 5), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235706, 5), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235706, 5) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.098-0400 s20016| 2018-03-16T17:28:26.096-0400 D ASIO [ConfigServerCatalogCacheLoader-0] startCommand: RemoteCommand 312 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:56.096-0400 cmd:{ find: "chunks", filter: { ns: "test.foo", lastmod: { $gte: Timestamp(0, 0) } }, sort: { lastmod: 1 }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235706, 5), t: 1 } }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.098-0400 s20016| 2018-03-16T17:28:26.096-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 312 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.099-0400 s20016| 2018-03-16T17:28:26.096-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 312 finished with response: { cursor: { firstBatch: [ { _id: "test.foo-x_MinKey", ns: "test.foo", min: { x: MinKey }, max: { x: MaxKey }, shard: "stale_mongos_updates_and_removes-rs0", lastmod: Timestamp(1, 0), lastmodEpoch: ObjectId('5aac36f988c1f30185b7b1df'), history: [ { validAfter: Timestamp(1521235705, 141), shard: "stale_mongos_updates_and_removes-rs0" } ] } ], id: 0, ns: "config.chunks" }, ok: 1.0, operationTime: Timestamp(1521235706, 5), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235706, 5), t: 1 }, lastOpVisible: { ts: Timestamp(1521235706, 5), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235706, 5), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235706, 5), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235706, 5) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.099-0400 s20016| 2018-03-16T17:28:26.096-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Refresh for collection test.foo took 1 ms and found version 1|0||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.099-0400 s20016| 2018-03-16T17:28:26.096-0400 D ASIO [conn1] startCommand: RemoteCommand 314 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36fa1085bf286ecf4d36'), x: 0.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f988c1f30185b7b1df') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.100-0400 s20016| 2018-03-16T17:28:26.096-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-9-0] Starting asynchronous command 314 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.105-0400 d20010| 2018-03-16T17:28:26.105-0400 I INDEX [ShardServerCatalogCacheLoader-0] build index on: config.cache.chunks.test.foo properties: { v: 2, key: { lastmod: 1 }, name: "lastmod_1", ns: "config.cache.chunks.test.foo" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.105-0400 d20010| 2018-03-16T17:28:26.105-0400 I INDEX [ShardServerCatalogCacheLoader-0] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.106-0400 s20016| 2018-03-16T17:28:26.105-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-9-0] Request 314 finished with response: { n: 1, opTime: { ts: Timestamp(1521235706, 6), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235706, 6), $gleStats: { lastOpTime: { ts: Timestamp(1521235706, 6), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235706, 6), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235706, 3), $configServerState: { opTime: { ts: Timestamp(1521235706, 5), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.106-0400 d20010| 2018-03-16T17:28:26.105-0400 I INDEX [ShardServerCatalogCacheLoader-0] build index done. scanned 0 total records. 0 secs [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.106-0400 s20016| 2018-03-16T17:28:26.106-0400 D ASIO [conn1] startCommand: RemoteCommand 316 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36fa1085bf286ecf4d37'), x: 0.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f988c1f30185b7b1df') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.106-0400 s20016| 2018-03-16T17:28:26.106-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-10-0] Starting asynchronous command 316 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.107-0400 s20016| 2018-03-16T17:28:26.106-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-10-0] Request 316 finished with response: { n: 1, opTime: { ts: Timestamp(1521235706, 9), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235706, 9), $gleStats: { lastOpTime: { ts: Timestamp(1521235706, 9), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235706, 9), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235706, 3), $configServerState: { opTime: { ts: Timestamp(1521235706, 5), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.107-0400 s20016| 2018-03-16T17:28:26.107-0400 D ASIO [conn1] startCommand: RemoteCommand 318 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36fa1085bf286ecf4d38'), x: 1.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f988c1f30185b7b1df') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.108-0400 s20016| 2018-03-16T17:28:26.107-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-11-0] Starting asynchronous command 318 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.108-0400 s20016| 2018-03-16T17:28:26.107-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-11-0] Request 318 finished with response: { n: 1, opTime: { ts: Timestamp(1521235706, 11), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235706, 11), $gleStats: { lastOpTime: { ts: Timestamp(1521235706, 11), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235706, 11), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235706, 3), $configServerState: { opTime: { ts: Timestamp(1521235706, 5), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.109-0400 s20016| 2018-03-16T17:28:26.108-0400 D ASIO [conn1] startCommand: RemoteCommand 320 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36fa1085bf286ecf4d39'), x: 1.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f988c1f30185b7b1df') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.109-0400 s20016| 2018-03-16T17:28:26.108-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-12-0] Starting asynchronous command 320 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.109-0400 s20016| 2018-03-16T17:28:26.109-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-12-0] Request 320 finished with response: { n: 1, opTime: { ts: Timestamp(1521235706, 12), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235706, 12), $gleStats: { lastOpTime: { ts: Timestamp(1521235706, 12), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235706, 12), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235706, 3), $configServerState: { opTime: { ts: Timestamp(1521235706, 5), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.110-0400 s20016| 2018-03-16T17:28:26.109-0400 D ASIO [conn1] startCommand: RemoteCommand 322 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36fa1085bf286ecf4d3a'), x: 2.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f988c1f30185b7b1df') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.110-0400 s20016| 2018-03-16T17:28:26.110-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-13-0] Starting asynchronous command 322 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.111-0400 s20016| 2018-03-16T17:28:26.110-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-13-0] Request 322 finished with response: { n: 1, opTime: { ts: Timestamp(1521235706, 13), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235706, 13), $gleStats: { lastOpTime: { ts: Timestamp(1521235706, 13), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235706, 13), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235706, 3), $configServerState: { opTime: { ts: Timestamp(1521235706, 5), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.111-0400 s20016| 2018-03-16T17:28:26.111-0400 D ASIO [conn1] startCommand: RemoteCommand 324 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36fa1085bf286ecf4d3b'), x: 2.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f988c1f30185b7b1df') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.111-0400 s20016| 2018-03-16T17:28:26.111-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-14-0] Starting asynchronous command 324 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.112-0400 s20016| 2018-03-16T17:28:26.111-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-14-0] Request 324 finished with response: { n: 1, opTime: { ts: Timestamp(1521235706, 14), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235706, 14), $gleStats: { lastOpTime: { ts: Timestamp(1521235706, 14), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235706, 14), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235706, 3), $configServerState: { opTime: { ts: Timestamp(1521235706, 5), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.112-0400 s20016| 2018-03-16T17:28:26.112-0400 D ASIO [conn1] startCommand: RemoteCommand 326 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36fa1085bf286ecf4d3c'), x: 3.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f988c1f30185b7b1df') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.113-0400 s20016| 2018-03-16T17:28:26.112-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-15-0] Starting asynchronous command 326 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.113-0400 s20016| 2018-03-16T17:28:26.112-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-15-0] Request 326 finished with response: { n: 1, opTime: { ts: Timestamp(1521235706, 15), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235706, 15), $gleStats: { lastOpTime: { ts: Timestamp(1521235706, 15), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235706, 15), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235706, 3), $configServerState: { opTime: { ts: Timestamp(1521235706, 5), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.114-0400 s20016| 2018-03-16T17:28:26.113-0400 D ASIO [conn1] startCommand: RemoteCommand 328 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36fa1085bf286ecf4d3d'), x: 3.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f988c1f30185b7b1df') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.114-0400 s20016| 2018-03-16T17:28:26.113-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-16-0] Starting asynchronous command 328 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.114-0400 s20016| 2018-03-16T17:28:26.114-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-16-0] Request 328 finished with response: { n: 1, opTime: { ts: Timestamp(1521235706, 16), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235706, 16), $gleStats: { lastOpTime: { ts: Timestamp(1521235706, 16), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235706, 16), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235706, 3), $configServerState: { opTime: { ts: Timestamp(1521235706, 5), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.115-0400 s20016| 2018-03-16T17:28:26.115-0400 D ASIO [conn1] startCommand: RemoteCommand 330 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36fa1085bf286ecf4d3e'), x: 4.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f988c1f30185b7b1df') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.115-0400 s20016| 2018-03-16T17:28:26.115-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-17-0] Starting asynchronous command 330 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.116-0400 s20016| 2018-03-16T17:28:26.115-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-17-0] Request 330 finished with response: { n: 1, opTime: { ts: Timestamp(1521235706, 17), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235706, 17), $gleStats: { lastOpTime: { ts: Timestamp(1521235706, 17), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235706, 17), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235706, 3), $configServerState: { opTime: { ts: Timestamp(1521235706, 5), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.116-0400 s20016| 2018-03-16T17:28:26.116-0400 D ASIO [conn1] startCommand: RemoteCommand 332 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36fa1085bf286ecf4d3f'), x: 4.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f988c1f30185b7b1df') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.116-0400 s20016| 2018-03-16T17:28:26.116-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-18-0] Starting asynchronous command 332 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.117-0400 s20016| 2018-03-16T17:28:26.116-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-18-0] Request 332 finished with response: { n: 1, opTime: { ts: Timestamp(1521235706, 18), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235706, 18), $gleStats: { lastOpTime: { ts: Timestamp(1521235706, 18), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235706, 18), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235706, 3), $configServerState: { opTime: { ts: Timestamp(1521235706, 5), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.117-0400 s20016| 2018-03-16T17:28:26.117-0400 D ASIO [conn1] startCommand: RemoteCommand 334 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36fa1085bf286ecf4d40'), x: 5.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f988c1f30185b7b1df') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.118-0400 s20016| 2018-03-16T17:28:26.117-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-19-0] Starting asynchronous command 334 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.118-0400 s20016| 2018-03-16T17:28:26.117-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-19-0] Request 334 finished with response: { n: 1, opTime: { ts: Timestamp(1521235706, 19), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235706, 19), $gleStats: { lastOpTime: { ts: Timestamp(1521235706, 19), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235706, 19), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235706, 3), $configServerState: { opTime: { ts: Timestamp(1521235706, 5), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.119-0400 s20016| 2018-03-16T17:28:26.118-0400 D ASIO [conn1] startCommand: RemoteCommand 336 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36fa1085bf286ecf4d41'), x: 5.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f988c1f30185b7b1df') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.119-0400 s20016| 2018-03-16T17:28:26.118-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-20-0] Connecting to ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.119-0400 s20016| 2018-03-16T17:28:26.119-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-20-0] Starting asynchronous command 337 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.119-0400 d20010| 2018-03-16T17:28:26.119-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56546 #62 (53 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.119-0400 d20010| 2018-03-16T17:28:26.119-0400 I NETWORK [conn62] received client metadata from 127.0.0.1:56546 conn62: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-20", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.120-0400 s20016| 2018-03-16T17:28:26.119-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-20-0] Successfully connected to ubuntu:20010, took 1ms (1 connections now open to ubuntu:20010) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.120-0400 s20016| 2018-03-16T17:28:26.119-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-20-0] Request 337 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.120-0400 s20016| 2018-03-16T17:28:26.119-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-20-0] Starting asynchronous command 336 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.120-0400 s20016| 2018-03-16T17:28:26.119-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-20-0] Request 336 finished with response: { n: 1, opTime: { ts: Timestamp(1521235706, 20), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235706, 20), $gleStats: { lastOpTime: { ts: Timestamp(1521235706, 20), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235706, 20), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235706, 3), $configServerState: { opTime: { ts: Timestamp(1521235706, 5), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.121-0400 s20016| 2018-03-16T17:28:26.120-0400 D ASIO [conn1] startCommand: RemoteCommand 339 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36fa1085bf286ecf4d42'), x: 6.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f988c1f30185b7b1df') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.121-0400 s20016| 2018-03-16T17:28:26.120-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-21-0] Connecting to ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.121-0400 s20016| 2018-03-16T17:28:26.120-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-21-0] Starting asynchronous command 340 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.121-0400 d20010| 2018-03-16T17:28:26.120-0400 I NETWORK [listener] connection accepted from 127.0.0.1:56548 #63 (54 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.121-0400 d20010| 2018-03-16T17:28:26.121-0400 I NETWORK [conn63] received client metadata from 127.0.0.1:56548 conn63: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-21", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.122-0400 s20016| 2018-03-16T17:28:26.121-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-21-0] Successfully connected to ubuntu:20010, took 1ms (1 connections now open to ubuntu:20010) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.122-0400 s20016| 2018-03-16T17:28:26.121-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-21-0] Request 340 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.122-0400 s20016| 2018-03-16T17:28:26.121-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-21-0] Starting asynchronous command 339 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.123-0400 s20016| 2018-03-16T17:28:26.121-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-21-0] Request 339 finished with response: { n: 1, opTime: { ts: Timestamp(1521235706, 21), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235706, 21), $gleStats: { lastOpTime: { ts: Timestamp(1521235706, 21), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235706, 21), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235706, 3), $configServerState: { opTime: { ts: Timestamp(1521235706, 5), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.123-0400 s20016| 2018-03-16T17:28:26.122-0400 D ASIO [conn1] startCommand: RemoteCommand 342 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36fa1085bf286ecf4d43'), x: 6.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f988c1f30185b7b1df') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.123-0400 s20016| 2018-03-16T17:28:26.122-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-22-0] Starting asynchronous command 342 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.124-0400 s20016| 2018-03-16T17:28:26.122-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-22-0] Request 342 finished with response: { n: 1, opTime: { ts: Timestamp(1521235706, 22), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235706, 22), $gleStats: { lastOpTime: { ts: Timestamp(1521235706, 22), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235706, 22), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235706, 3), $configServerState: { opTime: { ts: Timestamp(1521235706, 5), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.124-0400 s20016| 2018-03-16T17:28:26.123-0400 D ASIO [conn1] startCommand: RemoteCommand 344 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36fa1085bf286ecf4d44'), x: 7.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f988c1f30185b7b1df') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.124-0400 s20016| 2018-03-16T17:28:26.123-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-23-0] Starting asynchronous command 344 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.125-0400 s20016| 2018-03-16T17:28:26.124-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-23-0] Request 344 finished with response: { n: 1, opTime: { ts: Timestamp(1521235706, 23), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235706, 23), $gleStats: { lastOpTime: { ts: Timestamp(1521235706, 23), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235706, 23), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235706, 3), $configServerState: { opTime: { ts: Timestamp(1521235706, 5), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.127-0400 s20016| 2018-03-16T17:28:26.126-0400 D ASIO [conn1] startCommand: RemoteCommand 346 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36fa1085bf286ecf4d45'), x: 7.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f988c1f30185b7b1df') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.127-0400 s20016| 2018-03-16T17:28:26.126-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-24-0] Starting asynchronous command 346 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.127-0400 s20016| 2018-03-16T17:28:26.127-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-24-0] Request 346 finished with response: { n: 1, opTime: { ts: Timestamp(1521235706, 24), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235706, 24), $gleStats: { lastOpTime: { ts: Timestamp(1521235706, 24), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235706, 24), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235706, 3), $configServerState: { opTime: { ts: Timestamp(1521235706, 5), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.128-0400 s20016| 2018-03-16T17:28:26.128-0400 D ASIO [conn1] startCommand: RemoteCommand 348 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36fa1085bf286ecf4d46'), x: 8.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f988c1f30185b7b1df') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.128-0400 s20016| 2018-03-16T17:28:26.128-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-25-0] Starting asynchronous command 348 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.129-0400 s20016| 2018-03-16T17:28:26.128-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-25-0] Request 348 finished with response: { n: 1, opTime: { ts: Timestamp(1521235706, 25), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235706, 25), $gleStats: { lastOpTime: { ts: Timestamp(1521235706, 25), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235706, 25), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235706, 3), $configServerState: { opTime: { ts: Timestamp(1521235706, 5), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.129-0400 s20016| 2018-03-16T17:28:26.129-0400 D ASIO [conn1] startCommand: RemoteCommand 350 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36fa1085bf286ecf4d47'), x: 8.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f988c1f30185b7b1df') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.129-0400 s20016| 2018-03-16T17:28:26.129-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-26-0] Starting asynchronous command 350 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.130-0400 s20016| 2018-03-16T17:28:26.129-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-26-0] Request 350 finished with response: { n: 1, opTime: { ts: Timestamp(1521235706, 26), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235706, 26), $gleStats: { lastOpTime: { ts: Timestamp(1521235706, 26), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235706, 26), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235706, 3), $configServerState: { opTime: { ts: Timestamp(1521235706, 5), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.130-0400 s20016| 2018-03-16T17:28:26.130-0400 D ASIO [conn1] startCommand: RemoteCommand 352 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36fa1085bf286ecf4d48'), x: 9.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f988c1f30185b7b1df') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.131-0400 s20016| 2018-03-16T17:28:26.130-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-27-0] Starting asynchronous command 352 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.131-0400 s20016| 2018-03-16T17:28:26.130-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-27-0] Request 352 finished with response: { n: 1, opTime: { ts: Timestamp(1521235706, 27), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235706, 27), $gleStats: { lastOpTime: { ts: Timestamp(1521235706, 27), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235706, 27), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235706, 3), $configServerState: { opTime: { ts: Timestamp(1521235706, 5), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.132-0400 s20016| 2018-03-16T17:28:26.131-0400 D ASIO [conn1] startCommand: RemoteCommand 354 -- target:ubuntu:20010 db:test cmd:{ insert: "foo", bypassDocumentValidation: false, ordered: true, documents: [ { _id: ObjectId('5aac36fa1085bf286ecf4d49'), x: 9.0, fieldToUpdate: 0.0 } ], shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f988c1f30185b7b1df') ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.132-0400 s20016| 2018-03-16T17:28:26.131-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-28-0] Starting asynchronous command 354 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.132-0400 s20016| 2018-03-16T17:28:26.132-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-28-0] Request 354 finished with response: { n: 1, opTime: { ts: Timestamp(1521235706, 28), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235706, 28), $gleStats: { lastOpTime: { ts: Timestamp(1521235706, 28), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235706, 28), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235706, 3), $configServerState: { opTime: { ts: Timestamp(1521235706, 5), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.134-0400 Waiting for op with OpTime { "ts" : Timestamp(1521235706, 5), "t" : NumberLong(1) } to be committed on all secondaries [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.138-0400 s20016| 2018-03-16T17:28:26.137-0400 D ASIO [conn1] startCommand: RemoteCommand 356 -- target:ubuntu:20012 db:config cmd:{ find: "chunks", filter: { min: { x: MinKey }, max: { x: MaxKey } }, limit: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.138-0400 s20016| 2018-03-16T17:28:26.137-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-29-0] Connecting to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.138-0400 s20016| 2018-03-16T17:28:26.138-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-29-0] Starting asynchronous command 357 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.138-0400 c20012| 2018-03-16T17:28:26.138-0400 I NETWORK [listener] connection accepted from 127.0.0.1:60894 #45 (28 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.138-0400 c20012| 2018-03-16T17:28:26.138-0400 I NETWORK [conn45] received client metadata from 127.0.0.1:60894 conn45: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-29", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.139-0400 s20016| 2018-03-16T17:28:26.138-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-29-0] Successfully connected to ubuntu:20012, took 1ms (1 connections now open to ubuntu:20012) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.139-0400 s20016| 2018-03-16T17:28:26.138-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-29-0] Request 357 finished with response: {} [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.139-0400 s20016| 2018-03-16T17:28:26.138-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-29-0] Starting asynchronous command 356 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.140-0400 s20016| 2018-03-16T17:28:26.138-0400 D ASIO [NetworkInterfaceASIO-TaskExecutorPool-29-0] Request 356 finished with response: { cursor: { firstBatch: [ { _id: "test.foo-x_MinKey", ns: "test.foo", min: { x: MinKey }, max: { x: MaxKey }, shard: "stale_mongos_updates_and_removes-rs0", lastmod: Timestamp(1, 0), lastmodEpoch: ObjectId('5aac36f988c1f30185b7b1df'), history: [ { validAfter: Timestamp(1521235705, 141), shard: "stale_mongos_updates_and_removes-rs0" } ] } ], id: 0, ns: "config.chunks" }, ok: 1.0, operationTime: Timestamp(1521235706, 5), $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235706, 28), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235706, 5) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.140-0400 s20016| 2018-03-16T17:28:26.139-0400 I SHARDING [conn1] Refreshing chunks for collection test.foo based on version 1|0||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.140-0400 s20016| 2018-03-16T17:28:26.139-0400 D ASIO [ConfigServerCatalogCacheLoader-0] startCommand: RemoteCommand 359 -- target:ubuntu:20013 db:config expDate:2018-03-16T17:28:56.139-0400 cmd:{ find: "collections", filter: { _id: "test.foo" }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235706, 5), t: 1 } }, limit: 1, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.140-0400 s20016| 2018-03-16T17:28:26.139-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 359 on host ubuntu:20013 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.141-0400 s20016| 2018-03-16T17:28:26.139-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 359 finished with response: { cursor: { firstBatch: [ { _id: "test.foo", lastmodEpoch: ObjectId('5aac36f988c1f30185b7b1df'), lastmod: new Date(4294967296), dropped: false, key: { x: 1.0 }, unique: false, uuid: UUID("53837380-bed7-429b-ba96-f32fe994b394") } ], id: 0, ns: "config.collections" }, ok: 1.0, operationTime: Timestamp(1521235706, 5), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235706, 5), t: 1 }, lastOpVisible: { ts: Timestamp(1521235706, 5), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235706, 28), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235706, 5) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.141-0400 s20016| 2018-03-16T17:28:26.139-0400 D ASIO [ConfigServerCatalogCacheLoader-0] startCommand: RemoteCommand 361 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:56.139-0400 cmd:{ find: "chunks", filter: { ns: "test.foo", lastmod: { $gte: Timestamp(1, 0) } }, sort: { lastmod: 1 }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235706, 5), t: 1 } }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.142-0400 s20016| 2018-03-16T17:28:26.140-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 361 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.142-0400 s20016| 2018-03-16T17:28:26.140-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 361 finished with response: { cursor: { firstBatch: [ { _id: "test.foo-x_MinKey", ns: "test.foo", min: { x: MinKey }, max: { x: MaxKey }, shard: "stale_mongos_updates_and_removes-rs0", lastmod: Timestamp(1, 0), lastmodEpoch: ObjectId('5aac36f988c1f30185b7b1df'), history: [ { validAfter: Timestamp(1521235705, 141), shard: "stale_mongos_updates_and_removes-rs0" } ] } ], id: 0, ns: "config.chunks" }, ok: 1.0, operationTime: Timestamp(1521235706, 5), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235706, 5), t: 1 }, lastOpVisible: { ts: Timestamp(1521235706, 5), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235706, 5), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235706, 28), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235706, 5) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.143-0400 s20016| 2018-03-16T17:28:26.140-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Refresh for collection test.foo took 0 ms and found version 1|0||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.143-0400 s20016| 2018-03-16T17:28:26.140-0400 I COMMAND [conn1] Splitting chunk [{ x: MinKey }, { x: MaxKey }) in collection test.foo on shard stale_mongos_updates_and_removes-rs0 at key { x: 5.0 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.143-0400 s20016| 2018-03-16T17:28:26.140-0400 D ASIO [conn1] startCommand: RemoteCommand 363 -- target:ubuntu:20010 db:admin cmd:{ splitChunk: "test.foo", from: "stale_mongos_updates_and_removes-rs0", keyPattern: { x: 1.0 }, epoch: ObjectId('5aac36f988c1f30185b7b1df'), shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f988c1f30185b7b1df') ], min: { x: MinKey }, max: { x: MaxKey }, splitKeys: [ { x: 5.0 } ] } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.143-0400 s20016| 2018-03-16T17:28:26.140-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 363 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.144-0400 d20010| 2018-03-16T17:28:26.140-0400 I SHARDING [conn21] received splitChunk request: { splitChunk: "test.foo", from: "stale_mongos_updates_and_removes-rs0", keyPattern: { x: 1.0 }, epoch: ObjectId('5aac36f988c1f30185b7b1df'), shardVersion: [ Timestamp(1, 0), ObjectId('5aac36f988c1f30185b7b1df') ], min: { x: MinKey }, max: { x: MaxKey }, splitKeys: [ { x: 5.0 } ], $clusterTime: { clusterTime: Timestamp(1521235706, 28), signature: { hash: BinData(0, 793B00E09A3C5E443A29C280C281ADBBCD7368CA), keyId: 6533657568122765331 } }, $client: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" }, mongos: { host: "ubuntu:20016", client: "127.0.0.1:50986", version: "3.6.3" } }, $configServerState: { opTime: { ts: Timestamp(1521235706, 5), t: 1 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.145-0400 d20010| 2018-03-16T17:28:26.145-0400 I SHARDING [conn21] distributed lock 'test.foo' acquired for 'splitting chunk [{ x: MinKey }, { x: MaxKey }) in test.foo', ts : 5aac36faf7e5a0b40b8e358f [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.146-0400 c20012| 2018-03-16T17:28:26.146-0400 I SHARDING [conn30] about to log metadata event into changelog: { _id: "ubuntu-2018-03-16T17:28:26.146-0400-5aac36fa88c1f30185b7b225", server: "ubuntu", clientAddr: "127.0.0.1:60730", time: new Date(1521235706146), what: "split", ns: "test.foo", details: { before: { min: { x: MinKey }, max: { x: MaxKey }, lastmod: Timestamp(1, 0), lastmodEpoch: ObjectId('5aac36f988c1f30185b7b1df') }, left: { min: { x: MinKey }, max: { x: 5.0 }, lastmod: Timestamp(1, 1), lastmodEpoch: ObjectId('5aac36f988c1f30185b7b1df') }, right: { min: { x: 5.0 }, max: { x: MaxKey }, lastmod: Timestamp(1, 2), lastmodEpoch: ObjectId('5aac36f988c1f30185b7b1df') } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.157-0400 d20010| 2018-03-16T17:28:26.157-0400 I SHARDING [conn21] distributed lock with ts: 5aac36faf7e5a0b40b8e358f' unlocked. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.157-0400 s20016| 2018-03-16T17:28:26.157-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 363 finished with response: { ok: 1.0, operationTime: Timestamp(1521235706, 28), $gleStats: { lastOpTime: { ts: Timestamp(1521235705, 123), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235706, 32), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235706, 3), $configServerState: { opTime: { ts: Timestamp(1521235706, 32), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.158-0400 s20016| 2018-03-16T17:28:26.157-0400 I SHARDING [conn1] Refreshing chunks for collection test.foo based on version 1|0||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.158-0400 s20016| 2018-03-16T17:28:26.157-0400 D ASIO [ConfigServerCatalogCacheLoader-0] startCommand: RemoteCommand 365 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:56.157-0400 cmd:{ find: "collections", filter: { _id: "test.foo" }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235706, 32), t: 1 } }, limit: 1, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.158-0400 s20016| 2018-03-16T17:28:26.158-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 365 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.159-0400 s20016| 2018-03-16T17:28:26.158-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 365 finished with response: { cursor: { firstBatch: [ { _id: "test.foo", lastmodEpoch: ObjectId('5aac36f988c1f30185b7b1df'), lastmod: new Date(4294967296), dropped: false, key: { x: 1.0 }, unique: false, uuid: UUID("53837380-bed7-429b-ba96-f32fe994b394") } ], id: 0, ns: "config.collections" }, ok: 1.0, operationTime: Timestamp(1521235706, 32), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235706, 32), t: 1 }, lastOpVisible: { ts: Timestamp(1521235706, 32), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235706, 5), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235706, 32), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235706, 32) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.159-0400 s20016| 2018-03-16T17:28:26.158-0400 D ASIO [ConfigServerCatalogCacheLoader-0] startCommand: RemoteCommand 367 -- target:ubuntu:20014 db:config expDate:2018-03-16T17:28:56.158-0400 cmd:{ find: "chunks", filter: { ns: "test.foo", lastmod: { $gte: Timestamp(1, 0) } }, sort: { lastmod: 1 }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235706, 32), t: 1 } }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.159-0400 s20016| 2018-03-16T17:28:26.158-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 367 on host ubuntu:20014 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.160-0400 s20016| 2018-03-16T17:28:26.158-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 367 finished with response: { cursor: { firstBatch: [ { _id: "test.foo-x_MinKey", lastmod: Timestamp(1, 1), lastmodEpoch: ObjectId('5aac36f988c1f30185b7b1df'), ns: "test.foo", min: { x: MinKey }, max: { x: 5.0 }, shard: "stale_mongos_updates_and_removes-rs0", history: [ { validAfter: Timestamp(1521235705, 141), shard: "stale_mongos_updates_and_removes-rs0" } ] }, { _id: "test.foo-x_5.0", lastmod: Timestamp(1, 2), lastmodEpoch: ObjectId('5aac36f988c1f30185b7b1df'), ns: "test.foo", min: { x: 5.0 }, max: { x: MaxKey }, shard: "stale_mongos_updates_and_removes-rs0", history: [ { validAfter: Timestamp(1521235705, 141), shard: "stale_mongos_updates_and_removes-rs0" } ] } ], id: 0, ns: "config.chunks" }, ok: 1.0, operationTime: Timestamp(1521235706, 32), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235706, 32), t: 1 }, lastOpVisible: { ts: Timestamp(1521235706, 32), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235706, 32), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235706, 32) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.161-0400 s20016| 2018-03-16T17:28:26.158-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Refresh for collection test.foo took 0 ms and found version 1|2||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.161-0400 s20016| 2018-03-16T17:28:26.158-0400 D ASIO [conn1] startCommand: RemoteCommand 369 -- target:ubuntu:20012 db:admin cmd:{ _configsvrMoveChunk: 1, _id: "test.foo-x_MinKey", ns: "test.foo", min: { x: MinKey }, max: { x: 5.0 }, shard: "stale_mongos_updates_and_removes-rs0", lastmod: Timestamp(1, 2), lastmodEpoch: ObjectId('5aac36f988c1f30185b7b1df'), toShard: "stale_mongos_updates_and_removes-rs1", maxChunkSizeBytes: 67108864, secondaryThrottle: {}, waitForDelete: true, writeConcern: { w: "majority", wtimeout: 15000 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.161-0400 s20016| 2018-03-16T17:28:26.158-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 369 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.161-0400 c20012| 2018-03-16T17:28:26.160-0400 I SHARDING [conn23] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.162-0400 c20012| 2018-03-16T17:28:26.160-0400 I SHARDING [conn23] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.162-0400 c20012| 2018-03-16T17:28:26.160-0400 I SHARDING [conn23] xxx routing info needs refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.162-0400 c20012| 2018-03-16T17:28:26.160-0400 I SHARDING [conn23] xxx calling schedule collection refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.162-0400 c20012| 2018-03-16T17:28:26.160-0400 I SHARDING [conn23] xxx is incremental [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.162-0400 c20012| 2018-03-16T17:28:26.160-0400 I SHARDING [conn23] xxx existing routing info? [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.162-0400 c20012| 2018-03-16T17:28:26.160-0400 I SHARDING [conn23] 2|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.162-0400 c20012| 2018-03-16T17:28:26.160-0400 I SHARDING [conn23] Refreshing chunks for collection test.foo based on version 2|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.162-0400 c20012| 2018-03-16T17:28:26.160-0400 I SHARDING [conn23] xxx getting chunks since now [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.162-0400 c20012| 2018-03-16T17:28:26.160-0400 I SHARDING [conn23] xxx waiting if !refreshNot [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.163-0400 c20012| 2018-03-16T17:28:26.161-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh callback [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.163-0400 c20012| 2018-03-16T17:28:26.161-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh completed [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.163-0400 c20012| 2018-03-16T17:28:26.161-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Refresh for collection test.foo took 0 ms and found version 1|2||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.163-0400 c20012| 2018-03-16T17:28:26.161-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx marking as status ok [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.163-0400 c20012| 2018-03-16T17:28:26.161-0400 I SHARDING [conn23] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.163-0400 c20012| 2018-03-16T17:28:26.161-0400 I SHARDING [conn23] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.171-0400 c20012| 2018-03-16T17:28:26.171-0400 I SHARDING [conn23] distributed lock 'test.foo' acquired for 'Migrating chunk(s) in collection test.foo', ts : 5aac36ec88c1f30185b7ac69 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.172-0400 d20010| 2018-03-16T17:28:26.172-0400 I SHARDING [conn15] Starting chunk migration ns: test.foo, [{ x: MinKey }, { x: 5.0 }), fromShard: stale_mongos_updates_and_removes-rs0, toShard: stale_mongos_updates_and_removes-rs1 with expected collection version epoch 5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.172-0400 d20010| 2018-03-16T17:28:26.172-0400 I SHARDING [conn15] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.172-0400 d20010| 2018-03-16T17:28:26.172-0400 I SHARDING [conn15] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.172-0400 d20010| 2018-03-16T17:28:26.172-0400 I SHARDING [conn15] xxx routing info needs refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.172-0400 d20010| 2018-03-16T17:28:26.172-0400 I SHARDING [conn15] xxx calling schedule collection refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.172-0400 d20010| 2018-03-16T17:28:26.172-0400 I SHARDING [conn15] xxx is incremental [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.173-0400 d20010| 2018-03-16T17:28:26.172-0400 I SHARDING [conn15] xxx existing routing info? [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.173-0400 d20010| 2018-03-16T17:28:26.172-0400 I SHARDING [conn15] 1|0||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.173-0400 d20010| 2018-03-16T17:28:26.172-0400 I SHARDING [conn15] Refreshing chunks for collection test.foo based on version 1|0||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.173-0400 d20010| 2018-03-16T17:28:26.172-0400 I SHARDING [conn15] xxx getting chunks since now [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.173-0400 d20010| 2018-03-16T17:28:26.172-0400 I SHARDING [conn15] xxx getting chunks since [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.173-0400 d20010| 2018-03-16T17:28:26.172-0400 I SHARDING [conn15] xxx waiting if !refreshNot [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.173-0400 d20010| 2018-03-16T17:28:26.172-0400 I SHARDING [ShardServerCatalogCacheLoader-0] xxx primary and calling get chunks since for primary [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.173-0400 d20010| 2018-03-16T17:28:26.172-0400 I SHARDING [ShardServerCatalogCacheLoader-0] version is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.174-0400 d20010| 2018-03-16T17:28:26.172-0400 I SHARDING [ShardServerCatalogCacheLoader-0] 1|0||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.174-0400 d20010| 2018-03-16T17:28:26.173-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] refresh in primary [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.174-0400 d20010| 2018-03-16T17:28:26.173-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] 2 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.174-0400 d20010| 2018-03-16T17:28:26.173-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx collec and changed chunks max epoch or version doesn't equal max [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.199-0400 d20010| 2018-03-16T17:28:26.199-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Cache loader remotely refreshed for collection test.foo from collection version 1|0||5aac36f988c1f30185b7b1df and found collection version 1|2||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.200-0400 d20010| 2018-03-16T17:28:26.199-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Cache loader found enqueued metadata from 1|1||5aac36f988c1f30185b7b1df to 1|2||5aac36f988c1f30185b7b1df and persisted metadata from 1|0||5aac36f988c1f30185b7b1df to 1|0||5aac36f988c1f30185b7b1df, GTE cache version 1|0||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.200-0400 d20010| 2018-03-16T17:28:26.199-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx is okay [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.200-0400 d20010| 2018-03-16T17:28:26.199-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx finsihing refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.200-0400 d20010| 2018-03-16T17:28:26.199-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh callback [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.200-0400 d20010| 2018-03-16T17:28:26.199-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh completed [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.200-0400 d20010| 2018-03-16T17:28:26.199-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Refresh for collection test.foo took 27 ms and found version 1|2||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.201-0400 d20010| 2018-03-16T17:28:26.199-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx marking as status ok [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.201-0400 d20010| 2018-03-16T17:28:26.200-0400 I SHARDING [conn15] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.201-0400 d20010| 2018-03-16T17:28:26.200-0400 I SHARDING [conn15] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.201-0400 d20010| 2018-03-16T17:28:26.200-0400 I SHARDING [conn15] Updating collection metadata for test.foo from collection version: 1|0||5aac36f988c1f30185b7b1df, shard version: 1|0||5aac36f988c1f30185b7b1df to collection version: 1|2||5aac36f988c1f30185b7b1df, shard version: 1|2||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.201-0400 d20010| 2018-03-16T17:28:26.200-0400 I SHARDING [conn15] about to log metadata event into changelog: { _id: "ubuntu-2018-03-16T17:28:26.200-0400-5aac36faf7e5a0b40b8e359b", server: "ubuntu", clientAddr: "127.0.0.1:56424", time: new Date(1521235706200), what: "moveChunk.start", ns: "test.foo", details: { min: { x: MinKey }, max: { x: 5.0 }, from: "stale_mongos_updates_and_removes-rs0", to: "stale_mongos_updates_and_removes-rs1" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.205-0400 d20011| 2018-03-16T17:28:26.204-0400 I SHARDING [conn22] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.205-0400 d20011| 2018-03-16T17:28:26.205-0400 I SHARDING [conn22] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.205-0400 d20011| 2018-03-16T17:28:26.205-0400 I SHARDING [conn22] xxx routing info needs refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.205-0400 d20011| 2018-03-16T17:28:26.205-0400 I SHARDING [conn22] xxx calling schedule collection refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.205-0400 d20011| 2018-03-16T17:28:26.205-0400 I SHARDING [conn22] xxx is not incr [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.205-0400 d20011| 2018-03-16T17:28:26.205-0400 I SHARDING [conn22] xxx existing routing info? [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.205-0400 d20011| 2018-03-16T17:28:26.205-0400 I SHARDING [conn22] Refreshing chunks for collection test.foo based on version 0|0||000000000000000000000000 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.206-0400 d20011| 2018-03-16T17:28:26.205-0400 I SHARDING [conn22] xxx getting chunks since now [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.206-0400 d20011| 2018-03-16T17:28:26.205-0400 I SHARDING [conn22] xxx getting chunks since [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.206-0400 d20011| 2018-03-16T17:28:26.205-0400 I SHARDING [conn22] xxx waiting if !refreshNot [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.206-0400 d20011| 2018-03-16T17:28:26.205-0400 I SHARDING [ShardServerCatalogCacheLoader-0] xxx primary and calling get chunks since for primary [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.206-0400 d20011| 2018-03-16T17:28:26.205-0400 I SHARDING [ShardServerCatalogCacheLoader-0] version is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.206-0400 d20011| 2018-03-16T17:28:26.205-0400 I SHARDING [ShardServerCatalogCacheLoader-0] 0|0||000000000000000000000000 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.206-0400 d20011| 2018-03-16T17:28:26.205-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] refresh in primary [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.206-0400 d20011| 2018-03-16T17:28:26.205-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] 2 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.207-0400 d20011| 2018-03-16T17:28:26.205-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx collec and changed chunks max epoch or version doesn't equal max [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.306-0400 d20011| 2018-03-16T17:28:26.305-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Cache loader remotely refreshed for collection test.foo from collection version 0|0||000000000000000000000000 and found collection version 1|2||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.306-0400 d20011| 2018-03-16T17:28:26.305-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Cache loader found enqueued metadata from 1|1||5aac36f988c1f30185b7b1df to 1|2||5aac36f988c1f30185b7b1df and no persisted metadata, GTE cache version 0|0||000000000000000000000000 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.306-0400 d20011| 2018-03-16T17:28:26.305-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx is okay [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.306-0400 d20011| 2018-03-16T17:28:26.305-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx finsihing refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.306-0400 d20011| 2018-03-16T17:28:26.305-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh callback [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.306-0400 d20011| 2018-03-16T17:28:26.305-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh completed [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.307-0400 d20011| 2018-03-16T17:28:26.305-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Refresh for collection test.foo took 100 ms and found version 1|2||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.307-0400 d20011| 2018-03-16T17:28:26.305-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx marking as status ok [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.307-0400 d20011| 2018-03-16T17:28:26.305-0400 I SHARDING [conn22] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.307-0400 d20011| 2018-03-16T17:28:26.306-0400 I SHARDING [conn22] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.307-0400 d20011| 2018-03-16T17:28:26.306-0400 I SHARDING [conn22] Marking collection test.foo as sharded with collection version: 1|2||5aac36f988c1f30185b7b1df, shard version: 0|0||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.307-0400 d20011| 2018-03-16T17:28:26.306-0400 I STORAGE [ShardServerCatalogCacheLoader-0] createCollection: config.cache.chunks.test.foo with generated UUID: 28822cba-faa2-4e9a-be1d-529e7970297f [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.308-0400 d20011| 2018-03-16T17:28:26.306-0400 I COMMAND [conn22] command admin.$cmd appName: "MongoDB Shell" command: _recvChunkStart { _recvChunkStart: "test.foo", sessionId: "stale_mongos_updates_and_removes-rs0_stale_mongos_updates_and_removes-rs1_5aac36faf7e5a0b40b8e35a0", from: "stale_mongos_updates_and_removes-rs0/ubuntu:20010", fromShardName: "stale_mongos_updates_and_removes-rs0", toShardName: "stale_mongos_updates_and_removes-rs1", min: { x: MinKey }, max: { x: 5.0 }, shardKeyPattern: { x: 1.0 }, $clusterTime: { clusterTime: Timestamp(1521235706, 41), signature: { hash: BinData(0, 793B00E09A3C5E443A29C280C281ADBBCD7368CA), keyId: 6533657568122765331 } }, $configServerState: { opTime: { ts: Timestamp(1521235706, 38), t: 1 } }, $db: "admin" } numYields:0 reslen:327 locks:{ Global: { acquireCount: { r: 3, w: 1 } }, Database: { acquireCount: { r: 1, w: 1 } }, Collection: { acquireCount: { r: 1, W: 1 } } } protocol:op_msg 101ms [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.308-0400 d20011| 2018-03-16T17:28:26.306-0400 I SHARDING [migrateThread] Starting receiving end of migration of chunk { x: MinKey } -> { x: 5.0 } for collection test.foo from stale_mongos_updates_and_removes-rs0/ubuntu:20010 at epoch 5aac36f988c1f30185b7b1df with session id stale_mongos_updates_and_removes-rs0_stale_mongos_updates_and_removes-rs1_5aac36faf7e5a0b40b8e35a0 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.309-0400 d20011| 2018-03-16T17:28:26.306-0400 I STORAGE [migrateThread] createCollection: test.foo with provided UUID: 53837380-bed7-429b-ba96-f32fe994b394 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.344-0400 d20011| 2018-03-16T17:28:26.343-0400 I INDEX [ShardServerCatalogCacheLoader-0] build index on: config.cache.chunks.test.foo properties: { v: 2, key: { lastmod: 1 }, name: "lastmod_1", ns: "config.cache.chunks.test.foo" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.344-0400 d20011| 2018-03-16T17:28:26.343-0400 I INDEX [ShardServerCatalogCacheLoader-0] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.350-0400 d20011| 2018-03-16T17:28:26.350-0400 I INDEX [migrateThread] build index on: test.foo properties: { v: 2, key: { x: 1.0 }, name: "x_1", ns: "test.foo" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.350-0400 d20011| 2018-03-16T17:28:26.350-0400 I INDEX [migrateThread] building index using bulk method; build may temporarily use up to 500 megabytes of RAM [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.351-0400 d20011| 2018-03-16T17:28:26.350-0400 I SHARDING [migrateThread] Scheduling deletion of any documents in test.foo range [{ x: MinKey }, { x: 5.0 }) before migrating in a chunk covering the range [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.351-0400 d20011| 2018-03-16T17:28:26.350-0400 I SHARDING [Collection Range Deleter] No documents remain to delete in test.foo range [{ x: MinKey }, { x: 5.0 }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.351-0400 d20011| 2018-03-16T17:28:26.350-0400 I SHARDING [Collection Range Deleter] Waiting for majority replication of local deletions in test.foo range [{ x: MinKey }, { x: 5.0 }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.351-0400 d20011| 2018-03-16T17:28:26.350-0400 I INDEX [ShardServerCatalogCacheLoader-0] build index done. scanned 0 total records. 0 secs [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.353-0400 d20011| 2018-03-16T17:28:26.353-0400 I SHARDING [Collection Range Deleter] Finished deleting documents in test.foo range [{ x: MinKey }, { x: 5.0 }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.353-0400 d20011| 2018-03-16T17:28:26.353-0400 I SHARDING [migrateThread] Finished deleting test.foo range [{ x: MinKey }, { x: 5.0 }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.354-0400 d20011| 2018-03-16T17:28:26.354-0400 I SHARDING [migrateThread] Waiting for replication to catch up before entering critical section [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.357-0400 d20011| 2018-03-16T17:28:26.357-0400 I SHARDING [migrateThread] migrate commit succeeded flushing to secondaries for 'test.foo' { x: MinKey } -> { x: 5.0 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.358-0400 d20010| 2018-03-16T17:28:26.357-0400 I SHARDING [conn15] moveChunk data transfer progress: { waited: true, active: true, sessionId: "stale_mongos_updates_and_removes-rs0_stale_mongos_updates_and_removes-rs1_5aac36faf7e5a0b40b8e35a0", ns: "test.foo", from: "stale_mongos_updates_and_removes-rs0/ubuntu:20010", min: { x: MinKey }, max: { x: 5.0 }, shardKeyPattern: { x: 1.0 }, state: "steady", counts: { cloned: 10, clonedBytes: 560, catchup: 0, steady: 0 }, ok: 1.0, operationTime: Timestamp(1521235706, 61), $gleStats: { lastOpTime: { ts: Timestamp(1521235706, 44), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235706, 61), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235706, 61), $configServerState: { opTime: { ts: Timestamp(1521235706, 38), t: 1 } } } mem used: 0 documents remaining to clone: 0 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.359-0400 d20010| 2018-03-16T17:28:26.359-0400 I SHARDING [conn15] Migration successfully entered critical section [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.368-0400 d20011| 2018-03-16T17:28:26.367-0400 I SHARDING [migrateThread] migrate commit succeeded flushing to secondaries for 'test.foo' { x: MinKey } -> { x: 5.0 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.538-0400 c20013| 2018-03-16T17:28:26.538-0400 D REPL_HB [replexec-1] Sending heartbeat (requestId: 1168) to ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.539-0400 c20012| 2018-03-16T17:28:26.538-0400 D REPL_HB [conn9] Received heartbeat request from ubuntu:20013, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235706, 41), signature: { hash: BinData(0, 793B00E09A3C5E443A29C280C281ADBBCD7368CA), keyId: 6533657568122765331 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.539-0400 c20012| 2018-03-16T17:28:26.538-0400 D REPL_HB [conn9] Processed heartbeat from ubuntu:20013 and generated response, { ok: 1.0, electionTime: new Date(6533657559532830722), state: 1, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235706, 38), t: 1 }, opTime: { ts: Timestamp(1521235706, 38), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.540-0400 c20013| 2018-03-16T17:28:26.538-0400 D REPL_HB [replexec-3] Received response to heartbeat (requestId: 1168) from ubuntu:20012, { ok: 1.0, electionTime: new Date(6533657559532830722), state: 1, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235706, 38), t: 1 }, opTime: { ts: Timestamp(1521235706, 38), t: 1 }, operationTime: Timestamp(1521235706, 38), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235706, 38), t: 1 }, lastOpVisible: { ts: Timestamp(1521235706, 38), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235706, 41), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235706, 38) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.540-0400 c20013| 2018-03-16T17:28:26.539-0400 D REPL_HB [replexec-3] Scheduling heartbeat to ubuntu:20012 at 2018-03-16T21:28:28.539Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.540-0400 c20013| 2018-03-16T17:28:26.539-0400 D REPL_HB [replexec-0] Sending heartbeat (requestId: 1170) to ubuntu:20014, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.541-0400 c20014| 2018-03-16T17:28:26.539-0400 D REPL_HB [conn10] Received heartbeat request from ubuntu:20013, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235706, 41), signature: { hash: BinData(0, 793B00E09A3C5E443A29C280C281ADBBCD7368CA), keyId: 6533657568122765331 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.541-0400 c20014| 2018-03-16T17:28:26.539-0400 D REPL_HB [conn10] Processed heartbeat from ubuntu:20013 and generated response, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", syncingTo: "ubuntu:20012", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235706, 38), t: 1 }, opTime: { ts: Timestamp(1521235706, 38), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.542-0400 c20013| 2018-03-16T17:28:26.539-0400 D REPL_HB [replexec-1] Received response to heartbeat (requestId: 1170) from ubuntu:20014, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", syncingTo: "ubuntu:20012", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235706, 38), t: 1 }, opTime: { ts: Timestamp(1521235706, 38), t: 1 }, operationTime: Timestamp(1521235706, 38), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235706, 38), t: 1 }, lastOpVisible: { ts: Timestamp(1521235706, 38), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235706, 41), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235706, 38) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.542-0400 c20013| 2018-03-16T17:28:26.539-0400 D REPL_HB [replexec-1] Scheduling heartbeat to ubuntu:20014 at 2018-03-16T21:28:28.539Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.582-0400 c20014| 2018-03-16T17:28:26.582-0400 D REPL_HB [replexec-3] Sending heartbeat (requestId: 1159) to ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.582-0400 c20012| 2018-03-16T17:28:26.582-0400 D REPL_HB [conn8] Received heartbeat request from ubuntu:20014, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235706, 41), signature: { hash: BinData(0, 793B00E09A3C5E443A29C280C281ADBBCD7368CA), keyId: 6533657568122765331 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.583-0400 c20012| 2018-03-16T17:28:26.582-0400 D REPL_HB [conn8] Processed heartbeat from ubuntu:20014 and generated response, { ok: 1.0, electionTime: new Date(6533657559532830722), state: 1, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235706, 38), t: 1 }, opTime: { ts: Timestamp(1521235706, 38), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.584-0400 c20014| 2018-03-16T17:28:26.582-0400 D REPL_HB [replexec-1] Received response to heartbeat (requestId: 1159) from ubuntu:20012, { ok: 1.0, electionTime: new Date(6533657559532830722), state: 1, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235706, 38), t: 1 }, opTime: { ts: Timestamp(1521235706, 38), t: 1 }, operationTime: Timestamp(1521235706, 38), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235706, 38), t: 1 }, lastOpVisible: { ts: Timestamp(1521235706, 38), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235706, 41), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235706, 38) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.584-0400 c20014| 2018-03-16T17:28:26.582-0400 D REPL_HB [replexec-1] Scheduling heartbeat to ubuntu:20012 at 2018-03-16T21:28:28.582Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.584-0400 c20014| 2018-03-16T17:28:26.582-0400 D REPL_HB [replexec-4] Sending heartbeat (requestId: 1161) to ubuntu:20013, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.584-0400 c20013| 2018-03-16T17:28:26.583-0400 D REPL_HB [conn9] Received heartbeat request from ubuntu:20014, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235706, 41), signature: { hash: BinData(0, 793B00E09A3C5E443A29C280C281ADBBCD7368CA), keyId: 6533657568122765331 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.585-0400 c20013| 2018-03-16T17:28:26.583-0400 D REPL_HB [conn9] Processed heartbeat from ubuntu:20014 and generated response, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", syncingTo: "ubuntu:20012", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235706, 38), t: 1 }, opTime: { ts: Timestamp(1521235706, 38), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.585-0400 c20014| 2018-03-16T17:28:26.583-0400 D REPL_HB [replexec-0] Received response to heartbeat (requestId: 1161) from ubuntu:20013, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", syncingTo: "ubuntu:20012", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235706, 38), t: 1 }, opTime: { ts: Timestamp(1521235706, 38), t: 1 }, operationTime: Timestamp(1521235706, 38), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235706, 38), t: 1 }, lastOpVisible: { ts: Timestamp(1521235706, 38), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235706, 41), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235706, 38) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.586-0400 c20014| 2018-03-16T17:28:26.583-0400 D REPL_HB [replexec-0] Scheduling heartbeat to ubuntu:20013 at 2018-03-16T21:28:28.583Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.755-0400 d20011| 2018-03-16T17:28:26.754-0400 I SHARDING [migrateThread] about to log metadata event into changelog: { _id: "ubuntu-2018-03-16T17:28:26.754-0400-5aac36fa6a0ae3d58b640338", server: "ubuntu", clientAddr: "", time: new Date(1521235706754), what: "moveChunk.to", ns: "test.foo", details: { min: { x: MinKey }, max: { x: 5.0 }, step 1 of 6: 44, step 2 of 6: 2, step 3 of 6: 0, step 4 of 6: 0, step 5 of 6: 13, step 6 of 6: 386, note: "success" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.761-0400 d20011| 2018-03-16T17:28:26.760-0400 I COMMAND [conn22] command admin.$cmd appName: "MongoDB Shell" command: _recvChunkCommit { _recvChunkCommit: "test.foo", waitForSteadyOrDone: false, sessionId: "stale_mongos_updates_and_removes-rs0_stale_mongos_updates_and_removes-rs1_5aac36faf7e5a0b40b8e35a0", $clusterTime: { clusterTime: Timestamp(1521235706, 64), signature: { hash: BinData(0, 793B00E09A3C5E443A29C280C281ADBBCD7368CA), keyId: 6533657568122765331 } }, $configServerState: { opTime: { ts: Timestamp(1521235706, 38), t: 1 } }, $db: "admin" } numYields:0 reslen:569 locks:{} protocol:op_msg 401ms [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.767-0400 d20010| 2018-03-16T17:28:26.767-0400 I SHARDING [conn15] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.767-0400 d20010| 2018-03-16T17:28:26.767-0400 I SHARDING [conn15] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.767-0400 d20010| 2018-03-16T17:28:26.767-0400 I SHARDING [conn15] xxx routing info needs refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.768-0400 d20010| 2018-03-16T17:28:26.767-0400 I SHARDING [conn15] xxx calling schedule collection refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.768-0400 d20010| 2018-03-16T17:28:26.767-0400 I SHARDING [conn15] xxx is incremental [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.768-0400 d20010| 2018-03-16T17:28:26.767-0400 I SHARDING [conn15] xxx existing routing info? [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.768-0400 d20010| 2018-03-16T17:28:26.767-0400 I SHARDING [conn15] 1|2||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.768-0400 d20010| 2018-03-16T17:28:26.767-0400 I SHARDING [conn15] Refreshing chunks for collection test.foo based on version 1|2||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.768-0400 d20010| 2018-03-16T17:28:26.767-0400 I SHARDING [conn15] xxx getting chunks since now [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.768-0400 d20010| 2018-03-16T17:28:26.767-0400 I SHARDING [conn15] xxx getting chunks since [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.769-0400 d20010| 2018-03-16T17:28:26.767-0400 I SHARDING [conn15] xxx waiting if !refreshNot [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.769-0400 d20010| 2018-03-16T17:28:26.767-0400 I SHARDING [ShardServerCatalogCacheLoader-0] xxx primary and calling get chunks since for primary [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.769-0400 d20010| 2018-03-16T17:28:26.767-0400 I SHARDING [ShardServerCatalogCacheLoader-0] version is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.769-0400 d20010| 2018-03-16T17:28:26.767-0400 I SHARDING [ShardServerCatalogCacheLoader-0] 1|2||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.769-0400 d20010| 2018-03-16T17:28:26.768-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] refresh in primary [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.769-0400 d20010| 2018-03-16T17:28:26.768-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] 2 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.769-0400 d20010| 2018-03-16T17:28:26.768-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx collec and changed chunks max epoch or version doesn't equal max [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.804-0400 d20010| 2018-03-16T17:28:26.803-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Cache loader remotely refreshed for collection test.foo from collection version 1|2||5aac36f988c1f30185b7b1df and found collection version 2|1||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.804-0400 d20010| 2018-03-16T17:28:26.804-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Cache loader found enqueued metadata from 2|0||5aac36f988c1f30185b7b1df to 2|1||5aac36f988c1f30185b7b1df and persisted metadata from 1|2||5aac36f988c1f30185b7b1df to 1|2||5aac36f988c1f30185b7b1df, GTE cache version 1|2||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.804-0400 d20010| 2018-03-16T17:28:26.804-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx is okay [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.804-0400 d20010| 2018-03-16T17:28:26.804-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx finsihing refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.804-0400 d20010| 2018-03-16T17:28:26.804-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh callback [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.805-0400 d20010| 2018-03-16T17:28:26.804-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh completed [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.805-0400 d20010| 2018-03-16T17:28:26.804-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Refresh for collection test.foo took 36 ms and found version 2|1||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.805-0400 d20010| 2018-03-16T17:28:26.804-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx marking as status ok [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.805-0400 d20010| 2018-03-16T17:28:26.804-0400 I SHARDING [conn15] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.805-0400 d20010| 2018-03-16T17:28:26.804-0400 I SHARDING [conn15] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.806-0400 d20010| 2018-03-16T17:28:26.804-0400 I SHARDING [conn15] Updating collection metadata for test.foo from collection version: 1|2||5aac36f988c1f30185b7b1df, shard version: 1|2||5aac36f988c1f30185b7b1df to collection version: 2|1||5aac36f988c1f30185b7b1df, shard version: 2|1||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.806-0400 d20010| 2018-03-16T17:28:26.804-0400 I SHARDING [conn15] Migration succeeded and updated collection version to 2|1||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.806-0400 d20010| 2018-03-16T17:28:26.804-0400 I SHARDING [conn15] about to log metadata event into changelog: { _id: "ubuntu-2018-03-16T17:28:26.804-0400-5aac36faf7e5a0b40b8e35bd", server: "ubuntu", clientAddr: "127.0.0.1:56424", time: new Date(1521235706804), what: "moveChunk.commit", ns: "test.foo", details: { min: { x: MinKey }, max: { x: 5.0 }, from: "stale_mongos_updates_and_removes-rs0", to: "stale_mongos_updates_and_removes-rs1", counts: { cloned: 10, clonedBytes: 560, catchup: 0, steady: 0 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.809-0400 d20010| 2018-03-16T17:28:26.808-0400 I SHARDING [conn15] Scheduling immediate deletion of test.foo range [{ x: MinKey }, { x: 5.0 }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.809-0400 d20010| 2018-03-16T17:28:26.808-0400 I SHARDING [conn15] Waiting for cleanup of test.foo range [{ x: MinKey }, { x: 5.0 }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.809-0400 d20011| 2018-03-16T17:28:26.809-0400 I SHARDING [conn22] XXXX in set shard version cmd obj is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.810-0400 d20011| 2018-03-16T17:28:26.809-0400 I SHARDING [conn22] { setShardVersion: "test.foo", init: false, authoritative: false, configdb: "stale_mongos_updates_and_removes-configRS/ubuntu:20012,ubuntu:20013,ubuntu:20014", shard: "stale_mongos_updates_and_removes-rs1", shardHost: "ubuntu:20011", version: Timestamp(2, 1), versionEpoch: ObjectId('5aac36f988c1f30185b7b1df'), noConnectionVersioning: true, $clusterTime: { clusterTime: Timestamp(1521235706, 75), signature: { hash: BinData(0, 793B00E09A3C5E443A29C280C281ADBBCD7368CA), keyId: 6533657568122765331 } }, $client: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" }, mongos: { host: "ubuntu:20016", client: "127.0.0.1:50986", version: "3.6.3" } }, $configServerState: { opTime: { ts: Timestamp(1521235706, 75), t: 1 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.810-0400 d20011| 2018-03-16T17:28:26.809-0400 I SHARDING [conn22] XXXX requested version is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.810-0400 d20011| 2018-03-16T17:28:26.809-0400 I SHARDING [conn22] 2|1||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.810-0400 d20010| 2018-03-16T17:28:26.809-0400 I SHARDING [Collection Range Deleter] No documents remain to delete in test.foo range [{ x: MinKey }, { x: 5.0 }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.810-0400 d20010| 2018-03-16T17:28:26.809-0400 I SHARDING [Collection Range Deleter] Waiting for majority replication of local deletions in test.foo range [{ x: MinKey }, { x: 5.0 }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.812-0400 d20010| 2018-03-16T17:28:26.812-0400 I SHARDING [Collection Range Deleter] Finished deleting documents in test.foo range [{ x: MinKey }, { x: 5.0 }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.813-0400 d20010| 2018-03-16T17:28:26.812-0400 I SHARDING [conn15] about to log metadata event into changelog: { _id: "ubuntu-2018-03-16T17:28:26.812-0400-5aac36faf7e5a0b40b8e35be", server: "ubuntu", clientAddr: "127.0.0.1:56424", time: new Date(1521235706812), what: "moveChunk.from", ns: "test.foo", details: { min: { x: MinKey }, max: { x: 5.0 }, step 1 of 6: 0, step 2 of 6: 28, step 3 of 6: 106, step 4 of 6: 51, step 5 of 6: 403, step 6 of 6: 51, to: "stale_mongos_updates_and_removes-rs1", from: "stale_mongos_updates_and_removes-rs0", note: "success" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.817-0400 d20010| 2018-03-16T17:28:26.816-0400 I COMMAND [conn15] command admin.$cmd appName: "MongoDB Shell" command: moveChunk { moveChunk: "test.foo", shardVersion: [ Timestamp(1, 2), ObjectId('5aac36f988c1f30185b7b1df') ], epoch: ObjectId('5aac36f988c1f30185b7b1df'), configdb: "stale_mongos_updates_and_removes-configRS/ubuntu:20012,ubuntu:20013,ubuntu:20014", fromShard: "stale_mongos_updates_and_removes-rs0", toShard: "stale_mongos_updates_and_removes-rs1", min: { x: MinKey }, max: { x: 5.0 }, maxChunkSizeBytes: 67108864, waitForDelete: true, takeDistLock: false, $clusterTime: { clusterTime: Timestamp(1521235706, 34), signature: { hash: BinData(0, 793B00E09A3C5E443A29C280C281ADBBCD7368CA), keyId: 6533657568122765331 } }, $client: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" }, mongos: { host: "ubuntu:20016", client: "127.0.0.1:50986", version: "3.6.3" } }, $configServerState: { opTime: { ts: Timestamp(1521235706, 34), t: 1 } }, $db: "admin" } numYields:0 reslen:317 locks:{ Global: { acquireCount: { r: 34, w: 16 } }, Database: { acquireCount: { r: 9, w: 14, W: 2 } }, Collection: { acquireCount: { r: 9, w: 1, W: 6 } }, oplog: { acquireCount: { w: 7 } } } protocol:op_msg 644ms [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.822-0400 c20012| 2018-03-16T17:28:26.822-0400 I SHARDING [NetworkInterfaceASIO-ShardRegistry-0] distributed lock with ts: '5aac36ec88c1f30185b7ac69' and _id: 'test.foo' unlocked. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.822-0400 c20012| 2018-03-16T17:28:26.822-0400 I SHARDING [conn23] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.822-0400 c20012| 2018-03-16T17:28:26.822-0400 I SHARDING [conn23] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.822-0400 c20012| 2018-03-16T17:28:26.822-0400 I SHARDING [conn23] xxx routing info needs refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.823-0400 c20012| 2018-03-16T17:28:26.822-0400 I SHARDING [conn23] xxx calling schedule collection refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.823-0400 c20012| 2018-03-16T17:28:26.822-0400 I SHARDING [conn23] xxx is incremental [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.823-0400 c20012| 2018-03-16T17:28:26.822-0400 I SHARDING [conn23] xxx existing routing info? [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.823-0400 c20012| 2018-03-16T17:28:26.822-0400 I SHARDING [conn23] 1|2||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.823-0400 c20012| 2018-03-16T17:28:26.822-0400 I SHARDING [conn23] Refreshing chunks for collection test.foo based on version 1|2||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.823-0400 c20012| 2018-03-16T17:28:26.822-0400 I SHARDING [conn23] xxx getting chunks since now [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.823-0400 c20012| 2018-03-16T17:28:26.822-0400 I SHARDING [conn23] xxx waiting if !refreshNot [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.824-0400 c20012| 2018-03-16T17:28:26.822-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh callback [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.824-0400 c20012| 2018-03-16T17:28:26.822-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh completed [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.824-0400 c20012| 2018-03-16T17:28:26.822-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Refresh for collection test.foo took 0 ms and found version 2|1||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.824-0400 c20012| 2018-03-16T17:28:26.822-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx marking as status ok [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.824-0400 c20012| 2018-03-16T17:28:26.822-0400 I SHARDING [conn23] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.824-0400 c20012| 2018-03-16T17:28:26.822-0400 I SHARDING [conn23] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.827-0400 c20012| 2018-03-16T17:28:26.826-0400 I COMMAND [conn23] command admin.$cmd appName: "MongoDB Shell" command: _configsvrMoveChunk { _configsvrMoveChunk: 1, _id: "test.foo-x_MinKey", ns: "test.foo", min: { x: MinKey }, max: { x: 5.0 }, shard: "stale_mongos_updates_and_removes-rs0", lastmod: Timestamp(1, 2), lastmodEpoch: ObjectId('5aac36f988c1f30185b7b1df'), toShard: "stale_mongos_updates_and_removes-rs1", maxChunkSizeBytes: 67108864, secondaryThrottle: {}, waitForDelete: true, writeConcern: { w: "majority", wtimeout: 15000 }, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235706, 32), signature: { hash: BinData(0, 793B00E09A3C5E443A29C280C281ADBBCD7368CA), keyId: 6533657568122765331 } }, $client: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" }, mongos: { host: "ubuntu:20016", client: "127.0.0.1:50986", version: "3.6.3" } }, $configServerState: { opTime: { ts: Timestamp(1521235706, 32), t: 1 } }, $db: "admin" } numYields:0 reslen:462 locks:{ Global: { acquireCount: { r: 10, w: 6 } }, Database: { acquireCount: { r: 2, w: 6 } }, Collection: { acquireCount: { r: 2, w: 3 } }, oplog: { acquireCount: { w: 3 } } } protocol:op_msg 667ms [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.827-0400 s20016| 2018-03-16T17:28:26.826-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 369 finished with response: { ok: 1.0, operationTime: Timestamp(1521235706, 88), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235706, 88), t: 1 }, lastOpVisible: { ts: Timestamp(1521235706, 88), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235706, 88), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235706, 88), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235706, 88) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.828-0400 Waiting for op with OpTime { "ts" : Timestamp(1521235706, 88), "t" : NumberLong(1) } to be committed on all secondaries [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.832-0400 s20015| 2018-03-16T17:28:26.832-0400 I SHARDING [conn1] Refreshing chunks for collection test.foo based on version 1|0||5aac36f888c1f30185b7b0c4 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.833-0400 s20015| 2018-03-16T17:28:26.832-0400 D ASIO [ConfigServerCatalogCacheLoader-0] startCommand: RemoteCommand 66 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:56.832-0400 cmd:{ find: "collections", filter: { _id: "test.foo" }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235705, 93), t: 1 } }, limit: 1, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.833-0400 s20015| 2018-03-16T17:28:26.832-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 66 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.834-0400 s20015| 2018-03-16T17:28:26.832-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 66 finished with response: { cursor: { firstBatch: [ { _id: "test.foo", lastmodEpoch: ObjectId('5aac36f988c1f30185b7b1df'), lastmod: new Date(4294967296), dropped: false, key: { x: 1.0 }, unique: false, uuid: UUID("53837380-bed7-429b-ba96-f32fe994b394") } ], id: 0, ns: "config.collections" }, ok: 1.0, operationTime: Timestamp(1521235706, 88), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235706, 88), t: 1 }, lastOpVisible: { ts: Timestamp(1521235706, 88), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235705, 93), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235706, 88), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235706, 88) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.834-0400 s20015| 2018-03-16T17:28:26.833-0400 D ASIO [ConfigServerCatalogCacheLoader-0] startCommand: RemoteCommand 68 -- target:ubuntu:20013 db:config expDate:2018-03-16T17:28:56.833-0400 cmd:{ find: "chunks", filter: { ns: "test.foo", lastmod: { $gte: Timestamp(0, 0) } }, sort: { lastmod: 1 }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235706, 88), t: 1 } }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.834-0400 s20015| 2018-03-16T17:28:26.833-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 68 on host ubuntu:20013 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.835-0400 s20015| 2018-03-16T17:28:26.833-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 68 finished with response: { cursor: { firstBatch: [ { _id: "test.foo-x_MinKey", lastmod: Timestamp(2, 0), lastmodEpoch: ObjectId('5aac36f988c1f30185b7b1df'), ns: "test.foo", min: { x: MinKey }, max: { x: 5.0 }, shard: "stale_mongos_updates_and_removes-rs1", history: [ { validAfter: Timestamp(1521235706, 65), shard: "stale_mongos_updates_and_removes-rs1" }, { validAfter: Timestamp(1521235705, 141), shard: "stale_mongos_updates_and_removes-rs0" } ] }, { _id: "test.foo-x_5.0", lastmod: Timestamp(2, 1), lastmodEpoch: ObjectId('5aac36f988c1f30185b7b1df'), ns: "test.foo", min: { x: 5.0 }, max: { x: MaxKey }, shard: "stale_mongos_updates_and_removes-rs0" } ], id: 0, ns: "config.chunks" }, ok: 1.0, operationTime: Timestamp(1521235706, 88), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235706, 88), t: 1 }, lastOpVisible: { ts: Timestamp(1521235706, 88), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235706, 88), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235706, 88) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.835-0400 s20015| 2018-03-16T17:28:26.833-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Refresh for collection test.foo took 1 ms and found version 2|1||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.836-0400 s20015| 2018-03-16T17:28:26.833-0400 D ASIO [conn1] startCommand: RemoteCommand 70 -- target:ubuntu:20012 db:admin cmd:{ _configsvrMoveChunk: 1, _id: "test.foo-x_MinKey", ns: "test.foo", min: { x: MinKey }, max: { x: 5.0 }, shard: "stale_mongos_updates_and_removes-rs1", lastmod: Timestamp(2, 1), lastmodEpoch: ObjectId('5aac36f988c1f30185b7b1df'), toShard: "stale_mongos_updates_and_removes-rs0", maxChunkSizeBytes: 67108864, secondaryThrottle: {}, waitForDelete: true, writeConcern: { w: "majority", wtimeout: 15000 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.836-0400 s20015| 2018-03-16T17:28:26.833-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 70 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.836-0400 c20012| 2018-03-16T17:28:26.835-0400 I SHARDING [conn21] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.836-0400 c20012| 2018-03-16T17:28:26.835-0400 I SHARDING [conn21] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.836-0400 c20012| 2018-03-16T17:28:26.835-0400 I SHARDING [conn21] xxx routing info needs refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.836-0400 c20012| 2018-03-16T17:28:26.836-0400 I SHARDING [conn21] xxx calling schedule collection refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.837-0400 c20012| 2018-03-16T17:28:26.836-0400 I SHARDING [conn21] xxx is incremental [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.837-0400 c20012| 2018-03-16T17:28:26.836-0400 I SHARDING [conn21] xxx existing routing info? [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.837-0400 c20012| 2018-03-16T17:28:26.836-0400 I SHARDING [conn21] 2|1||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.837-0400 c20012| 2018-03-16T17:28:26.836-0400 I SHARDING [conn21] Refreshing chunks for collection test.foo based on version 2|1||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.837-0400 c20012| 2018-03-16T17:28:26.836-0400 I SHARDING [conn21] xxx getting chunks since now [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.837-0400 c20012| 2018-03-16T17:28:26.836-0400 I SHARDING [conn21] xxx waiting if !refreshNot [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.837-0400 c20012| 2018-03-16T17:28:26.836-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh callback [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.838-0400 c20012| 2018-03-16T17:28:26.836-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh completed [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.838-0400 c20012| 2018-03-16T17:28:26.836-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Refresh for collection test.foo took 0 ms and found version 2|1||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.838-0400 c20012| 2018-03-16T17:28:26.836-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx marking as status ok [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.838-0400 c20012| 2018-03-16T17:28:26.836-0400 I SHARDING [conn21] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.838-0400 c20012| 2018-03-16T17:28:26.836-0400 I SHARDING [conn21] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.845-0400 c20012| 2018-03-16T17:28:26.844-0400 I SHARDING [conn21] distributed lock 'test.foo' acquired for 'Migrating chunk(s) in collection test.foo', ts : 5aac36ec88c1f30185b7ac69 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.846-0400 d20011| 2018-03-16T17:28:26.845-0400 I SHARDING [conn15] Starting chunk migration ns: test.foo, [{ x: MinKey }, { x: 5.0 }), fromShard: stale_mongos_updates_and_removes-rs1, toShard: stale_mongos_updates_and_removes-rs0 with expected collection version epoch 5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.846-0400 d20011| 2018-03-16T17:28:26.845-0400 I SHARDING [conn15] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.846-0400 d20011| 2018-03-16T17:28:26.845-0400 I SHARDING [conn15] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.846-0400 d20011| 2018-03-16T17:28:26.845-0400 I SHARDING [conn15] xxx routing info needs refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.846-0400 d20011| 2018-03-16T17:28:26.845-0400 I SHARDING [conn15] xxx calling schedule collection refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.847-0400 d20011| 2018-03-16T17:28:26.845-0400 I SHARDING [conn15] xxx is incremental [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.847-0400 d20011| 2018-03-16T17:28:26.845-0400 I SHARDING [conn15] xxx existing routing info? [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.847-0400 d20011| 2018-03-16T17:28:26.845-0400 I SHARDING [conn15] 1|2||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.847-0400 d20011| 2018-03-16T17:28:26.845-0400 I SHARDING [conn15] Refreshing chunks for collection test.foo based on version 1|2||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.847-0400 d20011| 2018-03-16T17:28:26.845-0400 I SHARDING [conn15] xxx getting chunks since now [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.847-0400 d20011| 2018-03-16T17:28:26.845-0400 I SHARDING [conn15] xxx getting chunks since [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.847-0400 d20011| 2018-03-16T17:28:26.845-0400 I SHARDING [conn15] xxx waiting if !refreshNot [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.848-0400 d20011| 2018-03-16T17:28:26.846-0400 I SHARDING [ShardServerCatalogCacheLoader-0] xxx primary and calling get chunks since for primary [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.848-0400 d20011| 2018-03-16T17:28:26.846-0400 I SHARDING [ShardServerCatalogCacheLoader-0] version is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.848-0400 d20011| 2018-03-16T17:28:26.846-0400 I SHARDING [ShardServerCatalogCacheLoader-0] 1|2||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.848-0400 d20011| 2018-03-16T17:28:26.847-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] refresh in primary [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.848-0400 d20011| 2018-03-16T17:28:26.847-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] 2 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.848-0400 d20011| 2018-03-16T17:28:26.847-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx collec and changed chunks max epoch or version doesn't equal max [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.906-0400 d20011| 2018-03-16T17:28:26.906-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Cache loader remotely refreshed for collection test.foo from collection version 1|2||5aac36f988c1f30185b7b1df and found collection version 2|1||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.907-0400 d20011| 2018-03-16T17:28:26.906-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Cache loader found enqueued metadata from 2|0||5aac36f988c1f30185b7b1df to 2|1||5aac36f988c1f30185b7b1df and persisted metadata from 1|2||5aac36f988c1f30185b7b1df to 1|2||5aac36f988c1f30185b7b1df, GTE cache version 1|2||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.907-0400 d20011| 2018-03-16T17:28:26.906-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx is okay [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.907-0400 d20011| 2018-03-16T17:28:26.906-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx finsihing refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.907-0400 d20011| 2018-03-16T17:28:26.906-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh callback [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.907-0400 d20011| 2018-03-16T17:28:26.906-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh completed [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.908-0400 d20011| 2018-03-16T17:28:26.906-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Refresh for collection test.foo took 60 ms and found version 2|1||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.908-0400 d20011| 2018-03-16T17:28:26.906-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx marking as status ok [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.908-0400 d20011| 2018-03-16T17:28:26.906-0400 I SHARDING [conn15] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.908-0400 d20011| 2018-03-16T17:28:26.906-0400 I SHARDING [conn15] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.908-0400 d20011| 2018-03-16T17:28:26.907-0400 I SHARDING [conn15] Updating collection metadata for test.foo from collection version: 1|2||5aac36f988c1f30185b7b1df, shard version: 0|0||5aac36f988c1f30185b7b1df to collection version: 2|1||5aac36f988c1f30185b7b1df, shard version: 2|0||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.909-0400 d20011| 2018-03-16T17:28:26.907-0400 I SHARDING [conn15] about to log metadata event into changelog: { _id: "ubuntu-2018-03-16T17:28:26.907-0400-5aac36fa6a0ae3d58b640345", server: "ubuntu", clientAddr: "127.0.0.1:55074", time: new Date(1521235706907), what: "moveChunk.start", ns: "test.foo", details: { min: { x: MinKey }, max: { x: 5.0 }, from: "stale_mongos_updates_and_removes-rs1", to: "stale_mongos_updates_and_removes-rs0" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.920-0400 d20010| 2018-03-16T17:28:26.920-0400 I SHARDING [conn43] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.920-0400 d20010| 2018-03-16T17:28:26.920-0400 I SHARDING [conn43] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.920-0400 d20010| 2018-03-16T17:28:26.920-0400 I SHARDING [conn43] xxx routing info needs refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.920-0400 d20010| 2018-03-16T17:28:26.920-0400 I SHARDING [conn43] xxx calling schedule collection refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.920-0400 d20010| 2018-03-16T17:28:26.920-0400 I SHARDING [conn43] xxx is incremental [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.921-0400 d20010| 2018-03-16T17:28:26.920-0400 I SHARDING [conn43] xxx existing routing info? [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.921-0400 d20010| 2018-03-16T17:28:26.920-0400 I SHARDING [conn43] 2|1||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.921-0400 d20010| 2018-03-16T17:28:26.920-0400 I SHARDING [conn43] Refreshing chunks for collection test.foo based on version 2|1||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.921-0400 d20010| 2018-03-16T17:28:26.920-0400 I SHARDING [conn43] xxx getting chunks since now [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.921-0400 d20010| 2018-03-16T17:28:26.920-0400 I SHARDING [conn43] xxx getting chunks since [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.921-0400 d20010| 2018-03-16T17:28:26.920-0400 I SHARDING [conn43] xxx waiting if !refreshNot [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.921-0400 d20010| 2018-03-16T17:28:26.920-0400 I SHARDING [ShardServerCatalogCacheLoader-0] xxx primary and calling get chunks since for primary [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.922-0400 d20010| 2018-03-16T17:28:26.920-0400 I SHARDING [ShardServerCatalogCacheLoader-0] version is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.922-0400 d20010| 2018-03-16T17:28:26.920-0400 I SHARDING [ShardServerCatalogCacheLoader-0] 2|1||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.922-0400 d20010| 2018-03-16T17:28:26.921-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] refresh in primary [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.922-0400 d20010| 2018-03-16T17:28:26.921-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] 1 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.922-0400 d20010| 2018-03-16T17:28:26.921-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Cache loader remotely refreshed for collection test.foo from collection version 2|1||5aac36f988c1f30185b7b1df and found collection version 2|1||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.922-0400 d20010| 2018-03-16T17:28:26.921-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Cache loader found no enqueued metadata and persisted metadata from 2|1||5aac36f988c1f30185b7b1df to 2|1||5aac36f988c1f30185b7b1df, GTE cache version 2|1||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.923-0400 d20010| 2018-03-16T17:28:26.921-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx is okay [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.923-0400 d20010| 2018-03-16T17:28:26.921-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx finsihing refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.923-0400 d20010| 2018-03-16T17:28:26.921-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh callback [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.923-0400 d20010| 2018-03-16T17:28:26.921-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh completed [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.923-0400 d20010| 2018-03-16T17:28:26.921-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Refresh for collection test.foo took 1 ms and found version 2|1||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.923-0400 d20010| 2018-03-16T17:28:26.921-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx marking as status ok [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.924-0400 d20010| 2018-03-16T17:28:26.921-0400 I SHARDING [conn43] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.924-0400 d20010| 2018-03-16T17:28:26.921-0400 I SHARDING [conn43] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.924-0400 d20010| 2018-03-16T17:28:26.922-0400 I SHARDING [migrateThread] Starting receiving end of migration of chunk { x: MinKey } -> { x: 5.0 } for collection test.foo from stale_mongos_updates_and_removes-rs1/ubuntu:20011 at epoch 5aac36f988c1f30185b7b1df with session id stale_mongos_updates_and_removes-rs1_stale_mongos_updates_and_removes-rs0_5aac36fa6a0ae3d58b64034a [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.924-0400 d20010| 2018-03-16T17:28:26.922-0400 I NETWORK [migrateThread] Successfully connected to stale_mongos_updates_and_removes-rs1/ubuntu:20011 (1 connections now open to stale_mongos_updates_and_removes-rs1/ubuntu:20011 with a 0 second timeout) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.924-0400 d20011| 2018-03-16T17:28:26.922-0400 I NETWORK [listener] connection accepted from 127.0.0.1:55196 #27 (17 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.925-0400 d20011| 2018-03-16T17:28:26.922-0400 I NETWORK [conn27] received client metadata from 127.0.0.1:55196 conn27: { driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.925-0400 d20010| 2018-03-16T17:28:26.923-0400 I SHARDING [migrateThread] Scheduling deletion of any documents in test.foo range [{ x: MinKey }, { x: 5.0 }) before migrating in a chunk covering the range [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.925-0400 d20010| 2018-03-16T17:28:26.923-0400 I SHARDING [Collection Range Deleter] No documents remain to delete in test.foo range [{ x: MinKey }, { x: 5.0 }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.925-0400 d20010| 2018-03-16T17:28:26.923-0400 I SHARDING [Collection Range Deleter] Waiting for majority replication of local deletions in test.foo range [{ x: MinKey }, { x: 5.0 }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.925-0400 d20010| 2018-03-16T17:28:26.923-0400 I SHARDING [Collection Range Deleter] Finished deleting documents in test.foo range [{ x: MinKey }, { x: 5.0 }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.926-0400 d20010| 2018-03-16T17:28:26.923-0400 I SHARDING [migrateThread] Finished deleting test.foo range [{ x: MinKey }, { x: 5.0 }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.926-0400 d20010| 2018-03-16T17:28:26.924-0400 I SHARDING [migrateThread] Waiting for replication to catch up before entering critical section [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.926-0400 d20010| 2018-03-16T17:28:26.926-0400 I SHARDING [migrateThread] migrate commit succeeded flushing to secondaries for 'test.foo' { x: MinKey } -> { x: 5.0 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.927-0400 d20011| 2018-03-16T17:28:26.926-0400 I SHARDING [conn15] moveChunk data transfer progress: { waited: true, active: true, sessionId: "stale_mongos_updates_and_removes-rs1_stale_mongos_updates_and_removes-rs0_5aac36fa6a0ae3d58b64034a", ns: "test.foo", from: "stale_mongos_updates_and_removes-rs1/ubuntu:20011", min: { x: MinKey }, max: { x: 5.0 }, shardKeyPattern: { x: 1.0 }, state: "steady", counts: { cloned: 10, clonedBytes: 560, catchup: 0, steady: 0 }, ok: 1.0, operationTime: Timestamp(1521235706, 108), $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235706, 108), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235706, 108), $configServerState: { opTime: { ts: Timestamp(1521235706, 94), t: 1 } } } mem used: 0 documents remaining to clone: 0 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.928-0400 d20011| 2018-03-16T17:28:26.927-0400 I SHARDING [conn15] Migration successfully entered critical section [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:26.937-0400 d20010| 2018-03-16T17:28:26.936-0400 I SHARDING [migrateThread] migrate commit succeeded flushing to secondaries for 'test.foo' { x: MinKey } -> { x: 5.0 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.326-0400 d20010| 2018-03-16T17:28:27.325-0400 I SHARDING [migrateThread] about to log metadata event into changelog: { _id: "ubuntu-2018-03-16T17:28:27.325-0400-5aac36fbf7e5a0b40b8e35c8", server: "ubuntu", clientAddr: "", time: new Date(1521235707325), what: "moveChunk.to", ns: "test.foo", details: { min: { x: MinKey }, max: { x: 5.0 }, step 1 of 6: 1, step 2 of 6: 0, step 3 of 6: 1, step 4 of 6: 0, step 5 of 6: 12, step 6 of 6: 388, note: "success" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.331-0400 d20010| 2018-03-16T17:28:27.331-0400 I COMMAND [conn43] command admin.$cmd command: _recvChunkCommit { _recvChunkCommit: "test.foo", waitForSteadyOrDone: false, sessionId: "stale_mongos_updates_and_removes-rs1_stale_mongos_updates_and_removes-rs0_5aac36fa6a0ae3d58b64034a", $clusterTime: { clusterTime: Timestamp(1521235706, 110), signature: { hash: BinData(0, 793B00E09A3C5E443A29C280C281ADBBCD7368CA), keyId: 6533657568122765331 } }, $configServerState: { opTime: { ts: Timestamp(1521235706, 94), t: 1 } }, $db: "admin" } numYields:0 reslen:549 locks:{} protocol:op_msg 403ms [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.331-0400 d20011| 2018-03-16T17:28:27.331-0400 I SHARDING [conn15] Moving last chunk for the collection out [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.336-0400 d20011| 2018-03-16T17:28:27.336-0400 I SHARDING [conn15] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.336-0400 d20011| 2018-03-16T17:28:27.336-0400 I SHARDING [conn15] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.336-0400 d20011| 2018-03-16T17:28:27.336-0400 I SHARDING [conn15] xxx routing info needs refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.336-0400 d20011| 2018-03-16T17:28:27.336-0400 I SHARDING [conn15] xxx calling schedule collection refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.337-0400 d20011| 2018-03-16T17:28:27.336-0400 I SHARDING [conn15] xxx is incremental [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.337-0400 d20011| 2018-03-16T17:28:27.336-0400 I SHARDING [conn15] xxx existing routing info? [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.337-0400 d20011| 2018-03-16T17:28:27.336-0400 I SHARDING [conn15] 2|1||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.337-0400 d20011| 2018-03-16T17:28:27.336-0400 I SHARDING [conn15] Refreshing chunks for collection test.foo based on version 2|1||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.337-0400 d20011| 2018-03-16T17:28:27.336-0400 I SHARDING [conn15] xxx getting chunks since now [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.337-0400 d20011| 2018-03-16T17:28:27.336-0400 I SHARDING [conn15] xxx getting chunks since [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.337-0400 d20011| 2018-03-16T17:28:27.336-0400 I SHARDING [conn15] xxx waiting if !refreshNot [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.338-0400 d20011| 2018-03-16T17:28:27.336-0400 I SHARDING [ShardServerCatalogCacheLoader-0] xxx primary and calling get chunks since for primary [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.338-0400 d20011| 2018-03-16T17:28:27.336-0400 I SHARDING [ShardServerCatalogCacheLoader-0] version is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.338-0400 d20011| 2018-03-16T17:28:27.336-0400 I SHARDING [ShardServerCatalogCacheLoader-0] 2|1||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.338-0400 d20011| 2018-03-16T17:28:27.337-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] refresh in primary [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.338-0400 d20011| 2018-03-16T17:28:27.337-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] 2 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.338-0400 d20011| 2018-03-16T17:28:27.337-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx collec and changed chunks max epoch or version doesn't equal max [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.409-0400 d20011| 2018-03-16T17:28:27.409-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Cache loader remotely refreshed for collection test.foo from collection version 2|1||5aac36f988c1f30185b7b1df and found collection version 3|0||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.410-0400 d20011| 2018-03-16T17:28:27.409-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Cache loader found enqueued metadata from 2|1||5aac36f988c1f30185b7b1df to 3|0||5aac36f988c1f30185b7b1df and persisted metadata from 2|1||5aac36f988c1f30185b7b1df to 2|1||5aac36f988c1f30185b7b1df, GTE cache version 2|1||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.410-0400 d20011| 2018-03-16T17:28:27.409-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx is okay [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.410-0400 d20011| 2018-03-16T17:28:27.409-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx finsihing refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.410-0400 d20011| 2018-03-16T17:28:27.409-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh callback [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.410-0400 d20011| 2018-03-16T17:28:27.410-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh completed [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.411-0400 d20011| 2018-03-16T17:28:27.410-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Refresh for collection test.foo took 73 ms and found version 3|0||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.411-0400 d20011| 2018-03-16T17:28:27.410-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx marking as status ok [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.411-0400 d20011| 2018-03-16T17:28:27.410-0400 I SHARDING [conn15] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.411-0400 d20011| 2018-03-16T17:28:27.410-0400 I SHARDING [conn15] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.411-0400 d20011| 2018-03-16T17:28:27.410-0400 I SHARDING [conn15] Updating collection metadata for test.foo from collection version: 2|1||5aac36f988c1f30185b7b1df, shard version: 2|0||5aac36f988c1f30185b7b1df to collection version: 3|0||5aac36f988c1f30185b7b1df, shard version: 0|0||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.412-0400 d20011| 2018-03-16T17:28:27.410-0400 I SHARDING [conn15] Migration succeeded and updated collection version to 3|0||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.412-0400 d20011| 2018-03-16T17:28:27.410-0400 I SHARDING [conn15] about to log metadata event into changelog: { _id: "ubuntu-2018-03-16T17:28:27.410-0400-5aac36fb6a0ae3d58b640368", server: "ubuntu", clientAddr: "127.0.0.1:55074", time: new Date(1521235707410), what: "moveChunk.commit", ns: "test.foo", details: { min: { x: MinKey }, max: { x: 5.0 }, from: "stale_mongos_updates_and_removes-rs1", to: "stale_mongos_updates_and_removes-rs0", counts: { cloned: 10, clonedBytes: 560, catchup: 0, steady: 0 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.415-0400 d20011| 2018-03-16T17:28:27.414-0400 I SHARDING [conn15] Scheduling immediate deletion of test.foo range [{ x: MinKey }, { x: 5.0 }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.415-0400 d20011| 2018-03-16T17:28:27.414-0400 I SHARDING [conn15] Waiting for cleanup of test.foo range [{ x: MinKey }, { x: 5.0 }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.415-0400 d20010| 2018-03-16T17:28:27.415-0400 I SHARDING [conn43] XXXX in set shard version cmd obj is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.416-0400 d20010| 2018-03-16T17:28:27.415-0400 I SHARDING [conn43] { setShardVersion: "test.foo", init: false, authoritative: false, configdb: "stale_mongos_updates_and_removes-configRS/ubuntu:20012,ubuntu:20013,ubuntu:20014", shard: "stale_mongos_updates_and_removes-rs0", shardHost: "ubuntu:20010", version: Timestamp(3, 0), versionEpoch: ObjectId('5aac36f988c1f30185b7b1df'), noConnectionVersioning: true, $clusterTime: { clusterTime: Timestamp(1521235707, 11), signature: { hash: BinData(0, CEFAA93570840EC6E37F07AD52BBF4B956967DEE), keyId: 6533657568122765331 } }, $client: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" }, mongos: { host: "ubuntu:20015", client: "127.0.0.1:47840", version: "3.6.3" } }, $configServerState: { opTime: { ts: Timestamp(1521235707, 11), t: 1 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.416-0400 d20010| 2018-03-16T17:28:27.415-0400 I SHARDING [conn43] XXXX requested version is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.416-0400 d20010| 2018-03-16T17:28:27.415-0400 I SHARDING [conn43] 3|0||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.416-0400 d20010| 2018-03-16T17:28:27.415-0400 I SHARDING [conn43] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.416-0400 d20010| 2018-03-16T17:28:27.415-0400 I SHARDING [conn43] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.416-0400 d20010| 2018-03-16T17:28:27.415-0400 I SHARDING [conn43] xxx routing info needs refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.416-0400 d20010| 2018-03-16T17:28:27.415-0400 I SHARDING [conn43] xxx calling schedule collection refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.417-0400 d20010| 2018-03-16T17:28:27.415-0400 I SHARDING [conn43] xxx is incremental [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.417-0400 d20010| 2018-03-16T17:28:27.415-0400 I SHARDING [conn43] xxx existing routing info? [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.417-0400 d20010| 2018-03-16T17:28:27.415-0400 I SHARDING [conn43] 2|1||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.417-0400 d20010| 2018-03-16T17:28:27.415-0400 I SHARDING [conn43] Refreshing chunks for collection test.foo based on version 2|1||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.417-0400 d20010| 2018-03-16T17:28:27.415-0400 I SHARDING [conn43] xxx getting chunks since now [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.417-0400 d20010| 2018-03-16T17:28:27.415-0400 I SHARDING [conn43] xxx getting chunks since [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.417-0400 d20010| 2018-03-16T17:28:27.415-0400 I SHARDING [conn43] xxx waiting if !refreshNot [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.418-0400 d20010| 2018-03-16T17:28:27.415-0400 I SHARDING [ShardServerCatalogCacheLoader-0] xxx primary and calling get chunks since for primary [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.418-0400 d20010| 2018-03-16T17:28:27.415-0400 I SHARDING [ShardServerCatalogCacheLoader-0] version is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.418-0400 d20010| 2018-03-16T17:28:27.415-0400 I SHARDING [ShardServerCatalogCacheLoader-0] 2|1||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.418-0400 d20011| 2018-03-16T17:28:27.415-0400 I SHARDING [Collection Range Deleter] No documents remain to delete in test.foo range [{ x: MinKey }, { x: 5.0 }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.418-0400 d20011| 2018-03-16T17:28:27.415-0400 I SHARDING [Collection Range Deleter] Waiting for majority replication of local deletions in test.foo range [{ x: MinKey }, { x: 5.0 }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.418-0400 d20010| 2018-03-16T17:28:27.416-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] refresh in primary [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.419-0400 d20010| 2018-03-16T17:28:27.416-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] 2 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.419-0400 d20010| 2018-03-16T17:28:27.416-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx collec and changed chunks max epoch or version doesn't equal max [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.419-0400 d20011| 2018-03-16T17:28:27.418-0400 I SHARDING [Collection Range Deleter] Finished deleting documents in test.foo range [{ x: MinKey }, { x: 5.0 }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.419-0400 d20011| 2018-03-16T17:28:27.418-0400 I SHARDING [conn15] about to log metadata event into changelog: { _id: "ubuntu-2018-03-16T17:28:27.418-0400-5aac36fb6a0ae3d58b640369", server: "ubuntu", clientAddr: "127.0.0.1:55074", time: new Date(1521235707418), what: "moveChunk.from", ns: "test.foo", details: { min: { x: MinKey }, max: { x: 5.0 }, step 1 of 6: 0, step 2 of 6: 61, step 3 of 6: 15, step 4 of 6: 4, step 5 of 6: 404, step 6 of 6: 87, to: "stale_mongos_updates_and_removes-rs0", from: "stale_mongos_updates_and_removes-rs1", note: "success" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.424-0400 d20011| 2018-03-16T17:28:27.423-0400 I COMMAND [conn15] command admin.$cmd appName: "MongoDB Shell" command: moveChunk { moveChunk: "test.foo", shardVersion: [ Timestamp(2, 1), ObjectId('5aac36f988c1f30185b7b1df') ], epoch: ObjectId('5aac36f988c1f30185b7b1df'), configdb: "stale_mongos_updates_and_removes-configRS/ubuntu:20012,ubuntu:20013,ubuntu:20014", fromShard: "stale_mongos_updates_and_removes-rs1", toShard: "stale_mongos_updates_and_removes-rs0", min: { x: MinKey }, max: { x: 5.0 }, maxChunkSizeBytes: 67108864, waitForDelete: true, takeDistLock: false, $clusterTime: { clusterTime: Timestamp(1521235706, 90), signature: { hash: BinData(0, 793B00E09A3C5E443A29C280C281ADBBCD7368CA), keyId: 6533657568122765331 } }, $client: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" }, mongos: { host: "ubuntu:20015", client: "127.0.0.1:47840", version: "3.6.3" } }, $configServerState: { opTime: { ts: Timestamp(1521235706, 90), t: 1 } }, $db: "admin" } numYields:0 reslen:317 locks:{ Global: { acquireCount: { r: 32, w: 14 } }, Database: { acquireCount: { r: 9, w: 12, W: 2 } }, Collection: { acquireCount: { r: 9, w: 1, W: 6 } }, oplog: { acquireCount: { w: 5 } } } protocol:op_msg 578ms [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.427-0400 c20012| 2018-03-16T17:28:27.427-0400 I SHARDING [NetworkInterfaceASIO-ShardRegistry-0] distributed lock with ts: '5aac36ec88c1f30185b7ac69' and _id: 'test.foo' unlocked. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.427-0400 c20012| 2018-03-16T17:28:27.427-0400 I SHARDING [conn21] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.427-0400 c20012| 2018-03-16T17:28:27.427-0400 I SHARDING [conn21] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.427-0400 c20012| 2018-03-16T17:28:27.427-0400 I SHARDING [conn21] xxx routing info needs refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.428-0400 c20012| 2018-03-16T17:28:27.427-0400 I SHARDING [conn21] xxx calling schedule collection refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.428-0400 c20012| 2018-03-16T17:28:27.427-0400 I SHARDING [conn21] xxx is incremental [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.428-0400 c20012| 2018-03-16T17:28:27.427-0400 I SHARDING [conn21] xxx existing routing info? [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.428-0400 c20012| 2018-03-16T17:28:27.427-0400 I SHARDING [conn21] 2|1||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.428-0400 c20012| 2018-03-16T17:28:27.427-0400 I SHARDING [conn21] Refreshing chunks for collection test.foo based on version 2|1||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.428-0400 c20012| 2018-03-16T17:28:27.427-0400 I SHARDING [conn21] xxx getting chunks since now [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.428-0400 c20012| 2018-03-16T17:28:27.427-0400 I SHARDING [conn21] xxx waiting if !refreshNot [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.428-0400 c20012| 2018-03-16T17:28:27.427-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh callback [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.429-0400 c20012| 2018-03-16T17:28:27.427-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh completed [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.429-0400 c20012| 2018-03-16T17:28:27.427-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Refresh for collection test.foo took 0 ms and found version 3|0||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.429-0400 c20012| 2018-03-16T17:28:27.427-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx marking as status ok [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.429-0400 c20012| 2018-03-16T17:28:27.427-0400 I SHARDING [conn21] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.429-0400 c20012| 2018-03-16T17:28:27.427-0400 I SHARDING [conn21] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.436-0400 c20012| 2018-03-16T17:28:27.435-0400 I COMMAND [conn21] command admin.$cmd appName: "MongoDB Shell" command: _configsvrMoveChunk { _configsvrMoveChunk: 1, _id: "test.foo-x_MinKey", ns: "test.foo", min: { x: MinKey }, max: { x: 5.0 }, shard: "stale_mongos_updates_and_removes-rs1", lastmod: Timestamp(2, 1), lastmodEpoch: ObjectId('5aac36f988c1f30185b7b1df'), toShard: "stale_mongos_updates_and_removes-rs0", maxChunkSizeBytes: 67108864, secondaryThrottle: {}, waitForDelete: true, writeConcern: { w: "majority", wtimeout: 15000 }, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235706, 88), signature: { hash: BinData(0, 793B00E09A3C5E443A29C280C281ADBBCD7368CA), keyId: 6533657568122765331 } }, $client: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" }, mongos: { host: "ubuntu:20015", client: "127.0.0.1:47840", version: "3.6.3" } }, $configServerState: { opTime: { ts: Timestamp(1521235706, 88), t: 1 } }, $db: "admin" } numYields:0 reslen:462 locks:{ Global: { acquireCount: { r: 10, w: 6 } }, Database: { acquireCount: { r: 2, w: 6 } }, Collection: { acquireCount: { r: 2, w: 3 } }, oplog: { acquireCount: { w: 3 } } } protocol:op_msg 601ms [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.436-0400 s20015| 2018-03-16T17:28:27.435-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 70 finished with response: { ok: 1.0, operationTime: Timestamp(1521235707, 24), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235707, 24), t: 1 }, lastOpVisible: { ts: Timestamp(1521235707, 24), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235707, 24), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235707, 24), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235707, 24) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.436-0400 s20016| 2018-03-16T17:28:27.435-0400 I SHARDING [conn1] Refreshing chunks for collection test.foo based on version 1|2||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.437-0400 s20016| 2018-03-16T17:28:27.436-0400 D ASIO [ConfigServerCatalogCacheLoader-0] startCommand: RemoteCommand 371 -- target:ubuntu:20014 db:config expDate:2018-03-16T17:28:57.436-0400 cmd:{ find: "collections", filter: { _id: "test.foo" }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235706, 88), t: 1 } }, limit: 1, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.437-0400 s20016| 2018-03-16T17:28:27.436-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 371 on host ubuntu:20014 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.438-0400 s20016| 2018-03-16T17:28:27.436-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 371 finished with response: { cursor: { firstBatch: [ { _id: "test.foo", lastmodEpoch: ObjectId('5aac36f988c1f30185b7b1df'), lastmod: new Date(4294967296), dropped: false, key: { x: 1.0 }, unique: false, uuid: UUID("53837380-bed7-429b-ba96-f32fe994b394") } ], id: 0, ns: "config.collections" }, ok: 1.0, operationTime: Timestamp(1521235707, 24), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235707, 24), t: 1 }, lastOpVisible: { ts: Timestamp(1521235707, 24), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235707, 24), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235707, 24) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.438-0400 s20016| 2018-03-16T17:28:27.436-0400 D ASIO [ConfigServerCatalogCacheLoader-0] startCommand: RemoteCommand 373 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:57.436-0400 cmd:{ find: "chunks", filter: { ns: "test.foo", lastmod: { $gte: Timestamp(1, 2) } }, sort: { lastmod: 1 }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235707, 24), t: 1 } }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.438-0400 s20016| 2018-03-16T17:28:27.436-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 373 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.439-0400 s20016| 2018-03-16T17:28:27.436-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 373 finished with response: { cursor: { firstBatch: [ { _id: "test.foo-x_5.0", lastmod: Timestamp(2, 1), lastmodEpoch: ObjectId('5aac36f988c1f30185b7b1df'), ns: "test.foo", min: { x: 5.0 }, max: { x: MaxKey }, shard: "stale_mongos_updates_and_removes-rs0" }, { _id: "test.foo-x_MinKey", lastmod: Timestamp(3, 0), lastmodEpoch: ObjectId('5aac36f988c1f30185b7b1df'), ns: "test.foo", min: { x: MinKey }, max: { x: 5.0 }, shard: "stale_mongos_updates_and_removes-rs0", history: [ { validAfter: Timestamp(1521235707, 1), shard: "stale_mongos_updates_and_removes-rs0" }, { validAfter: Timestamp(1521235706, 65), shard: "stale_mongos_updates_and_removes-rs1" }, { validAfter: Timestamp(1521235705, 141), shard: "stale_mongos_updates_and_removes-rs0" } ] } ], id: 0, ns: "config.chunks" }, ok: 1.0, operationTime: Timestamp(1521235707, 24), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235707, 24), t: 1 }, lastOpVisible: { ts: Timestamp(1521235707, 24), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235706, 88), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235707, 24), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235707, 24) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.440-0400 s20016| 2018-03-16T17:28:27.437-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Refresh for collection test.foo took 1 ms and found version 3|0||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.440-0400 s20016| 2018-03-16T17:28:27.437-0400 I SHARDING [conn1] Refreshing chunks for collection test.foo based on version 3|0||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.440-0400 s20016| 2018-03-16T17:28:27.437-0400 D ASIO [ConfigServerCatalogCacheLoader-0] startCommand: RemoteCommand 375 -- target:ubuntu:20013 db:config expDate:2018-03-16T17:28:57.437-0400 cmd:{ find: "collections", filter: { _id: "test.foo" }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235707, 24), t: 1 } }, limit: 1, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.440-0400 s20016| 2018-03-16T17:28:27.437-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 375 on host ubuntu:20013 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.441-0400 s20016| 2018-03-16T17:28:27.437-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 375 finished with response: { cursor: { firstBatch: [ { _id: "test.foo", lastmodEpoch: ObjectId('5aac36f988c1f30185b7b1df'), lastmod: new Date(4294967296), dropped: false, key: { x: 1.0 }, unique: false, uuid: UUID("53837380-bed7-429b-ba96-f32fe994b394") } ], id: 0, ns: "config.collections" }, ok: 1.0, operationTime: Timestamp(1521235707, 24), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235707, 24), t: 1 }, lastOpVisible: { ts: Timestamp(1521235707, 24), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235707, 24), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235707, 24) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.441-0400 s20016| 2018-03-16T17:28:27.437-0400 D ASIO [ConfigServerCatalogCacheLoader-0] startCommand: RemoteCommand 377 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:57.437-0400 cmd:{ find: "chunks", filter: { ns: "test.foo", lastmod: { $gte: Timestamp(3, 0) } }, sort: { lastmod: 1 }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235707, 24), t: 1 } }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.442-0400 s20016| 2018-03-16T17:28:27.437-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 377 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.443-0400 s20016| 2018-03-16T17:28:27.437-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 377 finished with response: { cursor: { firstBatch: [ { _id: "test.foo-x_MinKey", lastmod: Timestamp(3, 0), lastmodEpoch: ObjectId('5aac36f988c1f30185b7b1df'), ns: "test.foo", min: { x: MinKey }, max: { x: 5.0 }, shard: "stale_mongos_updates_and_removes-rs0", history: [ { validAfter: Timestamp(1521235707, 1), shard: "stale_mongos_updates_and_removes-rs0" }, { validAfter: Timestamp(1521235706, 65), shard: "stale_mongos_updates_and_removes-rs1" }, { validAfter: Timestamp(1521235705, 141), shard: "stale_mongos_updates_and_removes-rs0" } ] } ], id: 0, ns: "config.chunks" }, ok: 1.0, operationTime: Timestamp(1521235707, 24), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235707, 24), t: 1 }, lastOpVisible: { ts: Timestamp(1521235707, 24), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235706, 88), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235707, 24), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235707, 24) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.443-0400 s20016| 2018-03-16T17:28:27.438-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Refresh for collection test.foo took 0 ms and found version 3|0||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.443-0400 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.443-0400 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.443-0400 ---- [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.443-0400 xxx will fail below at do remove [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.443-0400 ---- [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.443-0400 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.443-0400 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.443-0400 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.443-0400 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.443-0400 ---- [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.443-0400 xxx top multi stale [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.444-0400 ---- [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.444-0400 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.444-0400 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.444-0400 s20016| 2018-03-16T17:28:27.439-0400 D SHARDING [conn1] trying to acquire new distributed lock for test-movePrimary ( lock timeout : 900000 ms, ping interval : 30000 ms, process : ubuntu:20016:1521235700:-4633782234644071543 ) with lockSessionID: 5aac36fbebcdfa1d2b617ddc, why: dropCollection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.444-0400 s20016| 2018-03-16T17:28:27.439-0400 D ASIO [conn1] startCommand: RemoteCommand 379 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:57.439-0400 cmd:{ findAndModify: "locks", query: { _id: "test-movePrimary", state: 0 }, update: { $set: { ts: ObjectId('5aac36fbebcdfa1d2b617ddc'), state: 2, who: "ubuntu:20016:1521235700:-4633782234644071543:conn1", process: "ubuntu:20016:1521235700:-4633782234644071543", when: new Date(1521235707439), why: "dropCollection" } }, upsert: true, new: true, writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.445-0400 s20016| 2018-03-16T17:28:27.439-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 379 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.445-0400 s20016| 2018-03-16T17:28:27.443-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 379 finished with response: { lastErrorObject: { n: 1, updatedExisting: true }, value: { _id: "test-movePrimary", state: 2, process: "ubuntu:20016:1521235700:-4633782234644071543", ts: ObjectId('5aac36fbebcdfa1d2b617ddc'), when: new Date(1521235707439), who: "ubuntu:20016:1521235700:-4633782234644071543:conn1", why: "dropCollection" }, ok: 1.0, operationTime: Timestamp(1521235707, 25), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235707, 25), t: 1 }, lastOpVisible: { ts: Timestamp(1521235707, 25), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235707, 25), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235707, 25), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235707, 25) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.446-0400 s20016| 2018-03-16T17:28:27.443-0400 I SHARDING [conn1] distributed lock 'test-movePrimary' acquired for 'dropCollection', ts : 5aac36fbebcdfa1d2b617ddc [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.446-0400 s20016| 2018-03-16T17:28:27.443-0400 D SHARDING [conn1] trying to acquire new distributed lock for test ( lock timeout : 900000 ms, ping interval : 30000 ms, process : ubuntu:20016:1521235700:-4633782234644071543 ) with lockSessionID: 5aac36fbebcdfa1d2b617ddd, why: dropCollection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.446-0400 s20016| 2018-03-16T17:28:27.443-0400 D ASIO [conn1] startCommand: RemoteCommand 381 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:57.443-0400 cmd:{ findAndModify: "locks", query: { _id: "test", state: 0 }, update: { $set: { ts: ObjectId('5aac36fbebcdfa1d2b617ddd'), state: 2, who: "ubuntu:20016:1521235700:-4633782234644071543:conn1", process: "ubuntu:20016:1521235700:-4633782234644071543", when: new Date(1521235707443), why: "dropCollection" } }, upsert: true, new: true, writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.447-0400 s20016| 2018-03-16T17:28:27.443-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 381 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.448-0400 s20016| 2018-03-16T17:28:27.447-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 381 finished with response: { lastErrorObject: { n: 1, updatedExisting: true }, value: { _id: "test", state: 2, process: "ubuntu:20016:1521235700:-4633782234644071543", ts: ObjectId('5aac36fbebcdfa1d2b617ddd'), when: new Date(1521235707443), who: "ubuntu:20016:1521235700:-4633782234644071543:conn1", why: "dropCollection" }, ok: 1.0, operationTime: Timestamp(1521235707, 26), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235707, 26), t: 1 }, lastOpVisible: { ts: Timestamp(1521235707, 26), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235707, 26), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235707, 26), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235707, 26) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.448-0400 s20016| 2018-03-16T17:28:27.447-0400 I SHARDING [conn1] distributed lock 'test' acquired for 'dropCollection', ts : 5aac36fbebcdfa1d2b617ddd [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.449-0400 s20016| 2018-03-16T17:28:27.447-0400 D SHARDING [conn1] trying to acquire new distributed lock for test.foo ( lock timeout : 900000 ms, ping interval : 30000 ms, process : ubuntu:20016:1521235700:-4633782234644071543 ) with lockSessionID: 5aac36fbebcdfa1d2b617dde, why: dropCollection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.449-0400 s20016| 2018-03-16T17:28:27.447-0400 D ASIO [conn1] startCommand: RemoteCommand 383 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:57.447-0400 cmd:{ findAndModify: "locks", query: { _id: "test.foo", state: 0 }, update: { $set: { ts: ObjectId('5aac36fbebcdfa1d2b617dde'), state: 2, who: "ubuntu:20016:1521235700:-4633782234644071543:conn1", process: "ubuntu:20016:1521235700:-4633782234644071543", when: new Date(1521235707447), why: "dropCollection" } }, upsert: true, new: true, writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.449-0400 s20016| 2018-03-16T17:28:27.447-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 383 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.454-0400 s20016| 2018-03-16T17:28:27.453-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 383 finished with response: { lastErrorObject: { n: 1, updatedExisting: true }, value: { _id: "test.foo", state: 2, process: "ubuntu:20016:1521235700:-4633782234644071543", ts: ObjectId('5aac36fbebcdfa1d2b617dde'), when: new Date(1521235707447), who: "ubuntu:20016:1521235700:-4633782234644071543:conn1", why: "dropCollection" }, ok: 1.0, operationTime: Timestamp(1521235707, 27), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235707, 27), t: 1 }, lastOpVisible: { ts: Timestamp(1521235707, 27), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235707, 27), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235707, 27), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235707, 27) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.454-0400 s20016| 2018-03-16T17:28:27.453-0400 I SHARDING [conn1] distributed lock 'test.foo' acquired for 'dropCollection', ts : 5aac36fbebcdfa1d2b617dde [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.454-0400 s20016| 2018-03-16T17:28:27.453-0400 I SHARDING [conn1] about to log metadata event into changelog: { _id: "ubuntu-2018-03-16T17:28:27.453-0400-5aac36fbebcdfa1d2b617ddf", server: "ubuntu", clientAddr: "127.0.0.1:50986", time: new Date(1521235707453), what: "dropCollection.start", ns: "test.foo", details: {} } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.455-0400 s20016| 2018-03-16T17:28:27.453-0400 D ASIO [conn1] startCommand: RemoteCommand 385 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:57.453-0400 cmd:{ insert: "changelog", bypassDocumentValidation: false, ordered: true, documents: [ { _id: "ubuntu-2018-03-16T17:28:27.453-0400-5aac36fbebcdfa1d2b617ddf", server: "ubuntu", clientAddr: "127.0.0.1:50986", time: new Date(1521235707453), what: "dropCollection.start", ns: "test.foo", details: {} } ], writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.455-0400 s20016| 2018-03-16T17:28:27.453-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 385 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.457-0400 s20016| 2018-03-16T17:28:27.457-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 385 finished with response: { n: 1, opTime: { ts: Timestamp(1521235707, 28), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235707, 28), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235707, 28), t: 1 }, lastOpVisible: { ts: Timestamp(1521235707, 28), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235707, 28), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235707, 28), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235707, 28) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.458-0400 s20016| 2018-03-16T17:28:27.457-0400 D ASIO [conn1] startCommand: RemoteCommand 387 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:57.457-0400 cmd:{ find: "shards", readConcern: { level: "majority", afterOpTime: { ts: Timestamp(1521235707, 28), t: 1 } }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.458-0400 s20016| 2018-03-16T17:28:27.457-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 387 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.459-0400 s20016| 2018-03-16T17:28:27.457-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 387 finished with response: { cursor: { firstBatch: [ { _id: "stale_mongos_updates_and_removes-rs0", host: "stale_mongos_updates_and_removes-rs0/ubuntu:20010", state: 1 }, { _id: "stale_mongos_updates_and_removes-rs1", host: "stale_mongos_updates_and_removes-rs1/ubuntu:20011", state: 1 } ], id: 0, ns: "config.shards" }, ok: 1.0, operationTime: Timestamp(1521235707, 28), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235707, 28), t: 1 }, lastOpVisible: { ts: Timestamp(1521235707, 28), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235707, 28), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235707, 28), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235707, 28) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.459-0400 s20016| 2018-03-16T17:28:27.457-0400 D SHARDING [conn1] dropCollection test.foo started [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.459-0400 s20016| 2018-03-16T17:28:27.457-0400 D ASIO [conn1] startCommand: RemoteCommand 389 -- target:ubuntu:20010 db:test cmd:{ drop: "foo" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.459-0400 s20016| 2018-03-16T17:28:27.457-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 389 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.459-0400 d20010| 2018-03-16T17:28:27.457-0400 I COMMAND [conn21] CMD: drop test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.460-0400 d20010| 2018-03-16T17:28:27.457-0400 I STORAGE [conn21] dropCollection: test.foo (53837380-bed7-429b-ba96-f32fe994b394) - renaming to drop-pending collection: test.system.drop.1521235707i29t1.foo with drop optime { ts: Timestamp(1521235707, 29), t: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.460-0400 s20016| 2018-03-16T17:28:27.460-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 389 finished with response: { ns: "test.foo", nIndexesWas: 2, ok: 1.0, operationTime: Timestamp(1521235707, 29), $gleStats: { lastOpTime: { ts: Timestamp(1521235707, 29), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235707, 29), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235706, 108), $configServerState: { opTime: { ts: Timestamp(1521235707, 28), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.460-0400 s20016| 2018-03-16T17:28:27.460-0400 D ASIO [conn1] startCommand: RemoteCommand 391 -- target:ubuntu:20011 db:test cmd:{ drop: "foo" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.460-0400 s20016| 2018-03-16T17:28:27.460-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 391 on host ubuntu:20011 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.461-0400 d20011| 2018-03-16T17:28:27.460-0400 I COMMAND [conn21] CMD: drop test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.461-0400 d20011| 2018-03-16T17:28:27.460-0400 I STORAGE [conn21] dropCollection: test.foo (53837380-bed7-429b-ba96-f32fe994b394) - renaming to drop-pending collection: test.system.drop.1521235707i30t1.foo with drop optime { ts: Timestamp(1521235707, 30), t: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.463-0400 s20016| 2018-03-16T17:28:27.462-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 391 finished with response: { ns: "test.foo", nIndexesWas: 2, ok: 1.0, operationTime: Timestamp(1521235707, 30), $gleStats: { lastOpTime: { ts: Timestamp(1521235707, 30), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235707, 30), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235707, 21), $configServerState: { opTime: { ts: Timestamp(1521235707, 28), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.463-0400 s20016| 2018-03-16T17:28:27.462-0400 D SHARDING [conn1] dropCollection test.foo shard data deleted [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.463-0400 s20016| 2018-03-16T17:28:27.462-0400 D ASIO [conn1] startCommand: RemoteCommand 393 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:57.462-0400 cmd:{ delete: "chunks", bypassDocumentValidation: false, ordered: true, deletes: [ { q: { ns: "test.foo" }, limit: 0 } ], writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.463-0400 s20016| 2018-03-16T17:28:27.462-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 393 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.470-0400 s20016| 2018-03-16T17:28:27.469-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 393 finished with response: { n: 2, opTime: { ts: Timestamp(1521235707, 32), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235707, 32), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235707, 32), t: 1 }, lastOpVisible: { ts: Timestamp(1521235707, 32), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235707, 32), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235707, 32), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235707, 32) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.470-0400 s20016| 2018-03-16T17:28:27.469-0400 D SHARDING [conn1] dropCollection test.foo chunk data deleted [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.470-0400 s20016| 2018-03-16T17:28:27.469-0400 D ASIO [conn1] startCommand: RemoteCommand 395 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:57.469-0400 cmd:{ update: "collections", bypassDocumentValidation: false, ordered: true, updates: [ { q: { _id: "test.foo" }, u: { _id: "test.foo", lastmodEpoch: ObjectId('000000000000000000000000'), lastmod: new Date(1521235707469), dropped: true }, multi: false, upsert: false } ], writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.471-0400 s20016| 2018-03-16T17:28:27.469-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 395 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.475-0400 s20016| 2018-03-16T17:28:27.474-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 395 finished with response: { n: 1, nModified: 1, opTime: { ts: Timestamp(1521235707, 33), t: 1 }, electionId: ObjectId('7fffffff0000000000000001'), ok: 1.0, operationTime: Timestamp(1521235707, 33), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235707, 33), t: 1 }, lastOpVisible: { ts: Timestamp(1521235707, 33), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235707, 33), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235707, 33), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235707, 33) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.475-0400 s20016| 2018-03-16T17:28:27.474-0400 D SHARDING [conn1] dropCollection test.foo collection marked as dropped [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.476-0400 s20016| 2018-03-16T17:28:27.474-0400 D ASIO [conn1] startCommand: RemoteCommand 397 -- target:ubuntu:20010 db:admin cmd:{ setShardVersion: "test.foo", init: false, authoritative: true, configdb: "stale_mongos_updates_and_removes-configRS/ubuntu:20012,ubuntu:20013,ubuntu:20014", shard: "stale_mongos_updates_and_removes-rs0", shardHost: "stale_mongos_updates_and_removes-rs0/ubuntu:20010", version: Timestamp(0, 0), versionEpoch: ObjectId('000000000000000000000000'), noConnectionVersioning: true } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.476-0400 s20016| 2018-03-16T17:28:27.474-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 397 on host ubuntu:20010 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.476-0400 d20010| 2018-03-16T17:28:27.474-0400 I SHARDING [conn21] XXXX in set shard version cmd obj is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.477-0400 d20010| 2018-03-16T17:28:27.475-0400 I SHARDING [conn21] { setShardVersion: "test.foo", init: false, authoritative: true, configdb: "stale_mongos_updates_and_removes-configRS/ubuntu:20012,ubuntu:20013,ubuntu:20014", shard: "stale_mongos_updates_and_removes-rs0", shardHost: "stale_mongos_updates_and_removes-rs0/ubuntu:20010", version: Timestamp(0, 0), versionEpoch: ObjectId('000000000000000000000000'), noConnectionVersioning: true, $clusterTime: { clusterTime: Timestamp(1521235707, 33), signature: { hash: BinData(0, CEFAA93570840EC6E37F07AD52BBF4B956967DEE), keyId: 6533657568122765331 } }, $client: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "0.0.0" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "17.10" }, mongos: { host: "ubuntu:20016", client: "127.0.0.1:50986", version: "3.6.3" } }, $configServerState: { opTime: { ts: Timestamp(1521235707, 33), t: 1 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.477-0400 d20010| 2018-03-16T17:28:27.475-0400 I SHARDING [conn21] XXXX requested version is [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.477-0400 d20010| 2018-03-16T17:28:27.475-0400 I SHARDING [conn21] 0|0||000000000000000000000000 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.477-0400 d20010| 2018-03-16T17:28:27.475-0400 I SHARDING [conn21] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.477-0400 d20010| 2018-03-16T17:28:27.475-0400 I SHARDING [conn21] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.477-0400 d20010| 2018-03-16T17:28:27.475-0400 I SHARDING [conn21] xxx routing info needs refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.477-0400 d20010| 2018-03-16T17:28:27.475-0400 I SHARDING [conn21] xxx waiting if !refreshNot [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.507-0400 d20010| 2018-03-16T17:28:27.506-0400 I REPL [replication-0] Completing collection drop for test.system.drop.1521235707i29t1.foo with drop optime { ts: Timestamp(1521235707, 29), t: 1 } (notification optime: { ts: Timestamp(1521235707, 29), t: 1 }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.507-0400 d20010| 2018-03-16T17:28:27.506-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Cache loader remotely refreshed for collection test.foo from collection version 2|1||5aac36f988c1f30185b7b1df and found collection version 3|0||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.507-0400 d20010| 2018-03-16T17:28:27.507-0400 I STORAGE [replication-0] Finishing collection drop for test.system.drop.1521235707i29t1.foo (53837380-bed7-429b-ba96-f32fe994b394). [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.508-0400 d20010| 2018-03-16T17:28:27.507-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Cache loader found enqueued metadata from 2|1||5aac36f988c1f30185b7b1df to 3|0||5aac36f988c1f30185b7b1df and persisted metadata from 2|1||5aac36f988c1f30185b7b1df to 2|1||5aac36f988c1f30185b7b1df, GTE cache version 2|1||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.508-0400 d20010| 2018-03-16T17:28:27.507-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx is okay [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.508-0400 d20010| 2018-03-16T17:28:27.507-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx finsihing refresh [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.508-0400 d20010| 2018-03-16T17:28:27.507-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh callback [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.508-0400 d20010| 2018-03-16T17:28:27.507-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx on refresh completed [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.508-0400 d20010| 2018-03-16T17:28:27.507-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] Refresh for collection test.foo took 92 ms and found version 3|0||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.509-0400 d20010| 2018-03-16T17:28:27.507-0400 I SHARDING [ConfigServerCatalogCacheLoader-0] xxx marking as status ok [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.509-0400 d20010| 2018-03-16T17:28:27.507-0400 I SHARDING [conn43] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.509-0400 d20010| 2018-03-16T17:28:27.507-0400 I SHARDING [conn43] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.509-0400 d20010| 2018-03-16T17:28:27.507-0400 I SHARDING [conn21] xxx looking at collection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.509-0400 d20010| 2018-03-16T17:28:27.507-0400 I SHARDING [conn43] Updating collection metadata for test.foo from collection version: 2|1||5aac36f988c1f30185b7b1df, shard version: 2|1||5aac36f988c1f30185b7b1df to collection version: 3|0||5aac36f988c1f30185b7b1df, shard version: 3|0||5aac36f988c1f30185b7b1df [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.509-0400 d20010| 2018-03-16T17:28:27.507-0400 I SHARDING [conn21] test.foo [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.510-0400 s20016| 2018-03-16T17:28:27.507-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 397 finished with response: { oldVersion: Timestamp(0, 0), oldVersionEpoch: ObjectId('000000000000000000000000'), ns: "test.foo", globalVersion: Timestamp(3, 0), globalVersionEpoch: ObjectId('5aac36f988c1f30185b7b1df'), reloadConfig: true, version: Timestamp(0, 0), versionEpoch: ObjectId('000000000000000000000000'), origVersion: Timestamp(0, 0), origVersionEpoch: ObjectId('000000000000000000000000'), ok: 0.0, errmsg: "requested shard version differs from config shard version for test.foo, requested version is 0|0||000000000000000000000000 but found version 3|0||5aac...", operationTime: Timestamp(1521235707, 35), $gleStats: { lastOpTime: { ts: Timestamp(1521235707, 29), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235707, 36), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235707, 29), $configServerState: { opTime: { ts: Timestamp(1521235707, 33), t: 1 } } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.510-0400 s20016| 2018-03-16T17:28:27.507-0400 D - [conn1] User Assertion: 8:requested shard version differs from config shard version for test.foo, requested version is 0|0||000000000000000000000000 but found version 3|0||5aac36f988c1f30185b7b1df src/mongo/s/commands/cluster_drop_cmd.cpp 109 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.511-0400 s20016| 2018-03-16T17:28:27.507-0400 D ASIO [conn1] startCommand: RemoteCommand 399 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:57.507-0400 cmd:{ findAndModify: "locks", query: { ts: ObjectId('5aac36fbebcdfa1d2b617dde') }, update: { $set: { state: 0 } }, writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.511-0400 s20016| 2018-03-16T17:28:27.508-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 399 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.512-0400 d20011| 2018-03-16T17:28:27.512-0400 I REPL [replication-0] Completing collection drop for test.system.drop.1521235707i30t1.foo with drop optime { ts: Timestamp(1521235707, 30), t: 1 } (notification optime: { ts: Timestamp(1521235707, 30), t: 1 }) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.512-0400 d20011| 2018-03-16T17:28:27.512-0400 I STORAGE [replication-0] Finishing collection drop for test.system.drop.1521235707i30t1.foo (53837380-bed7-429b-ba96-f32fe994b394). [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.513-0400 s20016| 2018-03-16T17:28:27.512-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 399 finished with response: { lastErrorObject: { n: 1, updatedExisting: true }, value: { _id: "test.foo", state: 2, process: "ubuntu:20016:1521235700:-4633782234644071543", ts: ObjectId('5aac36fbebcdfa1d2b617dde'), when: new Date(1521235707447), who: "ubuntu:20016:1521235700:-4633782234644071543:conn1", why: "dropCollection" }, ok: 1.0, operationTime: Timestamp(1521235707, 37), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235707, 37), t: 1 }, lastOpVisible: { ts: Timestamp(1521235707, 37), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235707, 37), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235707, 37), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235707, 37) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.513-0400 s20016| 2018-03-16T17:28:27.512-0400 I SHARDING [conn1] distributed lock with ts: 5aac36fbebcdfa1d2b617dde' unlocked. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.514-0400 s20016| 2018-03-16T17:28:27.512-0400 D ASIO [conn1] startCommand: RemoteCommand 401 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:57.512-0400 cmd:{ findAndModify: "locks", query: { ts: ObjectId('5aac36fbebcdfa1d2b617ddd') }, update: { $set: { state: 0 } }, writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.514-0400 s20016| 2018-03-16T17:28:27.512-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 401 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.525-0400 s20016| 2018-03-16T17:28:27.523-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 401 finished with response: { lastErrorObject: { n: 1, updatedExisting: true }, value: { _id: "test", state: 2, process: "ubuntu:20016:1521235700:-4633782234644071543", ts: ObjectId('5aac36fbebcdfa1d2b617ddd'), when: new Date(1521235707443), who: "ubuntu:20016:1521235700:-4633782234644071543:conn1", why: "dropCollection" }, ok: 1.0, operationTime: Timestamp(1521235707, 38), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235707, 38), t: 1 }, lastOpVisible: { ts: Timestamp(1521235707, 38), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235707, 38), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235707, 38), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235707, 38) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.525-0400 s20016| 2018-03-16T17:28:27.524-0400 I SHARDING [conn1] distributed lock with ts: 5aac36fbebcdfa1d2b617ddd' unlocked. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.525-0400 s20016| 2018-03-16T17:28:27.524-0400 D ASIO [conn1] startCommand: RemoteCommand 403 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:28:57.524-0400 cmd:{ findAndModify: "locks", query: { ts: ObjectId('5aac36fbebcdfa1d2b617ddc') }, update: { $set: { state: 0 } }, writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.525-0400 s20016| 2018-03-16T17:28:27.524-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 403 on host ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.529-0400 s20016| 2018-03-16T17:28:27.527-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 403 finished with response: { lastErrorObject: { n: 1, updatedExisting: true }, value: { _id: "test-movePrimary", state: 2, process: "ubuntu:20016:1521235700:-4633782234644071543", ts: ObjectId('5aac36fbebcdfa1d2b617ddc'), when: new Date(1521235707439), who: "ubuntu:20016:1521235700:-4633782234644071543:conn1", why: "dropCollection" }, ok: 1.0, operationTime: Timestamp(1521235707, 39), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235707, 39), t: 1 }, lastOpVisible: { ts: Timestamp(1521235707, 39), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: { ts: Timestamp(1521235707, 39), t: 1 }, electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235707, 39), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235707, 39) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.529-0400 s20016| 2018-03-16T17:28:27.527-0400 I SHARDING [conn1] distributed lock with ts: 5aac36fbebcdfa1d2b617ddc' unlocked. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.529-0400 2018-03-16T17:28:27.528-0400 E QUERY [js] Error: drop failed: { [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.529-0400 "ok" : 0, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.529-0400 "errmsg" : "requested shard version differs from config shard version for test.foo, requested version is 0|0||000000000000000000000000 but found version 3|0||5aac36f988c1f30185b7b1df", [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.529-0400 "code" : 8, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.529-0400 "codeName" : "UnknownError", [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.529-0400 "$clusterTime" : { [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.529-0400 "clusterTime" : Timestamp(1521235707, 39), [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.530-0400 "signature" : { [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.530-0400 "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="), [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.530-0400 "keyId" : NumberLong(0) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.530-0400 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.530-0400 }, [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.530-0400 "operationTime" : Timestamp(1521235707, 39) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.530-0400 } : [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.530-0400 _getErrorWithCode@src/mongo/shell/utils.js:25:13 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.530-0400 DBCollection.prototype.drop@src/mongo/shell/collection.js:702:1 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.530-0400 resetCollection@jstests/sharding/stale_mongos_updates_and_removes.js:26:16 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.530-0400 makeStaleMongosTargetMultipleShards@jstests/sharding/stale_mongos_updates_and_removes.js:51:9 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.531-0400 checkAllRemoveQueries/doRemove@jstests/sharding/stale_mongos_updates_and_removes.js:97:13 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.531-0400 checkAllRemoveQueries@jstests/sharding/stale_mongos_updates_and_removes.js:117:9 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.531-0400 @jstests/sharding/stale_mongos_updates_and_removes.js:223:5 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.531-0400 @jstests/sharding/stale_mongos_updates_and_removes.js:20:2 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.531-0400 failed to load: jstests/sharding/stale_mongos_updates_and_removes.js [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.531-0400 s20016| 2018-03-16T17:28:27.528-0400 I CONTROL [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.531-0400 s20016| 2018-03-16T17:28:27.529-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.532-0400 s20016| 2018-03-16T17:28:27.529-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.532-0400 s20016| 2018-03-16T17:28:27.529-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.532-0400 s20016| 2018-03-16T17:28:27.529-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.532-0400 s20016| 2018-03-16T17:28:27.529-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.532-0400 s20016| 2018-03-16T17:28:27.529-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.532-0400 s20016| 2018-03-16T17:28:27.529-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.533-0400 s20016| 2018-03-16T17:28:27.529-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.533-0400 s20016| 2018-03-16T17:28:27.529-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.533-0400 s20016| 2018-03-16T17:28:27.529-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.533-0400 s20016| 2018-03-16T17:28:27.529-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.533-0400 s20016| 2018-03-16T17:28:27.529-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.533-0400 s20016| 2018-03-16T17:28:27.529-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.533-0400 s20016| 2018-03-16T17:28:27.529-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.534-0400 s20016| 2018-03-16T17:28:27.529-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.534-0400 s20016| 2018-03-16T17:28:27.530-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.534-0400 s20016| 2018-03-16T17:28:27.530-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.534-0400 s20016| 2018-03-16T17:28:27.530-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.534-0400 s20016| 2018-03-16T17:28:27.530-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.534-0400 s20016| 2018-03-16T17:28:27.530-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.534-0400 s20016| 2018-03-16T17:28:27.530-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.535-0400 s20016| 2018-03-16T17:28:27.530-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.535-0400 s20016| 2018-03-16T17:28:27.530-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.535-0400 s20016| 2018-03-16T17:28:27.530-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.535-0400 s20016| 2018-03-16T17:28:27.530-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.535-0400 s20016| 2018-03-16T17:28:27.530-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.535-0400 s20016| 2018-03-16T17:28:27.530-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.536-0400 s20016| 2018-03-16T17:28:27.530-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.536-0400 s20016| 2018-03-16T17:28:27.530-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.536-0400 s20016| 2018-03-16T17:28:27.530-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.536-0400 s20016| 2018-03-16T17:28:27.530-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.536-0400 s20016| 2018-03-16T17:28:27.530-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.536-0400 s20016| 2018-03-16T17:28:27.530-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.536-0400 s20016| 2018-03-16T17:28:27.531-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.537-0400 s20016| 2018-03-16T17:28:27.531-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.537-0400 s20016| 2018-03-16T17:28:27.531-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.537-0400 s20016| 2018-03-16T17:28:27.531-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.537-0400 s20016| 2018-03-16T17:28:27.531-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.537-0400 s20016| 2018-03-16T17:28:27.531-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.537-0400 s20016| 2018-03-16T17:28:27.531-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.537-0400 s20016| 2018-03-16T17:28:27.531-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.538-0400 s20016| 2018-03-16T17:28:27.531-0400 D SHARDING [signalProcessingThread] ShardingCatalogClientImpl::shutDown() called. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.538-0400 s20016| 2018-03-16T17:28:27.531-0400 W SHARDING [signalProcessingThread] error encountered while cleaning up distributed ping entry for ubuntu:20016:1521235700:-4633782234644071543 :: caused by :: InterruptedAtShutdown: interrupted at shutdown [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.538-0400 s20016| 2018-03-16T17:28:27.531-0400 I FTDC [signalProcessingThread] Shutting down full-time diagnostic data capture [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.538-0400 s20016| 2018-03-16T17:28:27.531-0400 I CONTROL [signalProcessingThread] shutting down with code:0 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.538-0400 c20012| 2018-03-16T17:28:27.533-0400 I NETWORK [conn45] end connection 127.0.0.1:60894 (27 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.538-0400 d20010| 2018-03-16T17:28:27.533-0400 I NETWORK [conn63] end connection 127.0.0.1:56548 (53 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.539-0400 d20010| 2018-03-16T17:28:27.533-0400 I NETWORK [conn62] end connection 127.0.0.1:56546 (52 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.539-0400 c20012| 2018-03-16T17:28:27.533-0400 I NETWORK [conn44] end connection 127.0.0.1:60880 (26 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.539-0400 d20011| 2018-03-16T17:28:27.533-0400 I NETWORK [conn26] end connection 127.0.0.1:55188 (16 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.539-0400 d20011| 2018-03-16T17:28:27.533-0400 I NETWORK [conn24] end connection 127.0.0.1:55184 (14 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.539-0400 d20011| 2018-03-16T17:28:27.533-0400 I NETWORK [conn25] end connection 127.0.0.1:55186 (15 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.539-0400 d20010| 2018-03-16T17:28:27.533-0400 I NETWORK [conn61] end connection 127.0.0.1:56534 (51 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.540-0400 d20010| 2018-03-16T17:28:27.533-0400 I NETWORK [conn60] end connection 127.0.0.1:56532 (50 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.540-0400 d20010| 2018-03-16T17:28:27.533-0400 I NETWORK [conn59] end connection 127.0.0.1:56530 (49 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.540-0400 d20010| 2018-03-16T17:28:27.533-0400 I NETWORK [conn58] end connection 127.0.0.1:56528 (48 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.540-0400 d20010| 2018-03-16T17:28:27.533-0400 I NETWORK [conn56] end connection 127.0.0.1:56524 (47 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.540-0400 d20010| 2018-03-16T17:28:27.533-0400 I NETWORK [conn55] end connection 127.0.0.1:56522 (46 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.540-0400 d20010| 2018-03-16T17:28:27.533-0400 I NETWORK [conn52] end connection 127.0.0.1:56516 (42 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.540-0400 d20010| 2018-03-16T17:28:27.533-0400 I NETWORK [conn53] end connection 127.0.0.1:56518 (44 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.541-0400 d20010| 2018-03-16T17:28:27.533-0400 I NETWORK [conn54] end connection 127.0.0.1:56520 (43 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.541-0400 d20010| 2018-03-16T17:28:27.533-0400 I NETWORK [conn51] end connection 127.0.0.1:56514 (41 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.541-0400 d20010| 2018-03-16T17:28:27.533-0400 I NETWORK [conn47] end connection 127.0.0.1:56506 (37 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.541-0400 d20010| 2018-03-16T17:28:27.533-0400 I NETWORK [conn45] end connection 127.0.0.1:56502 (35 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.541-0400 d20010| 2018-03-16T17:28:27.533-0400 I NETWORK [conn44] end connection 127.0.0.1:56500 (34 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.541-0400 d20010| 2018-03-16T17:28:27.533-0400 I NETWORK [conn48] end connection 127.0.0.1:56508 (38 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.541-0400 c20012| 2018-03-16T17:28:27.533-0400 I NETWORK [conn43] end connection 127.0.0.1:60832 (25 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.542-0400 d20010| 2018-03-16T17:28:27.534-0400 I NETWORK [conn41] end connection 127.0.0.1:56486 (33 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.542-0400 d20010| 2018-03-16T17:28:27.533-0400 I NETWORK [conn46] end connection 127.0.0.1:56504 (36 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.542-0400 d20010| 2018-03-16T17:28:27.534-0400 I NETWORK [conn38] end connection 127.0.0.1:56480 (30 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.542-0400 d20010| 2018-03-16T17:28:27.533-0400 I NETWORK [conn49] end connection 127.0.0.1:56510 (39 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.542-0400 d20010| 2018-03-16T17:28:27.534-0400 I NETWORK [conn36] end connection 127.0.0.1:56476 (28 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.542-0400 d20010| 2018-03-16T17:28:27.534-0400 I NETWORK [conn40] end connection 127.0.0.1:56484 (32 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.542-0400 d20010| 2018-03-16T17:28:27.534-0400 I NETWORK [conn39] end connection 127.0.0.1:56482 (31 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.543-0400 d20010| 2018-03-16T17:28:27.533-0400 I NETWORK [conn50] end connection 127.0.0.1:56512 (40 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.543-0400 d20010| 2018-03-16T17:28:27.534-0400 I NETWORK [conn30] end connection 127.0.0.1:56464 (22 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.543-0400 d20010| 2018-03-16T17:28:27.534-0400 I NETWORK [conn29] end connection 127.0.0.1:56462 (21 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.543-0400 d20010| 2018-03-16T17:28:27.534-0400 I NETWORK [conn28] end connection 127.0.0.1:56460 (20 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.543-0400 d20010| 2018-03-16T17:28:27.534-0400 I NETWORK [conn34] end connection 127.0.0.1:56472 (26 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.543-0400 d20010| 2018-03-16T17:28:27.534-0400 I NETWORK [conn27] end connection 127.0.0.1:56458 (19 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.544-0400 d20010| 2018-03-16T17:28:27.534-0400 I NETWORK [conn26] end connection 127.0.0.1:56456 (18 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.544-0400 d20010| 2018-03-16T17:28:27.534-0400 I NETWORK [conn31] end connection 127.0.0.1:56466 (23 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.544-0400 d20010| 2018-03-16T17:28:27.534-0400 I NETWORK [conn24] end connection 127.0.0.1:56452 (17 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.544-0400 d20010| 2018-03-16T17:28:27.534-0400 I NETWORK [conn25] end connection 127.0.0.1:56454 (15 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.544-0400 d20011| 2018-03-16T17:28:27.534-0400 I NETWORK [conn21] end connection 127.0.0.1:55090 (13 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.544-0400 d20010| 2018-03-16T17:28:27.534-0400 I NETWORK [conn35] end connection 127.0.0.1:56474 (27 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.544-0400 d20011| 2018-03-16T17:28:27.534-0400 I NETWORK [conn20] end connection 127.0.0.1:55086 (12 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.545-0400 d20010| 2018-03-16T17:28:27.534-0400 I NETWORK [conn33] end connection 127.0.0.1:56470 (25 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.545-0400 d20010| 2018-03-16T17:28:27.534-0400 I NETWORK [conn32] end connection 127.0.0.1:56468 (24 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.545-0400 d20010| 2018-03-16T17:28:27.534-0400 I NETWORK [conn37] end connection 127.0.0.1:56478 (29 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.545-0400 c20013| 2018-03-16T17:28:27.534-0400 I NETWORK [conn14] end connection 127.0.0.1:47076 (12 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.545-0400 c20012| 2018-03-16T17:28:27.534-0400 I NETWORK [conn25] end connection 127.0.0.1:60686 (24 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.545-0400 d20010| 2018-03-16T17:28:27.534-0400 I NETWORK [conn23] end connection 127.0.0.1:56450 (16 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.545-0400 d20010| 2018-03-16T17:28:27.533-0400 I NETWORK [conn57] end connection 127.0.0.1:56526 (45 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.546-0400 d20010| 2018-03-16T17:28:27.534-0400 I NETWORK [conn22] end connection 127.0.0.1:56448 (14 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.546-0400 d20010| 2018-03-16T17:28:27.534-0400 I NETWORK [conn21] end connection 127.0.0.1:56444 (13 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.546-0400 c20012| 2018-03-16T17:28:27.534-0400 I NETWORK [conn26] end connection 127.0.0.1:60688 (23 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.546-0400 d20010| 2018-03-16T17:28:27.534-0400 I NETWORK [conn20] end connection 127.0.0.1:56440 (12 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.546-0400 c20012| 2018-03-16T17:28:27.534-0400 I NETWORK [conn23] end connection 127.0.0.1:60682 (21 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.546-0400 c20012| 2018-03-16T17:28:27.534-0400 I NETWORK [conn24] end connection 127.0.0.1:60684 (22 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.546-0400 c20014| 2018-03-16T17:28:27.534-0400 I NETWORK [conn16] end connection 127.0.0.1:44498 (13 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.547-0400 c20014| 2018-03-16T17:28:27.534-0400 I NETWORK [conn15] end connection 127.0.0.1:44464 (12 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.547-0400 c20013| 2018-03-16T17:28:27.534-0400 I NETWORK [conn13] end connection 127.0.0.1:47044 (11 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.547-0400 c20012| 2018-03-16T17:28:27.534-0400 I NETWORK [conn22] end connection 127.0.0.1:60676 (20 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.547-0400 d20011| 2018-03-16T17:28:27.534-0400 I CONTROL [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.547-0400 d20011| 2018-03-16T17:28:27.534-0400 I NETWORK [signalProcessingThread] shutdown: going to close listening sockets... [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.547-0400 d20011| 2018-03-16T17:28:27.534-0400 I NETWORK [signalProcessingThread] removing socket file: /tmp/mongodb-20011.sock [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.548-0400 d20011| 2018-03-16T17:28:27.534-0400 I REPL [signalProcessingThread] shutdown: removing checkpointTimestamp collection... [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.548-0400 d20011| 2018-03-16T17:28:27.534-0400 I STORAGE [signalProcessingThread] Finishing collection drop for local.replset.checkpointTimestamp (b9cac20d-e504-4527-957c-2d50129882e2). [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.548-0400 d20011| 2018-03-16T17:28:27.536-0400 I REPL [signalProcessingThread] shutting down replication subsystems [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.548-0400 d20011| 2018-03-16T17:28:27.536-0400 I REPL [signalProcessingThread] Stopping replication reporter thread [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.548-0400 d20011| 2018-03-16T17:28:27.536-0400 I REPL [signalProcessingThread] Stopping replication fetcher thread [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.548-0400 d20011| 2018-03-16T17:28:27.536-0400 I REPL [signalProcessingThread] Stopping replication applier thread [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.555-0400 d20011| 2018-03-16T17:28:27.555-0400 I REPL [signalProcessingThread] Stopping replication storage threads [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.556-0400 d20011| 2018-03-16T17:28:27.555-0400 W SHARDING [signalProcessingThread] error encountered while cleaning up distributed ping entry for ubuntu:20011:1521235703:1064903155863675804 :: caused by :: ShutdownInProgress: Shutdown in progress [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.556-0400 d20011| 2018-03-16T17:28:27.556-0400 W SHARDING [signalProcessingThread] cant reload ShardRegistry :: caused by :: CallbackCanceled: Callback canceled [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.556-0400 d20011| 2018-03-16T17:28:27.556-0400 I FTDC [signalProcessingThread] Shutting down full-time diagnostic data capture [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.558-0400 d20011| 2018-03-16T17:28:27.558-0400 I STORAGE [WTOplogJournalThread] oplog journal thread loop shutting down [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.558-0400 d20011| 2018-03-16T17:28:27.558-0400 I STORAGE [signalProcessingThread] WiredTigerKVEngine shutting down [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.880-0400 d20011| 2018-03-16T17:28:27.880-0400 I STORAGE [signalProcessingThread] WiredTiger message [1521235707:880263][24218:0x7f41c2b86700], txn-recover: Main recovery loop: starting at 1/169728 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.944-0400 c20012| 2018-03-16T17:28:27.944-0400 I SHARDING [Balancer] caught exception while doing balance: interrupted at shutdown [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.944-0400 c20012| 2018-03-16T17:28:27.944-0400 I STORAGE [Balancer] createCollection: config.actionlog with generated UUID: a2454b57-c77b-48a4-ba8a-856bf8992f2c [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.962-0400 c20014| 2018-03-16T17:28:27.962-0400 I STORAGE [repl writer worker 14] createCollection: config.actionlog with provided UUID: a2454b57-c77b-48a4-ba8a-856bf8992f2c [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.962-0400 c20013| 2018-03-16T17:28:27.962-0400 I STORAGE [repl writer worker 1] createCollection: config.actionlog with provided UUID: a2454b57-c77b-48a4-ba8a-856bf8992f2c [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.977-0400 d20011| 2018-03-16T17:28:27.977-0400 I STORAGE [signalProcessingThread] WiredTiger message [1521235707:977617][24218:0x7f41c2b86700], txn-recover: Recovering log 1 through 2 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:27.988-0400 c20012| 2018-03-16T17:28:27.988-0400 I SHARDING [Balancer] about to log metadata event into actionlog: { _id: "ubuntu-2018-03-16T17:28:27.988-0400-5aac36fb88c1f30185b7b363", server: "ubuntu", clientAddr: "", time: new Date(1521235707988), what: "balancer.round", ns: "", details: { executionTimeMillis: 1, errorOccured: true, errmsg: "interrupted at shutdown" } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.038-0400 d20011| 2018-03-16T17:28:28.038-0400 I STORAGE [signalProcessingThread] WiredTiger message [1521235708:38652][24218:0x7f41c2b86700], txn-recover: Recovering log 2 through 2 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.071-0400 c20012| 2018-03-16T17:28:28.071-0400 D REPL_HB [replexec-1] Sending heartbeat (requestId: 120) to ubuntu:20013, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20012", fromId: 0, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.071-0400 c20012| 2018-03-16T17:28:28.071-0400 D REPL_HB [replexec-2] Sending heartbeat (requestId: 121) to ubuntu:20014, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20012", fromId: 0, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.072-0400 c20013| 2018-03-16T17:28:28.071-0400 D REPL_HB [conn3] Received heartbeat request from ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20012", fromId: 0, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235707, 41), signature: { hash: BinData(0, CEFAA93570840EC6E37F07AD52BBF4B956967DEE), keyId: 6533657568122765331 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.072-0400 c20013| 2018-03-16T17:28:28.071-0400 D REPL_HB [conn3] Processed heartbeat from ubuntu:20012 and generated response, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", syncingTo: "ubuntu:20012", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235707, 41), t: 1 }, opTime: { ts: Timestamp(1521235707, 41), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.073-0400 c20014| 2018-03-16T17:28:28.071-0400 D REPL_HB [conn4] Received heartbeat request from ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20012", fromId: 0, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235707, 41), signature: { hash: BinData(0, CEFAA93570840EC6E37F07AD52BBF4B956967DEE), keyId: 6533657568122765331 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.073-0400 c20014| 2018-03-16T17:28:28.071-0400 D REPL_HB [conn4] Processed heartbeat from ubuntu:20012 and generated response, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", syncingTo: "ubuntu:20012", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235707, 41), t: 1 }, opTime: { ts: Timestamp(1521235707, 41), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.074-0400 c20012| 2018-03-16T17:28:28.071-0400 D REPL_HB [replexec-0] Received response to heartbeat (requestId: 120) from ubuntu:20013, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", syncingTo: "ubuntu:20012", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235707, 41), t: 1 }, opTime: { ts: Timestamp(1521235707, 41), t: 1 }, operationTime: Timestamp(1521235707, 41), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235707, 41), t: 1 }, lastOpVisible: { ts: Timestamp(1521235707, 41), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235707, 41), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235707, 41) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.074-0400 c20012| 2018-03-16T17:28:28.071-0400 D REPL_HB [replexec-0] Scheduling heartbeat to ubuntu:20013 at 2018-03-16T21:28:30.071Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.075-0400 c20012| 2018-03-16T17:28:28.071-0400 D REPL_HB [replexec-1] Received response to heartbeat (requestId: 121) from ubuntu:20014, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", syncingTo: "ubuntu:20012", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235707, 41), t: 1 }, opTime: { ts: Timestamp(1521235707, 41), t: 1 }, operationTime: Timestamp(1521235707, 41), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235707, 41), t: 1 }, lastOpVisible: { ts: Timestamp(1521235707, 41), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235707, 41), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235707, 41) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.075-0400 c20012| 2018-03-16T17:28:28.071-0400 D REPL_HB [replexec-1] Scheduling heartbeat to ubuntu:20014 at 2018-03-16T21:28:30.071Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.096-0400 d20011| 2018-03-16T17:28:28.095-0400 I STORAGE [signalProcessingThread] WiredTiger message [1521235708:95783][24218:0x7f41c2b86700], txn-recover: Set global recovery timestamp: 0 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.190-0400 d20011| 2018-03-16T17:28:28.190-0400 I STORAGE [signalProcessingThread] shutdown: removing fs lock... [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.191-0400 d20011| 2018-03-16T17:28:28.190-0400 I CONTROL [signalProcessingThread] now exiting [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.191-0400 d20011| 2018-03-16T17:28:28.190-0400 I CONTROL [signalProcessingThread] shutting down with code:0 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.195-0400 d20010| 2018-03-16T17:28:28.195-0400 I NETWORK [conn43] end connection 127.0.0.1:56498 (11 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.196-0400 d20010| 2018-03-16T17:28:28.195-0400 I NETWORK [conn42] end connection 127.0.0.1:56496 (10 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.196-0400 c20012| 2018-03-16T17:28:28.195-0400 I NETWORK [conn34] end connection 127.0.0.1:60750 (19 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.196-0400 c20014| 2018-03-16T17:28:28.195-0400 I NETWORK [conn22] end connection 127.0.0.1:44668 (11 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.196-0400 c20013| 2018-03-16T17:28:28.195-0400 I NETWORK [conn18] end connection 127.0.0.1:47118 (10 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.196-0400 d20010| 2018-03-16T17:28:28.195-0400 I NETWORK [conn13] end connection 127.0.0.1:56410 (9 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.196-0400 c20014| 2018-03-16T17:28:28.195-0400 I NETWORK [conn18] end connection 127.0.0.1:44532 (10 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.197-0400 c20012| 2018-03-16T17:28:28.195-0400 I NETWORK [conn32] end connection 127.0.0.1:60742 (18 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.197-0400 c20012| 2018-03-16T17:28:28.195-0400 I NETWORK [conn33] end connection 127.0.0.1:60748 (17 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.197-0400 c20013| 2018-03-16T17:28:28.195-0400 I NETWORK [conn17] end connection 127.0.0.1:47106 (9 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.197-0400 d20010| 2018-03-16T17:28:28.195-0400 I CONTROL [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.197-0400 d20010| 2018-03-16T17:28:28.195-0400 I NETWORK [signalProcessingThread] shutdown: going to close listening sockets... [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.197-0400 d20010| 2018-03-16T17:28:28.195-0400 I NETWORK [signalProcessingThread] removing socket file: /tmp/mongodb-20010.sock [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.197-0400 d20010| 2018-03-16T17:28:28.196-0400 I REPL [signalProcessingThread] shutdown: removing checkpointTimestamp collection... [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.198-0400 d20010| 2018-03-16T17:28:28.196-0400 I STORAGE [signalProcessingThread] Finishing collection drop for local.replset.checkpointTimestamp (dd78e7aa-9762-4900-a8bc-816f562b70d4). [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.201-0400 d20010| 2018-03-16T17:28:28.201-0400 I REPL [signalProcessingThread] shutting down replication subsystems [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.201-0400 d20010| 2018-03-16T17:28:28.201-0400 I REPL [signalProcessingThread] Stopping replication reporter thread [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.201-0400 d20010| 2018-03-16T17:28:28.201-0400 I REPL [signalProcessingThread] Stopping replication fetcher thread [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.201-0400 d20010| 2018-03-16T17:28:28.201-0400 I REPL [signalProcessingThread] Stopping replication applier thread [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.539-0400 c20013| 2018-03-16T17:28:28.539-0400 D REPL_HB [replexec-2] Sending heartbeat (requestId: 1386) to ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.539-0400 c20012| 2018-03-16T17:28:28.539-0400 D REPL_HB [conn9] Received heartbeat request from ubuntu:20013, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235707, 41), signature: { hash: BinData(0, CEFAA93570840EC6E37F07AD52BBF4B956967DEE), keyId: 6533657568122765331 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.540-0400 c20012| 2018-03-16T17:28:28.539-0400 D REPL_HB [conn9] Processed heartbeat from ubuntu:20013 and generated response, { ok: 1.0, electionTime: new Date(6533657559532830722), state: 1, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235707, 41), t: 1 }, opTime: { ts: Timestamp(1521235707, 41), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.540-0400 c20013| 2018-03-16T17:28:28.539-0400 D REPL_HB [replexec-0] Sending heartbeat (requestId: 1387) to ubuntu:20014, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.541-0400 c20013| 2018-03-16T17:28:28.539-0400 D REPL_HB [replexec-0] Received response to heartbeat (requestId: 1386) from ubuntu:20012, { ok: 1.0, electionTime: new Date(6533657559532830722), state: 1, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235707, 41), t: 1 }, opTime: { ts: Timestamp(1521235707, 41), t: 1 }, operationTime: Timestamp(1521235707, 41), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235707, 41), t: 1 }, lastOpVisible: { ts: Timestamp(1521235707, 41), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235707, 41), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235707, 41) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.541-0400 c20013| 2018-03-16T17:28:28.539-0400 D REPL_HB [replexec-0] Scheduling heartbeat to ubuntu:20012 at 2018-03-16T21:28:30.539Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.541-0400 c20014| 2018-03-16T17:28:28.539-0400 D REPL_HB [conn10] Received heartbeat request from ubuntu:20013, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235707, 41), signature: { hash: BinData(0, CEFAA93570840EC6E37F07AD52BBF4B956967DEE), keyId: 6533657568122765331 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.542-0400 c20014| 2018-03-16T17:28:28.540-0400 D REPL_HB [conn10] Processed heartbeat from ubuntu:20013 and generated response, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", syncingTo: "ubuntu:20012", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235707, 41), t: 1 }, opTime: { ts: Timestamp(1521235707, 41), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.542-0400 c20013| 2018-03-16T17:28:28.540-0400 D REPL_HB [replexec-2] Received response to heartbeat (requestId: 1387) from ubuntu:20014, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", syncingTo: "ubuntu:20012", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235707, 41), t: 1 }, opTime: { ts: Timestamp(1521235707, 41), t: 1 }, operationTime: Timestamp(1521235707, 41), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235707, 41), t: 1 }, lastOpVisible: { ts: Timestamp(1521235707, 41), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235707, 41), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235707, 41) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.543-0400 c20013| 2018-03-16T17:28:28.540-0400 D REPL_HB [replexec-2] Scheduling heartbeat to ubuntu:20014 at 2018-03-16T21:28:30.540Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.583-0400 c20014| 2018-03-16T17:28:28.582-0400 D REPL_HB [replexec-0] Sending heartbeat (requestId: 1377) to ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.583-0400 c20012| 2018-03-16T17:28:28.582-0400 D REPL_HB [conn8] Received heartbeat request from ubuntu:20014, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235707, 41), signature: { hash: BinData(0, CEFAA93570840EC6E37F07AD52BBF4B956967DEE), keyId: 6533657568122765331 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.584-0400 c20012| 2018-03-16T17:28:28.583-0400 D REPL_HB [conn8] Processed heartbeat from ubuntu:20014 and generated response, { ok: 1.0, electionTime: new Date(6533657559532830722), state: 1, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235707, 41), t: 1 }, opTime: { ts: Timestamp(1521235707, 41), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.584-0400 c20014| 2018-03-16T17:28:28.583-0400 D REPL_HB [replexec-3] Received response to heartbeat (requestId: 1377) from ubuntu:20012, { ok: 1.0, electionTime: new Date(6533657559532830722), state: 1, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235707, 41), t: 1 }, opTime: { ts: Timestamp(1521235707, 41), t: 1 }, operationTime: Timestamp(1521235707, 41), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235707, 41), t: 1 }, lastOpVisible: { ts: Timestamp(1521235707, 41), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('7fffffff0000000000000001') }, $clusterTime: { clusterTime: Timestamp(1521235707, 41), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235707, 41) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.585-0400 c20014| 2018-03-16T17:28:28.583-0400 D REPL_HB [replexec-3] Scheduling heartbeat to ubuntu:20012 at 2018-03-16T21:28:30.583Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.585-0400 c20014| 2018-03-16T17:28:28.583-0400 D REPL_HB [replexec-2] Sending heartbeat (requestId: 1379) to ubuntu:20013, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.585-0400 c20013| 2018-03-16T17:28:28.583-0400 D REPL_HB [conn9] Received heartbeat request from ubuntu:20014, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235707, 41), signature: { hash: BinData(0, CEFAA93570840EC6E37F07AD52BBF4B956967DEE), keyId: 6533657568122765331 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.586-0400 c20013| 2018-03-16T17:28:28.583-0400 D REPL_HB [conn9] Processed heartbeat from ubuntu:20014 and generated response, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", syncingTo: "ubuntu:20012", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235707, 41), t: 1 }, opTime: { ts: Timestamp(1521235707, 41), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.586-0400 c20014| 2018-03-16T17:28:28.583-0400 D REPL_HB [replexec-4] Received response to heartbeat (requestId: 1379) from ubuntu:20013, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", syncingTo: "ubuntu:20012", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235707, 41), t: 1 }, opTime: { ts: Timestamp(1521235707, 41), t: 1 }, operationTime: Timestamp(1521235707, 41), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235707, 41), t: 1 }, lastOpVisible: { ts: Timestamp(1521235707, 41), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235707, 41), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235707, 41) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.586-0400 c20014| 2018-03-16T17:28:28.584-0400 D REPL_HB [replexec-4] Scheduling heartbeat to ubuntu:20013 at 2018-03-16T21:28:30.583Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.951-0400 d20010| 2018-03-16T17:28:28.951-0400 I REPL [signalProcessingThread] Stopping replication storage threads [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.952-0400 d20010| 2018-03-16T17:28:28.952-0400 W SHARDING [signalProcessingThread] error encountered while cleaning up distributed ping entry for ubuntu:20010:1521235703:-3756998713987756063 :: caused by :: ShutdownInProgress: Shutdown in progress [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.953-0400 d20010| 2018-03-16T17:28:28.952-0400 W SHARDING [signalProcessingThread] cant reload ShardRegistry :: caused by :: CallbackCanceled: Callback canceled [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.953-0400 d20010| 2018-03-16T17:28:28.952-0400 I FTDC [signalProcessingThread] Shutting down full-time diagnostic data capture [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.955-0400 d20010| 2018-03-16T17:28:28.954-0400 I STORAGE [WTOplogJournalThread] oplog journal thread loop shutting down [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:28.955-0400 d20010| 2018-03-16T17:28:28.954-0400 I STORAGE [signalProcessingThread] WiredTigerKVEngine shutting down [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:29.316-0400 d20010| 2018-03-16T17:28:29.316-0400 I STORAGE [signalProcessingThread] WiredTiger message [1521235709:316372][24148:0x7f1ff3986700], txn-recover: Main recovery loop: starting at 1/185856 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:29.416-0400 d20010| 2018-03-16T17:28:29.415-0400 I STORAGE [signalProcessingThread] WiredTiger message [1521235709:415831][24148:0x7f1ff3986700], txn-recover: Recovering log 1 through 2 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:29.480-0400 d20010| 2018-03-16T17:28:29.480-0400 I STORAGE [signalProcessingThread] WiredTiger message [1521235709:480553][24148:0x7f1ff3986700], txn-recover: Recovering log 2 through 2 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:29.537-0400 d20010| 2018-03-16T17:28:29.537-0400 I STORAGE [signalProcessingThread] WiredTiger message [1521235709:537190][24148:0x7f1ff3986700], txn-recover: Set global recovery timestamp: 0 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:29.618-0400 d20010| 2018-03-16T17:28:29.617-0400 I STORAGE [signalProcessingThread] shutdown: removing fs lock... [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:29.618-0400 d20010| 2018-03-16T17:28:29.617-0400 I CONTROL [signalProcessingThread] now exiting [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:29.618-0400 d20010| 2018-03-16T17:28:29.617-0400 I CONTROL [signalProcessingThread] shutting down with code:0 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:29.623-0400 c20014| 2018-03-16T17:28:29.623-0400 I NETWORK [conn21] end connection 127.0.0.1:44622 (9 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:29.623-0400 c20014| 2018-03-16T17:28:29.623-0400 I NETWORK [conn17] end connection 127.0.0.1:44510 (8 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:29.623-0400 c20012| 2018-03-16T17:28:29.623-0400 I NETWORK [conn30] end connection 127.0.0.1:60730 (16 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:29.623-0400 c20013| 2018-03-16T17:28:29.623-0400 I NETWORK [conn16] end connection 127.0.0.1:47092 (8 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:29.624-0400 c20012| 2018-03-16T17:28:29.623-0400 I NETWORK [conn29] end connection 127.0.0.1:60728 (15 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:29.624-0400 c20012| 2018-03-16T17:28:29.623-0400 I NETWORK [conn28] end connection 127.0.0.1:60722 (14 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:29.624-0400 c20013| 2018-03-16T17:28:29.623-0400 I NETWORK [conn15] end connection 127.0.0.1:47086 (7 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:29.624-0400 c20012| 2018-03-16T17:28:29.623-0400 I CONTROL [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:29.624-0400 c20012| 2018-03-16T17:28:29.623-0400 I SHARDING [Balancer] CSRS balancer is now stopped [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:29.624-0400 c20012| 2018-03-16T17:28:29.623-0400 I NETWORK [signalProcessingThread] shutdown: going to close listening sockets... [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:29.624-0400 c20012| 2018-03-16T17:28:29.623-0400 I NETWORK [signalProcessingThread] removing socket file: /tmp/mongodb-20012.sock [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:29.624-0400 c20012| 2018-03-16T17:28:29.623-0400 I REPL [signalProcessingThread] shutdown: removing checkpointTimestamp collection... [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:29.625-0400 c20012| 2018-03-16T17:28:29.623-0400 I STORAGE [signalProcessingThread] Finishing collection drop for local.replset.checkpointTimestamp (9d960db0-e157-43e7-8353-1bc39bfd0fae). [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:29.625-0400 c20012| 2018-03-16T17:28:29.625-0400 I REPL [signalProcessingThread] shutting down replication subsystems [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:29.625-0400 c20012| 2018-03-16T17:28:29.625-0400 I REPL [signalProcessingThread] Stopping replication reporter thread [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:29.625-0400 c20012| 2018-03-16T17:28:29.625-0400 I REPL [signalProcessingThread] Stopping replication fetcher thread [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:29.625-0400 c20012| 2018-03-16T17:28:29.625-0400 I REPL [signalProcessingThread] Stopping replication applier thread [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:29.642-0400 c20012| 2018-03-16T17:28:29.641-0400 I REPL [signalProcessingThread] Stopping replication storage threads [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:29.642-0400 c20012| 2018-03-16T17:28:29.642-0400 W SHARDING [signalProcessingThread] cant reload ShardRegistry :: caused by :: CallbackCanceled: Callback canceled [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:29.642-0400 c20012| 2018-03-16T17:28:29.642-0400 I FTDC [signalProcessingThread] Shutting down full-time diagnostic data capture [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:29.645-0400 c20012| 2018-03-16T17:28:29.644-0400 I STORAGE [WTOplogJournalThread] oplog journal thread loop shutting down [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:29.645-0400 c20012| 2018-03-16T17:28:29.644-0400 I STORAGE [signalProcessingThread] WiredTigerKVEngine shutting down [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.001-0400 c20012| 2018-03-16T17:28:30.001-0400 I STORAGE [signalProcessingThread] WiredTiger message [1521235710:1016][24291:0x7f6a33604700], txn-recover: Main recovery loop: starting at 1/215168 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.108-0400 c20012| 2018-03-16T17:28:30.108-0400 I STORAGE [signalProcessingThread] WiredTiger message [1521235710:108431][24291:0x7f6a33604700], txn-recover: Recovering log 1 through 2 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.176-0400 c20012| 2018-03-16T17:28:30.176-0400 I STORAGE [signalProcessingThread] WiredTiger message [1521235710:176252][24291:0x7f6a33604700], txn-recover: Recovering log 2 through 2 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.238-0400 c20012| 2018-03-16T17:28:30.238-0400 I STORAGE [signalProcessingThread] WiredTiger message [1521235710:238509][24291:0x7f6a33604700], txn-recover: Set global recovery timestamp: 0 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.350-0400 c20012| 2018-03-16T17:28:30.350-0400 I STORAGE [signalProcessingThread] shutdown: removing fs lock... [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.350-0400 c20012| 2018-03-16T17:28:30.350-0400 I CONTROL [signalProcessingThread] now exiting [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.350-0400 c20012| 2018-03-16T17:28:30.350-0400 I CONTROL [signalProcessingThread] shutting down with code:0 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.355-0400 c20014| 2018-03-16T17:28:30.355-0400 I NETWORK [conn19] end connection 127.0.0.1:44548 (7 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.355-0400 c20013| 2018-03-16T17:28:30.355-0400 I NETWORK [conn19] end connection 127.0.0.1:47124 (6 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.355-0400 c20014| 2018-03-16T17:28:30.355-0400 I ASIO [NetworkInterfaceASIO-RS-0] Ending connection to host ubuntu:20012 due to bad connection status; 1 connections to that host remain open [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.356-0400 c20014| 2018-03-16T17:28:30.355-0400 I NETWORK [conn4] end connection 127.0.0.1:44404 (6 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.356-0400 c20014| 2018-03-16T17:28:30.355-0400 I REPL [replication-0] Restarting oplog query due to error: HostUnreachable: error in fetcher batch callback :: caused by :: End of file. Last fetched optime (with hash): { ts: Timestamp(1521235707, 41), t: 1 }[-4804795384625061556]. Restarts remaining: 3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.356-0400 c20013| 2018-03-16T17:28:30.355-0400 I ASIO [NetworkInterfaceASIO-RS-0] Ending connection to host ubuntu:20012 due to bad connection status; 1 connections to that host remain open [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.356-0400 c20013| 2018-03-16T17:28:30.355-0400 I NETWORK [conn3] end connection 127.0.0.1:46976 (5 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.356-0400 c20013| 2018-03-16T17:28:30.355-0400 I REPL [replication-0] Restarting oplog query due to error: HostUnreachable: error in fetcher batch callback :: caused by :: End of file. Last fetched optime (with hash): { ts: Timestamp(1521235707, 41), t: 1 }[-4804795384625061556]. Restarts remaining: 3 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.357-0400 c20014| 2018-03-16T17:28:30.355-0400 I ASIO [replication-0] dropping unhealthy pooled connection to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.357-0400 c20013| 2018-03-16T17:28:30.355-0400 I ASIO [replication-0] dropping unhealthy pooled connection to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.357-0400 c20013| 2018-03-16T17:28:30.355-0400 I ASIO [replication-0] after drop, pool was empty, going to spawn some connections [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.357-0400 c20014| 2018-03-16T17:28:30.355-0400 I ASIO [replication-0] after drop, pool was empty, going to spawn some connections [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.357-0400 c20014| 2018-03-16T17:28:30.355-0400 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.357-0400 c20013| 2018-03-16T17:28:30.355-0400 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.358-0400 c20014| 2018-03-16T17:28:30.355-0400 I REPL [replication-0] Scheduled new oplog query Fetcher source: ubuntu:20012 database: local query: { find: "oplog.rs", filter: { ts: { $gte: Timestamp(1521235707, 41) } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 60000, batchSize: 13981010, term: 1, readConcern: { afterClusterTime: Timestamp(1521235707, 41) } } query metadata: { $replData: 1, $oplogQueryData: 1, $readPreference: { mode: "secondaryPreferred" } } active: 1 findNetworkTimeout: 65000ms getMoreNetworkTimeout: 10000ms shutting down?: 0 first: 1 firstCommandScheduler: RemoteCommandRetryScheduler request: RemoteCommand 1382 -- target:ubuntu:20012 db:local cmd:{ find: "oplog.rs", filter: { ts: { $gte: Timestamp(1521235707, 41) } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 60000, batchSize: 13981010, term: 1, readConcern: { afterClusterTime: Timestamp(1521235707, 41) } } active: 1 callbackHandle.valid: 1 callbackHandle.cancelled: 0 attempt: 1 retryPolicy: RetryPolicyImpl maxAttempts: 1 maxTimeMillis: -1ms [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.359-0400 c20013| 2018-03-16T17:28:30.355-0400 I REPL [replication-0] Scheduled new oplog query Fetcher source: ubuntu:20012 database: local query: { find: "oplog.rs", filter: { ts: { $gte: Timestamp(1521235707, 41) } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 60000, batchSize: 13981010, term: 1, readConcern: { afterClusterTime: Timestamp(1521235707, 41) } } query metadata: { $replData: 1, $oplogQueryData: 1, $readPreference: { mode: "secondaryPreferred" } } active: 1 findNetworkTimeout: 65000ms getMoreNetworkTimeout: 10000ms shutting down?: 0 first: 1 firstCommandScheduler: RemoteCommandRetryScheduler request: RemoteCommand 1391 -- target:ubuntu:20012 db:local cmd:{ find: "oplog.rs", filter: { ts: { $gte: Timestamp(1521235707, 41) } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 60000, batchSize: 13981010, term: 1, readConcern: { afterClusterTime: Timestamp(1521235707, 41) } } active: 1 callbackHandle.valid: 1 callbackHandle.cancelled: 0 attempt: 1 retryPolicy: RetryPolicyImpl maxAttempts: 1 maxTimeMillis: -1ms [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.359-0400 c20013| 2018-03-16T17:28:30.355-0400 I CONTROL [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.359-0400 c20013| 2018-03-16T17:28:30.355-0400 I NETWORK [signalProcessingThread] shutdown: going to close listening sockets... [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.359-0400 c20013| 2018-03-16T17:28:30.355-0400 I NETWORK [signalProcessingThread] removing socket file: /tmp/mongodb-20013.sock [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.360-0400 c20013| 2018-03-16T17:28:30.355-0400 I ASIO [NetworkInterfaceASIO-RS-0] Failed to connect to ubuntu:20012 - HostUnreachable: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.360-0400 c20013| 2018-03-16T17:28:30.355-0400 I ASIO [NetworkInterfaceASIO-RS-0] Dropping all pooled connections to ubuntu:20012 due to failed operation on a connection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.360-0400 c20013| 2018-03-16T17:28:30.355-0400 I REPL [signalProcessingThread] shutdown: removing checkpointTimestamp collection... [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.360-0400 c20014| 2018-03-16T17:28:30.355-0400 I ASIO [NetworkInterfaceASIO-RS-0] Failed to connect to ubuntu:20012 - HostUnreachable: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.360-0400 c20013| 2018-03-16T17:28:30.355-0400 I REPL [replication-1] Restarting oplog query due to error: HostUnreachable: error in fetcher batch callback :: caused by :: Connection refused. Last fetched optime (with hash): { ts: Timestamp(1521235707, 41), t: 1 }[-4804795384625061556]. Restarts remaining: 2 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.360-0400 c20014| 2018-03-16T17:28:30.355-0400 I ASIO [NetworkInterfaceASIO-RS-0] Dropping all pooled connections to ubuntu:20012 due to failed operation on a connection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.361-0400 c20013| 2018-03-16T17:28:30.355-0400 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.361-0400 c20014| 2018-03-16T17:28:30.355-0400 I REPL [replication-1] Restarting oplog query due to error: HostUnreachable: error in fetcher batch callback :: caused by :: Connection refused. Last fetched optime (with hash): { ts: Timestamp(1521235707, 41), t: 1 }[-4804795384625061556]. Restarts remaining: 2 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.362-0400 c20013| 2018-03-16T17:28:30.355-0400 I REPL [replication-1] Scheduled new oplog query Fetcher source: ubuntu:20012 database: local query: { find: "oplog.rs", filter: { ts: { $gte: Timestamp(1521235707, 41) } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 60000, batchSize: 13981010, term: 1, readConcern: { afterClusterTime: Timestamp(1521235707, 41) } } query metadata: { $replData: 1, $oplogQueryData: 1, $readPreference: { mode: "secondaryPreferred" } } active: 1 findNetworkTimeout: 65000ms getMoreNetworkTimeout: 10000ms shutting down?: 0 first: 1 firstCommandScheduler: RemoteCommandRetryScheduler request: RemoteCommand 1393 -- target:ubuntu:20012 db:local cmd:{ find: "oplog.rs", filter: { ts: { $gte: Timestamp(1521235707, 41) } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 60000, batchSize: 13981010, term: 1, readConcern: { afterClusterTime: Timestamp(1521235707, 41) } } active: 1 callbackHandle.valid: 1 callbackHandle.cancelled: 0 attempt: 1 retryPolicy: RetryPolicyImpl maxAttempts: 1 maxTimeMillis: -1ms [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.362-0400 c20014| 2018-03-16T17:28:30.355-0400 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.363-0400 c20014| 2018-03-16T17:28:30.355-0400 I REPL [replication-1] Scheduled new oplog query Fetcher source: ubuntu:20012 database: local query: { find: "oplog.rs", filter: { ts: { $gte: Timestamp(1521235707, 41) } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 60000, batchSize: 13981010, term: 1, readConcern: { afterClusterTime: Timestamp(1521235707, 41) } } query metadata: { $replData: 1, $oplogQueryData: 1, $readPreference: { mode: "secondaryPreferred" } } active: 1 findNetworkTimeout: 65000ms getMoreNetworkTimeout: 10000ms shutting down?: 0 first: 1 firstCommandScheduler: RemoteCommandRetryScheduler request: RemoteCommand 1384 -- target:ubuntu:20012 db:local cmd:{ find: "oplog.rs", filter: { ts: { $gte: Timestamp(1521235707, 41) } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 60000, batchSize: 13981010, term: 1, readConcern: { afterClusterTime: Timestamp(1521235707, 41) } } active: 1 callbackHandle.valid: 1 callbackHandle.cancelled: 0 attempt: 1 retryPolicy: RetryPolicyImpl maxAttempts: 1 maxTimeMillis: -1ms [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.363-0400 c20013| 2018-03-16T17:28:30.355-0400 I STORAGE [signalProcessingThread] Finishing collection drop for local.replset.checkpointTimestamp (d2595055-7415-461b-84ca-bacd1861c40f). [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.363-0400 c20013| 2018-03-16T17:28:30.356-0400 I ASIO [NetworkInterfaceASIO-RS-0] Failed to connect to ubuntu:20012 - HostUnreachable: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.363-0400 c20013| 2018-03-16T17:28:30.356-0400 I ASIO [NetworkInterfaceASIO-RS-0] Dropping all pooled connections to ubuntu:20012 due to failed operation on a connection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.363-0400 c20014| 2018-03-16T17:28:30.356-0400 I ASIO [NetworkInterfaceASIO-RS-0] Failed to connect to ubuntu:20012 - HostUnreachable: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.363-0400 c20014| 2018-03-16T17:28:30.356-0400 I ASIO [NetworkInterfaceASIO-RS-0] Dropping all pooled connections to ubuntu:20012 due to failed operation on a connection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.364-0400 c20013| 2018-03-16T17:28:30.356-0400 I REPL [replication-0] Restarting oplog query due to error: HostUnreachable: error in fetcher batch callback :: caused by :: Connection refused. Last fetched optime (with hash): { ts: Timestamp(1521235707, 41), t: 1 }[-4804795384625061556]. Restarts remaining: 1 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.364-0400 c20013| 2018-03-16T17:28:30.356-0400 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.365-0400 c20013| 2018-03-16T17:28:30.356-0400 I REPL [replication-0] Scheduled new oplog query Fetcher source: ubuntu:20012 database: local query: { find: "oplog.rs", filter: { ts: { $gte: Timestamp(1521235707, 41) } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 60000, batchSize: 13981010, term: 1, readConcern: { afterClusterTime: Timestamp(1521235707, 41) } } query metadata: { $replData: 1, $oplogQueryData: 1, $readPreference: { mode: "secondaryPreferred" } } active: 1 findNetworkTimeout: 65000ms getMoreNetworkTimeout: 10000ms shutting down?: 0 first: 1 firstCommandScheduler: RemoteCommandRetryScheduler request: RemoteCommand 1395 -- target:ubuntu:20012 db:local cmd:{ find: "oplog.rs", filter: { ts: { $gte: Timestamp(1521235707, 41) } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 60000, batchSize: 13981010, term: 1, readConcern: { afterClusterTime: Timestamp(1521235707, 41) } } active: 1 callbackHandle.valid: 1 callbackHandle.cancelled: 0 attempt: 1 retryPolicy: RetryPolicyImpl maxAttempts: 1 maxTimeMillis: -1ms [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.365-0400 c20014| 2018-03-16T17:28:30.356-0400 I REPL [replication-0] Restarting oplog query due to error: HostUnreachable: error in fetcher batch callback :: caused by :: Connection refused. Last fetched optime (with hash): { ts: Timestamp(1521235707, 41), t: 1 }[-4804795384625061556]. Restarts remaining: 1 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.365-0400 c20014| 2018-03-16T17:28:30.356-0400 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.366-0400 c20014| 2018-03-16T17:28:30.356-0400 I REPL [replication-0] Scheduled new oplog query Fetcher source: ubuntu:20012 database: local query: { find: "oplog.rs", filter: { ts: { $gte: Timestamp(1521235707, 41) } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 60000, batchSize: 13981010, term: 1, readConcern: { afterClusterTime: Timestamp(1521235707, 41) } } query metadata: { $replData: 1, $oplogQueryData: 1, $readPreference: { mode: "secondaryPreferred" } } active: 1 findNetworkTimeout: 65000ms getMoreNetworkTimeout: 10000ms shutting down?: 0 first: 1 firstCommandScheduler: RemoteCommandRetryScheduler request: RemoteCommand 1386 -- target:ubuntu:20012 db:local cmd:{ find: "oplog.rs", filter: { ts: { $gte: Timestamp(1521235707, 41) } }, tailable: true, oplogReplay: true, awaitData: true, maxTimeMS: 60000, batchSize: 13981010, term: 1, readConcern: { afterClusterTime: Timestamp(1521235707, 41) } } active: 1 callbackHandle.valid: 1 callbackHandle.cancelled: 0 attempt: 1 retryPolicy: RetryPolicyImpl maxAttempts: 1 maxTimeMillis: -1ms [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.366-0400 c20013| 2018-03-16T17:28:30.356-0400 I ASIO [NetworkInterfaceASIO-RS-0] Failed to connect to ubuntu:20012 - HostUnreachable: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.366-0400 c20013| 2018-03-16T17:28:30.356-0400 I ASIO [NetworkInterfaceASIO-RS-0] Dropping all pooled connections to ubuntu:20012 due to failed operation on a connection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.366-0400 c20013| 2018-03-16T17:28:30.356-0400 I REPL [replication-1] Error returned from oplog query (no more query restarts left): HostUnreachable: error in fetcher batch callback :: caused by :: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.367-0400 c20014| 2018-03-16T17:28:30.356-0400 I ASIO [NetworkInterfaceASIO-RS-0] Failed to connect to ubuntu:20012 - HostUnreachable: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.367-0400 c20014| 2018-03-16T17:28:30.356-0400 I ASIO [NetworkInterfaceASIO-RS-0] Dropping all pooled connections to ubuntu:20012 due to failed operation on a connection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.367-0400 c20013| 2018-03-16T17:28:30.356-0400 W REPL [rsBackgroundSync] Fetcher stopped querying remote oplog with error: HostUnreachable: error in fetcher batch callback :: caused by :: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.367-0400 c20014| 2018-03-16T17:28:30.356-0400 I REPL [replication-1] Error returned from oplog query (no more query restarts left): HostUnreachable: error in fetcher batch callback :: caused by :: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.367-0400 c20014| 2018-03-16T17:28:30.356-0400 W REPL [rsBackgroundSync] Fetcher stopped querying remote oplog with error: HostUnreachable: error in fetcher batch callback :: caused by :: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.368-0400 c20014| 2018-03-16T17:28:30.356-0400 I REPL [rsBackgroundSync] could not find member to sync from [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.368-0400 c20014| 2018-03-16T17:28:30.356-0400 D REPL_HB [rsBackgroundSync] Cancelling all heartbeats. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.368-0400 c20014| 2018-03-16T17:28:30.356-0400 D REPL_HB [rsBackgroundSync] Scheduling heartbeat to ubuntu:20012 at 2018-03-16T21:28:30.356Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.368-0400 c20014| 2018-03-16T17:28:30.356-0400 D REPL_HB [rsBackgroundSync] Scheduling heartbeat to ubuntu:20013 at 2018-03-16T21:28:30.356Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.368-0400 c20014| 2018-03-16T17:28:30.356-0400 D REPL_HB [replexec-2] Sending heartbeat (requestId: 1388) to ubuntu:20013, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.368-0400 c20014| 2018-03-16T17:28:30.356-0400 D REPL_HB [replexec-3] Sending heartbeat (requestId: 1389) to ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.369-0400 c20014| 2018-03-16T17:28:30.356-0400 I ASIO [replexec-3] dropping unhealthy pooled connection to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.369-0400 c20014| 2018-03-16T17:28:30.356-0400 I ASIO [replexec-3] after drop, pool was empty, going to spawn some connections [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.369-0400 c20014| 2018-03-16T17:28:30.356-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.369-0400 c20013| 2018-03-16T17:28:30.356-0400 D REPL_HB [conn9] Received heartbeat request from ubuntu:20014, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235707, 41), signature: { hash: BinData(0, CEFAA93570840EC6E37F07AD52BBF4B956967DEE), keyId: 6533657568122765331 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.369-0400 c20013| 2018-03-16T17:28:30.356-0400 D REPL_HB [conn9] Processed heartbeat from ubuntu:20014 and generated response, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", syncingTo: "ubuntu:20012", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235707, 41), t: 1 }, opTime: { ts: Timestamp(1521235707, 41), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.370-0400 c20014| 2018-03-16T17:28:30.356-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Failed to connect to ubuntu:20012 - HostUnreachable: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.370-0400 c20014| 2018-03-16T17:28:30.356-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to ubuntu:20012 due to failed operation on a connection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.370-0400 c20014| 2018-03-16T17:28:30.356-0400 I REPL_HB [replexec-4] Error in heartbeat (requestId: 1389) to ubuntu:20012, response status: HostUnreachable: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.370-0400 c20014| 2018-03-16T17:28:30.356-0400 D REPL_HB [replexec-4] Scheduling heartbeat to ubuntu:20012 at 2018-03-16T21:28:30.356Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.370-0400 c20014| 2018-03-16T17:28:30.356-0400 D REPL_HB [replexec-2] Sending heartbeat (requestId: 1391) to ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.371-0400 c20014| 2018-03-16T17:28:30.356-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.371-0400 c20014| 2018-03-16T17:28:30.356-0400 D REPL_HB [replexec-4] Received response to heartbeat (requestId: 1388) from ubuntu:20013, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", syncingTo: "ubuntu:20012", term: 1, primaryId: 0, durableOpTime: { ts: Timestamp(1521235707, 41), t: 1 }, opTime: { ts: Timestamp(1521235707, 41), t: 1 }, operationTime: Timestamp(1521235707, 41), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235707, 41), t: 1 }, lastOpVisible: { ts: Timestamp(1521235707, 41), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: 0, syncSourceIndex: 0 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235707, 41), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235707, 41) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.371-0400 c20014| 2018-03-16T17:28:30.356-0400 D REPL_HB [replexec-4] Scheduling heartbeat to ubuntu:20013 at 2018-03-16T21:28:30.856Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.372-0400 c20014| 2018-03-16T17:28:30.357-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Failed to connect to ubuntu:20012 - HostUnreachable: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.372-0400 c20014| 2018-03-16T17:28:30.357-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to ubuntu:20012 due to failed operation on a connection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.372-0400 c20014| 2018-03-16T17:28:30.357-0400 I REPL_HB [replexec-3] Error in heartbeat (requestId: 1391) to ubuntu:20012, response status: HostUnreachable: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.372-0400 c20014| 2018-03-16T17:28:30.357-0400 D REPL_HB [replexec-3] Scheduling heartbeat to ubuntu:20012 at 2018-03-16T21:28:30.357Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.372-0400 c20014| 2018-03-16T17:28:30.357-0400 D REPL_HB [replexec-3] Sending heartbeat (requestId: 1394) to ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.372-0400 c20014| 2018-03-16T17:28:30.357-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.373-0400 c20014| 2018-03-16T17:28:30.357-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Failed to connect to ubuntu:20012 - HostUnreachable: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.373-0400 c20014| 2018-03-16T17:28:30.357-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to ubuntu:20012 due to failed operation on a connection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.373-0400 c20014| 2018-03-16T17:28:30.357-0400 I REPL_HB [replexec-1] Error in heartbeat (requestId: 1394) to ubuntu:20012, response status: HostUnreachable: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.373-0400 c20014| 2018-03-16T17:28:30.357-0400 I REPL [replexec-1] Member ubuntu:20012 is now in state RS_DOWN [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.373-0400 c20014| 2018-03-16T17:28:30.357-0400 D REPL_HB [replexec-1] Scheduling heartbeat to ubuntu:20012 at 2018-03-16T21:28:30.857Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.373-0400 c20013| 2018-03-16T17:28:30.358-0400 I REPL [signalProcessingThread] shutting down replication subsystems [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.374-0400 c20013| 2018-03-16T17:28:30.358-0400 I REPL [rsBackgroundSync] could not find member to sync from [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.374-0400 c20013| 2018-03-16T17:28:30.358-0400 D REPL_HB [rsBackgroundSync] Cancelling all heartbeats. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.374-0400 c20013| 2018-03-16T17:28:30.358-0400 D REPL_HB [rsBackgroundSync] Scheduling heartbeat to ubuntu:20012 at 2018-03-16T21:28:30.358Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.374-0400 c20013| 2018-03-16T17:28:30.358-0400 D REPL_HB [rsBackgroundSync] Scheduling heartbeat to ubuntu:20014 at 2018-03-16T21:28:30.358Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.375-0400 c20013| 2018-03-16T17:28:30.358-0400 I REPL [signalProcessingThread] Stopping replication reporter thread [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.375-0400 c20013| 2018-03-16T17:28:30.358-0400 D REPL_HB [replexec-1] Sending heartbeat (requestId: 1397) to ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.376-0400 c20013| 2018-03-16T17:28:30.358-0400 I REPL [SyncSourceFeedback] SyncSourceFeedback error sending update to ubuntu:20012: CallbackCanceled: Reporter no longer valid [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.376-0400 c20013| 2018-03-16T17:28:30.358-0400 I ASIO [replexec-1] dropping unhealthy pooled connection to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.376-0400 c20013| 2018-03-16T17:28:30.358-0400 I ASIO [replexec-1] after drop, pool was empty, going to spawn some connections [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.376-0400 c20013| 2018-03-16T17:28:30.358-0400 I REPL [signalProcessingThread] Stopping replication fetcher thread [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.376-0400 c20013| 2018-03-16T17:28:30.358-0400 I REPL [signalProcessingThread] Stopping replication applier thread [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.376-0400 c20013| 2018-03-16T17:28:30.358-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.376-0400 c20013| 2018-03-16T17:28:30.358-0400 D REPL_HB [replexec-2] Sending heartbeat (requestId: 1399) to ubuntu:20014, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.377-0400 c20013| 2018-03-16T17:28:30.358-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Failed to connect to ubuntu:20012 - HostUnreachable: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.377-0400 c20013| 2018-03-16T17:28:30.358-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to ubuntu:20012 due to failed operation on a connection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.377-0400 c20014| 2018-03-16T17:28:30.358-0400 D REPL_HB [conn10] Received heartbeat request from ubuntu:20013, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235707, 41), signature: { hash: BinData(0, CEFAA93570840EC6E37F07AD52BBF4B956967DEE), keyId: 6533657568122765331 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.378-0400 c20014| 2018-03-16T17:28:30.359-0400 D REPL_HB [conn10] Processed heartbeat from ubuntu:20013 and generated response, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, durableOpTime: { ts: Timestamp(1521235707, 41), t: 1 }, opTime: { ts: Timestamp(1521235707, 41), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.378-0400 c20013| 2018-03-16T17:28:30.359-0400 I REPL_HB [replexec-0] Error in heartbeat (requestId: 1397) to ubuntu:20012, response status: HostUnreachable: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.378-0400 c20013| 2018-03-16T17:28:30.359-0400 D REPL_HB [replexec-0] Scheduling heartbeat to ubuntu:20012 at 2018-03-16T21:28:30.359Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.378-0400 c20013| 2018-03-16T17:28:30.359-0400 D REPL_HB [replexec-0] Sending heartbeat (requestId: 1400) to ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.378-0400 c20013| 2018-03-16T17:28:30.359-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.379-0400 c20013| 2018-03-16T17:28:30.359-0400 D REPL_HB [replexec-1] Received response to heartbeat (requestId: 1399) from ubuntu:20014, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, durableOpTime: { ts: Timestamp(1521235707, 41), t: 1 }, opTime: { ts: Timestamp(1521235707, 41), t: 1 }, operationTime: Timestamp(1521235707, 41), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235707, 41), t: 1 }, lastOpVisible: { ts: Timestamp(1521235707, 41), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: -1, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235707, 41), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235707, 41) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.379-0400 c20013| 2018-03-16T17:28:30.359-0400 D REPL_HB [replexec-1] Scheduling heartbeat to ubuntu:20014 at 2018-03-16T21:28:30.859Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.379-0400 c20013| 2018-03-16T17:28:30.359-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Failed to connect to ubuntu:20012 - HostUnreachable: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.379-0400 c20013| 2018-03-16T17:28:30.359-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to ubuntu:20012 due to failed operation on a connection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.380-0400 c20013| 2018-03-16T17:28:30.359-0400 I REPL_HB [replexec-2] Error in heartbeat (requestId: 1400) to ubuntu:20012, response status: HostUnreachable: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.380-0400 c20013| 2018-03-16T17:28:30.359-0400 D REPL_HB [replexec-2] Scheduling heartbeat to ubuntu:20012 at 2018-03-16T21:28:30.359Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.380-0400 c20013| 2018-03-16T17:28:30.359-0400 D REPL_HB [replexec-3] Sending heartbeat (requestId: 1403) to ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.380-0400 c20013| 2018-03-16T17:28:30.359-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.380-0400 c20013| 2018-03-16T17:28:30.359-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Failed to connect to ubuntu:20012 - HostUnreachable: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.380-0400 c20013| 2018-03-16T17:28:30.359-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to ubuntu:20012 due to failed operation on a connection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.381-0400 c20013| 2018-03-16T17:28:30.359-0400 I REPL_HB [replexec-0] Error in heartbeat (requestId: 1403) to ubuntu:20012, response status: HostUnreachable: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.381-0400 c20013| 2018-03-16T17:28:30.359-0400 I REPL [replexec-0] Member ubuntu:20012 is now in state RS_DOWN [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.381-0400 c20013| 2018-03-16T17:28:30.359-0400 D REPL_HB [replexec-0] Scheduling heartbeat to ubuntu:20012 at 2018-03-16T21:28:30.859Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.857-0400 s20015| 2018-03-16T17:28:30.856-0400 D ASIO [Uptime reporter] startCommand: RemoteCommand 72 -- target:ubuntu:20012 db:config expDate:2018-03-16T17:29:00.856-0400 cmd:{ update: "mongos", bypassDocumentValidation: false, ordered: true, updates: [ { q: { _id: "ubuntu:20015" }, u: { $set: { _id: "ubuntu:20015", ping: new Date(1521235710856), up: 10, waiting: true, mongoVersion: "3.6.3" } }, multi: false, upsert: true } ], writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.857-0400 s20015| 2018-03-16T17:28:30.856-0400 I ASIO [Uptime reporter] dropping unhealthy pooled connection to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.859-0400 s20015| 2018-03-16T17:28:30.856-0400 I ASIO [Uptime reporter] dropping unhealthy pooled connection to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.859-0400 s20015| 2018-03-16T17:28:30.856-0400 I ASIO [Uptime reporter] dropping unhealthy pooled connection to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.859-0400 s20015| 2018-03-16T17:28:30.856-0400 I ASIO [Uptime reporter] after drop, pool was empty, going to spawn some connections [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.859-0400 s20015| 2018-03-16T17:28:30.856-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Connecting to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.860-0400 c20014| 2018-03-16T17:28:30.857-0400 D REPL_HB [replexec-2] Sending heartbeat (requestId: 1396) to ubuntu:20013, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.860-0400 c20013| 2018-03-16T17:28:30.857-0400 D REPL_HB [conn9] Received heartbeat request from ubuntu:20014, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235707, 41), signature: { hash: BinData(0, CEFAA93570840EC6E37F07AD52BBF4B956967DEE), keyId: 6533657568122765331 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.860-0400 s20015| 2018-03-16T17:28:30.857-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Failed to connect to ubuntu:20012 - HostUnreachable: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.861-0400 c20013| 2018-03-16T17:28:30.857-0400 D REPL_HB [conn9] Processed heartbeat from ubuntu:20014 and generated response, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, durableOpTime: { ts: Timestamp(1521235707, 41), t: 1 }, opTime: { ts: Timestamp(1521235707, 41), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.861-0400 s20015| 2018-03-16T17:28:30.857-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Request 73 finished with response: HostUnreachable: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.861-0400 s20015| 2018-03-16T17:28:30.857-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Dropping all pooled connections to ubuntu:20012 due to failed operation on a connection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.861-0400 s20015| 2018-03-16T17:28:30.857-0400 D ASIO [NetworkInterfaceASIO-ShardRegistry-0] Failed to get connection from pool for request 72: HostUnreachable: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.861-0400 c20014| 2018-03-16T17:28:30.857-0400 D REPL_HB [replexec-4] Sending heartbeat (requestId: 1397) to ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.862-0400 s20015| 2018-03-16T17:28:30.857-0400 I NETWORK [NetworkInterfaceASIO-ShardRegistry-0] Marking host ubuntu:20012 as failed :: caused by :: HostUnreachable: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.862-0400 s20015| 2018-03-16T17:28:30.857-0400 D SHARDING [NetworkInterfaceASIO-ShardRegistry-0] Error processing the remote request, not updating operationTime or gLE [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.862-0400 s20015| 2018-03-16T17:28:30.857-0400 D SHARDING [Uptime reporter] Batch write command to config failed with retriable error and will be retried :: caused by :: HostUnreachable: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.862-0400 c20014| 2018-03-16T17:28:30.857-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.862-0400 s20015| 2018-03-16T17:28:30.857-0400 D NETWORK [Uptime reporter] Starting new refresh of replica set stale_mongos_updates_and_removes-configRS [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.863-0400 c20014| 2018-03-16T17:28:30.857-0400 D REPL_HB [replexec-0] Received response to heartbeat (requestId: 1396) from ubuntu:20013, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, durableOpTime: { ts: Timestamp(1521235707, 41), t: 1 }, opTime: { ts: Timestamp(1521235707, 41), t: 1 }, operationTime: Timestamp(1521235707, 41), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235707, 41), t: 1 }, lastOpVisible: { ts: Timestamp(1521235707, 41), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: -1, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235707, 41), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235707, 41) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.863-0400 s20015| 2018-03-16T17:28:30.857-0400 D NETWORK [Uptime reporter] polling for status of connection to 127.0.1.1:20012, event detected [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.863-0400 c20014| 2018-03-16T17:28:30.857-0400 D REPL_HB [replexec-0] Scheduling heartbeat to ubuntu:20013 at 2018-03-16T21:28:31.357Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.864-0400 s20015| 2018-03-16T17:28:30.857-0400 I NETWORK [Uptime reporter] Socket closed remotely, no longer connected (idle 10 secs, remote host 127.0.1.1:20012) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.864-0400 s20015| 2018-03-16T17:28:30.857-0400 D NETWORK [Uptime reporter] creating new connection to:ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.864-0400 c20014| 2018-03-16T17:28:30.857-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Failed to connect to ubuntu:20012 - HostUnreachable: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.864-0400 c20014| 2018-03-16T17:28:30.857-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to ubuntu:20012 due to failed operation on a connection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.864-0400 s20015| 2018-03-16T17:28:30.857-0400 W NETWORK [Uptime reporter] Failed to connect to 127.0.1.1:20012, in(checking socket for error after poll), reason: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.865-0400 s20015| 2018-03-16T17:28:30.857-0400 D - [Uptime reporter] User Assertion: 40356:connection pool: connect failed ubuntu:20012 : couldn't connect to server ubuntu:20012, connection attempt failed src/mongo/client/connpool.cpp 395 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.865-0400 c20014| 2018-03-16T17:28:30.857-0400 I REPL_HB [replexec-3] Error in heartbeat (requestId: 1397) to ubuntu:20012, response status: HostUnreachable: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.865-0400 c20014| 2018-03-16T17:28:30.857-0400 D REPL_HB [replexec-3] Scheduling heartbeat to ubuntu:20012 at 2018-03-16T21:28:30.857Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.865-0400 c20014| 2018-03-16T17:28:30.857-0400 D REPL_HB [replexec-3] Sending heartbeat (requestId: 1400) to ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.865-0400 c20014| 2018-03-16T17:28:30.857-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.866-0400 s20015| 2018-03-16T17:28:30.857-0400 D NETWORK [Uptime reporter] polling for status of connection to 127.0.1.1:20013, no events [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.866-0400 c20014| 2018-03-16T17:28:30.857-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Failed to connect to ubuntu:20012 - HostUnreachable: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.866-0400 c20014| 2018-03-16T17:28:30.857-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to ubuntu:20012 due to failed operation on a connection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.866-0400 c20014| 2018-03-16T17:28:30.857-0400 I REPL_HB [replexec-2] Error in heartbeat (requestId: 1400) to ubuntu:20012, response status: HostUnreachable: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.866-0400 c20014| 2018-03-16T17:28:30.857-0400 D REPL_HB [replexec-2] Scheduling heartbeat to ubuntu:20012 at 2018-03-16T21:28:30.857Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.867-0400 c20014| 2018-03-16T17:28:30.857-0400 D REPL_HB [replexec-2] Sending heartbeat (requestId: 1402) to ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.867-0400 c20014| 2018-03-16T17:28:30.858-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.867-0400 s20015| 2018-03-16T17:28:30.858-0400 D NETWORK [Uptime reporter] polling for status of connection to 127.0.1.1:20014, no events [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.867-0400 c20014| 2018-03-16T17:28:30.858-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Failed to connect to ubuntu:20012 - HostUnreachable: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.867-0400 c20014| 2018-03-16T17:28:30.858-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to ubuntu:20012 due to failed operation on a connection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.867-0400 c20014| 2018-03-16T17:28:30.858-0400 I REPL_HB [replexec-0] Error in heartbeat (requestId: 1402) to ubuntu:20012, response status: HostUnreachable: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.868-0400 c20014| 2018-03-16T17:28:30.858-0400 D REPL_HB [replexec-0] Scheduling heartbeat to ubuntu:20012 at 2018-03-16T21:28:31.358Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.868-0400 s20015| 2018-03-16T17:28:30.858-0400 W NETWORK [Uptime reporter] Unable to reach primary for set stale_mongos_updates_and_removes-configRS [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.868-0400 c20013| 2018-03-16T17:28:30.859-0400 D REPL_HB [replexec-1] Sending heartbeat (requestId: 1405) to ubuntu:20014, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.868-0400 c20014| 2018-03-16T17:28:30.859-0400 D REPL_HB [conn10] Received heartbeat request from ubuntu:20013, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235707, 41), signature: { hash: BinData(0, CEFAA93570840EC6E37F07AD52BBF4B956967DEE), keyId: 6533657568122765331 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.869-0400 c20014| 2018-03-16T17:28:30.859-0400 D REPL_HB [conn10] Processed heartbeat from ubuntu:20013 and generated response, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, durableOpTime: { ts: Timestamp(1521235707, 41), t: 1 }, opTime: { ts: Timestamp(1521235707, 41), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.869-0400 c20013| 2018-03-16T17:28:30.859-0400 D REPL_HB [replexec-2] Sending heartbeat (requestId: 1406) to ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.869-0400 c20013| 2018-03-16T17:28:30.859-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.870-0400 c20013| 2018-03-16T17:28:30.859-0400 D REPL_HB [replexec-3] Received response to heartbeat (requestId: 1405) from ubuntu:20014, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, durableOpTime: { ts: Timestamp(1521235707, 41), t: 1 }, opTime: { ts: Timestamp(1521235707, 41), t: 1 }, operationTime: Timestamp(1521235707, 41), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235707, 41), t: 1 }, lastOpVisible: { ts: Timestamp(1521235707, 41), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: -1, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235707, 41), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235707, 41) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.870-0400 c20013| 2018-03-16T17:28:30.859-0400 D REPL_HB [replexec-3] Scheduling heartbeat to ubuntu:20014 at 2018-03-16T21:28:31.359Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.870-0400 c20013| 2018-03-16T17:28:30.859-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Failed to connect to ubuntu:20012 - HostUnreachable: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.870-0400 c20013| 2018-03-16T17:28:30.859-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to ubuntu:20012 due to failed operation on a connection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.871-0400 c20013| 2018-03-16T17:28:30.860-0400 I REPL_HB [replexec-0] Error in heartbeat (requestId: 1406) to ubuntu:20012, response status: HostUnreachable: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.871-0400 c20013| 2018-03-16T17:28:30.860-0400 D REPL_HB [replexec-0] Scheduling heartbeat to ubuntu:20012 at 2018-03-16T21:28:30.860Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.871-0400 c20013| 2018-03-16T17:28:30.860-0400 D REPL_HB [replexec-0] Sending heartbeat (requestId: 1409) to ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.871-0400 c20013| 2018-03-16T17:28:30.860-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.871-0400 c20013| 2018-03-16T17:28:30.860-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Failed to connect to ubuntu:20012 - HostUnreachable: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.872-0400 c20013| 2018-03-16T17:28:30.860-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to ubuntu:20012 due to failed operation on a connection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.872-0400 c20013| 2018-03-16T17:28:30.860-0400 I REPL_HB [replexec-2] Error in heartbeat (requestId: 1409) to ubuntu:20012, response status: HostUnreachable: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.872-0400 c20013| 2018-03-16T17:28:30.860-0400 D REPL_HB [replexec-2] Scheduling heartbeat to ubuntu:20012 at 2018-03-16T21:28:30.860Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.872-0400 c20013| 2018-03-16T17:28:30.860-0400 D REPL_HB [replexec-2] Sending heartbeat (requestId: 1411) to ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.872-0400 c20013| 2018-03-16T17:28:30.860-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.872-0400 c20013| 2018-03-16T17:28:30.860-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Failed to connect to ubuntu:20012 - HostUnreachable: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.873-0400 c20013| 2018-03-16T17:28:30.860-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to ubuntu:20012 due to failed operation on a connection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.873-0400 c20013| 2018-03-16T17:28:30.860-0400 I REPL_HB [replexec-1] Error in heartbeat (requestId: 1411) to ubuntu:20012, response status: HostUnreachable: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:30.873-0400 c20013| 2018-03-16T17:28:30.860-0400 D REPL_HB [replexec-1] Scheduling heartbeat to ubuntu:20012 at 2018-03-16T21:28:31.360Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.357-0400 c20014| 2018-03-16T17:28:31.357-0400 D REPL_HB [replexec-1] Sending heartbeat (requestId: 1404) to ubuntu:20013, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.358-0400 c20013| 2018-03-16T17:28:31.357-0400 D REPL_HB [conn9] Received heartbeat request from ubuntu:20014, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235707, 41), signature: { hash: BinData(0, CEFAA93570840EC6E37F07AD52BBF4B956967DEE), keyId: 6533657568122765331 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.358-0400 c20013| 2018-03-16T17:28:31.357-0400 D REPL_HB [conn9] Processed heartbeat from ubuntu:20014 and generated response, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, durableOpTime: { ts: Timestamp(1521235707, 41), t: 1 }, opTime: { ts: Timestamp(1521235707, 41), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.359-0400 c20014| 2018-03-16T17:28:31.358-0400 D REPL_HB [replexec-3] Received response to heartbeat (requestId: 1404) from ubuntu:20013, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, durableOpTime: { ts: Timestamp(1521235707, 41), t: 1 }, opTime: { ts: Timestamp(1521235707, 41), t: 1 }, operationTime: Timestamp(1521235707, 41), $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235707, 41), t: 1 }, lastOpVisible: { ts: Timestamp(1521235707, 41), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: -1, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235707, 41), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235707, 41) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.359-0400 c20014| 2018-03-16T17:28:31.358-0400 D REPL_HB [replexec-3] Scheduling heartbeat to ubuntu:20013 at 2018-03-16T21:28:31.858Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.359-0400 c20014| 2018-03-16T17:28:31.358-0400 D REPL_HB [replexec-4] Sending heartbeat (requestId: 1406) to ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.360-0400 c20014| 2018-03-16T17:28:31.358-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.360-0400 c20014| 2018-03-16T17:28:31.358-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Failed to connect to ubuntu:20012 - HostUnreachable: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.360-0400 c20014| 2018-03-16T17:28:31.358-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to ubuntu:20012 due to failed operation on a connection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.360-0400 c20014| 2018-03-16T17:28:31.358-0400 I REPL_HB [replexec-2] Error in heartbeat (requestId: 1406) to ubuntu:20012, response status: HostUnreachable: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.360-0400 c20014| 2018-03-16T17:28:31.358-0400 D REPL_HB [replexec-2] Scheduling heartbeat to ubuntu:20012 at 2018-03-16T21:28:31.358Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.361-0400 c20014| 2018-03-16T17:28:31.358-0400 D REPL_HB [replexec-2] Sending heartbeat (requestId: 1408) to ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.361-0400 c20014| 2018-03-16T17:28:31.358-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.361-0400 s20015| 2018-03-16T17:28:31.358-0400 D NETWORK [Uptime reporter] Starting new refresh of replica set stale_mongos_updates_and_removes-configRS [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.361-0400 s20015| 2018-03-16T17:28:31.358-0400 D NETWORK [Uptime reporter] creating new connection to:ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.361-0400 c20014| 2018-03-16T17:28:31.358-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Failed to connect to ubuntu:20012 - HostUnreachable: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.361-0400 c20014| 2018-03-16T17:28:31.358-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to ubuntu:20012 due to failed operation on a connection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.362-0400 c20014| 2018-03-16T17:28:31.358-0400 I REPL_HB [replexec-1] Error in heartbeat (requestId: 1408) to ubuntu:20012, response status: HostUnreachable: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.362-0400 c20014| 2018-03-16T17:28:31.358-0400 D REPL_HB [replexec-1] Scheduling heartbeat to ubuntu:20012 at 2018-03-16T21:28:31.358Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.362-0400 c20014| 2018-03-16T17:28:31.358-0400 D REPL_HB [replexec-1] Sending heartbeat (requestId: 1410) to ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.362-0400 c20014| 2018-03-16T17:28:31.358-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.362-0400 s20015| 2018-03-16T17:28:31.358-0400 W NETWORK [Uptime reporter] Failed to connect to 127.0.1.1:20012, in(checking socket for error after poll), reason: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.363-0400 s20015| 2018-03-16T17:28:31.358-0400 D - [Uptime reporter] User Assertion: 40356:connection pool: connect failed ubuntu:20012 : couldn't connect to server ubuntu:20012, connection attempt failed src/mongo/client/connpool.cpp 395 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.363-0400 c20014| 2018-03-16T17:28:31.359-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Failed to connect to ubuntu:20012 - HostUnreachable: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.363-0400 c20014| 2018-03-16T17:28:31.359-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to ubuntu:20012 due to failed operation on a connection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.363-0400 c20014| 2018-03-16T17:28:31.359-0400 I REPL_HB [replexec-4] Error in heartbeat (requestId: 1410) to ubuntu:20012, response status: HostUnreachable: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.363-0400 c20014| 2018-03-16T17:28:31.359-0400 D REPL_HB [replexec-4] Scheduling heartbeat to ubuntu:20012 at 2018-03-16T21:28:31.859Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.363-0400 c20013| 2018-03-16T17:28:31.359-0400 I REPL [signalProcessingThread] Stopping replication storage threads [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.364-0400 s20015| 2018-03-16T17:28:31.359-0400 W NETWORK [Uptime reporter] Unable to reach primary for set stale_mongos_updates_and_removes-configRS [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.364-0400 c20013| 2018-03-16T17:28:31.359-0400 D REPL_HB [replexec-0] Sending heartbeat (requestId: 1413) to ubuntu:20014, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.364-0400 c20014| 2018-03-16T17:28:31.360-0400 D REPL_HB [conn10] Received heartbeat request from ubuntu:20013, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20013", fromId: 1, term: 1, $replData: 1, $clusterTime: { clusterTime: Timestamp(1521235707, 41), signature: { hash: BinData(0, CEFAA93570840EC6E37F07AD52BBF4B956967DEE), keyId: 6533657568122765331 } }, $db: "admin" } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.364-0400 c20014| 2018-03-16T17:28:31.360-0400 D REPL_HB [conn10] Processed heartbeat from ubuntu:20013 and generated response, { ok: 1.0, state: 2, v: 2, hbmsg: "", set: "stale_mongos_updates_and_removes-configRS", term: 1, durableOpTime: { ts: Timestamp(1521235707, 41), t: 1 }, opTime: { ts: Timestamp(1521235707, 41), t: 1 } } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.365-0400 c20013| 2018-03-16T17:28:31.360-0400 D REPL_HB [replexec-1] Received response to heartbeat (requestId: 1413) from ubuntu:20014 but the heartbeat was cancelled. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.365-0400 c20013| 2018-03-16T17:28:31.360-0400 W SHARDING [signalProcessingThread] cant reload ShardRegistry :: caused by :: CallbackCanceled: Callback canceled [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.365-0400 c20013| 2018-03-16T17:28:31.360-0400 I FTDC [signalProcessingThread] Shutting down full-time diagnostic data capture [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.365-0400 c20013| 2018-03-16T17:28:31.362-0400 I STORAGE [WTOplogJournalThread] oplog journal thread loop shutting down [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.365-0400 c20013| 2018-03-16T17:28:31.362-0400 I STORAGE [signalProcessingThread] WiredTigerKVEngine shutting down [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.674-0400 c20013| 2018-03-16T17:28:31.673-0400 I STORAGE [signalProcessingThread] WiredTiger message [1521235711:673958][24322:0x7fc7d60dc700], txn-recover: Main recovery loop: starting at 1/343936 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.772-0400 c20013| 2018-03-16T17:28:31.772-0400 I STORAGE [signalProcessingThread] WiredTiger message [1521235711:772430][24322:0x7fc7d60dc700], txn-recover: Recovering log 1 through 2 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.837-0400 c20013| 2018-03-16T17:28:31.836-0400 I STORAGE [signalProcessingThread] WiredTiger message [1521235711:836687][24322:0x7fc7d60dc700], txn-recover: Recovering log 2 through 2 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.858-0400 c20014| 2018-03-16T17:28:31.858-0400 D REPL_HB [replexec-0] Sending heartbeat (requestId: 1412) to ubuntu:20013, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.859-0400 c20014| 2018-03-16T17:28:31.858-0400 D REPL_HB [replexec-2] Received response to heartbeat (requestId: 1412) from ubuntu:20013, { operationTime: Timestamp(1521235707, 41), ok: 0.0, errmsg: "interrupted at shutdown", code: 11600, codeName: "InterruptedAtShutdown", $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235707, 41), t: 1 }, lastOpVisible: { ts: Timestamp(1521235707, 41), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: -1, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235707, 41), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235707, 41) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.859-0400 c20014| 2018-03-16T17:28:31.858-0400 I REPL_HB [replexec-2] Error in heartbeat (requestId: 1412) to ubuntu:20013, response status: InterruptedAtShutdown: interrupted at shutdown [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.859-0400 c20014| 2018-03-16T17:28:31.858-0400 D REPL_HB [replexec-2] Scheduling heartbeat to ubuntu:20013 at 2018-03-16T21:28:31.858Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.859-0400 c20014| 2018-03-16T17:28:31.858-0400 D REPL_HB [replexec-2] Sending heartbeat (requestId: 1414) to ubuntu:20013, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.860-0400 c20014| 2018-03-16T17:28:31.858-0400 D REPL_HB [replexec-1] Received response to heartbeat (requestId: 1414) from ubuntu:20013, { operationTime: Timestamp(1521235707, 41), ok: 0.0, errmsg: "interrupted at shutdown", code: 11600, codeName: "InterruptedAtShutdown", $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235707, 41), t: 1 }, lastOpVisible: { ts: Timestamp(1521235707, 41), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: -1, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235707, 41), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235707, 41) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.860-0400 c20014| 2018-03-16T17:28:31.858-0400 I REPL_HB [replexec-1] Error in heartbeat (requestId: 1414) to ubuntu:20013, response status: InterruptedAtShutdown: interrupted at shutdown [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.860-0400 c20014| 2018-03-16T17:28:31.858-0400 D REPL_HB [replexec-1] Scheduling heartbeat to ubuntu:20013 at 2018-03-16T21:28:31.858Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.860-0400 c20014| 2018-03-16T17:28:31.858-0400 D REPL_HB [replexec-1] Sending heartbeat (requestId: 1416) to ubuntu:20013, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.861-0400 c20014| 2018-03-16T17:28:31.858-0400 D REPL_HB [replexec-0] Received response to heartbeat (requestId: 1416) from ubuntu:20013, { operationTime: Timestamp(1521235707, 41), ok: 0.0, errmsg: "interrupted at shutdown", code: 11600, codeName: "InterruptedAtShutdown", $replData: { term: 1, lastOpCommitted: { ts: Timestamp(1521235707, 41), t: 1 }, lastOpVisible: { ts: Timestamp(1521235707, 41), t: 1 }, configVersion: 2, replicaSetId: ObjectId('5aac36ef88c1f30185b7ac6f'), primaryIndex: -1, syncSourceIndex: -1 }, $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('000000000000000000000000') }, $clusterTime: { clusterTime: Timestamp(1521235707, 41), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lastCommittedOpTime: Timestamp(1521235707, 41) } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.861-0400 c20014| 2018-03-16T17:28:31.858-0400 I REPL_HB [replexec-0] Error in heartbeat (requestId: 1416) to ubuntu:20013, response status: InterruptedAtShutdown: interrupted at shutdown [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.861-0400 c20014| 2018-03-16T17:28:31.858-0400 I REPL [replexec-0] Member ubuntu:20013 is now in state RS_DOWN [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.862-0400 c20014| 2018-03-16T17:28:31.858-0400 D REPL_HB [replexec-0] Scheduling heartbeat to ubuntu:20013 at 2018-03-16T21:28:32.358Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.862-0400 c20014| 2018-03-16T17:28:31.859-0400 D REPL_HB [replexec-3] Sending heartbeat (requestId: 1418) to ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.862-0400 c20014| 2018-03-16T17:28:31.859-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.862-0400 c20014| 2018-03-16T17:28:31.859-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Failed to connect to ubuntu:20012 - HostUnreachable: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.862-0400 c20014| 2018-03-16T17:28:31.859-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to ubuntu:20012 due to failed operation on a connection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.862-0400 c20014| 2018-03-16T17:28:31.859-0400 I REPL_HB [replexec-2] Error in heartbeat (requestId: 1418) to ubuntu:20012, response status: HostUnreachable: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.863-0400 c20014| 2018-03-16T17:28:31.859-0400 D REPL_HB [replexec-2] Scheduling heartbeat to ubuntu:20012 at 2018-03-16T21:28:31.859Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.863-0400 c20014| 2018-03-16T17:28:31.859-0400 D REPL_HB [replexec-2] Sending heartbeat (requestId: 1420) to ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.863-0400 c20014| 2018-03-16T17:28:31.859-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.863-0400 c20014| 2018-03-16T17:28:31.859-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Failed to connect to ubuntu:20012 - HostUnreachable: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.863-0400 c20014| 2018-03-16T17:28:31.859-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to ubuntu:20012 due to failed operation on a connection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.864-0400 c20014| 2018-03-16T17:28:31.859-0400 I REPL_HB [replexec-1] Error in heartbeat (requestId: 1420) to ubuntu:20012, response status: HostUnreachable: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.864-0400 c20014| 2018-03-16T17:28:31.859-0400 D REPL_HB [replexec-1] Scheduling heartbeat to ubuntu:20012 at 2018-03-16T21:28:31.859Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.864-0400 c20014| 2018-03-16T17:28:31.859-0400 D REPL_HB [replexec-1] Sending heartbeat (requestId: 1422) to ubuntu:20012, { replSetHeartbeat: "stale_mongos_updates_and_removes-configRS", configVersion: 2, hbv: 1, from: "ubuntu:20014", fromId: 2, term: 1 } [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.864-0400 s20015| 2018-03-16T17:28:31.859-0400 D NETWORK [Uptime reporter] Starting new refresh of replica set stale_mongos_updates_and_removes-configRS [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.864-0400 c20014| 2018-03-16T17:28:31.859-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.864-0400 s20015| 2018-03-16T17:28:31.859-0400 D NETWORK [Uptime reporter] creating new connection to:ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.864-0400 c20014| 2018-03-16T17:28:31.859-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Failed to connect to ubuntu:20012 - HostUnreachable: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.865-0400 c20014| 2018-03-16T17:28:31.859-0400 I ASIO [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to ubuntu:20012 due to failed operation on a connection [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.865-0400 c20014| 2018-03-16T17:28:31.859-0400 I REPL_HB [replexec-3] Error in heartbeat (requestId: 1422) to ubuntu:20012, response status: HostUnreachable: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.865-0400 c20014| 2018-03-16T17:28:31.859-0400 D REPL_HB [replexec-3] Scheduling heartbeat to ubuntu:20012 at 2018-03-16T21:28:32.359Z [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.865-0400 s20015| 2018-03-16T17:28:31.859-0400 W NETWORK [Uptime reporter] Failed to connect to 127.0.1.1:20012, in(checking socket for error after poll), reason: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.865-0400 s20015| 2018-03-16T17:28:31.860-0400 D - [Uptime reporter] User Assertion: 40356:connection pool: connect failed ubuntu:20012 : couldn't connect to server ubuntu:20012, connection attempt failed src/mongo/client/connpool.cpp 395 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.866-0400 s20015| 2018-03-16T17:28:31.860-0400 I NETWORK [Uptime reporter] Marking host ubuntu:20013 as failed :: caused by :: CommandFailed: Failed to execute 'ismaster' command [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.866-0400 s20015| 2018-03-16T17:28:31.860-0400 W NETWORK [Uptime reporter] Unable to reach primary for set stale_mongos_updates_and_removes-configRS [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.895-0400 c20013| 2018-03-16T17:28:31.894-0400 I STORAGE [signalProcessingThread] WiredTiger message [1521235711:894908][24322:0x7fc7d60dc700], txn-recover: Set global recovery timestamp: 0 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.993-0400 c20013| 2018-03-16T17:28:31.993-0400 I STORAGE [signalProcessingThread] shutdown: removing fs lock... [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.993-0400 c20013| 2018-03-16T17:28:31.993-0400 I CONTROL [signalProcessingThread] now exiting [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.993-0400 c20013| 2018-03-16T17:28:31.993-0400 I CONTROL [signalProcessingThread] shutting down with code:0 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.999-0400 c20014| 2018-03-16T17:28:31.998-0400 I NETWORK [conn11] end connection 127.0.0.1:44426 (5 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.999-0400 c20014| 2018-03-16T17:28:31.998-0400 I NETWORK [conn10] end connection 127.0.0.1:44418 (4 connections now open) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.999-0400 c20014| 2018-03-16T17:28:31.998-0400 I CONTROL [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.999-0400 c20014| 2018-03-16T17:28:31.999-0400 I NETWORK [signalProcessingThread] shutdown: going to close listening sockets... [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.999-0400 c20014| 2018-03-16T17:28:31.999-0400 I NETWORK [signalProcessingThread] removing socket file: /tmp/mongodb-20014.sock [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:31.999-0400 c20014| 2018-03-16T17:28:31.999-0400 I REPL [signalProcessingThread] shutdown: removing checkpointTimestamp collection... [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:32.000-0400 c20014| 2018-03-16T17:28:31.999-0400 I STORAGE [signalProcessingThread] Finishing collection drop for local.replset.checkpointTimestamp (079a37a7-d913-4e65-99ef-8f2907ef336e). [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:32.000-0400 c20014| 2018-03-16T17:28:32.000-0400 I REPL [signalProcessingThread] shutting down replication subsystems [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:32.001-0400 c20014| 2018-03-16T17:28:32.000-0400 I REPL [signalProcessingThread] Stopping replication reporter thread [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:32.001-0400 c20014| 2018-03-16T17:28:32.000-0400 I REPL [SyncSourceFeedback] SyncSourceFeedback error sending update to ubuntu:20012: CallbackCanceled: Reporter no longer valid [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:32.001-0400 c20014| 2018-03-16T17:28:32.001-0400 I REPL [signalProcessingThread] Stopping replication fetcher thread [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:32.001-0400 c20014| 2018-03-16T17:28:32.001-0400 I REPL [signalProcessingThread] Stopping replication applier thread [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:32.357-0400 c20014| 2018-03-16T17:28:32.357-0400 I REPL [signalProcessingThread] Stopping replication storage threads [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:32.358-0400 c20014| 2018-03-16T17:28:32.358-0400 W SHARDING [signalProcessingThread] cant reload ShardRegistry :: caused by :: CallbackCanceled: Callback canceled [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:32.359-0400 c20014| 2018-03-16T17:28:32.358-0400 I FTDC [signalProcessingThread] Shutting down full-time diagnostic data capture [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:32.360-0400 c20014| 2018-03-16T17:28:32.360-0400 I STORAGE [WTOplogJournalThread] oplog journal thread loop shutting down [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:32.361-0400 c20014| 2018-03-16T17:28:32.360-0400 I STORAGE [signalProcessingThread] WiredTigerKVEngine shutting down [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:32.361-0400 s20015| 2018-03-16T17:28:32.360-0400 D NETWORK [Uptime reporter] Starting new refresh of replica set stale_mongos_updates_and_removes-configRS [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:32.361-0400 s20015| 2018-03-16T17:28:32.360-0400 D NETWORK [Uptime reporter] creating new connection to:ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:32.361-0400 s20015| 2018-03-16T17:28:32.361-0400 W NETWORK [Uptime reporter] Failed to connect to 127.0.1.1:20012, in(checking socket for error after poll), reason: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:32.361-0400 s20015| 2018-03-16T17:28:32.361-0400 D - [Uptime reporter] User Assertion: 40356:connection pool: connect failed ubuntu:20012 : couldn't connect to server ubuntu:20012, connection attempt failed src/mongo/client/connpool.cpp 395 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:32.362-0400 s20015| 2018-03-16T17:28:32.361-0400 I NETWORK [Uptime reporter] Marking host ubuntu:20014 as failed :: caused by :: CommandFailed: Failed to execute 'ismaster' command [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:32.362-0400 s20015| 2018-03-16T17:28:32.361-0400 D NETWORK [Uptime reporter] SocketException: remote: (NONE):0 error: SocketException socket exception [CLOSED] server [127.0.1.1:20013] [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:32.362-0400 s20015| 2018-03-16T17:28:32.361-0400 D - [Uptime reporter] User Assertion: 6:network error while attempting to run command 'ismaster' on host 'ubuntu:20013' src/mongo/client/dbclient.cpp 238 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:32.362-0400 s20015| 2018-03-16T17:28:32.361-0400 I NETWORK [Uptime reporter] Detected bad connection created at 1521235700793678 microSec, clearing pool for ubuntu:20013 of 0 connections [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:32.362-0400 s20015| 2018-03-16T17:28:32.361-0400 I NETWORK [Uptime reporter] Dropping all pooled connections to ubuntu:20013(with timeout of 5 seconds) [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:32.363-0400 s20015| 2018-03-16T17:28:32.361-0400 I NETWORK [Uptime reporter] Ending connection to host ubuntu:20013(with timeout of 5 seconds) due to bad connection status; 0 connections to that host remain open [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:32.363-0400 s20015| 2018-03-16T17:28:32.361-0400 W NETWORK [Uptime reporter] Unable to reach primary for set stale_mongos_updates_and_removes-configRS [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:32.363-0400 s20015| 2018-03-16T17:28:32.361-0400 I NETWORK [Uptime reporter] Cannot reach any nodes for set stale_mongos_updates_and_removes-configRS. Please check network connectivity and the status of the set. This has happened for 1 checks in a row. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:32.726-0400 c20014| 2018-03-16T17:28:32.726-0400 I STORAGE [signalProcessingThread] WiredTiger message [1521235712:726719][24353:0x7f836d4a9700], txn-recover: Main recovery loop: starting at 1/342784 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:32.834-0400 c20014| 2018-03-16T17:28:32.833-0400 I STORAGE [signalProcessingThread] WiredTiger message [1521235712:833832][24353:0x7f836d4a9700], txn-recover: Recovering log 1 through 2 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:32.862-0400 s20015| 2018-03-16T17:28:32.861-0400 D NETWORK [Uptime reporter] Starting new refresh of replica set stale_mongos_updates_and_removes-configRS [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:32.862-0400 s20015| 2018-03-16T17:28:32.861-0400 D NETWORK [Uptime reporter] creating new connection to:ubuntu:20012 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:32.862-0400 s20015| 2018-03-16T17:28:32.862-0400 W NETWORK [Uptime reporter] Failed to connect to 127.0.1.1:20012, in(checking socket for error after poll), reason: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:32.862-0400 s20015| 2018-03-16T17:28:32.862-0400 D - [Uptime reporter] User Assertion: 40356:connection pool: connect failed ubuntu:20012 : couldn't connect to server ubuntu:20012, connection attempt failed src/mongo/client/connpool.cpp 395 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:32.862-0400 s20015| 2018-03-16T17:28:32.862-0400 D NETWORK [Uptime reporter] creating new connection to:ubuntu:20013 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:32.863-0400 s20015| 2018-03-16T17:28:32.862-0400 W NETWORK [Uptime reporter] Failed to connect to 127.0.1.1:20013, in(checking socket for error after poll), reason: Connection refused [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:32.863-0400 s20015| 2018-03-16T17:28:32.862-0400 D - [Uptime reporter] User Assertion: 40356:connection pool: connect failed ubuntu:20013 : couldn't connect to server ubuntu:20013, connection attempt failed src/mongo/client/connpool.cpp 395 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:32.863-0400 s20015| 2018-03-16T17:28:32.862-0400 W NETWORK [Uptime reporter] Unable to reach primary for set stale_mongos_updates_and_removes-configRS [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:32.863-0400 s20015| 2018-03-16T17:28:32.862-0400 I NETWORK [Uptime reporter] Cannot reach any nodes for set stale_mongos_updates_and_removes-configRS. Please check network connectivity and the status of the set. This has happened for 2 checks in a row. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:32.901-0400 c20014| 2018-03-16T17:28:32.901-0400 I STORAGE [signalProcessingThread] WiredTiger message [1521235712:901086][24353:0x7f836d4a9700], txn-recover: Recovering log 2 through 2 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:32.964-0400 c20014| 2018-03-16T17:28:32.964-0400 I STORAGE [signalProcessingThread] WiredTiger message [1521235712:964594][24353:0x7f836d4a9700], txn-recover: Set global recovery timestamp: 0 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:33.058-0400 c20014| 2018-03-16T17:28:33.058-0400 I STORAGE [signalProcessingThread] shutdown: removing fs lock... [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:33.059-0400 c20014| 2018-03-16T17:28:33.058-0400 I CONTROL [signalProcessingThread] now exiting [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:33.059-0400 c20014| 2018-03-16T17:28:33.058-0400 I CONTROL [signalProcessingThread] shutting down with code:0 [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:33.063-0400 s20015| 2018-03-16T17:28:33.063-0400 I CONTROL [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:33.064-0400 s20015| 2018-03-16T17:28:33.063-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:33.064-0400 s20015| 2018-03-16T17:28:33.063-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:33.064-0400 s20015| 2018-03-16T17:28:33.064-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:33.064-0400 s20015| 2018-03-16T17:28:33.064-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:33.064-0400 s20015| 2018-03-16T17:28:33.064-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:33.064-0400 s20015| 2018-03-16T17:28:33.064-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:33.065-0400 s20015| 2018-03-16T17:28:33.064-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:33.065-0400 s20015| 2018-03-16T17:28:33.064-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:33.065-0400 s20015| 2018-03-16T17:28:33.064-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:33.065-0400 s20015| 2018-03-16T17:28:33.064-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:33.065-0400 s20015| 2018-03-16T17:28:33.064-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:33.065-0400 s20015| 2018-03-16T17:28:33.064-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:33.065-0400 s20015| 2018-03-16T17:28:33.064-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:33.066-0400 s20015| 2018-03-16T17:28:33.064-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:33.066-0400 s20015| 2018-03-16T17:28:33.064-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:33.066-0400 s20015| 2018-03-16T17:28:33.064-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:33.066-0400 s20015| 2018-03-16T17:28:33.064-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:33.066-0400 s20015| 2018-03-16T17:28:33.064-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:33.066-0400 s20015| 2018-03-16T17:28:33.064-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:33.066-0400 s20015| 2018-03-16T17:28:33.064-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:33.066-0400 s20015| 2018-03-16T17:28:33.064-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:33.067-0400 s20015| 2018-03-16T17:28:33.064-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:33.067-0400 s20015| 2018-03-16T17:28:33.064-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:33.067-0400 s20015| 2018-03-16T17:28:33.065-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:33.067-0400 s20015| 2018-03-16T17:28:33.065-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:33.067-0400 s20015| 2018-03-16T17:28:33.065-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:33.067-0400 s20015| 2018-03-16T17:28:33.065-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:33.067-0400 s20015| 2018-03-16T17:28:33.065-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:33.068-0400 s20015| 2018-03-16T17:28:33.065-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:33.068-0400 s20015| 2018-03-16T17:28:33.065-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:33.068-0400 s20015| 2018-03-16T17:28:33.065-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:33.068-0400 s20015| 2018-03-16T17:28:33.065-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:33.068-0400 s20015| 2018-03-16T17:28:33.065-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:33.068-0400 s20015| 2018-03-16T17:28:33.065-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:33.068-0400 s20015| 2018-03-16T17:28:33.065-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:33.069-0400 s20015| 2018-03-16T17:28:33.065-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:33.069-0400 s20015| 2018-03-16T17:28:33.065-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:33.069-0400 s20015| 2018-03-16T17:28:33.065-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:33.069-0400 s20015| 2018-03-16T17:28:33.065-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:33.069-0400 s20015| 2018-03-16T17:28:33.065-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:33.069-0400 s20015| 2018-03-16T17:28:33.065-0400 D ASIO [signalProcessingThread] NetworkInterfaceASIO shutdown successfully [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:33.069-0400 s20015| 2018-03-16T17:28:33.065-0400 D SHARDING [signalProcessingThread] ShardingCatalogClientImpl::shutDown() called. [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:33.070-0400 s20015| 2018-03-16T17:28:33.066-0400 W SHARDING [signalProcessingThread] error encountered while cleaning up distributed ping entry for ubuntu:20015:1521235700:-7388722962896244705 :: caused by :: InterruptedAtShutdown: interrupted at shutdown [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:33.070-0400 s20015| 2018-03-16T17:28:33.066-0400 I FTDC [signalProcessingThread] Shutting down full-time diagnostic data capture [js_test:stale_mongos_updates_and_removes] 2018-03-16T17:28:33.070-0400 s20015| 2018-03-16T17:28:33.066-0400 I CONTROL [signalProcessingThread] shutting down with code:0 [executor:js_test:job0] 2018-03-16T17:28:33.074-0400 stale_mongos_updates_and_removes.js ran in 28.57 seconds. [executor:js_test:job0] 2018-03-16T17:28:33.074-0400 JSTest jstests/sharding/stale_mongos_updates_and_removes.js failed, so stopping... [executor:js_test:job0] 2018-03-16T17:28:33.074-0400 Received a StopExecution exception: JSTest jstests/sharding/stale_mongos_updates_and_removes.js failed. [executor] 2018-03-16T17:28:33.081-0400 Summary: 1 test(s) ran in 28.58 seconds (0 succeeded, 0 were skipped, 1 failed, 0 errored) The following tests failed (with exit code): jstests/sharding/stale_mongos_updates_and_removes.js (253) [resmoke] 2018-03-16T17:28:33.081-0400 ================================================================================ [resmoke] 2018-03-16T17:28:33.081-0400 Summary of sharding_last_stable_mongos_and_mixed_shards suite: 1 test(s) ran in 28.58 seconds (0 succeeded, 0 were skipped, 1 failed, 0 errored) The following tests failed (with exit code): jstests/sharding/stale_mongos_updates_and_removes.js (253) 1 test(s) ran in 28.58 seconds (0 succeeded, 0 were skipped, 1 failed, 0 errored) js_tests: 1 test(s) ran in 28.58 seconds (0 succeeded, 0 were skipped, 1 failed, 0 errored) The following tests failed (with exit code): jstests/sharding/stale_mongos_updates_and_removes.js (253)