[SERVER-44823] Sharding support for $planCacheStats Created: 25/Nov/19  Updated: 29/Oct/23  Resolved: 09/Dec/19

Status: Closed
Project: Core Server
Component/s: Querying
Affects Version/s: None
Fix Version/s: 4.3.3

Type: Improvement Priority: Major - P3
Reporter: David Storch Assignee: David Storch
Resolution: Fixed Votes: 0
Labels: qexec-team
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by DRIVERS-805 Sharding support for $planCacheStats Closed
Documented
is documented by DOCS-13247 Investigate changes in SERVER-44701: ... Closed
Related
related to SERVER-34633 Allow $currentOp to retrieve operatio... Closed
related to SERVER-45032 Allow $planCacheStats to target every... Closed
Backwards Compatibility: Fully Compatible
Sprint: Query 2019-12-16
Participants:

 Description   

Currently, an aggregate operation which reads the plan cache as a "virtual collection" using $planCacheStats is not supported when connected to mongos:

mongos> db.testColl.aggregate([{$planCacheStats: {}}])
2019-11-25T10:55:59.025-0500 E  QUERY    [js] uncaught exception: Error: command failed: {
	"ok" : 0,
	"errmsg" : "$planCacheStats cannot be executed against a MongoS.",
	"code" : 50932,
	"codeName" : "Location50932",
	"operationTime" : Timestamp(1574697353, 1),
	"$clusterTime" : {
		"clusterTime" : Timestamp(1574697353, 1),
		"signature" : {
			"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
			"keyId" : NumberLong(0)
		}
	}
} : aggregate failed :
_getErrorWithCode@src/mongo/shell/utils.js:25:13
doassert@src/mongo/shell/assert.js:18:14
_assertCommandWorked@src/mongo/shell/assert.js:612:17
assert.commandWorked@src/mongo/shell/assert.js:702:16
DB.prototype._runAggregate@src/mongo/shell/db.js:266:5
DBCollection.prototype.aggregate@src/mongo/shell/collection.js:1012:12
@(shell):1:1

Instead, users are expected to connect directly to the mongod of interest in order to examine that node's plan cache.

In some environments (e.g. Atlas), it may be difficult or discouraged to connect directly to a shardsvr. Furthermore, some users may wish to examine the plan caches on all nodes before drilling down into particular nodes of interest. Therefore, we should add support for $planCacheStats issued via a mongos. The most sensible behavior for such an operation would be to return the union of the plan cache entries from every shardsvr node in the cluster (as opposed to obeying the read preference and returning the plan caches for a particular node in each shard). This may require some work in the sharding infrastructure to allow an aggregate operation to target every node. The current infrastructure typically assumes that at most one host in each shard is targeted.

Finally, in order to allow users to filter, sort, group, etc. based on the host, we should augment each plan cache entry document in the result set with host:port information in the case of a sharded $planCacheStats.



 Comments   
Comment by Githook User [ 09/Dec/19 ]

Author:

{'name': 'David Storch', 'username': 'dstorch', 'email': 'david.storch@mongodb.com'}

Message: SERVER-44823 Add sharding support for $planCacheStats.

When a $planCacheStats pipeline is delivered to a mongos, it
will be forwarded to one host in each shard following the
normal host targeting rules for aggregate operations. Each
of the documents returned to the client will be augmented
with "shard" and "host" fields, giving the name of the shard
and the host:port string from which the plan cache metadata
originated.

Allowing clients to collect plan cache information from
every shardsvr in the cluster is left as future work.
Branch: master
https://github.com/mongodb/mongo/commit/01e9fd6d0ec033089ba2f45a440619d3a08c9d9f

Comment by David Storch [ 09/Dec/19 ]

The changes planned for this ticket will allow $planCacheStats to be issued against a mongos, but will not implement behavior in which the mongos targets every data-bearing node in the cluster. Instead, mongos will target a single host from every shard using the normal read preference rules for host selection. I have filed SERVER-45032 to track the follow-up work of allowing users to gather plan cache stats from every node in every shard by running a single $planCacheStats aggregate operation against a mongos.

This change adds a new "host" field to every document returned by $planCacheStats, which will contain the "host:port" string of the mongod from which the cache entry document originated. This makes it easy for users to understand which node's cache they are reading when connected to a replica set, and also disambiguates cache entries which may have come from different hosts when connected to a mongos. When the $planCacheStats operation is run through a mongos, each cache entry document will additionally contain a "shard" field which contains the shard name from which the document originated. This can be used in a similar fashion to "host". For example, $planCacheStats queries can use MQL to sort, group, or filter the results by shard name.

The following example demonstrates what $planCacheStats output looks like in a sharded scenario. I created a collection testDb.source which has chunks on two shards:

MongoDB Enterprise mongos> use testDb
switched to db testDb
MongoDB Enterprise mongos> sh.status()
--- Sharding Status ---
  sharding version: {
  	"_id" : 1,
  	"minCompatibleVersion" : 5,
  	"currentVersion" : 6,
  	"clusterId" : ObjectId("5deebc4fd7936beba0f5e30d")
  }
  shards:
        {  "_id" : "testCluster-rs0",  "host" : "testCluster-rs0/storchbox:20000",  "state" : 1 }
        {  "_id" : "testCluster-rs1",  "host" : "testCluster-rs1/storchbox:20001",  "state" : 1 }
...
  databases:
        {  "_id" : "config",  "primary" : "config",  "partitioned" : true }
                config.system.sessions
                        shard key: { "_id" : 1 }
                        unique: false
                        balancing: true
                        chunks:
                                testCluster-rs0	1
                        { "_id" : { "$minKey" : 1 } } -->> { "_id" : { "$maxKey" : 1 } } on : testCluster-rs0 Timestamp(1, 0)
        {  "_id" : "testDb",  "primary" : "testCluster-rs1",  "partitioned" : true,  "version" : {  "uuid" : UUID("0eded88a-1a28-49f2-8024-d254ca72e612"),  "lastMod" : 1 } }
                testDb.source
                        shard key: { "_id" : 1 }
                        unique: false
                        balancing: true
                        chunks:
                                testCluster-rs0	1
                                testCluster-rs1	1
                        { "_id" : { "$minKey" : 1 } } -->> { "_id" : 0 } on : testCluster-rs0 Timestamp(3, 0)
                        { "_id" : 0 } -->> { "_id" : { "$maxKey" : 1 } } on : testCluster-rs1 Timestamp(3, 1)

Next, created indexes and ran a query in order to produce a plan cache entry on the primary node of each shard:

MongoDB Enterprise mongos> db.source.createIndex({a: 1})
{
	"raw" : {
		"testCluster-rs1/storchbox:20001" : {
			"numIndexesBefore" : 2,
			"numIndexesAfter" : 2,
			"note" : "all indexes already exist",
			"ok" : 1
		},
		"testCluster-rs0/storchbox:20000" : {
			"createdCollectionAutomatically" : false,
			"numIndexesBefore" : 1,
			"numIndexesAfter" : 2,
			"commitQuorum" : 1,
			"ok" : 1
		}
	},
	"ok" : 1,
	"operationTime" : Timestamp(1575926963, 12),
	"$clusterTime" : {
		"clusterTime" : Timestamp(1575926963, 12),
		"signature" : {
			"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
			"keyId" : NumberLong(0)
		}
	}
}
MongoDB Enterprise mongos> db.source.createIndex({b: 1})
{
	"raw" : {
		"testCluster-rs0/storchbox:20000" : {
			"createdCollectionAutomatically" : false,
			"numIndexesBefore" : 2,
			"numIndexesAfter" : 3,
			"commitQuorum" : 1,
			"ok" : 1
		},
		"testCluster-rs1/storchbox:20001" : {
			"createdCollectionAutomatically" : false,
			"numIndexesBefore" : 2,
			"numIndexesAfter" : 3,
			"commitQuorum" : 1,
			"ok" : 1
		}
	},
	"ok" : 1,
	"operationTime" : Timestamp(1575926966, 2),
	"$clusterTime" : {
		"clusterTime" : Timestamp(1575926966, 2),
		"signature" : {
			"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
			"keyId" : NumberLong(0)
		}
	}
}
MongoDB Enterprise mongos> db.source.find({a: 1, b: 1})

Finally, I ran a query that returns the plan cache entries from the primary node of both shards. In particular, note the values of the "host" and "shard" fields for each of the two result documents:

MongoDB Enterprise mongos> db.source.aggregate([{$planCacheStats: {}}]).pretty()
{
	"createdFromQuery" : {
		"query" : {
			"a" : 1,
			"b" : 1
		},
		"sort" : {
 
		},
		"projection" : {
 
		}
	},
	"queryHash" : "43CAB4C5",
	"planCacheKey" : "CEC1F6AF",
	"isActive" : false,
	"works" : NumberLong(1),
	"cachedPlan" : {
		"stage" : "SHARDING_FILTER",
		"inputStage" : {
			"stage" : "FETCH",
			"filter" : {
				"b" : {
					"$eq" : 1
				}
			},
			"inputStage" : {
				"stage" : "IXSCAN",
				"keyPattern" : {
					"a" : 1
				},
				"indexName" : "a_1",
				"isMultiKey" : false,
				"multiKeyPaths" : {
					"a" : [ ]
				},
				"isUnique" : false,
				"isSparse" : false,
				"isPartial" : false,
				"indexVersion" : 2,
				"direction" : "forward",
				"indexBounds" : {
					"a" : [
						"[1.0, 1.0]"
					]
				}
			}
		}
	},
	"timeOfCreation" : ISODate("2019-12-09T21:30:26.753Z"),
	"creationExecStats" : [
              ...
	],
	"candidatePlanScores" : [
		1.0002,
		1.0002,
		1.0000999999999998
	],
	"indexFilterSet" : false,
	"host" : "storchbox:20000",
	"shard" : "testCluster-rs0"
}
{
	"createdFromQuery" : {
		"query" : {
			"a" : 1,
			"b" : 1
		},
		"sort" : {
 
		},
		"projection" : {
 
		}
	},
	"queryHash" : "43CAB4C5",
	"planCacheKey" : "CEC1F6AF",
	"isActive" : false,
	"works" : NumberLong(1),
	"cachedPlan" : {
		"stage" : "SHARDING_FILTER",
		"inputStage" : {
			"stage" : "FETCH",
			"filter" : {
				"b" : {
					"$eq" : 1
				}
			},
			"inputStage" : {
				"stage" : "IXSCAN",
				"keyPattern" : {
					"a" : 1
				},
				"indexName" : "a_1",
				"isMultiKey" : false,
				"multiKeyPaths" : {
					"a" : [ ]
				},
				"isUnique" : false,
				"isSparse" : false,
				"isPartial" : false,
				"indexVersion" : 2,
				"direction" : "forward",
				"indexBounds" : {
					"a" : [
						"[1.0, 1.0]"
					]
				}
			}
		}
	},
	"timeOfCreation" : ISODate("2019-12-09T21:30:26.754Z"),
	"creationExecStats" : [
		...
	],
	"candidatePlanScores" : [
		1.0002,
		1.0002,
		1.0000999999999998
	],
	"indexFilterSet" : false,
	"host" : "storchbox:20001",
	"shard" : "testCluster-rs1"
}

Comment by Kevin Pulo [ 26/Nov/19 ]

$currentOp could also benefit from being able to target all data-bearing members (see SERVER-34633).

Generated at Thu Feb 08 05:07:05 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.