RPC metadata (e.g. read preference) can be specified in a number of ways to a number of different RPC APIs. For example:

• In the metadata field of RemoteCommandRequest, which is passed directly to a TaskExecutor
  • ShardRemote uses this via its _appendMetadataForCommand
  • AsyncRequestSender also uses this
• Via the GenericArguments portion of an IDL-defined command struct
  • async_rpc functions this way
  • However, some ShardRemote methods accept IDL-defined command structs too, so it can accept metadata this way as well.
• By EgressMetadataHook implementations
  • VectorClockMetadataHook, ClientMetadataPropagationEgressHook, and RoutingTableCacheGossipMetadataHook are examples of this.
• Through ForwardableOperationMetadata, which is set on the opCtx
  • This sets things like the Comment, WriteBlockBypass, or authentication information on the opCtx.
  • comment in particular is set in the RemoteCommandRequest constructor
  • writeBlockBypass is serialized through the ClientMetadataPropagationEgressMetadataHook

These various ways can conflict with each other too (SERVER-97901), which requires repeated scans of BSON data to ensure that no duplicate keys are appended, which can be a performance concern. It's also sometimes difficult to determine precedence in these situations, which makes the code harder to reason about.

We should simplify this so there is preferably one unambiguous way to attach this kind of data. To start, we should try to expose fewer top-level RPC APIs that accept raw BSONObjs, and instead try to only use IDL-defined command structs. This will make duplicate checking/reassignment cheap and unambiguous.