Quoting the causal consistency spec:

The server reports its view of the current cluster time in the $clusterTime field in responses from the server and the driver participates in distributing the current cluster time to all nodes (called "gossipping the cluster time") by sending the highest $clusterTime it has seen so far in messages it sends to mongos servers. The current cluster time is a logical time, but is digitally signed to prevent malicious clients from propagating invalid cluster times. Cluster time is only used in replica sets and sharded clusters.

This definition only alludes to gossipping the cluster time to mongos servers, and says that cluster time is only used in replica sets or shard clusters with no mention of server/wire version.

According to the driver sessions spec on gossiping the cluster time:

Drivers MUST gossip the cluster time when connected to a deployment that uses cluster times.

A driver detects that it MUST participate in gossipping the cluster time when it sees a $clusterTime in a response received from a server.

Drivers MUST check that the server they are sending a command to supports $clusterTime before adding $clusterTime to the command. A server supports $clusterTime when the maxWireVersion >= 6.

Reading various bits of the driver sessions spec, we can conclude that $clusterTime is only supported by servers with maxWireVersion >= 6 that are also members of a replica set or shard cluster (not standalone servers).

The definition in the causal consistency spec could be improved by stating this requirement explicitly, and not exclusively referring to mongos on the subject of gossipping.