3.5.x versions of the server have two implementations of the update subsystem: the "old" (3.4 and earlier) system, and the new system in src/mongo/db/update which is both more performant and supports more expressive array updates. The old and new systems have different behavior with respect to field ordering. In order to ensure that the field ordering is consistent across all nodes in the replica set, the primary and secondaries must use the same version of the update subsystem. The is achieved via the feature compatibility version mechanism. Users must set the feature compatibility version (FCV) to "3.6" in order to enable the new update system.

The FCV check, however, does not guarantee that a given update uses the same version of the update code on every node. Consider the following sequence of events:

1. A two node replica set is started. Both nodes are version 3.6 but have FCV "3.4".
2. The client concurrently issues an update and setFeatureCompatibilityVersion("3.6"). These operations take compatible locks, and therefore execute concurrently on the server.
3. The setFCV command writes its update to admin.system.version to the oplog at optime t.
4. After this oplog entry is written but before the in-memory FCV state changes, the update is logged with some optime greater than t. This uses the old update system, since the FCV in-memory state has not yet been changed.
5. The two oplog entries are applied on the secondary. Since the admin.system.version write has an earlier optime (and must be applied in its own batch), the update uses the new update system.

I was able to reproduce a dbhash mismatch against a two-node 3.5.x replica set by running two scripts concurrently from two shells connected to the primary node. The first script repeatedly issues an update with two $set's, that will result in different field ordering depending on which version of the update implementation is used:

(function() {
    "use strict";

    db.c.drop();
    for (var i = 0; i < 1000; i++) {
        assert.writeOK(db.c.insert({_id: i}));
        assert.writeOK(db.c.update({_id: i}, {$set: {b: 1, a: 1}}));
    }
}());

The second script repeatedly sets the FCV from "3.4" to "3.6" and back again:

(function() {
    "use strict";

    while (true) {
        assert.commandWorked(db.adminCommand({setFeatureCompatibilityVersion: "3.4"}))
        assert.commandWorked(db.adminCommand({setFeatureCompatibilityVersion: "3.6"}))
    }
}());

After the first script completes, running the dbHash command against the test database on each node should show different hashes for test.c.