In some use cases inserts into the _id index are done in monotonically increasing _id order on the primary, and that is a case that WT optimizes for. However because of the way oplog entries are divided among the worker threads in SyncTail::fillWriterVectors on the secondary, the entries in the _id index may not be inserted in order on the secondary, and this is a less optimal path under WT. This can occur for example if multiple collections are being inserted into, one thread per collection, using a monotonic _id, such as ObjectId; in this case inserts into the _id index are in order in each WT index table on the primary, but not on the secondary. (This issue does not occur with multiple threads inserting into a single collection because the _id index inserts are not in order on either the primary or the secondary, so parity is maintained).

We can control whether inserts are ordered or disordered on the primary by either

• inserting into either 1 collection with 8 threads (disordered), or 8 separate collections with 8 threads (ordered)
• using either monotonic _ids (ordered) or random _ids (disordered)

Inserts are always disordered on the secondary, so by observing secondary performance relative to primary while varying ordering on the primary as above we can measure the relative impact of the disordered inserts on the secondary. Secondary processing rate relative to primary was measured by comparing number of entries inserted into oplog on secondary with number on primary at a particular point in time during the run:

                                        secondary/primary   relative
                                                            performance
8 collections, monotonic _ids		15273776/20179477 = 0.76           inserts ordered on primary, disordered on secondary
8 collections, random ids		17189861/20104144 = 0.86           inserts disordered on both primary and secondary
1 collection, monotonic _ids		17123525/20187676 = 0.85           inserts disordered on both primary and secondary

Note that performance of primary relative to secondary is reduced by either using random _ids, or by inserting into a single collection with multiple threads, both of which disorder the inserts into the _id index on the primary, to match the disordered inserts on the secondary.

Finally, we can achieve parity between the primary and the secondary by both

• incorporating the potential improvement of parallelizing oplog inserts (both bullets below)
• eliminating the issue described in this ticket by disordering the inserts on the primary by using a single collection (second bullet below)

                                          secondary/primary   relative
                                                              performance
  8 collections, monotonic _ids		18699167/20148811 = 0.93
  1 collection, monotonic _ids		20056367/20060385 = 0.99
  

It's unclear whether this issue can be addressed in the mongod layer, or whether an improvement in WT for the out-of-order insertion case is needed.