During steady state replication, due to the way that SECONDARY nodes group 'insert' oplog entries, the BSON maximum document size of 16MB may be exceeded if we try to group together a batch that include documents at or near the 16MB limit.
The predicate for how we delimit a group cuts off the group at the first op that exceeds the group size limit, but when we put these ops into a group here, it could put us over a maximum document size when we try to construct a BSON object here. This is due to the incorrect way that we measure the size of a batch. Given a list of ops, we start iterating from the second op, and initialize the batchSize to zero. Thus, we don't actually include the size of the first op in our measurement of the batch size, which is how we could be allowed to exceed document size limits when our ops are very close to 16MB.
The logic in place to handle insert grouping is somewhat hard to follow as is. This bug can be addressed by limiting batch sizes correctly by including the sizes of all ops, and making sure we delimit the group correctly based on this. Ideally, the grouping logic can be cleaned up at the same time to make it more clear that is correct.
A JS test reproducing this issue is attached. It reproduced the issue reliably running on MongoDB 3.5.7.