Description

This is a specific manifestation of SERVER-38174 that wasn't considered on that ticket.

Until (and including) 3.4, the supported and documented procedure to resize the oplog involved:

• restarting as a standalone,
• saving the last oplog entry,
• dropping the oplog,
• recreating the oplog,
• adding the saved oplog entry.

In 3.6 this was changed to a "live resize" using the replSetResizeOplog command.

However, if the old manual resize procedure is performed on an uncleanly shutdown 4.0 mongod, it can result in unapplied oplog entries being lost. This will cause the data on that node to no longer be consistent with the rest of the replica set (which can in turn lead to unspecified problems later on).

Reproducer:

• Start a 2 member replset.
• Run the attached check-confirmed.js script against the primary.
• kill -9 the primary.
• Restart the primary as a standalone.
• Confirm there are at least 2 unapplied oplog entries, ie. the _id values mentioned by the last >= 2 oplog entries have different (old) values in the collection.
• Manually resize the oplog.
• Return the member to the replset.
• Query for the _ids of the affected unapplied oplog entries (except the final oplog entry). The affected node will have the old value, whereas the other node will have the correct value.

Remediation:

Given that the old procedure has been around forever, it's reasonable to believe that experienced MongoDB admins:

• might not be aware of the change in documented procedure
• might perform the procedure on 4.0+ replica set members.

It's also not hard to imagine scenarios where this might be done after an unclean shutdown, for example, an admin wanting to reduce the size of the oplog after an out-of-disk-space crash.

This ticket is to request:

1. Entry in the Replica Sets section of the 4.0 Release Notes mentioning that manually resizing the oplog is not safe in 4.0, and therefore the NEW replSetResizeOplog procedure must be used instead.
2. Adjust the replSetResizeOplog entry in the Replica Sets section of the 3.6 Release Notes to mention that for storage engines which support this command, manually resizing the oplog is no longer supported.
3. Admonishments on the old procedure (3.4 and earlier branches) stating that it is not safe to perform on 4.0+ replica set members.
4. Admonishments on the new procedure (3.6 and later branches) stating that it is not safe to perform the OLD procedure on 4.0+ replica set members, and that the NEW replSetResizeOplog procedure MUST be used.

Scope of changes

Impact to Other Docs

MVP (Work and Date)

Resources (Scope or Design Docs, Invision, etc.)