[SERVER-84734] Oplog entries can come close to 16 MB + 16 KB size Created: 10/Jan/24  Updated: 16/Jan/24

Status: Open
Project: Core Server
Component/s: None
Affects Version/s: 6.0.0, 7.0.0
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Vishnu Kaushik Assignee: Backlog - Replication Team
Resolution: Unresolved Votes: 0
Labels: repl-shortlist
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Problem/Incident
Related
is related to SERVER-61891 Add document key to the insert oplog ... Closed
is related to SERVER-84722 Create undocumented server parameter ... Closed
Assigned Teams:
Replication
Operating System: ALL
Participants:

 Description   

The maximum size of a BSON document is 16 MB. Since insert oplog entries must contain the entire BSON document on insert, along with other metadata, we allow for 16 KB of buffer space for internal BSON documents - therefore the internal limit is 16 MB + 16 KB - otherwise no oplog entry would be able to contain an inserted BSON document of size 16 MB.

However, usually there isn't enough metadata in an oplog entry for it to actually reach the internal limit of 16 MB + 16 KB. Unfortunately, after SERVER-61891, it became possible for oplog entries to come close to the internal limit of 16 MB + 16 KB (when inserts with big _id s are performed). This isn't problematic at the time of writing the oplog, since the oplog entry is still under the limit, but it is at the time of reading the oplog entry.

When you read an oplog entry, the cursor response, which itself is an internal BSON and is therefore subject to the 16 MB + 16 KB, will contain the oplog entry, along with more metadata. Since the oplog entry itself has come close to the 16 MB + 16 KB limit, the cursor response is unable to hold the oplog entry along with more metadata as this causes the response to cross the 16 MB + 16 KB limit.

This can cause replication to stop, because the primary will not be able to create a response to send to the secondaries. See links.


Generated at Thu Feb 08 06:55:51 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.