[SERVER-16860] Secondaries of same spec cannot keep up with high primary write load Created: 15/Jan/15  Updated: 03/Dec/21  Resolved: 19/Mar/15

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: 2.6.7
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: John Page Assignee: John Page
Resolution: Incomplete Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

If I put a high continuous write load onto a a replica set I fall behind on the oplog until I fall off the end.

All three nodes are the same (High) spec - using write concern defaults and a multi threaded insert only workload ( OPLOG, Journal and Collection on same Drive). They are all interconnected by the same network, as is the client.

I know I could use a higher write concern but this is a simple and likely test scenario and I would expect the replication mechanism to be as fast as any external loader.



 Comments   
Comment by John Page [ 15/Jan/15 ]

Sorry 2.6.7

Comment by Eric Milkie [ 15/Jan/15 ]

john.page
In order to determine if this is a duplicate of SERVER-16517, we need to know what version of mongod you are running for this test.

Comment by John Page [ 15/Jan/15 ]

3 AWS EBS Instances - type C4.2XL
2 x 1TB SSD Disk with 4000 PIOPS as RAID 0.

Custom Java loader pushing in records with 10 fields, _id index only, 32 threads with batches of 512, average size or record 2K
Insert rate on Primary - 11,000->13,000 per second (measured with mongostat at 5 minute intervals).

Insert rate on Secondaries (x2) 8,000-9,000 per second.

Once 800GB loaded - oplog too old and secondaries failed.

Don't have any other logs as resynced secondaries manually.

Comment by Scott Hernandez (Inactive) [ 15/Jan/15 ]

Please provide your reproduction, logs, stats, version information, configuration and any other information about your tests including the detailed results.

Generated at Thu Feb 08 03:42:30 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.