[SERVER-29450] Lower OplogFetcher and CollectionCloner batch sizes in tests by default Created: 05/Jun/17  Updated: 30/Oct/23  Resolved: 28/Aug/18

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: None
Fix Version/s: 4.1.3

Type: Task Priority: Major - P3
Reporter: Judah Schvimer Assignee: Tess Avitabile (Inactive)
Resolution: Fixed Votes: 0
Labels: neweng
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-29196 collection cloner only sets batchSize... Closed
depends on SERVER-29400 Make OplogFetcher and CollectionClone... Closed
Backwards Compatibility: Fully Compatible
Sprint: Repl 2018-09-10
Participants:

 Description   

We currently barely test our batching code. Lowering the batch sizes would improve this.



 Comments   
Comment by Githook User [ 28/Aug/18 ]

Author:

{'name': 'Tess Avitabile', 'email': 'tess.avitabile@mongodb.com', 'username': 'tessavitabile'}

Message: SERVER-29450 Lower OplogFetcher and CollectionCloner batch sizes in tests by default
Branch: master
https://github.com/mongodb/mongo/commit/bf8b4ce9815b4514a72e4e547e5032806ab1d66e

Comment by Tess Avitabile (Inactive) [ 27/Aug/18 ]

We have some coverage for multiple batches in our unit tests. See this OplogFetcher test and this CollectionCloner test. For this ticket, I will change the replica_sets_initsync_jscore_passthrough and replica_sets_initsync_static_jscore_passthrough suites to use lower values for bgSyncOplogFetcherBatchSize and initialSyncOplogFetcherBatchSize. I will also update initial_sync_capped_index.js to take advantage of initialSyncOplogFetcherBatchSize in order to use a smaller collection and document size.

Comment by Tess Avitabile (Inactive) [ 08/Aug/18 ]

Got it. Then I will leave this in the epic to make sure that not all results are returned in the first batch in our tests.

Comment by Judah Schvimer [ 08/Aug/18 ]

I think I was wrong. I still think we could benefit from testing our batching logic more since most of our batch sizes in initial sync and normal oplog fetching are bigger than anything we test.

Comment by Tess Avitabile (Inactive) [ 08/Aug/18 ]

judah.schvimer, how would this have helped us find SERVER-34898? Oplog fetching and collection cloning is not done with readConcern majority, so I must be missing something.

Comment by Gregory McKeon (Inactive) [ 15/May/18 ]

Since we're likely to switch the OplogFetcher and CollectionCloner to use exhaust cursors, we may not do this.

Comment by William Schultz (Inactive) [ 10/May/18 ]

Another alternative here is to add a separate passthrough suite that runs with lower batch sizes, if we still want to maintain test coverage of default batch sizes.

Comment by Judah Schvimer [ 10/May/18 ]

This may have caught SERVER-34898 sooner.

Generated at Thu Feb 08 04:20:54 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.