[SERVER-28984] Improve Chunk Migration performance for clusters with high-latency Secondaries. Created: 26/Apr/17  Updated: 31/May/17  Resolved: 29/Apr/17

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 3.2.11
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: James Reitz Assignee: Kaloian Manassiev
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates SERVER-23340 Turn off moveChunk secondaryThrottle ... Closed
Participants:

 Description   

The Problem:

For customers who have their availability zones geographically quite far away, secondaries can have an unavoidable high latency connection to the primary, even though the data-rates are well within needs. Round-trip times as measured by ping can be in the 40-100ms range.

In the above environment, chunk migration is terribly slow. Chunk transfer rates in our cluster average a measly 4k bytes/sec. Furthermore, range-deletes for removing the documents that belonged to a chunk that was just moved are also terribly slow. The rest of Mongo works well in our cluster. In fact, our secondaries' Op-time lag are typically less than 1-second. Standing up local secondary servers (i.e., close to the primary servers) improves chunk migration rates and range-delete rates by at least two orders of magnitude, but this has a tremendous cost, effectively doubling the number of secondary servers we require.

A Possible Solution:

It appears from our experience that when the balancer calls moveChunk and/or rangeDeletes, it is using secondary-throttling by default with a write concern of at least 2. Why couldn't MongoDB support a "high-latency-tolerant" Chunk Migration mode; where calls to moveChunk and rangeDeletes for chunk migration, would use a write-concern of 1 (i.e., secondary-throttling disabled) except for the final write or delete?
The final write or delete for each chunk could use secondary throttling with a write-concern of 2. I believe this would yield a huge improvement in chunk migration performance for high-latency environments. What are the down sides to this solution? I really can't think of any.



 Comments   
Comment by Kaloian Manassiev [ 29/Apr/17 ]

Thanks for confirming your server version jimreitz. I am going to close this ticket as duplicate of SERVER-23340.

Best regards,
-Kal.

Comment by James Reitz [ 29/Apr/17 ]

@Kaloian Manassiev, I think you're right. We've been using 3.2.11. I didn't realize 3.4 had those changes.

Thanks @Dan Pasette, I'll keep that in mind. Sometimes compression actually makes latency issues worse because it can add more latency to packet transmission. But it's highly dependent on the implementation.

Comment by Daniel Pasette (Inactive) [ 27/Apr/17 ]

Another improvement made in v3.4 was compression of the wire protocol for intra-cluster data transfer (SERVER-3018). Especially in globally distributed clusters where bandwidth is limited (or expensive) this can be a big performance and cost-saving win. Please note, that it is not enabled by default. Docs are here.

Comment by Kaloian Manassiev [ 27/Apr/17 ]

Hi jimreitz,

I believe what you are requesting duplicates SERVER-23340. Which version of the server are you using and which storage engine?

Starting in version 3.4, if you are using the WiredTiger storage engine, the default write concern for migrations is w:1. The only time where majority write concern is used at the end of the migration.

Historically, the default w:2 write concern was used as a way to achieve throttling and reduce the performance impact of migrations due to database level locking in MMAP V1.

Best regards,
-Kal.

Generated at Thu Feb 08 04:19:36 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.