[SERVER-32885] Overlap chunk clone application on the donor with fetching documents from the recipient Created: 25/Jan/18 Updated: 30/Oct/23 Resolved: 19/Mar/18 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | None |
| Fix Version/s: | 3.4.15, 3.6.4, 3.7.4 |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Kaloian Manassiev | Assignee: | Blake Oler |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||
| Backport Requested: |
v3.6, v3.4
|
||||||||||||||||
| Sprint: | Sharding 2018-02-12, Sharding 2018-02-26, Sharding 2018-03-12, Sharding 2018-03-26 | ||||||||||||||||
| Participants: | |||||||||||||||||
| Case: | (copied to CRM) | ||||||||||||||||
| Description |
|
The current chunk clone logic (as part of moveChunk), ping-pongs serially between the donor fetching chunk documents and the recipient applying them. Since both sequences are potentially slow and might be bringing cold data from cache, there could be benefit in overlapping them. By putting this loop in a separate thread we can overlap that sequence and possibly double the clone speed. This can be implemented by spawning a ThreadPool at the beginning of the clone phase and putting a fixed-size BlockingQueue (similar to the JournalWriter) between the migrate driver thread and the worker thread. It is possible that such aggressive clone might not be appropriate for the MMAP V1 storage engine, so this optimization should only be done for WT. |
| Comments |
| Comment by Githook User [ 22/Mar/18 ] |
|
Author: {'email': 'blake.oler@mongodb.com', 'name': 'Blake Oler', 'username': 'BlakeIsBlake'}Message: (cherry picked from dfa03b56b8334a82b223e34f02fc5cbfead4dca2) |
| Comment by Githook User [ 20/Mar/18 ] |
|
Author: {'email': 'blake.oler@mongodb.com', 'name': 'Blake Oler', 'username': 'BlakeIsBlake'}Message: (cherry picked from dfa03b56b8334a82b223e34f02fc5cbfead4dca2) |
| Comment by Githook User [ 16/Mar/18 ] |
|
Author: {'email': 'blake.oler@mongodb.com', 'name': 'Blake Oler', 'username': 'BlakeIsBlake'}Message: Repushed to update error codes |
| Comment by Githook User [ 16/Mar/18 ] |
|
Author: {'email': 'blake.oler@mongodb.com', 'name': 'Blake Oler', 'username': 'BlakeIsBlake'}Message: Revert " This reverts commit 1848f026aa094d6d143dc3b664a272fbd64155e5. |
| Comment by Githook User [ 16/Mar/18 ] |
|
Author: {'email': 'blake.oler@mongodb.com', 'name': 'Blake Oler', 'username': 'BlakeIsBlake'}Message: |