[SERVER-32885] Overlap chunk clone application on the donor with fetching documents from the recipient Created: 25/Jan/18  Updated: 30/Oct/23  Resolved: 19/Mar/18

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: None
Fix Version/s: 3.4.15, 3.6.4, 3.7.4

Type: Improvement Priority: Major - P3
Reporter: Kaloian Manassiev Assignee: Blake Oler
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Related
is related to SERVER-27725 Use batch insert when migrating chunks Closed
Backwards Compatibility: Fully Compatible
Backport Requested:
v3.6, v3.4
Sprint: Sharding 2018-02-12, Sharding 2018-02-26, Sharding 2018-03-12, Sharding 2018-03-26
Participants:
Case:

 Description   

The current chunk clone logic (as part of moveChunk), ping-pongs serially between the donor fetching chunk documents and the recipient applying them. Since both sequences are potentially slow and might be bringing cold data from cache, there could be benefit in overlapping them.

By putting this loop in a separate thread we can overlap that sequence and possibly double the clone speed.

This can be implemented by spawning a ThreadPool at the beginning of the clone phase and putting a fixed-size BlockingQueue (similar to the JournalWriter) between the migrate driver thread and the worker thread.

It is possible that such aggressive clone might not be appropriate for the MMAP V1 storage engine, so this optimization should only be done for WT.



 Comments   
Comment by Githook User [ 22/Mar/18 ]

Author:

{'email': 'blake.oler@mongodb.com', 'name': 'Blake Oler', 'username': 'BlakeIsBlake'}

Message: SERVER-32885 Overlap chunk clone application on the donor with fetching documents from the recipient

(cherry picked from dfa03b56b8334a82b223e34f02fc5cbfead4dca2)
Branch: v3.4
https://github.com/mongodb/mongo/commit/7195da1959bf12fe2497e385b4b6579f43712930

Comment by Githook User [ 20/Mar/18 ]

Author:

{'email': 'blake.oler@mongodb.com', 'name': 'Blake Oler', 'username': 'BlakeIsBlake'}

Message: SERVER-32885 Overlap chunk clone application on the donor with fetching documents from the recipient

(cherry picked from dfa03b56b8334a82b223e34f02fc5cbfead4dca2)
Branch: v3.6
https://github.com/mongodb/mongo/commit/a350cb535b2c21aee665635b32b5a07013cc3c37

Comment by Githook User [ 16/Mar/18 ]

Author:

{'email': 'blake.oler@mongodb.com', 'name': 'Blake Oler', 'username': 'BlakeIsBlake'}

Message: SERVER-32885 Overlap chunk clone application on the donor with fetching documents from the recipient

Repushed to update error codes
Branch: master
https://github.com/mongodb/mongo/commit/a2698593ede55dd3aaaedc882beec5aa30ff056e

Comment by Githook User [ 16/Mar/18 ]

Author:

{'email': 'blake.oler@mongodb.com', 'name': 'Blake Oler', 'username': 'BlakeIsBlake'}

Message: Revert "SERVER-32885 Overlap chunk clone application on the donor with fetching documents from the recipient"

This reverts commit 1848f026aa094d6d143dc3b664a272fbd64155e5.
Branch: master
https://github.com/mongodb/mongo/commit/1cac032e38456b8cbb11921183198e43bb59c95e

Comment by Githook User [ 16/Mar/18 ]

Author:

{'email': 'blake.oler@mongodb.com', 'name': 'Blake Oler', 'username': 'BlakeIsBlake'}

Message: SERVER-32885 Overlap chunk clone application on the donor with fetching documents from the recipient
Branch: master
https://github.com/mongodb/mongo/commit/1848f026aa094d6d143dc3b664a272fbd64155e5

Generated at Thu Feb 08 04:31:36 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.