[SERVER-61126] move_chunks is extremely slow Created: 30/Oct/21  Updated: 06/Dec/22  Resolved: 01/Nov/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 4.4.9
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Daniele Tessaro Assignee: Backlog - Triage Team
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File Migrated bytes per second-1.png     PNG File Migrated bytes per second.png     PNG File Migrated docs per second.png    
Assigned Teams:
Server Triage
Operating System: ALL
Participants:

 Description   

We have a sharded cluster with 2 shards Shard A and Shard B, each in primary + 2 replicas configuration.
Shard A and B are in two different zones so that Shard A should contain recent data up to a cutoff and ShardB historical data.
Shard A is a machine with SD drive, shardB has a normal disk (since it is supposed to have lower usage patterns)

Initially there was only Shard A which grew to almost 7GB and around 600k chunks, with standard max chunk size.
We then attached ShardB, hoping to be able to move all the historical data in a relatively quick amount of time.

Since ShardB has been attached, the balancer has started moving chunks from A to B. The speed was already initially pretty bad but we have noticed that it got much worse with time, as you can see here:

At this speed it will take probably years to move the amount of data that we need to move.
Note that we've also tried to:
1. Completely stop the traffic to the cluster. This improved things a bit but didn't make a huge difference.
2. Merge chunks to increase their size. This is not helping. The last few very bad datapoints you can see in the charts are due to a series of chunks that are currently being moved that have an average size of 200Mb. When those chunks will be done we hope things will get slightly better.

A transfer rate of 200k/s seems really really low, so we basically want to know what kind of options we have.
Have we hit some kind of intrinsic bottleneck? How can we debug this issue?
Any help would be appreciated.



 Comments   
Comment by Daniele Tessaro [ 02/Nov/21 ]

Thanks, I will do. Unfortunately though I see a lot of people reporting similar issues and very few responses to their questions. I'm wondering if there's any plan to improve speed of this move chunks process.

Comment by Edwin Zhou [ 01/Nov/21 ]

Hi daniele@spiketrap.io,

Thanks for your report. For this issue we'd like to encourage you to start by asking our community for help by posting on the MongoDB Developer Community Forums.

If the discussion there leads you to suspect a bug in the MongoDB server, then we'd want to investigate it as a possible bug here in the SERVER project.

Best,
Edwin

Comment by Daniele Tessaro [ 30/Oct/21 ]

Just want to add that, in order to compute those charts, I've been looking into the `changelog` collection.
Here you can see a snippet of the last 10 entries in the changelog, from which I've only selected a few interesting fields:

[
    [0] {
        "details" => {
             "min" => {
                "time" => 2016-05-17 18:19:51 UTC
            },
             "max" => {
                "time" => 2016-05-18 01:02:38 UTC
            },
            "from" => "raw_shard_arsn0",
              "to" => "raw_shard_brsn0"
        },
           "time" => 2021-10-30 02:02:32.781 UTC,
           "what" => "moveChunk.start"
    },
    [1] {
        "details" => {
               "min" => {
                "time" => 2016-05-17 18:19:51 UTC
            },
               "max" => {
                "time" => 2016-05-18 01:02:38 UTC
            },
              "from" => "raw_shard_arsn0",
                "to" => "raw_shard_brsn0",
            "counts" => {
                     "cloned" => 253337,
                "clonedBytes" => 189371712,
                    "catchup" => 0,
                     "steady" => 0
            }
        },
           "time" => 2021-10-30 02:25:05.898 UTC,
           "what" => "moveChunk.commit"
    },
    [2] {
        "details" => {
             "min" => {
                "time" => 2016-05-18 01:02:38 UTC
            },
             "max" => {
                "time" => 2016-05-18 13:45:57 UTC
            },
            "from" => "raw_shard_arsn0",
              "to" => "raw_shard_brsn0"
        },
           "time" => 2021-10-30 02:25:09.023 UTC,
           "what" => "moveChunk.start"
    },
    [3] {
        "details" => {
               "min" => {
                "time" => 2016-05-18 01:02:38 UTC
            },
               "max" => {
                "time" => 2016-05-18 13:45:57 UTC
            },
              "from" => "raw_shard_arsn0",
                "to" => "raw_shard_brsn0",
            "counts" => {
                     "cloned" => 254066,
                "clonedBytes" => 197992651,
                    "catchup" => 0,
                     "steady" => 0
            }
        },
           "time" => 2021-10-30 03:06:17.675 UTC,
           "what" => "moveChunk.commit"
    },
    [4] {
        "details" => {
             "min" => {
                "time" => 2016-05-18 13:45:57 UTC
            },
             "max" => {
                "time" => 2016-05-18 21:37:13 UTC
            },
            "from" => "raw_shard_arsn0",
              "to" => "raw_shard_brsn0"
        },
           "time" => 2021-10-30 03:06:20.698 UTC,
           "what" => "moveChunk.start"
    },
    [5] {
        "details" => {
               "min" => {
                "time" => 2016-05-18 13:45:57 UTC
            },
               "max" => {
                "time" => 2016-05-18 21:37:13 UTC
            },
              "from" => "raw_shard_arsn0",
                "to" => "raw_shard_brsn0",
            "counts" => {
                     "cloned" => 254073,
                "clonedBytes" => 196977214,
                    "catchup" => 0,
                     "steady" => 0
            }
        },
           "time" => 2021-10-30 03:46:53.514 UTC,
           "what" => "moveChunk.commit"
    },
    [6] {
        "details" => {
             "min" => {
                "time" => 2016-05-18 21:37:13 UTC
            },
             "max" => {
                "time" => 2016-05-19 09:48:14 UTC
            },
            "from" => "raw_shard_arsn0",
              "to" => "raw_shard_brsn0"
        },
           "time" => 2021-10-30 03:46:56.675 UTC,
           "what" => "moveChunk.start"
    },
    [7] {
        "details" => {
             "min" => {
                "time" => 2016-05-18 21:37:13 UTC
            },
             "max" => {
                "time" => 2016-05-19 09:48:14 UTC
            },
            "from" => "raw_shard_arsn0",
              "to" => "raw_shard_brsn0"
        },
           "time" => 2021-10-30 04:19:40.161 UTC,
           "what" => "moveChunk.start"
    },
    [8] {
        "details" => {
               "min" => {
                "time" => 2016-05-18 21:37:13 UTC
            },
               "max" => {
                "time" => 2016-05-19 09:48:14 UTC
            },
              "from" => "raw_shard_arsn0",
                "to" => "raw_shard_brsn0",
            "counts" => {
                     "cloned" => 265207,
                "clonedBytes" => 203379400,
                    "catchup" => 0,
                     "steady" => 0
            }
        },
           "time" => 2021-10-30 04:54:20.133 UTC,
           "what" => "moveChunk.commit"
    },
    [9] {
        "details" => {
             "min" => {
                "time" => 2016-05-19 09:48:14 UTC
            },
             "max" => {
                "time" => 2016-05-19 20:10:00 UTC
            },
            "from" => "raw_shard_arsn0",
              "to" => "raw_shard_brsn0"
        },
           "time" => 2021-10-30 04:54:23.57 UTC,
           "what" => "moveChunk.start"
    }
]

Generated at Thu Feb 08 05:51:36 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.