[SERVER-14704] Chunk migration become (linearly?) slower as collection grows Created: 27/Jul/14 Updated: 10/Dec/14 Resolved: 19/Aug/14 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | 2.6.3 |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Vincent | Assignee: | Ramon Fernandez Marina |
| Resolution: | Done | Votes: | 1 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
| Participants: |
| Description |
|
In my use case I've multiple databases with the same "schemas" and type of data. I've noticed that chunk migration becomes slower and slower, in correlation with the collection size. For small databases/collections, migrating a chunk is generally done in less than 20 seconds while for my bigger collections it takes 1800 seconds in average (sometimes more than 1 hour), with all nuances between them (I've about 35 identical databases, with all sizes). Chunks have roughly the same size and number of documents in all cases, with exactly the same indexes. Updates/Inserts are happening, but at a slow pace (I'd say less than 10 updates/inserts per hour are happening on the chunk being migrated). All disks are regular SATA (because of dataset size). Exemple of a low migration: Data do not fit in RAM (but indexes does). iotop tells me that both the sender and the recipient are performing a lot of writes (both stuck at 100%). Magnitudes more than what is being transmitted.
DSK | sda | busy 100% | read 152 | write 2664 | MBr/s 0.36 | MBw/s 11.84 | avio 3.55 ms | You can probably find more insights in my MMS account: https://mms.mongodb.com/host/cluster/51a2dc5c7fe227e9f188c509/52bb9a10e4b0256ace50e0d3 Have a look to the log extract for a typical overview of chunk migration speed. |
| Comments |
| Comment by Vincent [ 14/Nov/14 ] | |
|
For reference, I solved this issue by switching to servers with less RAM (32GB instead of 96GB) but equipped with SSD disks. It dramatically improved performances of my application and chunk migrations are now blazing fast, even if my data doesn't fit in RAM (not even my indexes). | |
| Comment by Ramon Fernandez Marina [ 19/Aug/14 ] | |
|
tubededentifrice, thanks for sending your config database. There are several factors that can affect migrations, for example:
After examining the data you sent we haven't found any evidence of a bug in MongoDB. Since the SERVER project is for reporting bugs or feature suggestions for the MongoDB server and tools, I would recommend that you post your questions on the mongodb-user group, where you can reach a wide audience of MongoDB experts. Regards, | |
| Comment by Vincent [ 30/Jul/14 ] | |
|
It looks like the writes on the receiving side are caused by application updates thrown, not by the chunk migration, my apologies. | |
| Comment by Vincent [ 30/Jul/14 ] | |
|
Note that I already had this problem when I switched from 1 to 5 and then 5 to 3 shards (migration took really forever, with about 1 chunk (64 MB at the time...) moved per 24 hour – the hardware used was much less powerful at the time). | |
| Comment by Vincent [ 29/Jul/14 ] | |
|
Hi Ramon, 2. Nope, initially it was a simple RS without sharding. Then I had 5 shards, then 3 and now I'm moving all the data to have only 1 shard (maybe 2 shards then, etc.). Chunks are moved using moveChunk commands (because I can only have 1 draining shard, and it would be dumb to have the chunks moved to the other shard I want to remove!) 3. I had some "jumbo" chunks in the past, which were able to move with 256MB. Beside this, the doc state(d ?) that chunk migration was more efficient with big chunks and put less stress on mongos (I had an issue with this too...) at the cost of a less evenly balanced cluster and more "painful" migrations, which I don't really care about. (moving a 256MB chunk take less time than moving 4x 64MB chunks, isn't?) 4. I keep a "tail -f" on the logs => shardKeyPattern: { _id: 1.0 }, state: "clone", counts: { cloned: 24470, clonedBytes: 73203286, catchup: 0, steady: 0 }; the numbers I wrote may not be 100% accurate but gives a good idea of the reality. Edit: I've attached the dump to this ticket, restricted to project users | |
| Comment by Ramon Fernandez Marina [ 29/Jul/14 ] | |
|
tubededentifrice, we'll need more information to determine whether there's a bug here:
Thanks, | |
| Comment by Vincent [ 27/Jul/14 ] | |
|
I forgot to mention: the 3rd shard (the one that is not yet involved in chunk migration but still holds ~30% of the "big" collection I'm moving) is resting: which makes me conclude it's not a mater of what other operations are being performed in the database, but only a mater of the chunks being migrated. And also: all filesystems are ext4 |