[SERVER-67301] Balancer may perform one unnecessary migration for a completely balanced collection Created: 15/Jun/22  Updated: 29/Oct/23  Resolved: 27/Jul/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 6.0.3, 6.1.0-rc0

Type: Task Priority: Major - P3
Reporter: Pierlauro Sciarelli Assignee: Silvia Surroca
Resolution: Fixed Votes: 0
Labels: shardingemea-qw
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Duplicate
duplicates SERVER-67171 insert_with_data_size_aware_balancing... Closed
Problem/Incident
Related
is related to SERVER-68777 BalancerCollectionStatus may report b... Closed
Backwards Compatibility: Fully Compatible
Backport Requested:
v6.0
Sprint: Sharding EMEA 2022-07-11, Sharding EMEA 2022-07-25, Sharding EMEA 2022-08-08
Participants:
Linked BF Score: 35
Story Points: 3

 Description   

Preamble

As part of a range deletion:

  1. A batch of documents belonging to the orphaned range is deleted (each document in the batch is deleted in a single delete).
  2. The counter of orphans is updated accordingly, triggering also an update of the orphans counter on the BalancerStatsRegistry.

Note that 1-2 are not happening in the same storage transaction, and - even if they would - the update of the in-memory stats registry would still not happen atomically.

The balancer is considering the cluster balanced if the following condition is met:

size(most loaded shard) <= size(least loaded shard) + 2 * chunkSize

 

Problem

If the range deletion happens on an already balanced collection while a range deletion is ongoing, it may happen that the balancer retrieves statistics from shards between bullets 1 and 2. This may trigger an unnecessary migration (no big deal as the chunk would be later migrated back, but still it's an unnecessary operation).

Example

Given a chunk size of 128MB for a given collection, if the collection size between between two shards A and B differs less than 256MB the balancer should not move any data.

However, it could happen the following:

  • ShardA has 400MB, ShardB has 0MB
  • Move a range of 128MB from A to B
  • Move a range of 128MB from A to B
  • ShardB receives a bit of inserts, let's say 24MB

At this point the collection is balanced because (280 - 144 = 136) is less than (2 * chunkSize = 256) :

  • ShardA still has 400MB: 144MB (actual data size) plus 256MB (orphans)
  • ShardB has 280MB (a bit more of 256MB, accounting also for some inserts)

The range deleter kicks in:

  • Starts deleting the orphans for one range...
  • The range deleter has so far deleted 125MB
  • The balancer asks all the shards what's their data size:
    • Shard A will reply (400 [original size] - 125 [docs already deleted]) - (128 * 2 [tracked number of orphaned docs])
      So it replies 400 - 125 - 256 = 19MB
    • Shard B will reply 280MB
  • The balancer performs its calculation on the chunkSize: (280 - 19 > 2 * chunkSize) ? YES because (261 > 256)
  • A range is wrongly moved from shard B to shard A


 Comments   
Comment by Githook User [ 14/Oct/22 ]

Author:

{'name': 'Silvia Surroca', 'email': 'silvia.surroca@mongodb.com', 'username': 'silviasuhu'}

Message: SERVER-67301 Balancer may perform one unnecessary migration for a completely balanced collection
Branch: v6.0
https://github.com/mongodb/mongo/commit/74d6780ecbd503e6b5e762d1939de9c2a7654f9e

Comment by Githook User [ 27/Jul/22 ]

Author:

{'name': 'Silvia Surroca', 'email': 'silvia.surroca@mongodb.com', 'username': 'silviasuhu'}

Message: SERVER-67301 Balancer may perform one unnecessary migration for a completely balanced collection
Branch: master
https://github.com/mongodb/mongo/commit/4c4fe953305a928228a0f3d320092ae8929255a3

Generated at Thu Feb 08 06:07:47 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.