[SERVER-44062] Add zone operations concurrency testing for refineCollectionShardKey Created: 17/Oct/19  Updated: 29/Oct/23  Resolved: 01/Nov/19

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: None
Fix Version/s: 4.3.1

Type: Task Priority: Major - P3
Reporter: Blake Oler Assignee: Blake Oler
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Backwards Compatibility: Fully Compatible
Sprint: Sharding 2019-11-04
Participants:

 Description   

Background

Concurrency tests for refining a collection shard key should be added to verify a refine atomically takes effect and does not interfere with concurrent zone operations.

In particular, workloads should be added that:

  1. Repeatedly refines a shard key with concurrent zone operations (updateZoneKeyRange, addShardToZone, removeShardFromZone) in addition to concurrent CRUD ops.
    • Note updateZoneRange is the most important to test, so testing with the other two zone commands can be split out and re-prioritized if necessary.

In addition to the default concurrency suites, these workloads should run in suites with the balancer enabled and failovers. It should be verified that the balancer will actually attempt to move chunks, possibly enforced by inserting enough data to imbalance the cluster, beginning with lopsided zones, and/or adding fsm stages that explicitly wait for a balancer round.

Proposed Solution

High-Level Explanation

This workload will be built upon the underlying idea of refining a list of collections controlled by a latch.

Each thread will have two zones and two zone ranges. The thread will bounce the zones in between two shards, and will bounce the ranges in between the two zones. This means that any given time, a zone will belong to only one shard and will have a specified range. While this is happening, CRUD operations borrowed from the broadcast update/delete transaction states will be operating in documents belonging to the ranges owned by the thread. Every so often, the collection shard key will be refined, and then we will move to the next collection.

Steps for setting up the workload

  1. Create a number of zones equal to twice the amount of threads, so that each thread may have two zones.
  2. Create a number of ranges equal to twice the amount of threads, so that each thread may take ownership of two ranges.
  3. Assign to each thread two zones and two ranges for those zones.
  4. Shard the collection with the initial zones, which will create the chunks automatically.
  5. Assign each zone to only one shard. If the balancer is on, wait for the chunks to be distributed.
  6. Fill owned ranges with documents to be used for CRUD ops.

Steps for thread initialization for the workload.

  1. Cache into memory each thread's assigned zones, assigned ranges, and assigned documents.

States

  • sendZoneToOtherShard() – Picks a random zone assigned to this thread, removes it from the current shard, and assigns it to the other shard. Verifies that the first shard no longer has the zone, and that the second shard has the zone. If the balancer is on, waits for the chunks to be moved.
  • swapZoneRange() – Removes the ranges from each of the zones assigned to this thread, and swaps them, such that each range is now assigned to the opposite zone. If the ranges are owned by different shards and the balancer is on, waits for the chunks to be moved.
  • refineCollectionShardKey() – Refines the collection's shard key and decreases the latch count for collections to refine.
  • update/delete in transaction states – States borrowed from the broadcast update/delete transaction states.


 Comments   
Comment by Githook User [ 31/Oct/19 ]

Author:

{'username': 'BlakeIsBlake', 'email': 'blake.oler@mongodb.com', 'name': 'Blake Oler'}

Message: SERVER-44062 Add zone operations concurrency testing for refineCollectionShardKey
Branch: master
https://github.com/mongodb/mongo/commit/ae9747e4572976c713f2bc2eb02436eae4f36352

Comment by Blake Oler [ 18/Oct/19 ]

It turns out we can't verify that the balancer has balanced zones correctly en masse at the moment, due to the lack of ability to see into the balancer's "quietness" or its verification that nodes are completely balanced with regards to zones. So we'll have to hold off on verifying zones are on the right shards until then.

Refer to SERVER-43990.

Comment by Blake Oler [ 18/Oct/19 ]
  1. swapZoneRange should be able to emulate this.
  2. You're right there, it's not super interesting. It's to gain a bit of coverage with zones in concurrency testing altogether, since we currently have no zone concurrency testing.
Comment by Kaloian Manassiev [ 18/Oct/19 ]

Will there be separate states to ensure that adding zones concurrently with refining the shard key ends up with all zones containing the correct number of fields? Or is it the intention that swapZoneRange will emulate that?

The sendZoneToOtherShard operation is an addShardToZone/removeShardFromZone combo, right? If this is the case, why is it interesting that it is tested in combination with refining the shard key given that it doesn't change the bounds? Maybe it has something to do with the refine operation potentially changing more than just the zone boundaries and overwriting the shard.

Otherwise LGTM.

Comment by Blake Oler [ 17/Oct/19 ]

jack.mulrow,

  1. I was going to mix in the CRUD ops on top of a new workload, which should be easy to do with the way that I've pulled out the broadcast states. It'll be easier not to implement the CRUD ops at all, as this will cut down the work required for the workload.
  2. Added a comment on SERVER-43099.
  3. Got it.
Comment by Jack Mulrow [ 17/Oct/19 ]

blake.oler, the approach LGTM. This should give us really useful coverage.

A couple small notes:

  1. We might not need the CRUD ops since we're really only testing that chunks are migrated to the right shards, not how the migrations actually affect concurrent activity, but if it's easier to implement by extending one of the transaction workloads then I'm fine with keeping CRUD ops in.
  2. When SERVER-43099 re-enables a random balancer policy, we might need to do work to disable the randomness for this workload. I'm not sure we can do anything about that now, but I'd put a quick comment on that ticket for whoever ends up implementing it.
  3. I thought it might be tricky to handle a refine during swapZoneRange(), but we already implicitly extend tag boundaries from a user that don't include all shard key fields here, so I don't think this will be a problem, just mentioning in case it might save some investigation later.
Comment by Blake Oler [ 17/Oct/19 ]

jack.mulrow Wrote up a proposal for a workload for zone operations. Can I get an LGTM?

Generated at Thu Feb 08 05:04:54 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.