[SERVER-30083] Schedule orphan range deletions sooner after migration Created: 10/Jul/17  Updated: 30/Oct/23  Resolved: 28/Sep/17

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: None
Fix Version/s: 3.6.0-rc0

Type: Task Priority: Major - P3
Reporter: Dianna Hohensee (Inactive) Assignee: Nathan Myers
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Related
related to SERVER-29745 Range deletion after moving away a ch... Closed
Backwards Compatibility: Fully Compatible
Sprint: Sharding 2017-07-31, Sharding 2017-08-21, Sharding 2017-09-11, Sharding 2017-10-02
Participants:
Linked BF Score: 0

 Description   

This commit, which was subsequently reverted, caused a test hang in CollectionRangeDeleter::DeleteNotification::waitStatus: https://github.com/mongodb/mongo/commit/344bf6e257e1427bc594bacac3f5983c2bdeaacf The hang can be seen in this build https://evergreen.mongodb.com/task/mongodb_mongo_master_enterprise_rhel_62_64_bit_slow1_344bf6e257e1427bc594bacac3f5983c2bdeaacf_17_07_07_12_44_23 and in the linked BF-5886. The CollectionRangeDeleter functions called in moveChunk were changed in the above commit. It seems to have unwittingly surfaced a bug.

The original sequence of range deleter calls was

  • DeleteNotification notification = cleanUpRange()
  • notification.abandon()
  • waitForClean()

The failed commit did

  • DeleteNotification notification = cleanUpRange()
  • notification.waitStatus() <-- hangs non-deterministically, though frequent enough to be reproducible

There's no corresponding "Finished deleting mr_during_migrate.coll range ...." message after the donor finishes the migration and starts waiting in the hung test logs. And one of the thread dumps has CollectionRangeDeleter::DeleteNotification::waitStatus in it.



 Comments   
Comment by Githook User [ 28/Sep/17 ]

Author:

{'email': 'nathan.myers@10gen.com', 'name': 'Nathan Myers', 'username': 'nathan-myers-mongo'}

Message: SERVER-30083 Schedule range deletions sooner
Branch: master
https://github.com/mongodb/mongo/commit/19c025d79ca43cc97d577a11817957e4b1112645

Comment by Nathan Myers [ 27/Sep/17 ]

The problem occurs when the old metadata is still around (and maps the orphan range) at the time deletion is requested, so instead of scheduling its deletion immediately, we would attach the deletion job to the current metadata. The solution is to find the newest metadata that might depend on the range (which is the oldest place permissible) and drop it there. Then, when that metadata expires, the deletions are scheduled right then, not later.

This would show up in tests that stall waiting for a range deletion to complete. Those that don't wait would find it deleted shortly after.

Comment by Nathan Myers [ 17/Jul/17 ]

Sure enough, if it doesn't get a refresh, it throws and never gets to the point where it gets stuck. Hmm.

Comment by Dianna Hohensee (Inactive) [ 17/Jul/17 ]

I didn't look too closely, but if the MigrationSourceManager returns an error – like for failing to refresh --, the uassert in move_chunk_command.cpp catches it here andthen we never reach range deletion, no?

Comment by Nathan Myers [ 17/Jul/17 ]

Studying the code, it appears that MigrationSourceManager:: commitChunkMetadataOnConfig can fail to get a refreshed shard metadata. After it returns, if you construct a ScopedCollectionMetadata, it might still have the deleted range, and if you hold that stale SCM you will wait forever for the deletion to complete.

Comment by Dianna Hohensee (Inactive) [ 13/Jul/17 ]

Finally determined that the hang was caused by holding a ScopedCollectionMetadata object while scheduling and waiting to range deletion. The solution was to stop holding it – it wasn't necessary, anyway, so it's probably better not to hold on to it anyway. However, holding that scoped object of the latest metadata should not have held up range deletion of an unused range from an old metadata version: it's a bug that clean up was never scheduled. I suspect the error is either related to this not evaluating to true for some reason when clean up is first requested, or the ScopedCollectionMetadata destructor code that should schedule cleanup when old metadata is released.

Generated at Thu Feb 08 04:22:38 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.