[SERVER-8598] Add command to cleanup orphaned data created by failed chunk migrations Created: 15/Feb/13  Updated: 18/Feb/14  Resolved: 12/Aug/13

Status: Closed
Project: Core Server
Component/s: Admin, Sharding, Tools
Affects Version/s: None
Fix Version/s: 2.5.2

Type: Bug Priority: Major - P3
Reporter: Daniel Pasette (Inactive) Assignee: Greg Studer
Resolution: Done Votes: 6
Labels: cleanup, commands, sharding
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by SERVER-8838 removeRange() implicitly re-creates a... Closed
is depended on by DOCS-1838 Document cleanup orphan data cmd Closed
Duplicate
duplicates SERVER-8724 Add a command to clean up left over c... Closed
Related
related to SERVER-11104 cleanupOrphaned doesn't verify starti... Closed
related to SERVER-11277 cleanupOrphaned does nothing on empty... Closed
related to SERVER-11376 cleanupOrphaned errMsg and log contai... Closed
related to SERVER-5931 Secondary reads in sharded clusters n... Closed
is related to SERVER-6210 Clean up data left behind on shards b... Closed
is related to SERVER-8836 sharding/remove2.js failing on Window... Closed
is related to SERVER-11227 cleanupOrphaned parses secondaryThrot... Closed
is related to SERVER-5931 Secondary reads in sharded clusters n... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Participants:

 Description   

Provide a built in function to clean up orphaned data in a collection.



 Comments   
Comment by Vincent [ 18/Feb/14 ]

What is the correct way to achieve the same as this command in 2.4.x?
In fact I don't know how to perform an operation (delete) on a specific shard from a mongos, or accessing the chunk list from the shell of a shard.

Comment by auto [ 10/Oct/13 ]

Author:

{u'username': u'ajdavis', u'name': u'A. Jesse Jiryu Davis', u'email': u'jesse@mongodb.com'}

Message: SERVER-8598 Failpoints for testing moveChunk

Aids testing cleanupOrphaned and mergeChunks.

Signed-off-by: Matt Kangas <matt.kangas@mongodb.com>
Branch: master
https://github.com/mongodb/mongo/commit/dbae94b4a6a83ad32817038259f1d2d6ac5314f1

Comment by auto [ 21/Aug/13 ]

Author:

{u'username': u'gregstuder', u'name': u'Greg Studer', u'email': u'greg@10gen.com'}

Message: SERVER-8598 cleanup_orphaned_cmd_hashed.js minor test improvement
Branch: master
https://github.com/mongodb/mongo/commit/5f86ad54f492a352c589dd949706c0e820c2a453

Comment by auto [ 21/Aug/13 ]

Author:

{u'username': u'gregstuder', u'name': u'Greg Studer', u'email': u'greg@10gen.com'}

Message: SERVER-8598 test cleanup of orphaned data in hashed sharded collection
Branch: master
https://github.com/mongodb/mongo/commit/252f42781db4b1019433968b7293287d18b2ec40

Comment by auto [ 13/Aug/13 ]

Author:

{u'username': u'gregstuder', u'name': u'Greg Studer', u'email': u'greg@10gen.com'}

Message: SERVER-8598 correct auth settings for new cleanupOrphaned command
Branch: master
https://github.com/mongodb/mongo/commit/5c94494146e1f38c81a497ac7300b65d85456cc7

Comment by auto [ 09/Aug/13 ]

Author:

{u'username': u'gregstuder', u'name': u'Greg Studer', u'email': u'greg@10gen.com'}

Message: SERVER-8598 minor test and logging fixes for orphaned data cleanup
Branch: master
https://github.com/mongodb/mongo/commit/98117924c40a32d2f7be8f446c39a4b46268d537

Comment by Kevin J. Rice [ 06/Aug/13 ]

Also needs documentation change to describe how to call this function, whether it locks the collection on that shard, etc.

Comment by auto [ 06/Aug/13 ]

Author:

{u'username': u'tadmarshall', u'name': u'Tad Marshall', u'email': u'tad@10gen.com'}

Message: SERVER-8598 Visual Studio – add db/commands/cleanup_orphaned_cmd.cpp
Branch: master
https://github.com/mongodb/mongo/commit/cee23aa7958874aca663d7de4c83804b9c80468d

Comment by auto [ 06/Aug/13 ]

Author:

{u'username': u'gregstuder', u'name': u'Greg Studer', u'email': u'greg@10gen.com'}

Message: SERVER-8598 only unprotect pending range on unsuccessful TO migration
Branch: master
https://github.com/mongodb/mongo/commit/58d68a95c050daceee6c0122a72b8e43c5157261

Comment by auto [ 06/Aug/13 ]

Author:

{u'username': u'gregstuder', u'name': u'Greg Studer', u'email': u'greg@10gen.com'}

Message: SERVER-8598 merge newest pending chunks from stored metadata with newest chunks on reload
Branch: master
https://github.com/mongodb/mongo/commit/25e934ac1a262e90ad555235bf5b2de6ddb246a0

Comment by auto [ 06/Aug/13 ]

Author:

{u'username': u'gregstuder', u'name': u'Greg Studer', u'email': u'greg@10gen.com'}

Message: SERVER-8598 command to iteratively cleanup orphaned data
Branch: master
https://github.com/mongodb/mongo/commit/3bbe57a1e825a98a0c6e2601ac87978f02faecd6

Comment by auto [ 28/Jul/13 ]

Author:

{u'username': u'gregstuder', u'name': u'Greg Studer', u'email': u'greg@10gen.com'}

Message: SERVER-8598 on drop of chunks call GLE to ensure chunks actually dropped
Branch: master
https://github.com/mongodb/mongo/commit/1a072fc6cce47088804db327d98d4c56d5416450

Comment by Greg Studer [ 28/Jul/13 ]

Author:

{u'username': u'gregstuder', u'name': u'Greg Studer', u'email': u'greg@10gen.com'}

Message: SERVER-8958 better logging for mongos_validate_writes.js
Branch: master
https://github.com/mongodb/mongo/commit/10602c474190fdbbfc15ec2028b1b0b105b6aa9d

Note transposed 8958/8598

Comment by auto [ 27/Jul/13 ]

Author:

{u'username': u'gregstuder', u'name': u'Greg Studer', u'email': u'greg@10gen.com'}

Message: SERVER-8598 buildbot collection_metadata_test epoch validation now required for clonePlus/Minus chunk
Branch: master
https://github.com/mongodb/mongo/commit/edeab0376dc35946fe1b5fd0c505af20fb91c8bb

Comment by auto [ 26/Jul/13 ]

Author:

{u'username': u'gregstuder', u'name': u'Greg Studer', u'email': u'greg@10gen.com'}

Message: SERVER-8598 replace trySetVersion with epoch-safe refresh behavior
Branch: master
https://github.com/mongodb/mongo/commit/01296c1617fa8b003967c63dbbc1af911d6cd223

Comment by auto [ 19/Jul/13 ]

Author:

{u'username': u'gregstuder', u'name': u'Greg Studer', u'email': u'greg@10gen.com'}

Message: SERVER-8598 no global lock in SSV
Branch: master
https://github.com/mongodb/mongo/commit/1cb08f64a9e9087712b57197911d182a0220a9b9

Comment by auto [ 19/Jul/13 ]

Author:

{u'username': u'gregstuder', u'name': u'Greg Studer', u'email': u'greg@10gen.com'}

Message: SERVER-8598 minor fixes for metadata reload at zero version
Branch: master
https://github.com/mongodb/mongo/commit/c04a24af25d1dac21eab524bd3ee146bddc303d3

Comment by auto [ 19/Jul/13 ]

Author:

{u'username': u'gregstuder', u'name': u'Greg Studer', u'email': u'greg@10gen.com'}

Message: SERVER-8598 register pending chunks with chunk manager
Branch: master
https://github.com/mongodb/mongo/commit/fe0cfee0effbf9fbb8200ecfac332d8b52c165c2

Comment by auto [ 10/Jul/13 ]

Author:

{u'username': u'gregstuder', u'name': u'Greg Studer', u'email': u'greg@10gen.com'}

Message: SERVER-8598 fix compile duplicate it in d_state.cpp
Branch: master
https://github.com/mongodb/mongo/commit/1556be30b24b7a00038182e233a52e6c20385c26

Comment by auto [ 10/Jul/13 ]

Author:

{u'username': u'gregstuder', u'name': u'Greg Studer', u'email': u'greg@10gen.com'}

Message: SERVER-8598 link pending to ShardingState and RangeDeleter
Branch: master
https://github.com/mongodb/mongo/commit/c293028c1516b170e0fc5218f9e25c89f290daa5

Comment by auto [ 10/Jul/13 ]

Author:

{u'username': u'gregstuder', u'name': u'Greg Studer', u'email': u'greg@10gen.com'}

Message: SERVER-8598 allow CollectionManager to track pending chunks

Also includes fixes for MetadataLoader
Branch: master
https://github.com/mongodb/mongo/commit/01b912eb0897f490159f27be3142bd82d8806206

Comment by auto [ 17/Jun/13 ]

Author:

{u'username': u'gregstuder', u'name': u'Greg Studer', u'email': u'greg@10gen.com'}

Message: SERVER-8598 minor fixup from review
Branch: master
https://github.com/mongodb/mongo/commit/c676251944be837ebe285e12ff701e1594c2f70b

Comment by auto [ 17/Jun/13 ]

Author:

{u'username': u'gregstuder', u'name': u'Greg Studer', u'email': u'greg@10gen.com'}

Message: SERVER-8598 preserve semantics of exception-on-conn-failure in trySetVersion
Branch: master
https://github.com/mongodb/mongo/commit/88b7299f433e6b7bcae6df68a021fee91dc21dd3

Comment by auto [ 17/Jun/13 ]

Author:

{u'username': u'gregstuder', u'name': u'Greg Studer', u'email': u'greg@10gen.com'}

Message: SERVER-8598 CollectionManager -> CollectionMetadata refactor, no behavior change
Branch: master
https://github.com/mongodb/mongo/commit/432d1c9546340633dd5f0ef07259708a75f4a87e

Comment by auto [ 17/Jun/13 ]

Author:

{u'username': u'gregstuder', u'name': u'Greg Studer', u'email': u'greg@10gen.com'}

Message: SERVER-8598 hook up CollectionManager in place of ShardChunkManager
Branch: master
https://github.com/mongodb/mongo/commit/1459353f995228932bf5077aa4da52d8d68a6871

Comment by auto [ 04/Jun/13 ]

Author:

{u'username': u'gregstuder', u'name': u'Greg Studer', u'email': u'greg@10gen.com'}

Message: SERVER-8598 minor CollectionManager refactor cleanup pre-integration
Branch: master
https://github.com/mongodb/mongo/commit/ecc9f99bf487191c628c8c36fb9606304a790d31

Comment by auto [ 04/Jun/13 ]

Author:

{u'username': u'gregstuder', u'name': u'Greg Studer', u'email': u'greg@10gen.com'}

Message: SERVER-8598 ConfigVersion is not the same as ChunkVersion, cleanup
Branch: master
https://github.com/mongodb/mongo/commit/34f701faec271c454612a8b992efbae7f9070a96

Comment by auto [ 04/Jun/13 ]

Author:

{u'username': u'gregstuder', u'name': u'Greg Studer', u'email': u'greg@10gen.com'}

Message: SERVER-8598 push key expression evaluation out of ShardChunkManager
Branch: master
https://github.com/mongodb/mongo/commit/4cb7655e3da3466265ace5d6cf44c55b931f3d15

Comment by auto [ 22/May/13 ]

Author:

{u'username': u'renctan', u'name': u'Randolph Tan', u'email': u'randolph@10gen.com'}

Message: SERVER-8598 Add command to cleanup orphaned data created by failed chunk migrations

Moved range deleter files from s/ to db/
Branch: master
https://github.com/mongodb/mongo/commit/9d11c79f61a09496b50ee7e1eee34221b61ea842

Comment by auto [ 22/May/13 ]

Author:

{u'username': u'renctan', u'name': u'Randolph Tan', u'email': u'randolph@10gen.com'}

Message: SERVER-8598 Add command to cleanup orphaned data created by failed chunk migrations

Step 4: Delete documents when migration failed or was aborted at destination shard.
Branch: master
https://github.com/mongodb/mongo/commit/9e29cae9a0293d8aaa31dda4b74e19cbc0ec112a

Comment by auto [ 22/May/13 ]

Author:

{u'username': u'renctan', u'name': u'Randolph Tan', u'email': u'randolph@10gen.com'}

Message: SERVER-8598 Add command to cleanup orphaned data created by failed chunk migrations

Step 3: Replace OldDataCleanup with RangeDeleter
Branch: master
https://github.com/mongodb/mongo/commit/52e081c9bc5268dddcc3bf7e1df87266d7ea2ec3

Comment by auto [ 16/May/13 ]

Author:

{u'date': u'2013-05-16T20:00:07Z', u'name': u'Greg Studer', u'email': u'greg@10gen.com'}

Message: SERVER-8836 remove2.js wait for deletion on migration until the resolution of SERVER-8598
Branch: master
https://github.com/mongodb/mongo/commit/fe027497e73ea04b82077b9650876c40ea85b6ce

Comment by auto [ 18/Apr/13 ]

Author:

{u'date': u'2013-04-11T21:05:08Z', u'name': u'Randolph Tan', u'email': u'randolph@10gen.com'}

Message: SERVER-8598 Add command to cleanup orphaned data created by failed chunk migrations

Added a way to pass index specs for deletion.
Branch: master
https://github.com/mongodb/mongo/commit/e8129b807c8ac31b34c9c0cb9b5f4d76052453f3

Comment by Randolph Tan [ 18/Apr/13 ]

Author:

{u'date': u'2013-04-18T15:46:45Z', u'name': u'Randolph Tan', u'email': u'randolph@10gen.com'}

Message: SERVER-8589 Fixed race in range_deleter_test

Made sure to increment the counter after the event happened, not before.
Branch: master
https://github.com/mongodb/mongo/commit/8720e07600694c4390faab186ad5da8bdbe49a3a

Comment by auto [ 18/Apr/13 ]

Author:

{u'date': u'2013-04-18T10:06:24Z', u'name': u'Tad Marshall', u'email': u'tad@10gen.com'}

Message: SERVER-8598 Visual Studio – add new files
Branch: master
https://github.com/mongodb/mongo/commit/7a24fedbb3cdd55b2d204143a0cc605e391aa884

Comment by auto [ 18/Apr/13 ]

Author:

{u'date': u'2013-04-18T09:17:42Z', u'name': u'Tad Marshall', u'email': u'tad@10gen.com'}

Message: SERVER-8598 Fix Windows compile
Branch: master
https://github.com/mongodb/mongo/commit/957dda895f9bc4d481b1e40add92db9a33b46233

Comment by auto [ 17/Apr/13 ]

Author:

{u'date': u'2013-04-01T17:52:58Z', u'name': u'Randolph Tan', u'email': u'randolph@10gen.com'}

Message: SERVER-8598 Add command to cleanup orphaned data created by failed chunk migrations

Step 2: Implement range deleter.
Branch: master
https://github.com/mongodb/mongo/commit/61158f0c43993c321176fc0ecf47eb90c6c35dfe

Comment by auto [ 17/Apr/13 ]

Author:

{u'date': u'2013-03-29T14:26:07Z', u'name': u'Randolph Tan', u'email': u'randolph@10gen.com'}

Message: SERVER-8598 Add command to cleanup orphaned data created by failed chunk migrations

Step 1: Refactor bson range helpers.
Branch: master
https://github.com/mongodb/mongo/commit/2e3f2b38accbaf7ba47d39c04dc77caf9d9ed611

Comment by Randolph Tan [ 27/Feb/13 ]

Pseudo code in mongos:

SHARDED->commandOp(

{ cleanupOrphanData: <ns> }

)

Pseudo code in mongod:

if (isMigrationToHereActive) return false;

migrateStatus.prepare(); // so no chunks can be sent to this shard

for (chunk in ChunksIDontOwn) do
cleanUpOldData(chunk);
end

migrateStatus.setActive(false); // allow chunk migrations to this shard again

Notes:

  • ShardChunkManager only contains information about chunks the shard owns, so we need to manually query for the chunks.
  • I'm not too comfortable about reusing migrateStatus for locking. Should we create a new one?

Some things to consider:

  • secondary throttle?
Generated at Thu Feb 08 03:17:52 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.