[SERVER-29788] Log moveChunk counts in changelog Created: 22/Jun/17  Updated: 06/Dec/22  Resolved: 24/Oct/17

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Kevin Pulo Assignee: [DO NOT USE] Backlog - Sharding Team
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates SERVER-28670 Add sharding metadata refresh metrics... Closed
Related
Assigned Teams:
Sharding
Backwards Compatibility: Fully Compatible
Participants:

 Description   

Currently the counts of migrated documents are only recorded in the mongod logs, and not in config.changelog:

2017-06-12T19:05:49.897+0000 I SHARDING [conn1028707] moveChunk data transfer progress: { active: true, sessionId: "...", ns: "...", from: ".../...:27018,...:27018,...:27018", min: { ... }, max: { ... }, shardKeyPattern: { ... }, state: "steady", counts: { cloned: 53195, clonedBytes: 68335232, catchup: 0, steady: 0 }, ok: 1.0 } mem used: 0 documents remaining to clone: 0
2017-06-12T19:05:49.955+0000 I SHARDING [conn1028707] Migration successfully entered critical section
2017-06-12T19:05:49.965+0000 I SHARDING [conn1028707] Refreshing chunks for collection ... based on version ...|...||...
2017-06-12T19:05:49.982+0000 I SHARDING [CatalogCacheLoader-4626] Refresh for collection ... took 17 ms and found version ...|...||...
2017-06-12T19:05:49.992+0000 I SHARDING [conn1028707] Refreshing metadata for collection ... from collection version: ...|...||..., shard version: ...|...||... to collection version: ...|...||..., shard version: ...|...||...
2017-06-12T19:05:49.992+0000 I SHARDING [conn1028707] Migration succeeded and updated collection version to ...|...||...
2017-06-12T19:05:49.992+0000 I SHARDING [conn1028707] about to log metadata event into changelog: { _id: "...", server: "...", clientAddr: "...", time: new Date(...), what: "moveChunk.commit", ns: "...", details: { min: { ... }, max: { ... }, from: "...", to: "..." } }

It would be useful if the final counts could also be saved in the moveChunk.commit changelog document, ie:

 {
         "_id" : "...",
         "server" : "...",
         "clientAddr" : "...",
         "time" : ISODate(...),
         "what" : "moveChunk.commit",
         "ns" : "...",
         "details" : {
                 "min" : {
                         ...
                 },
                 "max" : {
                         ...
                 },
                 "from" : "...",
                 "to" : "...",
+                "counts" : {
+                        "cloned" : 53195,
+                        "clonedBytes" : 68335232,
+                        "catchup" : 0,
+                        "steady" : 0
+                }
         }
 }



 Comments   
Comment by Kaloian Manassiev [ 24/Oct/17 ]

Closing in lieu of SERVER-28670.

Comment by Kaloian Manassiev [ 17/Jul/17 ]

I don't think including these counts in the changelog document would actually have helped pinpoint the cause for slow migrations, because they only show the amount of data which was transferred during the clone stage, but neither show the duration, nor do they show how long did the catch-up take.

In addition I just saw that the catchup and steady values are always zero, because we never update them on the recipient shard, but I'll file a separate ticket for that.

I vote that we close this ticket in place of SERVER-28670 unless you think it might be beneficial to also include the migrations' duration as part of the changelog entry?

Generated at Thu Feb 08 04:21:49 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.