-
Type:
Task
-
Resolution: Fixed
-
Priority:
Major - P3
-
Affects Version/s: None
-
Component/s: Schema Management, Statistics
-
Storage Engines - Foundations
-
1,216.011
-
SE Foundations - 2026-10-13, SE Foundations - 2026-10-27, SE Foundations - 2026-11-10
-
5
Add statistics to schema operations with epochs:
- publish-queue depth,
- publish-queue cancellations,
- publish-queue deferrals,
- violations,
- etc...
Add a dedicated verbose category.
Note: Part of this ticket is to figure out which statistics are useful. Do not add metrics without clear understanding why they are needed.
Note: Consider updating Grafana dashboard(s) once the stats are in place.
Statistics to implement
Research done; WT-18949 (verbose category) and WT-18954 (queue log before API-violation panics) are merged.
Queue counts cover creates and drops only; the per-checkpoint metadata updates are excluded. Operations queued before the stable schema epoch was set (legacy) are not counted.
| Statistic | Type | Why |
|---|---|---|
| disagg: schema operations currently awaiting publish | gauge | The application owes a publish. Unpublished tables stay in cache, and their drops get EBUSY. |
| disagg: oldest schema operation currently awaiting publish, age (secs) | gauge | A stuck publish shows here even when the count is small. |
| disagg: schema operations currently awaiting a checkpoint | gauge | The stable schema epoch, the checkpoint, or the follower pickup is lagging. |
| disagg: oldest schema operation currently awaiting a checkpoint, age since publish (secs) | gauge | How long that lag has lasted. |
| transaction: transaction global stable disaggregated schema epoch | gauge | Compare with the stable timestamp, which is the same clock in MongoDB. |
| transaction: transaction global last checkpoint disaggregated schema epoch | gauge | Which schema operations the last checkpoint covers. |
| disagg: unpublished schema operations canceled by a drop | counter | Creates dropped before they were ever published. |
| disagg: drops refused because the table holds unpublished data | counter | The drop failure count cannot tell this EBUSY apart from the others. |
| checkpoint: shared metadata queue drain most recent time (msecs) | gauge | The drain holds the schema lock, so creates and drops wait for it. |
The first four are the Grafana candidates; the rest are for FTDC.
Not added:
- Violation counters. Every violation panics, or MongoDB aborts on the publish error, so the count never reaches FTDC;
WT-18954logs them instead. - Deferrals. checkpoint_disagg_metadata_unstable already counts them.
- Per-type enqueue, prune and clear counts. The gauges show their effect.
- Per-table statistics. Too costly at large table counts.
Cost: publish and cancel count during the queue walk they already make, enqueue counts in O(1), and the checkpoint drain, follower prune and clear recount. The statistics reader takes no lock. Measured on a release build with 5,000 queued operations: about 2-3 µs per publish, about 0.4% of a create plus its publish. A separate recount walk measured about +70% per publish.
Follow-up outside WT: a Grafana row, plus a CLOUDP whitelist entry for the four gauges.