[SERVER-60455] serverStatus metrics for external sorter Created: 04/Oct/21  Updated: 29/Oct/23  Resolved: 15/Jul/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 6.0.4, 6.1.0-rc0

Type: Improvement Priority: Major - P3
Reporter: Louis Williams Assignee: Shin Yee Tan
Resolution: Fixed Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
is depended on by COMPASS-5993 Investigate changes in SERVER-60455: ... Closed
Documented
is documented by DOCS-15518 [SERVER] Investigate changes in SERVE... Closed
Related
is related to SERVER-67703 Report sorter stats for queries in se... Closed
is related to SERVER-67903 Move existing sorter statistics to so... Closed
Backwards Compatibility: Fully Compatible
Backport Requested:
v6.0
Sprint: Execution Team 2021-12-13, Execution Team 2022-01-24, Execution Team 2022-02-21, Execution Team 2022-03-07, Execution Team 2022-03-21, Execution Team 2022-04-04, Execution Team 2022-05-02, Execution Team 2022-05-16, Execution Team 2022-05-30, Execution Team 2022-06-13, Execution Team 2022-06-27, Execution Team 2022-07-11, Execution Team 2022-07-25
Participants:

 Description   

It would be useful to capture the following diagnostic information in serverStatus about the external sorter:

  • Number of sorters in use
  • Number of sorter spills
  • Amount of data spilled
  • Average amount of data written per spill
  • Amount of unsorted data buffered in memory
  • Amount of data buffered in memory for merging


 Comments   
Comment by Githook User [ 23/Nov/22 ]

Author:

{'name': 'Shin Yee Tan', 'email': 'shinyee.tan@mongodb.com', 'username': 'shinyeet'}

Message: SERVER-60455 serverStatus metrics for external sorter

(cherry picked from commit dc6803c67067003e6575fdd57e7c387fbcb8b23b)
Branch: v6.0
https://github.com/mongodb/mongo/commit/06729d7c8624609bf417324c3e11c4f6d4641a5d

Comment by Githook User [ 15/Jul/22 ]

Author:

{'name': 'Shin Yee Tan', 'email': 'shinyee.tan@mongodb.com', 'username': 'shinyeet'}

Message: SERVER-60455 serverStatus metrics for external sorter
Branch: master
https://github.com/mongodb/mongo/commit/dc6803c67067003e6575fdd57e7c387fbcb8b23b

Comment by Bruce Lucas (Inactive) [ 15/Jun/22 ]

shinyee.tan@mongodb.com  thanks for the update. That all makes sense.

Comment by Shin Yee Tan [ 14/Jun/22 ]

Hey bruce.lucas@mongodb.com, regarding your questions:

  • We'll be reporting the number of sorter spills cumulative of all index creates and reporting the cumulative amount of data of these spills
  • We won't be computing the average amount of data per spill anymore because as you said, we'll be able to see this over whatever interval with serverStatus
  • For this ticket, we will have counters (for reasons stated in this ticket) for the number of sorters, number of spills, and amount of data spilled. We will file a separate ticket to investigate adding metrics for the unsorted data buffered in memory and data buffered in memory for merging.

We'll also be narrowing the scope of this ticket to external sorter usage only during index builds as queries have a separate stats reporting. As we're interested in the stats cumulative for all index creates, we'll report the diagnostics in serverStatus.

 

Comment by Bruce Lucas (Inactive) [ 05/Oct/21 ]

A few questions:

  • number of sorter spills is cumulative for all index creates?
  • ditto for amount of data?
  • "average amount of data written per spill" - average over what? Typically serverStatus has cumulative metrics so an average can be computed over whatever interval is under consideration (e.g. sample interval in t2, etc.)
  • the others are instantaneous numbers?

Generally if you're interested in stats per index create, it might be better to include these in the log line at the end of the index build instead of in serverStatus.

Generated at Thu Feb 08 05:49:51 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.