[SERVER-79260] Avoid relying on fast count for change_collection_server_stats.js and multitenant_pre_image_expired_document_remover.js Created: 24/Jul/23 Updated: 29/Oct/23 Resolved: 29/Aug/23 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | 7.2.0-rc0 |
| Type: | Task | Priority: | Major - P3 |
| Reporter: | Haley Connelly | Assignee: | Miquel Munoz (Inactive) |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||
| Sprint: | Execution EMEA Team 2023-08-07, Execution EMEA Team 2023-09-04 | ||||||||||||
| Participants: | |||||||||||||
| Linked BF Score: | 0 | ||||||||||||
| Description |
|
change_collection_server_stats.js and multitenant_pre_image_expired_document_remover.js both rely on count() to test the number of pre-images/ change collection entries. count() uses fast-count; fast-count is not guaranteed to be correct for unreplicated, untimestamped collections which rely on WT truncate. WT truncate doesn't keep track of the amount of documents truncated (with the benefit of extremely fast document removal). For this reason, these collections rely on CollectionTruncateMarkers to estimate how much data / the amount of documents exist with a given range of the collections. These initial estimated numbers are used by truncate to update the fast-count. CollectionTruncateMarkers for change-collections and pre-images collections are lazily initialized at startup, and are estimates. For this reason, the tests which rely on fast-count should opt for counting the number of change-collection entries and pre-images by fetching the collection and counting the number of entries explicitly (an example with pre-images).
Old description: At some level, we expect initial fast count estimates to be off around startup because the markers are initialised as estimates. However, we should investigate whether or not we would expect the counts to match within the time an assert.soon completes and explore the limitations of the estimation process. |
| Comments |
| Comment by Githook User [ 29/Aug/23 ] |
|
Author: {'name': 'Miquel Munoz', 'email': 'miquelmunozz@gmail.com', 'username': 'mimugara'}Message: |
| Comment by Josef Ahmad [ 25/Jul/23 ] |
|
By design, fast count isn't expected to be accurate on oplog-like tables like the change stream pre-images table. Test code that requires accurate count should iterate-count the table. |
| Comment by Haley Connelly [ 24/Jul/23 ] |
|
The same issue is extremely likely to happen in change_collection_server_stats.js |