[SERVER-62748] spill_to_disk.js fails burn-in tests with "WT_CACHE_FULL: operation would overflow cache" error Created: 19/Jan/22 Updated: 09/Jun/23 |
|
| Status: | Backlog |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Minor - P4 |
| Reporter: | Irina Yatsenko (Inactive) | Assignee: | Backlog - Query Execution |
| Resolution: | Unresolved | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||
| Assigned Teams: |
Query Execution
|
||||||||
| Sprint: | QE 2022-02-07, QE 2022-02-21, QE 2022-01-24 | ||||||||
| Participants: | |||||||||
| Linked BF Score: | 124 | ||||||||
| Description |
|
The test creates up to 200MB of test data and that seems to stress the WT cache when the test is run in repeat mode on Enterprise RHEL 8.0 (inMemory) for the burn-in check. https://jira.mongodb.org/browse/SERVER-61300 ran into this as well but the fix for that ticket hasn't fully addressed the problem as it's still happening: https://spruce.mongodb.com/task/mongodb_mongo_master_enterprise_rhel_80_64_bit_inmem_required_display_burn_in_tests_patch_2a763ca4ea023341983ca8f5e89fa214a56e331c_61e74f570305b91e5b10596e_22_01_18_23_40_26/execution-tasks?execution=0&sorts=STATUS%3AASC Affected suites: [j0:s0:prim] | 2022-01-19T06:47:56.628+00:00 W STORAGE 6148401 [JournalFlusher] "The JournalFlusher encountered an error attempting to flush data to disk","attr": {"JournalFlusherError":"ExceededMemoryLimit: -31807: WT_CACHE_FULL: operation would overflow cache"}[js_test:spill_to_disk] assert: write failed with error: { [js_test:spill_to_disk] } [js_test:spill_to_disk] } : |
| Comments |
| Comment by Mihai Andrei [ 19/Jan/22 ] |
|
That sounds fine to me, but I wonder whether we should spend some time to better understand why the JournalFlusher is running out of memory so quickly? Also, this seems to be an issue only on the burn in tests task, but not when run once regularly in a suite. |
| Comment by Kyle Suarez [ 19/Jan/22 ] |
|
mihai.andrei, taking a look at the linked patch build, it looks like cleaning every 20 executions still isn't enough. irina.yatsenko has suggested reducing the amount of memory used by the test, to avoid stressing memory-constrained systems like the inMemory storage engine builder. Perhaps we could dial down the size of the documents, as well as internalQuerySlotBasedExecutionHashAggApproxMemoryUseInBytesBeforeSpill, to maybe half of the current size? Even more? |