[SERVER-75814] MDB compact commands running in testing aren't actually exercising WT compact because the data size is too small and WT skips it Created: 06/Apr/23  Updated: 04/Dec/23

Status: Open
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Major - P3
Reporter: Dianna Hohensee (Inactive) Assignee: Backlog - Storage Engines Team
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-79869 The WiredTiger event handler is not u... Closed
related to SERVER-70201 Make compact killable Closed
Assigned Teams:
Storage Engines
Sprint: StorEng - Defined Pipeline, 2023-11-28 - Anthill Tiger
Participants:
Story Points: 5

 Description   

WT::compact skips compaction if the data size is too small
"skipping because the file size must be greater than 1MB"

[js_test:index_version_autoupgrade] d50050| {"t":{"$date":"2023-04-06T20:06:52.860+00:00"},"s":"I",  "c":"WTCMPCT",  "id":22430,   "ctx":"conn1","msg":"WiredTiger message","attr":{"message":{"ts_sec":1680811612,"ts_usec":860738,"thread":"11893:0x7f3546e11700","session_dhandle_name":"file:collection-15--5349796768648330457.wt","session_name":"WT_SESSION.compact","category":"WT_VERB_COMPACT","category_id":7,"verbose_level":"INFO","verbose_level_id":0,"msg":"collection-15--5349796768648330457.wt: skipping because the file size must be greater than 1MB: 4096B."}}}
 

MDB layer JS tests calling the compact command aren't actually testing compaction because compaction is skipped due to too little data. For reference this is the WT::compact code.

This is the list of effected tests:

jstests/noPassthrough/index_version_autoupgrade.js.
jstests/replsets/read_committed_with_catalog_changes.js
jstests/concurrency/fsm_workloads/compact.js
jstests/concurrency/fsm_workloads/compact_while_creating_indexes.js

There are other tests call the MDB compact command, but the intent in those tests are things like auth, allowed to run on a secondary, view namespace handling, etc.



 Comments   
Comment by Sean Watt [ 29/Nov/23 ]

I've taken a look through the jstests that use the compact command and many of them just test the usage of the server API and don't intend on compact doing any particular work.

In response to the tests linked in the description:

  • index_version_auoupgrade.js
    • Tests whether various MongoDB commands automatically upgrade the index version of existing indexes when they are rebuilt on a collection.
    • Since this is testing server functionality when calling commands, I don't think it's necessary for compact to do work.
  • read_committed_with_catalog_changes
    • It's possible to make compact do some work here. I'll continue investigating this one.
  • compact.js
    • This has been updated in SERVER-80120 and is now the main functional test for the compact algorithm in the jstests.
  • compact_while_creating_indexes
    • This one is currently disabled and should be fixed in SERVER-81114. That fix should also consider ensuring compact rewrites data.

A few other compact tests to consider are:

  • interupt_compact_commands.js
    • This one already creates large enough free space.
  • compact_keep_index.js
    • This is a simple one and could be combined into the existing `compact.js` test.

In summary the only tests worthwhile changing are

  • read_committed_with_catalog_changes
  • compact_keep_index.js
Comment by Etienne Petrel [ 18/Jul/23 ]

As keith.smith@mongodb.com said above, we have a set of tests dedicated to compact. However, with PM-3091, we would like MDB to introduce more testing in this area, especially with the new API that will be implemented. gregory.wlodarek@mongodb.com is the POC for those new tests.

Generated at Thu Feb 08 06:31:08 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.