[SERVER-61182] [inMemory] collmod_convert_to_unique.js fails under in-memory storage engine Created: 02/Nov/21  Updated: 10/Jan/24

Status: Backlog
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Benety Goh Assignee: Backlog - Storage Engines Team
Resolution: Unresolved Votes: 0
Labels: refinement
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-82431 Complete TODO listed in SERVER-61182 Closed
is related to SERVER-83314 Fail collMod command converting index... Blocked
is related to SERVER-61158 Convert a non-unique index to a uniqu... Closed
Assigned Teams:
Storage Engines
Operating System: ALL
Sprint: StorEng - Refinement Pipeline, Asparagus-StorEng - 2023-10-31
Participants:
Story Points: 2

 Description   

This test was introduced in SERVER-61158 to support collMod enhancements for indexes. The index spec is updated successfully in the index catalog but the WiredTiger storage engine fails on an invariant because the metadata changes, made with the the WT_SESSION::alter() function, did not go through as expected. We hit an invariant as we refresh the index using the WiredTigerUniqueIndex class.



 Comments   
Comment by Alexander Gorrod [ 01/Dec/23 ]

Thanks yuhong.zhang@mongodb.com. steve.kuhn@mongodb.com could you please create a project titled "Altering table metadata without exclusive access in WiredTiger" (or similar), so we can get more specific about how much work would be involved on the WiredTiger side to support it, and find a way to get it scheduled.

Comment by Yuhong Zhang [ 16/Nov/23 ]

alexander.gorrod@mongodb.com alter is used by collMod which is a replicated command used to modify collection metadata. This can cause a problem when we have a replica set with a mix of nodes running in-memory and on-disk storage engine. Currently there is really not a good solution to this problem so we would like to check again if it's possible to fix alter for in-memory engine.

Comment by Alexander Gorrod [ 25/Oct/23 ]

This is not simple to fix. The alter operation requires exclusive access to a data handle, which generally involves closing the handle and flushing associated content from the cache. It is not possible to do that when running with an in-memory only configuration, as the cache is the only place where the data exists.

It should (on some level) be safe for the alter to proceed without needing exclusive access, since it does not need to be concerned about the durability of the change (there is no durability). It's quite a lot of work to add a different concurrency control mechanism to handle alter commands, and given that this has remained as-is for a fair while without user-complaint, I'm inclined to close it out. The engineering effort required isn't justified given the minimal end-user consequence.

benety.goh@mongodb.com please re-open with additional context if this is the wrong decision.

Comment by Etienne Petrel [ 13/Dec/21 ]

Error when running the repo:

[js_test:collmod_convert_to_unique] [jsTest] ----
[js_test:collmod_convert_to_unique] [jsTest] Duplicate key error from failed conversion: {
[js_test:collmod_convert_to_unique] [jsTest] 	"ok" : 0,
[js_test:collmod_convert_to_unique] [jsTest] 	"errmsg" : "Failed to convert index to unique. firstRecordId: RecordId(1); firstDoc: { _id: 1.0, a: 100.0 }; secondRecordIdRecordId(2); secondDoc: { _id: 2.0, a: 100.0 }; collectionUUID: decc0ccf-78aa-426b-87ad-5037dabc61a8 :: caused by :: E11000 duplicate key error collection: test.collmod_convert_to_unique index: a_1 dup key: { a: 100.0 }",
[js_test:collmod_convert_to_unique] [jsTest] 	"code" : 11000,
[js_test:collmod_convert_to_unique] [jsTest] 	"codeName" : "DuplicateKey",
[js_test:collmod_convert_to_unique] [jsTest] 	"keyPattern" : {
[js_test:collmod_convert_to_unique] [jsTest] 		"a" : 1
[js_test:collmod_convert_to_unique] [jsTest] 	},
[js_test:collmod_convert_to_unique] [jsTest] 	"keyValue" : {
[js_test:collmod_convert_to_unique] [jsTest] 		"a" : 100
[js_test:collmod_convert_to_unique] [jsTest] 	}
[js_test:collmod_convert_to_unique] [jsTest] }
[js_test:collmod_convert_to_unique] [jsTest] ---- 

I dug into the code and I found this ticket WT-3310 that introduces the following changes where an alter command is ignored in the in-memory configuration. This comment from agorrod suspected this code might need to be changed at some point.

I removed those few lines from the code and the test runs successfully:

     /* In-memory ignores alter operations. */
-    if (F_ISSET(S2C(session), WT_CONN_IN_MEMORY))
-        goto err;
+    // if (F_ISSET(S2C(session), WT_CONN_IN_MEMORY))
+    //    goto err; 

It was a very quick investigation, we probably need to assess the impact(s) of allowing the alter command to be executed when WT is configured to be in memory.

 

Comment by Deepti Hasija [ 25/Nov/21 ]

Scope: Investigate and create a WT ticket if there is a fix required at the WT end.

Generated at Thu Feb 08 05:51:46 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.