[SERVER-68122] Investigate replicating the collection WiredTiger config string during initial sync Created: 19/Jul/22  Updated: 14/Dec/23  Resolved: 19/Jan/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 4.4.15
Fix Version/s: 4.4.19, 5.0.15, 6.3.0-rc0, 6.0.5

Type: Bug Priority: Major - P3
Reporter: Shreyas Kalyan Assignee: Yujin Kang Park
Resolution: Fixed Votes: 9
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
is depended on by SERVER-72229 Implement FCV upgrade step to remove ... Closed
is depended on by TOOLS-3233 Investigate changes in SERVER-68122: ... Closed
Related
related to SERVER-83830 On Enterprise build creating a collec... Closed
related to SERVER-73183 WiredTigerKVEngine::getSanitizedStora... Closed
related to SERVER-81069 Cleanup mitigations for replicated en... Backlog
related to SERVER-72227 Investigate how ESE WiredTiger config... Closed
is related to SERVER-82205 Make shell dbHash comparison to exclu... Closed
is related to SERVER-79496 Ban encryption options being provided... Closed
Assigned Teams:
Storage Execution
Backwards Compatibility: Minor Change
Operating System: ALL
Backport Requested:
v6.2, v6.0, v5.0, v4.4
Sprint: Execution Team 2023-01-23
Participants:
Case:
Linked BF Score: 120

 Description   

A customer has tried to add a mongod with encryption disabled to a replica set in which the other members have encryption enabled. When running getCollectionInfos() on the members of the replica set, some of the collections have the options.storageEngine.wiredTiger.configString option set. This string has "encryption=(keyid=\"admin\",name=AES256-CBC)" set as part of the complete option set. When new member initiates initial sync, it attempts to replicate the collection in its entirety, including the configString. WiredTiger then realizes the conflict between the collection and the mongod config settings and throws BadValue: 22: Invalid argument.

This behavior can be replicated by manually setting the configString for a collection like this -

db.createCollection(
"users",
{ storageEngine: { wiredTiger: { configString: "access_pattern_hint=none,allocation_size=4KB,app_metadata=(formatVersion=1),assert=(commit_timestamp=none,read_timestamp=none),block_allocation=best,block_compressor=snappy,cache_resident=false,checksum=on,colgroups=,collator=,columns=,dictionary=0,encryption=(keyid=\"admin\",name=AES256-CBC),exclusive=false,extractor=,format=btree,huffman_key=,huffman_value=,ignore_in_memory_cache_size=false,immutable=false,internal_item_max=0,internal_key_max=0,internal_key_truncate=true,internal_page_max=4KB,key_format=q,key_gap=10,leaf_item_max=0,leaf_key_max=0,leaf_page_max=32KB,leaf_value_max=64MB,log=(enabled=false),lsm=(auto_throttle=true,bloom=true,bloom_bit_count=16,bloom_config=,bloom_hash_count=8,bloom_oldest=false,chunk_count_limit=0,chunk_max=5GB,chunk_size=10MB,merge_custom(prefix=,start_generation=0,suffix=),merge_max=15,merge_min=0),memory_page_image_max=0,memory_page_max=10m,os_cache_dirty_max=0,os_cache_max=0,prefix_compression=false,prefix_compression_min=4,source=,split_deepen_min_child=0,split_deepen_per_child=0,split_pct=90,type=file,value_format=u,type=file" }} })

Initially investigated on a 4.4 mongod, have not verified whether this behavior has been fixed in a 5.0+ mongod.

These strings have been seen set on system collections, which we do not believe were manually created using create.

Because ESE is intended to be configured on a per-node basis, for upgrade/downgrade reasons, we do not expect these options to be replicated in the catalog. Nodes processing collections with these properties set should ignore the durable options and only respect the options defined in their configuration files.



 Comments   
Comment by Githook User [ 10/Feb/23 ]

Author:

{'name': 'Yu Jin Kang Park', 'email': 'yujin.kang@mongodb.com', 'username': 'ykangpark'}

Message: SERVER-68122 Remove encryption from storageOptions on secondary replication

(cherry picked from commit ef120ac5552574fb9b84d36d842ead8519588c31)
Branch: v5.0
https://github.com/mongodb/mongo/commit/27e0a3c4030eb970fe7f2c542a51dc7cdf2d706a

Comment by Githook User [ 09/Feb/23 ]

Author:

{'name': 'Yu Jin Kang Park', 'email': 'yujin.kang@mongodb.com', 'username': 'ykangpark'}

Message: SERVER-68122 Remove encryption from storageOptions on secondary replication

(cherry picked from commit ef120ac5552574fb9b84d36d842ead8519588c31)
Branch: v6.0
https://github.com/mongodb/mongo/commit/050aefc4638149ab67ff720aa5f4b9d22e1defd7

Comment by Githook User [ 25/Jan/23 ]

Author:

{'name': 'Yu Jin Kang Park', 'email': 'yujin.kang@mongodb.com', 'username': 'ykangpark'}

Message: SERVER-68122 Remove encryption from storageOptions on secondary replication

(cherry picked from commit ef120ac5552574fb9b84d36d842ead8519588c31)
Branch: v4.4
https://github.com/mongodb/mongo/commit/055c340fd1ba35182333097f3090dc77b3ca2983

Comment by Githook User [ 25/Jan/23 ]

Author:

{'name': 'Yu Jin Kang Park', 'email': 'yujin.kang@mongodb.com', 'username': 'ykangpark'}

Message: SERVER-68122 Add ignore_replicated_encryption_options.js

(cherry picked from commit f17b7ab250bd497a18e44848036874c6916518e3)
Branch: v4.4
https://github.com/10gen/mongo-enterprise-modules/commit/e8f49c84a0cbc5a5c52df1f96aba2b8222c6854e

Comment by Githook User [ 19/Jan/23 ]

Author:

{'name': 'Yu Jin Kang Park', 'email': 'yujin.kang@mongodb.com', 'username': 'ykangpark'}

Message: SERVER-68122 Remove encryption from storageOptions on secondary replication
Branch: master
https://github.com/mongodb/mongo/commit/ef120ac5552574fb9b84d36d842ead8519588c31

Comment by Githook User [ 19/Jan/23 ]

Author:

{'name': 'Yu Jin Kang Park', 'email': 'yujin.kang@mongodb.com', 'username': 'ykangpark'}

Message: SERVER-68122 Add ignore_replicated_encryption_options.js
Branch: master
https://github.com/10gen/mongo-enterprise-modules/commit/f17b7ab250bd497a18e44848036874c6916518e3

Comment by Jean da Silva [ 26/Jul/22 ]

Hi team, 

I'm sharing that I also fail on a similar condition on 4.4.15.

Initial-sync breaks, and node halts while trying to add a new unencrypted node into an encrypted Replica Set.

 

Then on the log, we see the collection is created with the encryption option on the replica node:

{"t":{"$date":"2022-07-25T11:56:10.965-07:00"},"s":"I",  "c":"STORAGE",  "id":20320,   "ctx":"ReplCoordExtern-1","msg":"createCollection","attr":{"namespace":"admin.system.version","uuidDisposition":"provided","
uuid":{"uuid":{"$uuid":"29cac414-440d-4659-9e84-a9c20395e5d4"}},"options":{"uuid":{"$uuid":"29cac414-440d-4659-9e84-a9c20395e5d4"},"storageEngine":{"wiredTiger":{"configString":"access_pattern_hint=none,allocati
on_size=4KB,app_metadata=(formatVersion=1),assert=(commit_timestamp=none,read_timestamp=none),block_allocation=best,block_compressor=snappy,cache_resident=false,checksum=on,colgroups=,collator=,columns=,dictionary=0,encryption=(keyid=\"admin\",name=AES256-CBC)

Failing with the following message:

{"t":{"$date":"2022-07-25T11:56:10.967-07:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"ReplCoordExtern-1","msg":"WiredTiger error","attr":{"error":22,"message":"[1658775370:967624][63382:0x7f7c2531c700], file:admin/collection-21-8793244868447355919.wt, WT_SESSION.create: __encryptor_confchk, 372: unknown encryptor 'AES256-CBC': Invalid argument"}}
 
{"t":{"$date":"2022-07-25T11:56:10.967-07:00"},"s":"I",  "c":"INITSYNC", "id":21077,   "ctx":"ReplCoordExtern-1","msg":"Non-retryable error occurred during cloner stage","attr":{"cloner":"CollectionCloner","stage":"createCollection","error":{"code":2,"codeName":"BadValue","errmsg":"22: Invalid argument"}}}
 
{"t":{"$date":"2022-07-25T11:56:10.967-07:00"},"s":"E",  "c":"INITSYNC", "id":21149,   "ctx":"ReplCoordExtern-1","msg":"Collection clone failed","attr":{"namespace":"admin.system.version","error":"BadValue: 22: Invalid argument"}}

 

What bugs me is that is an admin.system collection, and in normal conditions and even encrypted, we don't see that configString information:

PRIMARY>  db.getCollectionInfos({"name": "system.version"})
[
    {
        "name" : "system.version",
        "type" : "collection",
        "options" : {
            
        },
        "info" : {
            "readOnly" : false,
            "uuid" : UUID("498d977d-ce2c-477c-ad34-43f2e0506ea8")
        },
        "idIndex" : {
            "v" : 2,
            "key" : {
                "_id" : 1
            },
            "name" : "_id_"
        }
    }
]

 

But in this case, we see that encryption parameter on configString as you mentioned before:

PRIMARY> db.getCollectionInfos({"name": "system.version"})
[
   {
      "name":"system.version",
      "type":"collection",
      "options":{
         "storageEngine":{
            "wiredTiger":{
               "configString":"access_pattern_hint=none,allocation_size=4KB,app_metadata=(formatVersion=1),assert=(commit_timestamp=none,read_timestamp=none),block_allocation=best,block_compressor=snappy,cache_resident=false,checksum=on,colgroups=,collator=,columns=,dictionary=0,encryption=(keyid=\"admin\",name=AES256-CBC),exclusive=false,extractor=,format=btree,huffman_key=,huffman_value=,ignore_in_memory_cache_size=false,immutable=false,internal_item_max=0,internal_key_max=0,internal_key_truncate=true,internal_page_max=4KB,key_format=q,key_gap=10,leaf_item_max=0,leaf_key_max=0,leaf_page_max=32KB,leaf_value_max=64MB,log=(enabled=false),lsm=(auto_throttle=true,bloom=true,bloom_bit_count=16,bloom_config=,bloom_hash_count=8,bloom_oldest=false,chunk_count_limit=0,chunk_max=5GB,chunk_size=10MB,merge_custom=(prefix=,start_generation=0,suffix=),merge_max=15,merge_min=0),memory_page_image_max=0,memory_page_max=10m,os_cache_dirty_max=0,os_cache_max=0,prefix_compression=false,prefix_compression_min=4,source=,split_deepen_min_child=0,split_deepen_per_child=0,split_pct=90,type=file,value_format=u,type=file"
            }
         }
      },
      "info":{
         "readOnly":false,
         "uuid":"UUID(""29cac414-440d-4659-9e84-a9c20395e5d4"")"
      },
      "idIndex":{
         "v":2,
         "key":{
            "_id":1
         },
         "name":"_id_",
         "ns":"admin.system.version"
      }
   }
]

 

 

 

Generated at Thu Feb 08 06:09:57 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.