Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-89286

Implicit FLE state collection creation fails when data placed outside the db primary shard

    • Type: Icon: Bug Bug
    • Resolution: Unresolved
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 6.0.0, 7.0.0, 8.0.0-rc0
    • Component/s: None
    • Catalog and Routing
    • ALL
    • CAR Team 2024-09-16, CAR Team 2024-09-30, CAR Team 2024-10-14, CAR Team 2024-10-28, CAR Team 2024-11-11, CAR Team 2024-11-25, CAR Team 2024-12-23, CAR Team 2025-01-06
    • 3

      Collection creation in transactions involving more than one shard is disallowed because there is no conflict detection machinery at prepare time for distributed transactions (SERVER-46105 should address that).

      Write operations on encrypted fields are transactionally updating both main collection (already existing) and state collections (lazily created). Because of that, the first insert on a collection with FLE options is triggering the creation of the necessary state collections within a transaction.

      Example of faulty flow:

      • Create FLE collection
      • Move it out the primary shard (moveChunk for sharded collection / moveCollection for unshareded collection)
      • Insert data
        • Write on the main collection will be routed to a shard different than the db primary
        • Write on state collection(s) will target the primary shard and implicitly create the collection
        • Transaction fails because trying to create collection in a distributed transaction

      Possible solutions:

      • Catch the error in fle write path and create FLE collections before retrying the transaction
      • Always create FLE state collections as part of the main collection creation

      Reproducible:

      const st = new ShardingTest({shards: 2, mongos: 1, rs: {nodes: 1}});
      const mongos = st.s0;
      const shard0 = st.shard0.shardName;
      const shard1 = st.shard1.shardName;
      
      const kDbName = 'test';
      const kCollName = 'coll';
      const kFullNs = kDbName + '.' + kCollName;
      
      const kms = {
          key: BinData(
              0,
              "/tu9jUCBqZdwCelwE/EAm/4WqdxrSMi04B8e9uAV+m30rI1J2nhKZZtQjdvsSCwuI4erR6IEcEK+5eGUAODv43NDNIR9QheT2edWFewUfHKsl9cnzTc86meIzOmYl6dr")
      };
      
      const csfleOpts = {
          kmsProviders: {
              local: kms,
          },
          keyVaultNamespace: kDbName + ".keystore",
          schemaMap: {},
      };
      
      var shell = Mongo(mongos.host, csfleOpts);
      var kv = shell.getKeyVault();
      
      const schema = {
          "fields": [
              {
                  "path": "foo",
                  "keyId": kv.createKey("local", "ignored"),
                  "bsonType": "string",
                  "queries": {"queryType": "equality"}
              },
          ]
      };
      
      // Create FLE collection, shard it and move unique range out of the db primary shard
      const db = shell.getDB(kDbName);
      assert.commandWorked(db.createCollection(kCollName, {encryptedFields: schema}));
      assert.commandWorked(mongos.adminCommand({shardCollection: kFullNs, key: {_id: 1}}));
      const toShard = st.getPrimaryShard("test").name == shard0 ? shard1 : shard0;
      assert.commandWorked(mongos.adminCommand({moveChunk: kFullNs, find: {_id: 0}, to: toShard}));
      
      // This insert will fail
      db.getCollection(kCollName).insertOne({foo: "foovalue"});
      
      st.stop();
      

            Assignee:
            robert.sander@mongodb.com Robert Sander
            Reporter:
            pierlauro.sciarelli@mongodb.com Pierlauro Sciarelli
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated: