Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-20407

findAndModify on mongoS upserts to the wrong shard

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: 3.0.6, 3.1.8
    • Fix Version/s: 3.0.8, 3.1.9
    • Component/s: Querying, Sharding
    • Labels:
      None
    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      ALL
    • Backport Completed:
    • Steps To Reproduce:
      Hide

      This is simplified jstests/sharding/find_and_modify_after_multi_write.js

          var st = new ShardingTest({ shards: 2, mongos: 2 });
       
          var testDB = st.s.getDB('test');
          testDB.dropDatabase();
       
          testDB.adminCommand({ enableSharding: 'test' });
          st.ensurePrimaryShard('test', 'shard0000');
          testDB.adminCommand({ shardCollection: 'test.user', key: { x: 1 }});
          testDB.adminCommand({ split: 'test.user', middle: { x: 0 }});
          testDB.adminCommand({ moveChunk: 'test.user', find: { x: 0 }, to: 'shard0001' });
       
          var testDB2 = st.s1.getDB('test');
          testDB2.user.insert({ x: 123456 });
       
          // Move chunk to bump version on a different mongos.
          testDB.adminCommand({ moveChunk: 'test.user', find: { x: 0 }, to: 'shard0000' });
       
          testDB2.user.update({}, { $inc: { y: 987654 }}, false, true);
       
          // Issue a targetted findAndModify and check that it was upserted to the right shard.
          testDB2.runCommand({ findAndModify: 'user', query: { x: 100 }, update: { $set: { y: 1 }}, upsert: true });
       
          st.d0.getDB('test').user.findOne({ x: 100 });
      

      The last line should return a record, but it does not because findAndModify updates the wrong shard. The query

          st.d1.getDB('test').user.findOne({ x: 100 });
      

      finds the record.

      Show
      This is simplified jstests/sharding/find_and_modify_after_multi_write.js var st = new ShardingTest({ shards: 2, mongos: 2 });   var testDB = st.s.getDB('test'); testDB.dropDatabase();   testDB.adminCommand({ enableSharding: 'test' }); st.ensurePrimaryShard('test', 'shard0000'); testDB.adminCommand({ shardCollection: 'test.user', key: { x: 1 }}); testDB.adminCommand({ split: 'test.user', middle: { x: 0 }}); testDB.adminCommand({ moveChunk: 'test.user', find: { x: 0 }, to: 'shard0001' });   var testDB2 = st.s1.getDB('test'); testDB2.user.insert({ x: 123456 });   // Move chunk to bump version on a different mongos. testDB.adminCommand({ moveChunk: 'test.user', find: { x: 0 }, to: 'shard0000' });   testDB2.user.update({}, { $inc: { y: 987654 }}, false, true);   // Issue a targetted findAndModify and check that it was upserted to the right shard. testDB2.runCommand({ findAndModify: 'user', query: { x: 100 }, update: { $set: { y: 1 }}, upsert: true });   st.d0.getDB('test').user.findOne({ x: 100 }); The last line should return a record, but it does not because findAndModify updates the wrong shard. The query st.d1.getDB('test').user.findOne({ x: 100 }); finds the record.

      Description

      findAndModify command is not able to detect stale shard version and as a result can update the wrong shard if the chunk on the shard that it has cached has been moved via a moveChunk command from different mongoS.
      Current test jstests/sharding/find_and_modify_after_multi_write.js masks the issue by executing findOne just before calling findAndModify. findOne updates the ssv on the connection.
      The new ClusterClientCursor code path does not set ssv per connection and hence will invalidate the existing workaround.

        Issue Links

          Activity

          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'kaloianm', u'name': u'Kaloian Manassiev', u'email': u'kaloian.manassiev@mongodb.com'}

          Message: SERVER-20407 Sharded findAndModify should retry on shard version mismatch
          Branch: master
          https://github.com/mongodb/mongo/commit/4821a88f0f92ea1f4ec3f46e71d5913ebd815fe5

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'kaloianm', u'name': u'Kaloian Manassiev', u'email': u'kaloian.manassiev@mongodb.com'} Message: SERVER-20407 Sharded findAndModify should retry on shard version mismatch Branch: master https://github.com/mongodb/mongo/commit/4821a88f0f92ea1f4ec3f46e71d5913ebd815fe5
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'kaloianm', u'name': u'Kaloian Manassiev', u'email': u'kaloian.manassiev@mongodb.com'}

          Message: SERVER-20407 Extend sharding 'after-multi-write' tests

          The fix for SERVER-20407 had removed the part, which establishes an
          existing version on the connection and thus reduced the coverage of these
          tests. This change adds check that if a stale version has already been
          established on the connection, it would be refreshed.
          Branch: master
          https://github.com/mongodb/mongo/commit/071a8bfa568f5d796803194531be6cb84a12760d

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'kaloianm', u'name': u'Kaloian Manassiev', u'email': u'kaloian.manassiev@mongodb.com'} Message: SERVER-20407 Extend sharding 'after-multi-write' tests The fix for SERVER-20407 had removed the part, which establishes an existing version on the connection and thus reduced the coverage of these tests. This change adds check that if a stale version has already been established on the connection, it would be refreshed. Branch: master https://github.com/mongodb/mongo/commit/071a8bfa568f5d796803194531be6cb84a12760d
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'kaloianm', u'name': u'Kaloian Manassiev', u'email': u'kaloian.manassiev@mongodb.com'}

          Message: SERVER-20407 Extend sharding 'after-multi-write' tests

          The fix for SERVER-20407 had removed the part, which establishes an
          existing version on the connection and thus reduced the coverage of these
          tests. This change adds check that if a stale version has already been
          established on the connection, it would be refreshed.
          Branch: master
          https://github.com/mongodb/mongo/commit/071a8bfa568f5d796803194531be6cb84a12760d

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'kaloianm', u'name': u'Kaloian Manassiev', u'email': u'kaloian.manassiev@mongodb.com'} Message: SERVER-20407 Extend sharding 'after-multi-write' tests The fix for SERVER-20407 had removed the part, which establishes an existing version on the connection and thus reduced the coverage of these tests. This change adds check that if a stale version has already been established on the connection, it would be refreshed. Branch: master https://github.com/mongodb/mongo/commit/071a8bfa568f5d796803194531be6cb84a12760d
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'kaloianm', u'name': u'Kaloian Manassiev', u'email': u'kaloian.manassiev@mongodb.com'}

          Message: SERVER-20407 Sharded findAndModify should retry on shard version mismatch

          This is an approximate backport of
          4821a88f0f92ea1f4ec3f46e71d5913ebd815fe5.
          Branch: v3.0
          https://github.com/mongodb/mongo/commit/3f62deef2f4cccc007387b2fb061c0814ecec7c8

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'kaloianm', u'name': u'Kaloian Manassiev', u'email': u'kaloian.manassiev@mongodb.com'} Message: SERVER-20407 Sharded findAndModify should retry on shard version mismatch This is an approximate backport of 4821a88f0f92ea1f4ec3f46e71d5913ebd815fe5. Branch: v3.0 https://github.com/mongodb/mongo/commit/3f62deef2f4cccc007387b2fb061c0814ecec7c8

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: