[SERVER-20407] findAndModify on mongoS upserts to the wrong shard Created: 14/Sep/15  Updated: 10/Dec/15  Resolved: 16/Sep/15

Status: Closed
Project: Core Server
Component/s: Querying, Sharding
Affects Version/s: 3.0.6, 3.1.8
Fix Version/s: 3.0.8, 3.1.9

Type: Bug Priority: Major - P3
Reporter: Misha Tyulenev Assignee: Kaloian Manassiev
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
is duplicated by SERVER-20634 Mongo 3.0.5 might lose update operati... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Completed:
Steps To Reproduce:

This is simplified jstests/sharding/find_and_modify_after_multi_write.js

    var st = new ShardingTest({ shards: 2, mongos: 2 });
 
    var testDB = st.s.getDB('test');
    testDB.dropDatabase();
 
    testDB.adminCommand({ enableSharding: 'test' });
    st.ensurePrimaryShard('test', 'shard0000');
    testDB.adminCommand({ shardCollection: 'test.user', key: { x: 1 }});
    testDB.adminCommand({ split: 'test.user', middle: { x: 0 }});
    testDB.adminCommand({ moveChunk: 'test.user', find: { x: 0 }, to: 'shard0001' });
 
    var testDB2 = st.s1.getDB('test');
    testDB2.user.insert({ x: 123456 });
 
    // Move chunk to bump version on a different mongos.
    testDB.adminCommand({ moveChunk: 'test.user', find: { x: 0 }, to: 'shard0000' });
 
    testDB2.user.update({}, { $inc: { y: 987654 }}, false, true);
 
    // Issue a targetted findAndModify and check that it was upserted to the right shard.
    testDB2.runCommand({ findAndModify: 'user', query: { x: 100 }, update: { $set: { y: 1 }}, upsert: true });
 
    st.d0.getDB('test').user.findOne({ x: 100 });

The last line should return a record, but it does not because findAndModify updates the wrong shard. The query

    st.d1.getDB('test').user.findOne({ x: 100 });

finds the record.

Participants:

 Description   

findAndModify command is not able to detect stale shard version and as a result can update the wrong shard if the chunk on the shard that it has cached has been moved via a moveChunk command from different mongoS.
Current test jstests/sharding/find_and_modify_after_multi_write.js masks the issue by executing findOne just before calling findAndModify. findOne updates the ssv on the connection.
The new ClusterClientCursor code path does not set ssv per connection and hence will invalidate the existing workaround.



 Comments   
Comment by Githook User [ 26/Oct/15 ]

Author:

{u'username': u'kaloianm', u'name': u'Kaloian Manassiev', u'email': u'kaloian.manassiev@mongodb.com'}

Message: SERVER-20407 Sharded findAndModify should retry on shard version mismatch

This is an approximate backport of
4821a88f0f92ea1f4ec3f46e71d5913ebd815fe5.
Branch: v3.0
https://github.com/mongodb/mongo/commit/3f62deef2f4cccc007387b2fb061c0814ecec7c8

Comment by Githook User [ 17/Sep/15 ]

Author:

{u'username': u'kaloianm', u'name': u'Kaloian Manassiev', u'email': u'kaloian.manassiev@mongodb.com'}

Message: SERVER-20407 Extend sharding 'after-multi-write' tests

The fix for SERVER-20407 had removed the part, which establishes an
existing version on the connection and thus reduced the coverage of these
tests. This change adds check that if a stale version has already been
established on the connection, it would be refreshed.
Branch: master
https://github.com/mongodb/mongo/commit/071a8bfa568f5d796803194531be6cb84a12760d

Comment by Githook User [ 17/Sep/15 ]

Author:

{u'username': u'kaloianm', u'name': u'Kaloian Manassiev', u'email': u'kaloian.manassiev@mongodb.com'}

Message: SERVER-20407 Extend sharding 'after-multi-write' tests

The fix for SERVER-20407 had removed the part, which establishes an
existing version on the connection and thus reduced the coverage of these
tests. This change adds check that if a stale version has already been
established on the connection, it would be refreshed.
Branch: master
https://github.com/mongodb/mongo/commit/071a8bfa568f5d796803194531be6cb84a12760d

Comment by Githook User [ 16/Sep/15 ]

Author:

{u'username': u'kaloianm', u'name': u'Kaloian Manassiev', u'email': u'kaloian.manassiev@mongodb.com'}

Message: SERVER-20407 Sharded findAndModify should retry on shard version mismatch
Branch: master
https://github.com/mongodb/mongo/commit/4821a88f0f92ea1f4ec3f46e71d5913ebd815fe5

Generated at Thu Feb 08 03:54:07 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.