[SERVER-39971] findAndModify in sharded transactions and readConcern “majority” miss existing record Created: 05/Mar/19  Updated: 27/Oct/23  Resolved: 11/Apr/19

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Asya Kamsky Assignee: Asya Kamsky
Resolution: Gone away Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates WT-4537 Fix WiredTiger cursor prev/next trave... Closed
Operating System: ALL
Participants:

 Description   

I created a sharded cluster with 1 mongos, 1 config server and three shards each with a single node replica set.

I pre-load the data and the contended table in question has this index:

{D_W_ID:1,  D_ID:1, D_NEXT_O_ID:1, D_TAX:1}, {unique:true}

in addition to _id index. The collection is sharded with shard key

 {D_W_ID:1, D_ID:1} 

The following two contended operations start majority of transactions:

db.coll.findAndModify({D_W_ID:1, D_ID:1}, /* query */
     {$inc:{D_NEXT_O_ID:1}}, /* update */
     {sort:{D_NEXT_O_ID:-1}}, /* sort */
     {fields:{_id:0, D_NEXT_O_ID:1, D_TAX:1}}, /* projection */
     {new:false}
});
Ddb.coll.findAndModify({D_W_ID:1, D_ID:1}, /* query */
     {$inc:{D_SOMEOTHER_FIELD:1}} /* update */
});

With readConcern “majority” every once in a while one of these two findAndModify commands matches 0 documents, which is impossible.

 

There are no inserts, deletes or balancing happening to this collection, only lots of findAndModify and a very occasional find.



 Comments   
Comment by Asya Kamsky [ 11/Apr/19 ]

Gone away as a result of WT-4537 being fixed.

Comment by Kaloian Manassiev [ 01/Apr/19 ]

Oh I wasn't sure if 4.1.9 is out yet or not, that's why I said latest nightly. We can close it then, thanks Asya.

Comment by Asya Kamsky [ 01/Apr/19 ]

I tried with 4.1.9 and I couldn’t. Should I try again with nightly? I assumed that the previously suspected issue that was fixed in 4.1.9 was it.

Comment by Kaloian Manassiev [ 01/Apr/19 ]

asya, can you try to reproduce this on the latest nightly build?

Comment by Asya Kamsky [ 14/Mar/19 ]

It looks like that’s fixed in 4.1.9 - once that’s out I can try to reproduce and if I cannot then can close this resolved as duplicate.

Comment by Tess Avitabile (Inactive) [ 08/Mar/19 ]

This could be related to WT-4537.

Comment by Asya Kamsky [ 05/Mar/19 ]

Here are some typical log lines:

command tpcc.DISTRICT command: findAndModify { findAndModify: "DISTRICT", query: { D_ID: 2, D_W_ID: 2, $comment: "NEW_ORDER" }, new: false, update: { $inc: { D_NEXT_O_ID: 1 } }, fields: { D_ID: 1, D_W_ID: 1, _id: 0, D_NEXT_O_ID: 1, D_TAX: 1 }, sort: { NO_O_ID: 1 }, upsert: false, startTransaction: true, readConcern: { afterClusterTime: Timestamp(1551799426, 22), level: "majority" }, txnNumber: 2, autocommit: false, $queryOptions: { $readPreference: { mode: "primary" } }, shardVersion: [ Timestamp(2, 0), ObjectId('5c7e934bdf239651bf93d780') ], coordinator: true, lsid: { id: UUID("cbf83769-6e90-46bc-b083-b0b5986f8cc0"), uid: BinData(0, E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855) }, $clusterTime: { clusterTime: Timestamp(1551799426, 61), signature: { hash: BinData(0, 667E34F25B54FC8688388B173D454147E40A4469), keyId: 6664925521173807120 } }, $client: { driver: { name: "PyMongo", version: "3.7.2" }, os: { type: "Linux", name: "Ubuntu 16.04 xenial", architecture: "x86_64", version: "4.15.0-1017-gcp" }, platform: "CPython 2.7.12.final.0", mongos: { host: "tpcc-client-2:30000", client: "127.0.0.1:45174", version: "4.1.8" } }, $configServerState: { opTime: { ts: Timestamp(1551799419, 501), t: 1 } }, $db: "tpcc" } planSummary: IXSCAN { D_W_ID: 1, D_ID: 1, D_NEXT_O_ID: 1, D_TAX: 1 } keysExamined:0 docsExamined:0 hasSortStage:1 nMatched:0 nModified:0 prepareReadConflicts:14 numYields:0 queryHash:CE1281A6 planCacheKey:AC5F2BA9 reslen:387 locks:{} protocol:op_msg 47ms
 
command tpcc.DISTRICT command: findAndModify { findAndModify: "DISTRICT", query: { D_ID: 10, D_W_ID: 3, $comment: "PAYMENT" }, new: false, update: { $inc: { D_YTD: 4402.88 } }, fields: { D_NAME: 1, D_STREET_1: 1, D_ZIP: 1, D_STATE: 1, D_CITY: 1, D_STREET_2: 1 }, upsert: false, startTransaction: true, readConcern: { level: "majority" }, txnNumber: 2, autocommit: false, $queryOptions: { $readPreference: { mode: "primary" } }, shardVersion: [ Timestamp(3, 0), ObjectId('5c7edf3721787375ddde6f6d') ], coordinator: true, lsid: { id: UUID("7b8fd330-746a-471e-90e5-f4ebb1c2a79c"), uid: BinData(0, E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855) }, $clusterTime: { clusterTime: Timestamp(1551819049, 34), signature: { hash: BinData(0, 6602C30F1E7C8387BF65C0866E2818512836FF73), keyId: 6664982283461591069 } }, $client: { driver: { name: "PyMongo", version: "3.7.2" }, os: { type: "Linux", name: "Ubuntu 16.04 xenial", architecture: "x86_64", version: "4.15.0-1017-gcp" }, platform: "CPython 2.7.12.final.0", mongos: { host: "tpcc-client-2:30000", client: "127.0.0.1:45586", version: "4.1.8" } }, $configServerState: { opTime: { ts: Timestamp(1551819048, 1), t: 1 } }, $db: "tpcc" } planSummary: IXSCAN { D_W_ID: 1, D_ID: 1, D_NEXT_O_ID: 1, D_TAX: 1 } keysExamined:0 docsExamined:0 nMatched:0 nModified:0 prepareReadConflicts:1 numYields:0 queryHash:A1074DDB planCacheKey:1EFA745D reslen:387 locks:{} protocol:op_msg 34ms 

Generated at Thu Feb 08 04:53:39 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.