[SERVER-37919] Unique index with partialFilterExpression option causes WriteConflict Created: 05/Nov/18  Updated: 27/Oct/23  Resolved: 30/Nov/18

Status: Closed
Project: Core Server
Component/s: Concurrency, Index Maintenance
Affects Version/s: 4.0.3, 4.1.4
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Dan Bolar Assignee: Xiangyu Yao (Inactive)
Resolution: Gone away Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File TransactionTest1.js    
Operating System: ALL
Steps To Reproduce:

'use strict';
 
const runTest = async () => {
 const MongoClient = require('mongodb').MongoClient;
 let client;
 let db;
 let s1, s2;
 try {
 console.log('Creating database');
 client = await MongoClient.connect('mongodb://localhost:27017');
 db = client.db('testdb');
 await db.dropDatabase();
 
 let collection = await db.createCollection('testcoll');
 await db.createIndex('testcoll', \{id: 1}, \{unique: true});
 await db.createIndex('testcoll', \{prop1: 1}, \{unique: true, partialFilterExpression: {prop1: {$exists: true}}});
 
 console.log('Inserting documents');
 s1 = client.startSession();
 s1.startTransaction();
 s2 = client.startSession();
 s2.startTransaction();
 
 await collection.insertOne(\{id:1}, \{session: s1});
 await collection.insertOne(\{id:2}, \{session: s2});
 
 await s1.commitTransaction();
 await s2.commitTransaction();
 await s1.endSession();
 await s2.endSession();
 
 console.log('Deleting documents');
 s1 = client.startSession();
 s1.startTransaction();
 s2 = client.startSession();
 s2.startTransaction();
 
 await collection.deleteOne(\{id:1}, \{session: s1});
 await collection.deleteOne(\{id:2}, \{session: s2});
 
 await s1.commitTransaction();
 await s2.commitTransaction();
 await s1.endSession();
 await s2.endSession();
 
 } catch (error) {
 if (s1 && s1.inTransaction()) {
 await s1.abortTransaction();
 s1.endSession();
 }
 if (s2 && s2.inTransaction()) {
 await s2.abortTransaction();
 s2.endSession();
 }
 throw error;
 }
};
 
runTest()
 .then (() => {
 process.exit(0);
 })
 .catch(err => {
 console.error(err);
 process.exit(1);
 });

Sprint: Storage NYC 2018-12-03
Participants:

 Description   

An attempt to delete two documents from the same collection from two different transactions results in a WriteConflict exception if the collection has a unique index with the partialFilterExpression option and the field for that index is left unset in both documents.



 Comments   
Comment by Xiangyu Yao (Inactive) [ 29/Nov/18 ]

Closing this ticket as the work to backport the fix to 4.0 is extensive, and an easy workaround exists: apps can retry transactions when encountering this situation.

Comment by Dan Bolar [ 27/Nov/18 ]

I completely uninstalled 4.0 and did a fresh install of 4.1 and the lock issue no longer occurs with the new data directory.

Comment by Xiangyu Yao (Inactive) [ 27/Nov/18 ]

Got it. Did you run the v4.1.4 binary with a new data directory or the same data directory that v4.0 binary used? In that case, v4.1.4 binary will read the WiredTiger's metadata set by v4.0 binary and use the old unique index key format.

Comment by Dan Bolar [ 27/Nov/18 ]

I was able to reproduce the issue on 4.1.4.

mongod --version
db version v4.1.4
git version: 2f4b5918497b09a226a3ec5dcff930edd52ea1e9
allocator: tcmalloc
modules: none
build environment:
distmod: 2012plus
distarch: x86_64
target_arch: x86_64

Comment by Xiangyu Yao (Inactive) [ 27/Nov/18 ]

Hi dbolar, I saw 4.1.4 is in 'Affects Versions'. Did you run the test on that version and get the failure? I tried running your repo on my local copy of 4.1.4 and didn't experience the failure.

Comment by Xiangyu Yao (Inactive) [ 27/Nov/18 ]

There are two behaviors which when combined will cause this problem:
1. When deleting a document, we unindex the fields unconditionally regardless of partial filters. The reason is because during upgrade/downgrade, the BSON matcher may change across versions and we want to make sure there is no dangling index key.
2. If we cannot find the index key when trying to remove it, we assume there is a concurrent background index build, so we insert the key and then remove it right away in order to trigger a write conflict with the index build. However, this logic erroneously triggers the write conflict in the case described in this ticket.

(This is a 4.0-only problem because in 4.2 we have a new unique index key format which has record id appended to the key; thus step 2 won't trigger a write conflict in the described case).

Generated at Thu Feb 08 04:47:25 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.