[SERVER-16624] Batch insert triggers deadlock-detection fassert Created: 22/Dec/14  Updated: 24/Dec/14  Resolved: 22/Dec/14

Status: Closed
Project: Core Server
Component/s: Concurrency, Storage
Affects Version/s: 2.8.0-rc3
Fix Version/s: 2.8.0-rc4

Type: Bug Priority: Critical - P2
Reporter: Kamran K. Assignee: Kaloian Manassiev
Resolution: Done Votes: 0
Labels: 28qa
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File full_bt.txt     File gdb.log.tar.gz    
Issue Links:
Related
is related to SERVER-16636 Deadlock detection should check cycle... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Participants:

 Description   

I've hit this invariant a few times while running multiple tests in parallel. The tests were run against the mmapv1 storage engine.

Backtrace:

#0  0x00007ffff7bcc20b in raise (sig=5) at ../nptl/sysdeps/unix/sysv/linux/pt-raise.c:37
#1  0x0000000001899b9f in mongo::mongo_breakpoint () at src/mongo/util/debug_util.cpp:58
#2  0x0000000001890394 in mongo::breakpoint () at src/mongo/util/debug_util.h:73
#3  0x000000000188f378 in mongo::fassertFailed (msgid=28591) at src/mongo/util/assert_util.cpp:163
#4  0x0000000001313037 in mongo::LockerImpl<true>::lockComplete (this=0x3ba32a00, resId=..., timeoutMs=4294967295, checkDeadlock=false) at src/mongo/db/concurrency/lock_state.cpp:623
#5  0x0000000001311878 in mongo::LockerImpl<true>::lock (this=0x3ba32a00, resId=..., mode=mongo::MODE_X, timeoutMs=4294967295, checkDeadlock=false) at src/mongo/db/concurrency/lock_state.cpp:352
#6  0x0000000001303caf in mongo::Lock::CollectionLock::CollectionLock (this=0x83b3050, lockState=0x3ba32a00, ns=..., mode=mongo::MODE_IX) at src/mongo/db/concurrency/d_concurrency.cpp:194
#7  0x00000000012fa021 in mongo::WriteBatchExecutor::ExecInsertsState::_lockAndCheckImpl (this=0x7fffc6de1e30, result=0x7fffc6de09a0, intentLock=true)
    at src/mongo/db/commands/write_commands/batch_executor.cpp:982
#8  0x00000000012fa6d2 in mongo::WriteBatchExecutor::ExecInsertsState::lockAndCheck (this=0x7fffc6de1e30, result=0x7fffc6de09a0) at src/mongo/db/commands/write_commands/batch_executor.cpp:1026
#9  0x00000000012fa871 in mongo::insertOne (state=0x7fffc6de1e30, result=0x7fffc6de09a0) at src/mongo/db/commands/write_commands/batch_executor.cpp:1054
#10 0x00000000012faafb in mongo::WriteBatchExecutor::execOneInsert (this=0x7fffc6de2240, state=0x7fffc6de1e30, error=0x7fffc6de1e08) at src/mongo/db/commands/write_commands/batch_executor.cpp:1083
#11 0x00000000012f94e0 in mongo::WriteBatchExecutor::execInserts (this=0x7fffc6de2240, request=..., errors=0x7fffc6de20c0) at src/mongo/db/commands/write_commands/batch_executor.cpp:874
#12 0x00000000012f8f6d in mongo::WriteBatchExecutor::bulkExecute (this=0x7fffc6de2240, request=..., upsertedIds=0x7fffc6de20e0, errors=0x7fffc6de20c0)
    at src/mongo/db/commands/write_commands/batch_executor.cpp:757
#13 0x00000000012f6f71 in mongo::WriteBatchExecutor::executeBatch (this=0x7fffc6de2240, request=..., response=0x7fffc6de2280) at src/mongo/db/commands/write_commands/batch_executor.cpp:266
#14 0x0000000001300d0e in mongo::WriteCmd::run (this=0x30e92c0, txn=0x7fffc6de37b0, dbName=..., cmdObj=..., options=0, errMsg=..., result=..., fromRepl=false)
    at src/mongo/db/commands/write_commands/write_commands.cpp:144
#15 0x000000000131c64f in mongo::_execCommand (txn=0x7fffc6de37b0, c=0x30e92c0, dbname=..., cmdObj=..., queryOptions=0, errmsg=..., result=..., fromRepl=false) at src/mongo/db/dbcommands.cpp:1231
#16 0x000000000131d5cc in mongo::Command::execCommand (txn=0x7fffc6de37b0, c=0x30e92c0, queryOptions=0, cmdns=0x37c0a014 "db0.$cmd", cmdObj=..., result=..., fromRepl=false)
    at src/mongo/db/dbcommands.cpp:1447
#17 0x000000000131deae in mongo::_runCommands (txn=0x7fffc6de37b0, ns=0x37c0a014 "db0.$cmd", _cmdobj=..., b=..., anObjBuilder=..., fromRepl=false, queryOptions=0) at src/mongo/db/dbcommands.cpp:1522
#18 0x000000000151fd70 in mongo::runCommands (txn=0x7fffc6de37b0, ns=0x37c0a014 "db0.$cmd", jsobj=..., curop=..., b=..., anObjBuilder=..., fromRepl=false, queryOptions=0) at src/mongo/db/query/find.cpp:131
#19 0x0000000001521a74 in mongo::runQuery (txn=0x7fffc6de37b0, m=..., q=..., curop=..., result=..., fromDBDirectClient=false) at src/mongo/db/query/find.cpp:550
#20 0x0000000001427d0f in mongo::receivedQuery (txn=0x7fffc6de37b0, c=..., dbresponse=..., m=..., fromDBDirectClient=false) at src/mongo/db/instance.cpp:224
#21 0x0000000001428e21 in mongo::assembleResponse (txn=0x7fffc6de37b0, m=..., dbresponse=..., remote=..., fromDBDirectClient=false) at src/mongo/db/instance.cpp:394
#22 0x000000000112e176 in mongo::MyMessageHandler::process (this=0x306c198, m=..., port=0x3422410, le=0x1fa6b630) at src/mongo/db/db.cpp:195
#23 0x00000000018b74e5 in mongo::PortMessageServer::handleIncomingMsg (arg=0x83b2390) at src/mongo/util/net/message_server_port.cpp:234
#24 0x00007ffff7bc4182 in start_thread (arg=0x7fffc6de4700) at pthread_create.c:312
#25 0x00007ffff6cc4efd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111


Lock output:

LockerId 990 encountered an unexpected deadlock. The server will shut down in order to prevent becoming unresponsive. The cycle of deadlocked threads is:
 
Locker 931 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 932 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [945, 965, 1018, 991, 1020, 948, 985, 963, 984, 1008, 947, 1028, 953, 1016, 1023, 1001, 987, 1013, 940, 934, 1010, 1011, 941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 934 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [1010, 1011, 941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 937 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [969, 980, 951, 1022, 988, 1012, 968, 938, 1007, 995, 955, 1027, 1031, 954, 978, 1021, 1014, 960, 1005, 950, 1009, 1015, 998, 989, 939, 974, 932, 945, 965, 1018, 991, 1020, 948, 985, 963, 984, 1008, 947, 1028, 953, 1016, 1023, 1001, 987, 1013, 940, 934, 1010, 1011, 941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 938 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [1007, 995, 955, 1027, 1031, 954, 978, 1021, 1014, 960, 1005, 950, 1009, 1015, 998, 989, 939, 974, 932, 945, 965, 1018, 991, 1020, 948, 985, 963, 984, 1008, 947, 1028, 953, 1016, 1023, 1001, 987, 1013, 940, 934, 1010, 1011, 941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 939 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [974, 932, 945, 965, 1018, 991, 1020, 948, 985, 963, 984, 1008, 947, 1028, 953, 1016, 1023, 1001, 987, 1013, 940, 934, 1010, 1011, 941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 940 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [934, 1010, 1011, 941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 941 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 944 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 945 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [965, 1018, 991, 1020, 948, 985, 963, 984, 1008, 947, 1028, 953, 1016, 1023, 1001, 987, 1013, 940, 934, 1010, 1011, 941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 946 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 947 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [1028, 953, 1016, 1023, 1001, 987, 1013, 940, 934, 1010, 1011, 941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 948 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [985, 963, 984, 1008, 947, 1028, 953, 1016, 1023, 1001, 987, 1013, 940, 934, 1010, 1011, 941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 950 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [1009, 1015, 998, 989, 939, 974, 932, 945, 965, 1018, 991, 1020, 948, 985, 963, 984, 1008, 947, 1028, 953, 1016, 1023, 1001, 987, 1013, 940, 934, 1010, 1011, 941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 951 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [1022, 988, 1012, 968, 938, 1007, 995, 955, 1027, 1031, 954, 978, 1021, 1014, 960, 1005, 950, 1009, 1015, 998, 989, 939, 974, 932, 945, 965, 1018, 991, 1020, 948, 985, 963, 984, 1008, 947, 1028, 953, 1016, 1023, 1001, 987, 1013, 940, 934, 1010, 1011, 941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 953 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [1016, 1023, 1001, 987, 1013, 940, 934, 1010, 1011, 941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 954 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [978, 1021, 1014, 960, 1005, 950, 1009, 1015, 998, 989, 939, 974, 932, 945, 965, 1018, 991, 1020, 948, 985, 963, 984, 1008, 947, 1028, 953, 1016, 1023, 1001, 987, 1013, 940, 934, 1010, 1011, 941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 955 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [1027, 1031, 954, 978, 1021, 1014, 960, 1005, 950, 1009, 1015, 998, 989, 939, 974, 932, 945, 965, 1018, 991, 1020, 948, 985, 963, 984, 1008, 947, 1028, 953, 1016, 1023, 1001, 987, 1013, 940, 934, 1010, 1011, 941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 959 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 960 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [1005, 950, 1009, 1015, 998, 989, 939, 974, 932, 945, 965, 1018, 991, 1020, 948, 985, 963, 984, 1008, 947, 1028, 953, 1016, 1023, 1001, 987, 1013, 940, 934, 1010, 1011, 941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 963 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [984, 1008, 947, 1028, 953, 1016, 1023, 1001, 987, 1013, 940, 934, 1010, 1011, 941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 964 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [1024, 996, 997, 973, ]
 
Locker 965 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [1018, 991, 1020, 948, 985, 963, 984, 1008, 947, 1028, 953, 1016, 1023, 1001, 987, 1013, 940, 934, 1010, 1011, 941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 967 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 968 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [938, 1007, 995, 955, 1027, 1031, 954, 978, 1021, 1014, 960, 1005, 950, 1009, 1015, 998, 989, 939, 974, 932, 945, 965, 1018, 991, 1020, 948, 985, 963, 984, 1008, 947, 1028, 953, 1016, 1023, 1001, 987, 1013, 940, 934, 1010, 1011, 941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 969 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [980, 951, 1022, 988, 1012, 968, 938, 1007, 995, 955, 1027, 1031, 954, 978, 1021, 1014, 960, 1005, 950, 1009, 1015, 998, 989, 939, 974, 932, 945, 965, 1018, 991, 1020, 948, 985, 963, 984, 1008, 947, 1028, 953, 1016, 1023, 1001, 987, 1013, 940, 934, 1010, 1011, 941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 970 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 973 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by []
 
Locker 974 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [932, 945, 965, 1018, 991, 1020, 948, 985, 963, 984, 1008, 947, 1028, 953, 1016, 1023, 1001, 987, 1013, 940, 934, 1010, 1011, 941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 978 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [1021, 1014, 960, 1005, 950, 1009, 1015, 998, 989, 939, 974, 932, 945, 965, 1018, 991, 1020, 948, 985, 963, 984, 1008, 947, 1028, 953, 1016, 1023, 1001, 987, 1013, 940, 934, 1010, 1011, 941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 980 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [951, 1022, 988, 1012, 968, 938, 1007, 995, 955, 1027, 1031, 954, 978, 1021, 1014, 960, 1005, 950, 1009, 1015, 998, 989, 939, 974, 932, 945, 965, 1018, 991, 1020, 948, 985, 963, 984, 1008, 947, 1028, 953, 1016, 1023, 1001, 987, 1013, 940, 934, 1010, 1011, 941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 981 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 982 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 983 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 984 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [1008, 947, 1028, 953, 1016, 1023, 1001, 987, 1013, 940, 934, 1010, 1011, 941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 985 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [963, 984, 1008, 947, 1028, 953, 1016, 1023, 1001, 987, 1013, 940, 934, 1010, 1011, 941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 986 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 987 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [1013, 940, 934, 1010, 1011, 941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 988 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [1012, 968, 938, 1007, 995, 955, 1027, 1031, 954, 978, 1021, 1014, 960, 1005, 950, 1009, 1015, 998, 989, 939, 974, 932, 945, 965, 1018, 991, 1020, 948, 985, 963, 984, 1008, 947, 1028, 953, 1016, 1023, 1001, 987, 1013, 940, 934, 1010, 1011, 941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 989 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [939, 974, 932, 945, 965, 1018, 991, 1020, 948, 985, 963, 984, 1008, 947, 1028, 953, 1016, 1023, 1001, 987, 1013, 940, 934, 1010, 1011, 941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 990 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [1007, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, 976, 1017, 1000, 1022, 1019, 988, 969, 951, 937, 952, 1012, 980, 938, 968, 1002, 993, ]
 
Locker 991 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [1020, 948, 985, 963, 984, 1008, 947, 1028, 953, 1016, 1023, 1001, 987, 1013, 940, 934, 1010, 1011, 941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 993 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [1007, ]
 
Locker 994 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 995 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [955, 1027, 1031, 954, 978, 1021, 1014, 960, 1005, 950, 1009, 1015, 998, 989, 939, 974, 932, 945, 965, 1018, 991, 1020, 948, 985, 963, 984, 1008, 947, 1028, 953, 1016, 1023, 1001, 987, 1013, 940, 934, 1010, 1011, 941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 996 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [997, 973, ]
 
Locker 997 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [973, ]
 
Locker 998 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [989, 939, 974, 932, 945, 965, 1018, 991, 1020, 948, 985, 963, 984, 1008, 947, 1028, 953, 1016, 1023, 1001, 987, 1013, 940, 934, 1010, 1011, 941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 999 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [964, 1024, 996, 997, 973, ]
 
Locker 1001 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [987, 1013, 940, 934, 1010, 1011, 941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 1005 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [950, 1009, 1015, 998, 989, 939, 974, 932, 945, 965, 1018, 991, 1020, 948, 985, 963, 984, 1008, 947, 1028, 953, 1016, 1023, 1001, 987, 1013, 940, 934, 1010, 1011, 941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 1006 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 1007 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [995, 955, 1027, 1031, 954, 978, 1021, 1014, 960, 1005, 950, 1009, 1015, 998, 989, 939, 974, 932, 945, 965, 1018, 991, 1020, 948, 985, 963, 984, 1008, 947, 1028, 953, 1016, 1023, 1001, 987, 1013, 940, 934, 1010, 1011, 941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 1008 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [947, 1028, 953, 1016, 1023, 1001, 987, 1013, 940, 934, 1010, 1011, 941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 1009 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [1015, 998, 989, 939, 974, 932, 945, 965, 1018, 991, 1020, 948, 985, 963, 984, 1008, 947, 1028, 953, 1016, 1023, 1001, 987, 1013, 940, 934, 1010, 1011, 941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 1010 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [1011, 941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 1011 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 1012 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [968, 938, 1007, 995, 955, 1027, 1031, 954, 978, 1021, 1014, 960, 1005, 950, 1009, 1015, 998, 989, 939, 974, 932, 945, 965, 1018, 991, 1020, 948, 985, 963, 984, 1008, 947, 1028, 953, 1016, 1023, 1001, 987, 1013, 940, 934, 1010, 1011, 941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 1013 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [940, 934, 1010, 1011, 941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 1014 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [960, 1005, 950, 1009, 1015, 998, 989, 939, 974, 932, 945, 965, 1018, 991, 1020, 948, 985, 963, 984, 1008, 947, 1028, 953, 1016, 1023, 1001, 987, 1013, 940, 934, 1010, 1011, 941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 1015 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [998, 989, 939, 974, 932, 945, 965, 1018, 991, 1020, 948, 985, 963, 984, 1008, 947, 1028, 953, 1016, 1023, 1001, 987, 1013, 940, 934, 1010, 1011, 941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 1016 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [1023, 1001, 987, 1013, 940, 934, 1010, 1011, 941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 1018 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [991, 1020, 948, 985, 963, 984, 1008, 947, 1028, 953, 1016, 1023, 1001, 987, 1013, 940, 934, 1010, 1011, 941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 1020 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [948, 985, 963, 984, 1008, 947, 1028, 953, 1016, 1023, 1001, 987, 1013, 940, 934, 1010, 1011, 941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 1021 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [1014, 960, 1005, 950, 1009, 1015, 998, 989, 939, 974, 932, 945, 965, 1018, 991, 1020, 948, 985, 963, 984, 1008, 947, 1028, 953, 1016, 1023, 1001, 987, 1013, 940, 934, 1010, 1011, 941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 1022 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [988, 1012, 968, 938, 1007, 995, 955, 1027, 1031, 954, 978, 1021, 1014, 960, 1005, 950, 1009, 1015, 998, 989, 939, 974, 932, 945, 965, 1018, 991, 1020, 948, 985, 963, 984, 1008, 947, 1028, 953, 1016, 1023, 1001, 987, 1013, 940, 934, 1010, 1011, 941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 1023 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [1001, 987, 1013, 940, 934, 1010, 1011, 941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 1024 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [996, 997, 973, ]
 
Locker 1027 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [1031, 954, 978, 1021, 1014, 960, 1005, 950, 1009, 1015, 998, 989, 939, 974, 932, 945, 965, 1018, 991, 1020, 948, 985, 963, 984, 1008, 947, 1028, 953, 1016, 1023, 1001, 987, 1013, 940, 934, 1010, 1011, 941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 1028 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [53, 1016, 1023, 1001, 987, 1013, 940, 934, 1010, 1011, 941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]
 
Locker 1029 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [999, 964, 1024, 996, 997, 973, ]
 
Locker 1031 waits for resource {11422299607588131462: Collection, 2198927570733355654} held by [954, 978, 1021, 1014, 960, 1005, 950, 1009, 1015, 998, 989, 939, 974, 932, 945, 965, 1018, 991, 1020, 948, 985, 963, 984, 1008, 947, 1028, 953, 1016, 1023, 1001, 987, 1013, 940, 934, 1010, 1011, 941, 981, 967, 959, 970, 983, 931, 994, 946, 986, 990, 944, 1006, 982, 1029, 999, 964, 1024, 996, 997, 973, ]


Version: 0ae940b456ceafc3a0d473714c390f95d3a04823



 Comments   
Comment by Githook User [ 22/Dec/14 ]

Author:

{u'username': u'kaloianm', u'name': u'Kaloian Manassiev', u'email': u'kaloian.manassiev@mongodb.com'}

Message: SERVER-16624 Do not use deadlock detection in invariant checking

The problem is that our deadlock detector does not check for deadlock
stability currently. It is possible that as the graph traversal
progresses, locks which we were waiting on earlier get freed and suddenly
become waiting on us, becoming counted in the cycle.

The fix would be to add check for deadlock stability by re-iterating the
discovered cycle. Until this is fixed, we cannot use the deadlock
deterministically for assertions, only for retries (which is what we
already do).
Branch: master
https://github.com/mongodb/mongo/commit/92982e76c034f864f4499a0145ce7a75dc682812

Comment by Kaloian Manassiev [ 22/Dec/14 ]

The problem is that our deadlock detector does not check for deadlock stability currently. It is possible that as the graph traversal progresses, locks which we were waiting on earlier get freed and suddenly become waiting on us, becoming counted in the cycle.

The fix would be to add check for deadlock stability by re-iterating the discovered cycle. Until this is fixed, we cannot use the deadlock deterministically for assertions, only for retries (which is what we already do).

Filed SERVER-16636. As part of this fix, I will just remove the invariant.

Generated at Thu Feb 08 03:41:43 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.