[SERVER-32441] 3.6 mongod crash on find with index and nested $and/$or Created: 21/Dec/17  Updated: 30/Oct/23  Resolved: 05/Jan/18

Status: Closed
Project: Core Server
Component/s: Index Maintenance
Affects Version/s: 3.6.0
Fix Version/s: 3.6.3, 3.7.1

Type: Bug Priority: Critical - P2
Reporter: Michael Jansen Assignee: David Storch
Resolution: Fixed Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File crash.js     File tc.json    
Issue Links:
Backports
Duplicate
is duplicated by SERVER-32552 Server crashes running tests from a C... Closed
Related
related to SERVER-32545 Improve PlanEnumerator logLevel 5 deb... Closed
is related to SERVER-13732 Predicates in top-level implicit AND ... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v3.6
Steps To Reproduce:

1. mongoimport -h 127.0.0.1 --port 27017 -d test -c tc --jsonArray --file ./tc.json
2. mongo test --eval 'db.tc.ensureIndex(

{ "visible" : 1, "pubVisible" : 1, "archive" : 1, "traded" : 1, "date" : -1 }

)'
3. mongo test ./crash.js

Sprint: Query 2018-01-01, Query 2018-01-15
Participants:
Case:

 Description   

When executing a find with $and $or which uses an index command mongod crashes. with Invariant failure childDestinations->second.size() == 1 src/mongo/db/query/index_tag.cpp 180

2017-12-21T17:31:18.162+0000 F -        [conn16] Invariant failure childDestinations->second.size() == 1 src/mongo/db/query/index_tag.cpp 180
2017-12-21T17:31:18.162+0000 F -        [conn16]
 
***aborting after invariant() failure
 
 
2017-12-21T17:31:18.185+0000 F -        [conn16] Got signal: 6 (Aborted).
 
 0x55e1cf4cbbf1 0x55e1cf4cae09 0x55e1cf4cb2ed 0x7f3401f885a0 0x7f3401bea1f7 0x7f3401beb8e8 0x55e1cdc5d13c 0x55e1cf0d1046 0x55e1cf0d13bf 0x55e1cf0d1d81 0x55e1cf10a3dc 0x55e1ce2b4a25 0x55e1ce2b990e 0x55e1ce2b9b7b 0x55e1cdf38495 0x55e1cef6d5b6 0x55e1cef684cf 0x55e1cdee4d1e 0x55e1cdee6dd4 0x55e1cdee7b27 0x55e1cdef414a 0x55e1cdeefce7 0x55e1cdef2f31 0x55e1cee20a32 0x55e1cdeeeb50 0x55e1cdef1092 0x55e1cdef198b 0x55e1cdeefd6d 0x55e1cdef2f31 0x55e1cee20f95 0x55e1cf385674 0x7f3401f80de5 0x7f3401cad30d
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"55E1CD2D2000","o":"21F9BF1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55E1CD2D2000","o":"21F8E09"},{"b":"55E1CD2D2000","o":"21F92ED"},{"b":"7F3401F79000","o":"F5A0"},{"b":"7F3401BB5000","o":"351F7","s":"gsignal"},{"b":"7F3401BB5000","o":"368E8","s":"abort"},{"b":"55E1CD2D2000","o":"98B13C","s":"_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j"},{"b":"55E1CD2D2000","o":"1DFF046"},{"b":"55E1CD2D2000","o":"1DFF3BF"},{"b":"55E1CD2D2000","o":"1DFFD81","s":"_ZN5mongo24prepareForAccessPlanningEPNS_15MatchExpressionE"},{"b":"55E1CD2D2000","o":"1E383DC","s":"_ZN5mongo12QueryPlanner4planERKNS_14CanonicalQueryERKNS_18QueryPlannerParamsEPSt6vectorIPNS_13QuerySolutionESaIS9_EE"},{"b":"55E1CD2D2000","o":"FE2A25"},{"b":"55E1CD2D2000","o":"FE790E","s":"_ZN5mongo11getExecutorEPNS_16OperationContextEPNS_10CollectionESt10unique_ptrINS_14CanonicalQueryESt14default_deleteIS5_EENS_12PlanExecutor11YieldPolicyEm"},{"b":"55E1CD2D2000","o":"FE7B7B","s":"_ZN5mongo15getExecutorFindEPNS_16OperationContextEPNS_10CollectionERKNS_15NamespaceStringESt10unique_ptrINS_14CanonicalQueryESt14default_deleteIS8_EENS_12PlanExecutor11YieldPolicyEm"},{"b":"55E1CD2D2000","o":"C66495"},{"b":"55E1CD2D2000","o":"1C9B5B6","s":"_ZN5mongo12BasicCommand11enhancedRunEPNS_16OperationContextERKNS_12OpMsgRequestERNS_14BSONObjBuilderE"},{"b":"55E1CD2D2000","o":"1C964CF","s":"_ZN5mongo7Command9publicRunEPNS_16OperationContextERKNS_12OpMsgRequestERNS_14BSONObjBuilderE"},{"b":"55E1CD2D2000","o":"C12D1E"},{"b":"55E1CD2D2000","o":"C14DD4"},{"b":"55E1CD2D2000","o":"C15B27","s":"_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE"},{"b":"55E1CD2D2000","o":"C2214A","s":"_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE"},{"b":"55E1CD2D2000","o":"C1DCE7","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"55E1CD2D2000","o":"C20F31"},{"b":"55E1CD2D2000","o":"1B4EA32","s":"_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsE"},{"b":"55E1CD2D2000","o":"C1CB50","s":"_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS0_9OwnershipE"},{"b":"55E1CD2D2000","o":"C1F092","s":"_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE"},{"b":"55E1CD2D2000","o":"C1F98B","s":"_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE"},{"b":"55E1CD2D2000","o":"C1DD6D","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"55E1CD2D2000","o":"C20F31"},{"b":"55E1CD2D2000","o":"1B4EF95"},{"b":"55E1CD2D2000","o":"20B3674"},{"b":"7F3401F79000","o":"7DE5"},{"b":"7F3401BB5000","o":"F830D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.6.0", "gitVersion" : "a57d8e71e6998a2d0afde7edc11bd23e5661c915", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.9.62-21.56.amzn1.x86_64", "version" : "#1 SMP Thu Nov 16 05:37:08 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "b" : "55E1CD2D2000", "elfType" : 3, "buildId" : "53CA7A16D73321BD64A98A032B3684F2AF56007D" }, { "b" : "7FFFF09B0000", "elfType" : 3, "buildId" : "604EB37C4DE5D4A1FD185A9AE5E81F3734734D93" }, { "b" : "7F3403188000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "0BED68388AE751ACB77DA9FBB6B63CAA5E4622A9" }, { "b" : "7F3402F17000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "FAADF98D6634B218EF976D389BCEA90C97FB8CD2" }, { "b" : "7F3402AB9000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "33F4F81F7FB0C7609BA4B12108C84505AB12D46F" }, { "b" : "7F34028B1000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "BA23B2F2723B7C9BF5FFE9A8AF9E50219C03EB0E" }, { "b" : "7F34026AD000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "2CC91EC12043BC0357CD502BBB4D5A4940BFFB1B" }, { "b" : "7F34023AB000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "31404F7BCB6B5A880B25E0007EA5C73BBEC87E16" }, { "b" : "7F3402195000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "AC58019512A5359B077D2610DEA4AD6CF14CAC53" }, { "b" : "7F3401F79000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "4136F256787CE855F91B70793154EBCFCE047CFA" }, { "b" : "7F3401BB5000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "FB411B5AB3944BD21963101275C0F9EB3F943BDC" }, { "b" : "7F34033A2000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "CC0DF41E2210C483EFF9C95EDB5F2AE014752C70" }, { "b" : "7F3401968000", "path" : "/usr/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "6FBBD34B86296FDF883FE5122017EC5CD3F98ED7" }, { "b" : "7F3401680000", "path" : "/usr/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "76429E6FD408BBB675798D6458F2735383710D0B" }, { "b" : "7F340147D000", "path" : "/usr/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "5C01209C5AE1B1714F19B07EB58F2A1274B69DC8" }, { "b" : "7F340124A000", "path" : "/usr/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "5B2A76F1EF91EDAA0494BE680CADAFE6489326E1" }, { "b" : "7F3401034000", "path": "/lib64/libz.so.1", "elfType" : 3, "buildId" : "89C6AF118B6B4FB6A73AE1813E2C8BDD722956D1" }, { "b" : "7F3400E26000", "path" : "/usr/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "3ACB59488C6D8DE0A1F4F1B0C290A570D9E42F3D" }, { "b" : "7F3400C23000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "37A58210FA50C91E09387765408A92909468D25B" }, { "b" : "7F3400A02000", "path" : "/usr/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "F5054DC94443326819FBF3065CFDF5E4726F57EE" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55e1cf4cbbf1]
 mongod(+0x21F8E09) [0x55e1cf4cae09]
 mongod(+0x21F92ED) [0x55e1cf4cb2ed]
 libpthread.so.0(+0xF5A0) [0x7f3401f885a0]
 libc.so.6(gsignal+0x37) [0x7f3401bea1f7]
 libc.so.6(abort+0x148) [0x7f3401beb8e8]
 mongod(_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j+0x0) [0x55e1cdc5d13c]
 mongod(+0x1DFF046) [0x55e1cf0d1046]
 mongod(+0x1DFF3BF) [0x55e1cf0d13bf]
 mongod(_ZN5mongo24prepareForAccessPlanningEPNS_15MatchExpressionE+0x11) [0x55e1cf0d1d81]
 mongod(_ZN5mongo12QueryPlanner4planERKNS_14CanonicalQueryERKNS_18QueryPlannerParamsEPSt6vectorIPNS_13QuerySolutionESaIS9_EE+0x1EEC) [0x55e1cf10a3dc]
 mongod(+0xFE2A25) [0x55e1ce2b4a25]
 mongod(_ZN5mongo11getExecutorEPNS_16OperationContextEPNS_10CollectionESt10unique_ptrINS_14CanonicalQueryESt14default_deleteIS5_EENS_12PlanExecutor11YieldPolicyEm+0x7E) [0x55e1ce2b990e]
 mongod(_ZN5mongo15getExecutorFindEPNS_16OperationContextEPNS_10CollectionERKNS_15NamespaceStringESt10unique_ptrINS_14CanonicalQueryESt14default_deleteIS8_EENS_12PlanExecutor11YieldPolicyEm+0x9B) [0x55e1ce2b9b7b]
 mongod(+0xC66495) [0x55e1cdf38495]
 mongod(_ZN5mongo12BasicCommand11enhancedRunEPNS_16OperationContextERKNS_12OpMsgRequestERNS_14BSONObjBuilderE+0x76) [0x55e1cef6d5b6]
 mongod(_ZN5mongo7Command9publicRunEPNS_16OperationContextERKNS_12OpMsgRequestERNS_14BSONObjBuilderE+0x1F) [0x55e1cef684cf]
 mongod(+0xC12D1E) [0x55e1cdee4d1e]
 mongod(+0xC14DD4) [0x55e1cdee6dd4]
 mongod(_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE+0x2B7) [0x55e1cdee7b27]
 mongod(_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE+0xBA) [0x55e1cdef414a]
 mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x97) [0x55e1cdeefce7]
 mongod(+0xC20F31) [0x55e1cdef2f31]
 mongod(_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsE+0x1A2) [0x55e1cee20a32]
 mongod(_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS0_9OwnershipE+0x150) [0x55e1cdeeeb50]
 mongod(_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE+0xAF2) [0x55e1cdef1092]
 mongod(_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE+0x23B) [0x55e1cdef198b]
 mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x11D) [0x55e1cdeefd6d]
 mongod(+0xC20F31) [0x55e1cdef2f31]
 mongod(+0x1B4EF95) [0x55e1cee20f95]
 mongod(+0x20B3674) [0x55e1cf385674]
 libpthread.so.0(+0x7DE5) [0x7f3401f80de5]
 libc.so.6(clone+0x6D) [0x7f3401cad30d]
-----  END BACKTRACE  -----



 Comments   
Comment by Douglas Reith [ 07/Feb/18 ]

I think Atlas users can't use

db.adminCommand({setParameter: 1, internalQueryPlannerEnableIndexIntersection: false});

 "errmsg" : "not authorized on admin to execute command { setParameter: 1.0, internalQueryPlannerEnableIndexIntersection: false, $clusterTime: { clusterTime: Timestamp(1517975630, 1), signature: { hash: BinData(0, 6CB3FD8753C669EA9EBE9F59825FD6920D0476FB), keyId: 6516678557379330049 } }, $db: \"admin\" }",

perhaps everyone on 3.6.2 on Atlas should have this set by default for now?

Comment by Githook User [ 05/Jan/18 ]

Author:

{'name': 'David Storch', 'username': 'dstorch', 'email': 'david.storch@10gen.com'}

Message: SERVER-32441 Fix invariant failure in $or planning.

This fix removes the planner's ability to generate
hash-based index intersection plans between two OR nodes---
an obscure code path that was disabled by default.

(cherry picked from commit 87c1731e3e077ca9e8b750ebe4b3fef534f1cbe6)
Branch: v3.6
https://github.com/mongodb/mongo/commit/cec131017b16e12b993ef8c90733feed4081fe0d

Comment by Githook User [ 05/Jan/18 ]

Author:

{'name': 'David Storch', 'username': 'dstorch', 'email': 'david.storch@10gen.com'}

Message: SERVER-32441 Fix invariant failure in $or planning.

This fix removes the planner's ability to generate
hash-based index intersection plans between two OR nodes---
an obscure code path that was disabled by default.
Branch: master
https://github.com/mongodb/mongo/commit/87c1731e3e077ca9e8b750ebe4b3fef534f1cbe6

Comment by Michael Jansen [ 21/Dec/17 ]

works! much appreciated. thank you!

Comment by David Storch [ 21/Dec/17 ]

The problem has to do with a bad interaction between the $or access planning improvements made in SERVER-13732 and index intersection. I still need to dig in further in order to determine how we can fix this, and exactly what kinds of queries are affected. However, there is an easy workaround: disabling index intersection. Most applications can safely disable index intersection with the setParameter command:

db.adminCommand({setParameter: 1, internalQueryPlannerEnableIndexIntersection: false});

This should prevent the issue until we can provide a fix in the 3.6 branch.

Comment by Michael Jansen [ 21/Dec/17 ]

thank you for looking into it. when you have the fix. I will use it immediately. thank you!

Comment by David Storch [ 21/Dec/17 ]

Hi michaeljansen, thanks for reporting this issue. I can reproduce using the steps provided, and am looking into the root cause. I also confirmed that this does not reproduce against 3.4, so it is a regression introduced in 3.6. It looks related to the work done under SERVER-13732 to improve access path selection for contained $or queries.

Generated at Thu Feb 08 04:30:15 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.