[SERVER-77185] Bug with implementation of $lookup in SBE Created: 16/May/23  Updated: 29/Oct/23  Resolved: 31/May/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 6.0.6
Fix Version/s: 7.1.0-rc0, 6.0.7, 7.0.0-rc4

Type: Bug Priority: Major - P3
Reporter: Felipe Gasper Assignee: Ivan Fefer
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Problem/Incident
Assigned Teams:
Query Execution
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v7.0, v6.3, v6.0
Steps To Reproduce:

Unknown; this was a BF from mongosync. These can be hard to reproduce, unfortunately.

Sprint: QE 2023-05-29, QE 2023-06-12
Participants:

 Description   

https://parsley.mongodb.com/resmoke/881fbc6b87d5bed8f95527de86abd954/all?bookmarks=0,80180&shareLine=73970

ASSERT   4457000 [conn36] "Tripwire assertion","attr":{"error":{"code":6373900,"codeName":"Location6373900","errmsg":"bufferIdx not found in record store"},"location":"{fileName:\"src/mongo/db/exec/sbe/stages/hash_lookup.cpp\", line:432, functionName:\"operator()\"}"}

Sending to Query Execution per mihai.andrei@mongodb.com.



 Comments   
Comment by Githook User [ 12/Jun/23 ]

Author:

{'name': 'Ivan Fefer', 'email': 'ivan.fefer@mongodb.com', 'username': 'Fefer-Ivan'}

Message: SERVER-77185 Fix incorrect buffer indexing in case of spilling in hash lookup
Branch: v7.0
https://github.com/mongodb/mongo/commit/373210451367852927bc7ba2a9e4e8b2083e236d

Comment by Githook User [ 12/Jun/23 ]

Author:

{'name': 'Ivan Fefer', 'email': 'ivan.fefer@mongodb.com', 'username': 'Fefer-Ivan'}

Message: SERVER-77185 Fix incorrect buffer indexing in case of spilling in hash lookup
Branch: v6.0
https://github.com/mongodb/mongo/commit/82f5a7b1df90c2b320fe67daf49ceb127944aaaf

Comment by Githook User [ 30/May/23 ]

Author:

{'name': 'Ivan Fefer', 'email': 'ivan.fefer@mongodb.com', 'username': 'Fefer-Ivan'}

Message: SERVER-77185 Fix incorrect buffer indexing in case of spilling in hash lookup
Branch: master
https://github.com/mongodb/mongo/commit/9adbd8f24d7b232943d560b2969580cd6ca97893

Comment by Ivan Fefer [ 24/May/23 ]

The bug is in buffered value spilling condition: https://github.com/mongodb/mongo/blob/ecf5b575b936f7bbf6d9b3a581a18760e6450ab3/src/mongo/db/exec/sbe/stages/hash_lookup.cpp#L343

HashLookup has _buffer vector in-memory and spilled RecordStore for the rest of the values.
When building the hash table, we number buffered values from 0 to N.
We assume that first M numbers fit in memory and then we spill.
For example, here we want to get buffered value _bufferIt, we check if _bufferIt < _buffer.size() and look in memory. Otherwise, we go to the RecordStore:
https://github.com/mongodb/mongo/blob/ecf5b575b936f7bbf6d9b3a581a18760e6450ab3/src/mongo/db/exec/sbe/stages/hash_lookup.cpp#L426

When we start spilling, we should always spill. Otherwise there is this bug:
Imagine we have to buffer three values: small_0, big_1, small_2. Small_0 and small_2 fit together in RAM, but big have to be spilled.
Current code will do the following:
1. Add small_0 to _buffer.
2. See that small_0 + big_1 do not fit and spill big.
3. See that small_0 + small_2 fit and will add small_2 to _buffer.
We will end up with _buffer.size() == 2 with small_0 and small_2 and spilled RecordStore with RID 1 with big_1.
This is wrong.

There are actually two bugs: 
1. When looking for big_1, we will check that 1 < _buffer.size() and return _buffer[1] == small_2.
2. When looking for small_2 we will check that 2 >= _buffer.size() and look it up in RecordStore, but it won't be there.

 

I will fix the bug, add a regression test.

 

Comment by Ivan Fefer [ 24/May/23 ]

I was able to reproduce the issue.
Looks like a bug around missing/null lookup fields.

Generated at Thu Feb 08 06:34:46 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.