[SERVER-56869] [SBE][ASAN jscore] heap-use-after-free during computed_projections.js Created: 11/May/21  Updated: 29/Oct/23  Resolved: 17/May/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 5.0.0-rc0

Type: Bug Priority: Major - P3
Reporter: Kyle Suarez Assignee: Ian Boros
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File jscore_asan.txt    
Issue Links:
Depends
is depended on by SERVER-52799 Make sbe the default execution engine... Closed
Related
is related to SERVER-56900 SBE sort stage may violate SBE copyOr... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Sprint: Query Execution 2021-05-17, Query Execution 2021-05-31
Participants:

 Description   

During this patch build to enable SBE by default, we hit a heap-use-after-free during the "ludicrous nesting torture test" in jstests/core/computed_projections.js:

[j3] =================================================================
[j3] ==120948==ERROR: AddressSanitizer: heap-use-after-free on address 0x6060002fb2f4 at pc 0x7fa517900524 bp 0x7fa4cf0bb540 sp 0x7fa4cf0bb538
[j3] READ of size 4 at 0x6060002fb2f4 thread T37 (conn6)
[j3]     #0 0x7fa517900523 in mongo::DataType::Handler<unsigned int, void>::unsafeLoad(unsigned int*, char const*, unsigned long*) /data/mci/233c861a4a4a55a5148357cf11c28002/src/src/mongo/base/data_type.h:70:17
[j3]     #1 0x7fa517900523 in void mongo::DataType::unsafeLoad<unsigned int>(unsigned int*, char const*, unsigned long*) /data/mci/233c861a4a4a55a5148357cf11c28002/src/src/mongo/base/data_type.h:153
[j3]     #2 0x7fa517900523 in mongo::DataType::Handler<mongo::LittleEndian<unsigned int>, void>::unsafeLoad(mongo::LittleEndian<unsigned int>*, char const*, unsigned long*) /data/mci/233c861a4a4a55a5148357cf11c28002/src/src/mongo/base/data_type_endian.h:90
[j3]     #3 0x7fa517900523 in void mongo::DataType::unsafeLoad<mongo::LittleEndian<unsigned int> >(mongo::LittleEndian<unsigned int>*, char const*, unsigned long*) /data/mci/233c861a4a4a55a5148357cf11c28002/src/src/mongo/base/data_type.h:153
[j3]     #4 0x7fa517900523 in mongo::ConstDataView const& mongo::ConstDataView::readInto<mongo::LittleEndian<unsigned int> >(mongo::LittleEndian<unsigned int>*, long) const /data/mci/233c861a4a4a55a5148357cf11c28002/src/src/mongo/base/data_view.h:53
[j3]     #5 0x7fa50502067c in mongo::LittleEndian<unsigned int> mongo::ConstDataView::read<mongo::LittleEndian<unsigned int> >(long) const /data/mci/5b39450893c745aa14863a3d17537c10/src/src/mongo/base/data_view.h:62:9
[j3]     #6 0x7fa50502067c in mongo::sbe::value::copyValue(mongo::sbe::value::TypeTags, unsigned long) /data/mci/5b39450893c745aa14863a3d17537c10/src/src/mongo/db/exec/sbe/values/value.h:1068
[j3]     #7 0x7fa50333fffe in mongo::sbe::value::OwnedValueAccessor::makeOwned() /data/mci/72ab04a1a9ea0fb5cd8990198a70f1c8/src/src/mongo/db/exec/sbe/values/slot.h:194:32
[j3]     #8 0x7fa502d76717 in mongo::sbe::ProjectStage::doSaveState() /data/mci/5b39450893c745aa14863a3d17537c10/src/src/mongo/db/exec/sbe/stages/project.cpp:155:18
[j3]     #9 0x7fa504e19495 in mongo::sbe::CanChangeState<mongo::sbe::PlanStage>::saveState() /data/mci/801c90706300d15577b3a7cfdb920ef2/src/src/mongo/db/exec/sbe/stages/stages.h:132:16
[j3]     #10 0x7fa504e195a9 in mongo::sbe::CanChangeState<mongo::sbe::PlanStage>::saveState() /data/mci/801c90706300d15577b3a7cfdb920ef2/src/src/mongo/db/exec/sbe/stages/stages.h:137:20
[j3]     #11 0x7fa504e195a9 in mongo::sbe::CanChangeState<mongo::sbe::PlanStage>::saveState() /data/mci/801c90706300d15577b3a7cfdb920ef2/src/src/mongo/db/exec/sbe/stages/stages.h:137:20
[j3]     #12 0x7fa504e195a9 in mongo::sbe::CanChangeState<mongo::sbe::PlanStage>::saveState() /data/mci/801c90706300d15577b3a7cfdb920ef2/src/src/mongo/db/exec/sbe/stages/stages.h:137:20
[j3]     #13 0x7fa504e195a9 in mongo::sbe::CanChangeState<mongo::sbe::PlanStage>::saveState() /data/mci/801c90706300d15577b3a7cfdb920ef2/src/src/mongo/db/exec/sbe/stages/stages.h:137:20
[j3]     #14 0x7fa504e195a9 in mongo::sbe::CanChangeState<mongo::sbe::PlanStage>::saveState() /data/mci/801c90706300d15577b3a7cfdb920ef2/src/src/mongo/db/exec/sbe/stages/stages.h:137:20
[j3]     #15 0x7fa504e195a9 in mongo::sbe::CanChangeState<mongo::sbe::PlanStage>::saveState() /data/mci/801c90706300d15577b3a7cfdb920ef2/src/src/mongo/db/exec/sbe/stages/stages.h:137:20
[j3]     #16 0x7fa504e195a9 in mongo::sbe::CanChangeState<mongo::sbe::PlanStage>::saveState() /data/mci/801c90706300d15577b3a7cfdb920ef2/src/src/mongo/db/exec/sbe/stages/stages.h:137:20
[j3]     #17 0x7fa504e195a9 in mongo::sbe::CanChangeState<mongo::sbe::PlanStage>::saveState() /data/mci/801c90706300d15577b3a7cfdb920ef2/src/src/mongo/db/exec/sbe/stages/stages.h:137:20
[j3]     #18 0x7fa504e195a9 in mongo::sbe::CanChangeState<mongo::sbe::PlanStage>::saveState() /data/mci/801c90706300d15577b3a7cfdb920ef2/src/src/mongo/db/exec/sbe/stages/stages.h:137:20
[j3]     #19 0x7fa504e195a9 in mongo::sbe::CanChangeState<mongo::sbe::PlanStage>::saveState() /data/mci/801c90706300d15577b3a7cfdb920ef2/src/src/mongo/db/exec/sbe/stages/stages.h:137:20
[j3]     #20 0x7fa504e195a9 in mongo::sbe::CanChangeState<mongo::sbe::PlanStage>::saveState() /data/mci/801c90706300d15577b3a7cfdb920ef2/src/src/mongo/db/exec/sbe/stages/stages.h:137:20
[j3]     #21 0x7fa504e195a9 in mongo::sbe::CanChangeState<mongo::sbe::PlanStage>::saveState() /data/mci/801c90706300d15577b3a7cfdb920ef2/src/src/mongo/db/exec/sbe/stages/stages.h:137:20
[j3]     #22 0x7fa504e195a9 in mongo::sbe::CanChangeState<mongo::sbe::PlanStage>::saveState() /data/mci/801c90706300d15577b3a7cfdb920ef2/src/src/mongo/db/exec/sbe/stages/stages.h:137:20
[j3]     #23 0x7fa504e195a9 in mongo::sbe::CanChangeState<mongo::sbe::PlanStage>::saveState() /data/mci/801c90706300d15577b3a7cfdb920ef2/src/src/mongo/db/exec/sbe/stages/stages.h:137:20
[j3]     #24 0x7fa504e195a9 in mongo::sbe::CanChangeState<mongo::sbe::PlanStage>::saveState() /data/mci/801c90706300d15577b3a7cfdb920ef2/src/src/mongo/db/exec/sbe/stages/stages.h:137:20
[j3]     #25 0x7fa504e61e74 in mongo::PlanYieldPolicySBE::saveState(mongo::OperationContext*) /data/mci/60e523f5a256ca21c1b14b893792fad5/src/src/mongo/db/query/plan_yield_policy_sbe.cpp:37:15
[j3]     #26 0x7fa5028a424b in mongo::PlanYieldPolicy::yieldOrInterrupt(mongo::OperationContext*, std::function<void ()>) /data/mci/72ab04a1a9ea0fb5cd8990198a70f1c8/src/src/mongo/db/query/plan_yield_policy.cpp:96:17
[j3]     #27 0x7fa503341dbc in mongo::sbe::CanInterrupt::checkForInterrupt(mongo::OperationContext*) /data/mci/72ab04a1a9ea0fb5cd8990198a70f1c8/src/src/mongo/db/exec/sbe/stages/stages.h:335:13
[j3]     #28 0x7fa50336f108 in mongo::sbe::ScanStage::getNext() /data/mci/72ab04a1a9ea0fb5cd8990198a70f1c8/src/src/mongo/db/exec/sbe/stages/scan.cpp:286:5
[j3]     #29 0x7fa502e04c50 in mongo::sbe::TraverseStage::getNext() /data/mci/5b39450893c745aa14863a3d17537c10/src/src/mongo/db/exec/sbe/stages/traverse.cpp:147:32
[j3]     #30 0x7fa504e15e2e in mongo::fetchNext(mongo::sbe::PlanStage*, mongo::sbe::value::SlotAccessor*, mongo::sbe::value::SlotAccessor*, mongo::BSONObj*, mongo::RecordId*, bool) /data/mci/801c90706300d15577b3a7cfdb920ef2/src/src/mongo/db/query/plan_executor_sbe.cpp:327:24
[j3]     #31 0x7fa504e13c1d in mongo::PlanExecutorSBE::getNext(mongo::BSONObj*, mongo::RecordId*) /data/mci/801c90706300d15577b3a7cfdb920ef2/src/src/mongo/db/query/plan_executor_sbe.cpp:235:13

The full trace has been omitted due to length but the full logs with ASAN output are attached to this ticket.



 Comments   
Comment by Githook User [ 17/May/21 ]

Author:

{'name': 'Ian Boros', 'email': 'ian.boros@mongodb.com', 'username': 'puppyofkosh'}

Message: SERVER-56869 Various fixes for use-after-free bugs in SBE
Branch: master
https://github.com/mongodb/mongo/commit/fcdc972cd6f64bfe710161fb3a0ffc4a5ca329d1

Comment by Ian Boros [ 12/May/21 ]

I think there are some serious issues with how yielding and SBE interact.

Consider the pseudo-SBE-plan:

Traverse [s1]
From:
    Filter
    Scan (recordSlot=s1)
 In:
    Traverse
    From:
        project [s2 = getField(s1, "xyz")]
    In:
        <Whatever>

Imagine the following happens:
(1) The scan returns a record, the filter passes and the inner side is run. The project stage s2 is now pointing to field "xyz" in the object.
(2) We call getNext() again. This time the filter does not pass for the first record that the scan produces. At this point s2 is pointing to garbage, since each call to next() on the SortedInterfaceCursor invalidates the unowned BSON returned from the previous call.
(3) The filter runs getNext() on the scan stage yet again. This time, the scan decides that it is time to yield. As part of saveState(), the project stage tries to copy the data stored in s2. But that data is already gone and is now garbage.

Martin and I are discussing potential work-arounds...

Comment by Ian Boros [ 12/May/21 ]

CC martin.neupauer – Looks like this is from some of the recent code to deal with yielding in SBE.

Generated at Thu Feb 08 05:40:25 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.