Multiplanner's totalWorks counter can overflow and trigger a tassert

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Fixed
    • Priority: Major - P3
    • 9.0.0-rc1, 8.3.8
    • Affects Version/s: None
    • Component/s: None
    • None
    • Query Optimization
    • Fully Compatible
    • ALL
    • v8.3
    • Hide

       

      The following unit test run in multi_plan_test.cpp proves that this is possible given enough works being performed.

      DEATH_TEST_REGEX_F(QueryStageMultiPlanDeathTest,                   PickBestPlanAssertsWhenTotalWorksOverflows,                   "Tripwire assertion.*11484502") {    const int N = 100;    for (int i = 0; i < N; ++i) {        insert(BSON("foo" << (i % 10)));    }
    addIndex(BSON("foo" << 1));
    auto coll = getCollection();    // Plan 0: IXScan over foo == 7.    unique_ptr<WorkingSet> sharedWs(new WorkingSet());    unique_ptr<PlanStage> ixScanRoot = getIxScanPlan(expCtx.get(), coll, sharedWs.get(), 7);
    // Plan 1: CollScan with matcher.    BSONObj filterObj = BSON("foo" << 7);    unique_ptr<MatchExpression> filter = makeMatchExpressionFromFilter(expCtx.get(), filterObj);    unique_ptr<PlanStage> collScanRoot =        getCollScanPlan(expCtx.get(), coll, sharedWs.get(), filter.get());
    auto cq = makeCanonicalQuery(opCtx.get(), nss, filterObj);
    unique_ptr<MultiPlanStage> mps = std::make_unique<MultiPlanStage>(        expCtx.get(), coll, cq.get(), plan_cache_util::ClassicPlanCacheWriter{opCtx.get(), coll});    mps->addPlan(createQuerySolution(), std::move(ixScanRoot), sharedWs.get());    mps->addPlan(createQuerySolution(), std::move(collScanRoot), sharedWs.get());
    auto* mpsPtr = mps.get();    auto planYieldPolicy = makeClassicYieldPolicy(opCtx.get(),                                                  nss,                                                  static_cast<PlanStage*>(mpsPtr),                                                  PlanYieldPolicy::YieldPolicy::INTERRUPT_ONLY);
    // Simulate the effect of prior trial work that legitimately accumulated close to INT_MAX    // works (as would happen scanning a huge, poorly-indexed collection).    auto* mpStats = const_cast<MultiPlanStats*>(        dynamic_cast<const MultiPlanStats*>(mpsPtr->getSpecificStats()));    mpStats->totalWorks = std::numeric_limits<int>::max() - 3;
    // An ordinary, small trial (mirrors MPSCanRunTrialsInBatches) that does not hit EOF or the    // results limit within 7 works per plan, so it runs the full budget: totalWorks increases by    // maxNumWorksPerPlan * numCandidatePlans == 7 * 2 == 14. Added to the pre-seeded value, this    // overflows the 32-bit counter.    trial_period::TrialPhaseConfig trialsConfig{.maxNumWorksPerPlan = 7, .targetNumResults = 100};    ASSERT_OK(mpsPtr->runTrials(planYieldPolicy.get(), trialsConfig));
    // totalWorks has wrapped around to a non-positive value despite real trial work having run.    ASSERT_LTE(mpStats->totalWorks, 0);
    // pickBestPlan() incorrectly tassert-fails, believing no trials were ever run.    ASSERT_THROWS_CODE(mpsPtr->pickBestPlan(), DBException, 11484502);}

       

      Show
        The following unit test run in multi_plan_test.cpp proves that this is possible given enough works being performed. DEATH_TEST_REGEX_F(QueryStageMultiPlanDeathTest, PickBestPlanAssertsWhenTotalWorksOverflows, "Tripwire assertion.*11484502" ) { const int N = 100; for ( int i = 0; i < N; ++i) { insert(BSON( "foo" << (i % 10))); } addIndex(BSON( "foo" << 1)); auto coll = getCollection(); // Plan 0: IXScan over foo == 7. unique_ptr<WorkingSet> sharedWs( new WorkingSet()); unique_ptr<PlanStage> ixScanRoot = getIxScanPlan(expCtx.get(), coll, sharedWs.get(), 7); // Plan 1: CollScan with matcher. BSONObj filterObj = BSON( "foo" << 7); unique_ptr<MatchExpression> filter = makeMatchExpressionFromFilter(expCtx.get(), filterObj); unique_ptr<PlanStage> collScanRoot = getCollScanPlan(expCtx.get(), coll, sharedWs.get(), filter.get()); auto cq = makeCanonicalQuery(opCtx.get(), nss, filterObj); unique_ptr<MultiPlanStage> mps = std::make_unique<MultiPlanStage>( expCtx.get(), coll, cq.get(), plan_cache_util::ClassicPlanCacheWriter{opCtx.get(), coll}); mps->addPlan(createQuerySolution(), std::move(ixScanRoot), sharedWs.get()); mps->addPlan(createQuerySolution(), std::move(collScanRoot), sharedWs.get()); auto* mpsPtr = mps.get(); auto planYieldPolicy = makeClassicYieldPolicy(opCtx.get(), nss, static_cast<PlanStage*>(mpsPtr), PlanYieldPolicy::YieldPolicy::INTERRUPT_ONLY); // Simulate the effect of prior trial work that legitimately accumulated close to INT_MAX // works (as would happen scanning a huge, poorly-indexed collection). auto* mpStats = const_cast<MultiPlanStats*>( dynamic_cast< const MultiPlanStats*>(mpsPtr->getSpecificStats())); mpStats->totalWorks = std::numeric_limits< int >::max() - 3; // An ordinary, small trial (mirrors MPSCanRunTrialsInBatches) that does not hit EOF or the // results limit within 7 works per plan, so it runs the full budget: totalWorks increases by // maxNumWorksPerPlan * numCandidatePlans == 7 * 2 == 14. Added to the pre-seeded value, this // overflows the 32-bit counter. trial_period::TrialPhaseConfig trialsConfig{.maxNumWorksPerPlan = 7, .targetNumResults = 100}; ASSERT_OK(mpsPtr->runTrials(planYieldPolicy.get(), trialsConfig)); // totalWorks has wrapped around to a non-positive value despite real trial work having run. ASSERT_LTE(mpStats->totalWorks, 0); // pickBestPlan() incorrectly tassert-fails, believing no trials were ever run. ASSERT_THROWS_CODE(mpsPtr->pickBestPlan(), DBException, 11484502);}  
    • 200
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      If multiplanning runs for enough works then it can overflow its totalWorks counter into a negative value that causes a tassert to fail.

      MultiPlanStats.totalWorks is defined as a 32 bit signed int and used to track the total number of works performed across all plans during multiplanning.

      https://github.com/10gen/mongo/blob/58a67c40bb8bbffd91566b8635b04ab0ee558065/src/mongo/db/exec/plan_stats.h#L771 

      If the total number of works over all the plans exceeds INT_MAX (around 2.1 billion, which seems possible for a very large collection with lots of candidate plans) then _specificStats.totalWorks overflows. pickBestPlan() has a tassert that checks _specificStats.totalWorks > 0, which will fail in this scenario.

      https://github.com/10gen/mongo/blob/309569b326aa82b34c7089e3f104124f5a2a3b0a/src/mongo/db/exec/classic/multi_plan.cpp#L441 

       

      One solution would be to define MultiPlanStats.totalWorks as a 64-bit unsigned integer, which increases the maximum value to 264. There’s no reason this counter should be signed, and this max value is large enough that we can be sure we’ll never overflow since this counter only lasts for the multiplanner lifetime of one query.

      Also add the value of totalWorks to the tassert log here so that if the diagnosis of AF-18444 turns out to be incorrect and the same bug reoccurs then we'll have that information.

      Consider backporting this into 8.3 as well for the sake of the logging information and future AFs.

            Assignee:
            Kartal Kaan Bozdogan
            Reporter:
            Natalie Hill
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: