Test resuming a primary-driven index build with >16 MiB of spilled range metadata

XMLWordPrintableJSON

    • Type: Task
    • Resolution: Gone away
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Storage Execution
    • Storage Execution 2026-08-03
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Overview

      Add a test that exercises the resumable primary-driven index build (PDIB) code path when the sorter has spilled enough ranges that the ranges array in IndexStateInfo exceeds the 16 MiB BSON document size limit.

      Background

      The IndexStateInfo IDL struct (src/mongo/db/index_builds/resumable_index_builds.idl) contains a ranges field typed as array. This array is serialized to BSON when persisting resumable index build state to disk. Each SorterRange record (src/mongo/db/sorter/sorter.idl) is small, but if the sorter is configured with a very low memory budget and the collection is large, thousands of small spills can accumulate causing the total BSON size of the ranges array to grow large — potentially exceeding the 16 MiB BSON document limit.

      Hybrid index builds already have a "large" variant (jstests/noPassthrough/index_builds/resumable_index_build_collection_scan_phase_large.js) that exercises this scenario. Primary-driven index builds have their own resume path (introduced via featureFlagResumablePrimaryDrivenIndexBuilds) and use the same IndexStateInfo serialization, but currently have no coverage for the >16 MiB range-metadata case.

      Scope of Work

      1. New Test File

      File to add:
      * jstests/noPassthrough/index_builds/resumable_primary_driven_index_build_scan_phase_large.js
      ** Uses PrimaryDrivenResumableIndexBuildTest from jstests/noPassthrough/libs/index_builds/primary_driven.js
      ** Sets maxIndexBuildMemoryUsageMegabytes to a very small value (e.g. 1 MB) and inserts enough large documents to drive the spilled-range count high enough that the cumulative ranges BSON exceeds 16 MiB
      ** Verifies that the index build resumes successfully across a primary step-up

      2. Any Required Source Changes

      * If the serialization path does not already handle oversized ranges arrays gracefully (e.g. by splitting or truncating), identify and fix the bug first, then add the test to prevent regression

      Acceptance Criteria

      * New test exercises the scan-phase resume of a PDIB with >16 MiB of spilled range metadata
      * Test passes end-to-end (build resumes, index validates, data hashes match)
      * Test is tagged appropriately (requires_otel_build, requires_persistence, requires_replication)

      Related Files

      * IDL: src/mongo/db/index_builds/resumable_index_builds.idl
      * IDL: src/mongo/db/sorter/sorter.idl
      * Library: jstests/noPassthrough/libs/index_builds/primary_driven.js
      * Reference test: jstests/noPassthrough/index_builds/resumable_index_build_collection_scan_phase_large.js
      * Similar PDIB tests: jstests/noPassthrough/index_builds/resumable_primary_driven_index_build_scan_phase.js

            Assignee:
            Gregory Noma
            Reporter:
            Alex Sarkesian
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: