Rollback-to-stable causes untimestamped _mdb_catalog writes for unfinished single-phase indexes

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • Storage Execution
    • ALL
    • Hide

      Repro summary:

      • One phase foreground index build on a non-empty colleciton writes ready: false at T_init and ready:true at T_commit as separately timestamped catalog updates.
      • The build commits locally on the step-up node, but T_commit is prevented from reaching a majority.
      • The node's stable timestamp remains pinned at T_init, so WiredTiger checkpoints the historical catalog image where the index entry still exists with ready:false
      • That drop updates _mdb_catalog outside the replicated timestamped path and crashes.
      Show
      Repro summary: One phase foreground index build on a non-empty colleciton writes ready: false at T_init and ready:true at T_commit as separately timestamped catalog updates. The build commits locally on the step-up node, but T_commit is prevented from reaching a majority. The node's stable timestamp remains pinned at T_init , so WiredTiger checkpoints the historical catalog image where the index entry still exists with ready:false After the node reconnects and rolls back, openCatalog() reads that stable image and reconcileCatalogAndIdents() drops the "unfinished" index. That drop updates _mdb_catalog outside the replicated timestamped path and crashes.
    • 0
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Cause
      Rollback reuses a path in reconcileCatalogAndIdents() that's appropriate for standalone or startup recovery assumptions, but not for rollback-to-stable on a replica set member.

      Summary
      During rollback to stable, openCatalog() loads the durable catalog as of the stable timestamp and then calls reconcileCatalogAndIdents(). If the stable checkpoint contains an index catalog entry with ready:false, reconcile currently treats it as an unfinished single-phase index build and drops it - mutating _mdb_catalog.

      Rollback is operating on a historical stable image of the replicated catalog, not performing standalone cleanup. Rollback attempts an untimestamped drop of the index, its update to the _mdb_catalog yields

       WiredTiger error message: __wt_txn_timestamp_usage_check: unexpected timestamp usage: no timestamp provided for an update to a table configured to always use timestamps once they are first used

      Note: this only impacts RTS for single phase index builds on non-empty collections - empty collections will not generate an incomplete foreground index build.
       
      Attached is a repro courtesy of joan.bruguera-mico@mongodb.com

        1. config_chunks_one_phase_index_build_rollback.js.js
          4 kB

            Assignee:
            Unassigned
            Reporter:
            Haley Connelly
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: