Unable to abandon replicated index build that is missing from catalog after unclean shutdown

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • Storage Execution
    • ALL
    • Execution Team 2024-10-28, Storage Execution 2025-06-09, Storage Execution 2025-06-23, Storage Execution 2025-08-04, Storage Execution 2025-08-18, Storage Execution 2025-09-01
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      In the following series of events:

      1. Index build starts. It is journaled and written to the oplog.
      2. Node is shutdown uncleanly
      3. A checkpoint was not taken

      The node can be left in a state at startup where the index build does not exist on the catalog but exists in the oplog. In this scenario, the oplog is replayed during startup and the index build is restarted.

      This can be a problem when a problematic index build is the cause of unclean shutdown as we'll likely end up in a restart loop (node crashes during startup from the index build, restarts, crashes again). If the index build is running into OOM errors, upgrading the instance size can allow the index build to finish during startup. 

      However if upsizing is a no-go and/or we want to give up on the index build, we currently do not offer a way to drop the index build after an unclean shutdown and the index exists on the oplog but not the catalog. We are unable to drop the index build in standalone because it is not reconciled to the catalog. recoverFromOplogAsStandalone=true is also unhelpful in this case as it is a read-only mode and thus unable to drop the index while the problematic index build is in progress (note: the command itself is also unable to get the node back into healthy state as the index build will likely not complete either).

      The request of this ticket is to provide an option to abandon index builds after unclean shutdowns.

            Assignee:
            Unassigned
            Reporter:
            Shin Yee Tan
            Votes:
            0 Vote for this issue
            Watchers:
            12 Start watching this issue

              Created:
              Updated: