Adjust step_up_create_drop_since_installed_checkpoint.js for the follower table-drop contract

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Fixed
    • Priority: Major - P3
    • 9.1.0-rc0
    • Affects Version/s: None
    • Component/s: Storage Engine API
    • None
    • Storage Engines - Foundations
    • Fully Compatible
    • ALL
    • 200
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      WT-18405 makes a table drop on a disaggregated follower fail with EBUSY while the table holds data that no installed checkpoint covers. An empty table is unaffected — only a dirty one is refused.

      step_up_create_drop_since_installed_checkpoint.js was incompatible with that contract in two ways. This ticket fixes both; no server change is needed.

      1. It dropped a table it had written to, while pinning checkpoint installs. The test deliberately holds the secondary's installed checkpoint below the create, so the table's writes were never covered and the follower's drop was refused indefinitely. The collection is now created empty instead. The test's purpose is unchanged — a queued CREATE/REMOVE pair still straddles the checkpoint's schema epoch at step-up, because both still sit above the pinned checkpoint. The index build is kept: on an empty collection it does not dirty its internal tables, so that coverage survives.

      2. It asserted the drop completed before step-up. Under the new contract a follower drop may legitimately be deferred until a covering checkpoint is installed, so completion is now asserted after step-up.

      Verification

      Both disagg variants, with and without the WT-18405 hunk applied to the vendored WiredTiger — the test passes in all four combinations (38-68s, versus hanging to a 180s step-up timeout before). The only other failure in those runs is oplog_truncation_time_based.js, BF-45907, pre-existing.

      Not fixed here

      Diagnosing this surfaced a real server-side hazard that this change does not address, and which should probably be tracked separately.

      A follower's replicated dropIdent is retried with no cap and no deadline on the oplog applier thread. Normally that is harmless: the leader only replicates the drop once a covering checkpoint exists, so on the follower that checkpoint's stable timestamp S precedes the dropIdent optime D, S is installable, and the drop succeeds.

      It deadlocks when a single oplog batch straddles S. Checkpoint selection requires checkpoint.stableTimestamp <= lastApplied, and lastApplied advances on batch completion rather than per entry. So a batch that begins before S and contains the drop at D holds lastApplied below S for its whole duration; S can never be selected for install; the drop spins forever; the batch never completes; and the node can no longer step up. The condition the retry waits for can only be produced by progress the retry itself blocks.

      There is no lock contention involved — the checkpoint installer runs on its own executor and simply has nothing eligible to select, so no lock diagnostic will surface it. Exposure scales with batch size against checkpoint cadence: a lagging follower catching up in large batches is far more exposed than a caught-up one. Not measured.

      Also note the background reaper skips timestamped drops unless the node holds a write intent, so on a follower the replicated apply is the only thing that ever drops an ident — deferring is not an escape either.

      Related

      SERVER-122277 proposes the opposite coupling — skip installing checkpoints while drops are pending. WT-18405 needs pickups to proceed in order to clear drops, so check the interaction.

            Assignee:
            Alex Blekhman
            Reporter:
            Alex Blekhman
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: