Adapt disagg log_server_startup_deleted_segment jstests to CMS GetLogSegmentList filtering DELETED segments (SLS pin d284e28d / SLS-10295)

XMLWordPrintableJSON

    • Type: Task
    • Resolution: Fixed
    • Priority: Major - P3
    • 9.1.0-rc0
    • Affects Version/s: None
    • Component/s: Replication
    • None
    • RSSD
    • Major Change
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Generated by the sls-pin-bump skill, operated by @awang01810.

      The weekly SLS pin bump (SERVER-131879, PR #59167) advanced the pinned SLS commit to d284e28d. Three disagg tests that exercise the "log segment deleted from log server" path began failing. They passed at the prior pin 35e15b72, so the SLS range 35e15b72..d284e28d is the only change. All three were confirmed from the bot verification patch 6a5fda8b task logs to share one primary cause:

      • pali/log_server_startup_deleted_segment.js:121 (disagg_storage_0)
      • pali/log_server_startup_deleted_segment_uncheckpointed_write.js:129 (disagg_storage_2)
      • pali/log_server_startup_deleted_segment_other_lcg.js:124 (disagg_storage_5)

      Each fails on the same assertion: assert failed : segment 0 should still be listed in CMS.

      Root cause (confirmed, intentional)

      The SLS change is SLS-10295 "update flag_listing defaults" (10gen/sls PR #6363, commit a15009c5e), inside the pin range. It flips the CMS flag cms_get_log_segment_list_filter_deleted_segments_enabled from default_value: false to default_value: true. The flag's own comment: "When enabled, GetLogSegmentList filters out DELETING and DELETED segments by default." CellMetadataService also exposes a request field include_deleted_segments for callers that still want them. So the behavior change is intentional; no SLS escalation is needed.

      Sequence from the task logs:

      1. The test marks segment 0 DELETED. RecordDeletedLogSegment succeeds (CMS log: Marked log segment as DELETED log_id=1 segment_id=0).
      2. The following {{GetLogSegmentList {log_id:1}

        }} now returns ONLY the OPEN segment 1; segment 0 is omitted. (Before deletion the same call returned segment 0 as SEALED, so the filtering is specific to the DELETED/DELETING state, not to non-open states generally.)

      3. getLogSegment(cms, 1, 0) (slstest.js:1770 -> getLogSegments:1738) returns undefined, and the existence assert fails.

      Fix (RSSD, harness adaptation)

      Have the harness request deleted segments where it needs them. getLogSegments (slstest.js:1738) currently sends {{

      { log_id: logId }

      }}; add include_deleted_segments: true for the deleted-segment lookups (or plumb it through options). This preserves each test's intent (verify the segment persists in the catalog as DELETED) rather than flipping the assertion.

      Old-pin-safety: the include_deleted_segments request field predates the pin range (it is read in the pre-SLS-10295 handler), so setting it is honored by the old SLS binary too. The fix should therefore be landable independently on master (the untouched bump PR #59167 then goes green and merges as-is). Confirm with a DSC patch of the fix applied WITHOUT the bump before relying on this.

      Secondary (latent, not the cause; optional hardening)

      LogService.GetLogServers on a not-found segment now returns a response whose grpc-status-details-bin trailer code disagrees with the grpc-status header, so grpcurl (v1.9.1) aborts with "grpc-status-details-bin mismatch". This did NOT fail these tests: the deletion-confirmation loop catches the throw and matches on the grpc-message substring, which is still present, so it passes and control reaches the primary failure above. The mongod client handles the same response correctly. Worth hardening (a loose substring match masking a grpcurl abort) but out of scope for the immediate fix.

      Verification

      • Local disagg repro (SLS Docker) of the three tests at pin d284e28d: red before, green after.
      • Confirm the post-restart recovery portion of each test (log_server_startup_deleted_segment.js:139-158 and siblings) actually runs and passes once the assertion is corrected. The tests currently die before reaching it.
      • Re-run #59167's required/disagg verification patch off the bump branch; expect green.

      Timeline

      Land before the next automated pin bump (cron Tuesdays 14:00 UTC; next 2026-07-28) so SERVER-131879 is not left red across a cron cycle.

            Assignee:
            Andrew Wang
            Reporter:
            Andrew Wang
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: