Report oplog collection's size and count only when size-based oplog truncation is enabled

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Fixed
    • Priority: Major - P3
    • 9.1.0-rc0
    • Affects Version/s: None
    • Component/s: None
    • None
    • Storage Execution
    • Fully Compatible
    • Storage Execution 2026-07-20
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Time-based oplog truncation does not update oplog size and count

      This is because time-based oplog truncation uses truncation markers with 0 values for numBytesDeleted and numDocsDeleted; these are the values read by the fast count system when deciding how much to decrement the fast count of the oplog collection by.

      Remote is `10gen/mongo`; I'll pin permalinks to the current HEAD (`5f0883770706ca57ca33df24edf62dba55345fc3`).

      Root cause: disagg fast count overcounts oplog on truncation

      On disaggregated storage, the oplog uses replicated fast count (the WiredTiger SizeStorer is disabled), so the oplog record count is maintained purely from the docsDeleted/bytesDeleted values carried on each truncateRange operation. The default disagg truncation path (time-based) hardcodes those deltas to 0, so truncation physically removes records but reports that zero records were deleted.

      Why the default path is hit

      featureFlagSizeBasedOplogTruncationForDisagg defaults to false, so ReplicatedOplogTruncationThread::_reclaimOplog takes the time-based branch (_reclaimOplogByTime) rather than the size-based one.

      Feature flag default — server_feature_flags.idl#L51-L57:

      featureFlagSizeBasedOplogTruncationForDisagg:
          ...
          cpp_varname: gFeatureFlagSizeBasedOplogTruncationForDisagg
          default: false
          fcv_gated: false
      

      Branch selection — replicated_oplog_truncation_thread.cpp#L32-L40:

      RecordId ReplicatedOplogTruncationThread::_reclaimOplog(OperationContext* opCtx,
                                                              RecordStore& rs,
                                                              RecordId mayTruncateUpTo) {
          if (gFeatureFlagSizeBasedOplogTruncationForDisagg.isEnabled()) {
              return _reclaimOplogByMarkers(opCtx, rs, mayTruncateUpTo);
          }
          return _reclaimOplogByTime(opCtx, rs, mayTruncateUpTo);
      }
      

      The zeroed delta

      _reclaimOplogByTime builds its marker from CollectionTruncateMarkers::newestExpiredRecord and passes marker->bytes / marker->records straight into truncateRangereplicated_oplog_truncation_thread.cpp#L114-L120:

      collection_internal::truncateRange(opCtx,
                                         autoOplog.getCollection(),
                                         firstRecordId,
                                         truncateMarker->lastRecord,
                                         truncateMarker->bytes,    // 0
                                         truncateMarker->records); // 0
      

      But newestExpiredRecord always returns a marker with records = 0, bytes = 0 — it only computes the lastRecord boundary, not the number of records in the range — collection_truncate_markers.cpp#L99-L101:

      // byte and record increments are only advisory even when there's a size storer to look at them.
      return Marker(/*.records =*/0, /*.bytes =*/0, record->id, expiryTime);
      

      That "only advisory" comment is true for attached storage, where WiredTiger's SizeStorer tracks the true count independently. It is not true for disagg, where docsDeleted on the replicated truncateRange entry is the only source of the count decrement.

      Resulting drift

      • truncateRange propagates the (zero) delta into the replicated fast count — collection_write_path.cpp#L1048-L1053:
        opCtx->getServiceContext()->getOpObserver()->onTruncateRange(
            opCtx, collection, minRecordId, maxRecordId, bytesDeleted, docsDeleted, opTime);
        if (isReplicatedFastCountEnabled(opCtx)) {
            UncommittedFastCountChange::getForWrite(opCtx).record(
                collection->ns(), collection->uuid(), -docsDeleted, -bytesDeleted);
        }
        
      • Inserting the truncateRange oplog entry itself bumps the oplog count by +1oplog.cpp#L272-L275:
        if (isReplicatedFastCountEnabled(opCtx)) {
            UncommittedFastCountChange::getForWrite(opCtx).record(
                oplogCollection->ns(), oplogCollection->uuid(), nRecords, totalLength);
        }
        
      • Net effect per truncation: +1 (marker entry) − 0 (deleted records).

      The oplog's replicated fast count therefore climbs by one on every truncation cycle and never accounts for the records actually removed. The value is consistent across primary, secondaries, and checkpoint/recovery scans (all read docsDeleted = 0 from the same oplog entry), so it is a stable, unbounded overcount rather than a node-to-node divergence. db.oplog.rs.count() / fast count reports far more records than are physically present.

      Scope

      • Affected: disagg + default time-based truncation (featureFlagSizeBasedOplogTruncationForDisagg = false).
      • Not affected: size-based mode (markers from createNewMarker carry real counts); attached/non-disagg storage (RFC off, SizeStorer authoritative).
      • The checkpoint/scan delta accounting itself is correct and tested; the defect is purely the 0,0 fed in on the default path.

      Secondary concern

      computeTruncationBound disables its RFC gate when no valid-as-of timestamp is persisted yet (findPersistedTimestampStoreTs(...).value_or(Timestamp::max())) — oplog_truncation.cpp#L151-L165. In size-based mode, a cold-start full oplog scan (seekAfter = Timestamp::min) would double-subtract surviving truncateRange deltas for the oplog, since the scan counts the oplog absolutely while truncateRange is a relative delta. Real but much narrower window than the primary bug.

      Fix direction

      newestExpiredRecord must return the actual record/byte count in the [firstRecord, lastRecord] range when replicated fast count is in play (or _reclaimOplogByTime must compute the true deleted count before calling truncateRange). An accurate docsDeleted/bytesDeleted is required on the replicated path since there is no SizeStorer fallback.

       

      Note: the permalinks point at 10gen/mongo pinned to commit 5f08837 so the line numbers stay stable. If you'd rather link to a branch tip (e.g. master) instead of a pinned commit, I can swap them.

            Assignee:
            Yuhong Zhang
            Reporter:
            Damian Wasilewicz
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: