Recovery fails with EINVAL after salvaging leftover journal tail on macOS

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Minor - P4
    • None
    • Affects Version/s: None
    • Component/s: Logging
    • None
    • Storage Engines - Persistence
    • 35.195
    • None
    • None

      FCBIS mixed-version sync on macOS crashed in recovery with EINVAL after salvaging leftover bytes in a preallocated journal tail. The leftover-tail NOTICE is longstanding on macOS; this ticket is the subsequent recovery failure after salvage.

      Seen in BF-45282: fcbis_mixed_version_sync.js on enterprise-macos. An FCBIS node on latest FCV copied files from a last-LTS replica set. The FCV downgrade close/reopen then failed:

      journal/WiredTigerLog.0000000002: log record at position 33037184 has record len corruption 0x0
      log file journal/WiredTigerLog.0000000002 corrupted at position 33037184, truncated.
      log file journal/WiredTigerLog.0000000003 corrupted at position 69933952, truncated.
      int __wt_txn_recover(...):1462:Recovery failed  error_str: Invalid argument  error_code:22
      

      MongoDB leaves log.zero_fill off, so journal preallocation uses __wt_fextend. Linux fallocate zeros that tail. macOS has no fallocate and falls back to ftruncate, which is sparse and can leave leftover non-zero bytes in the unused 100MB tail. Recovery reports those as record len corruption 0x0.

      The first reopen after FCBIS is a backup restore (WT_CONN_WAS_BACKUP) and _log_has_hole skips record verification. The extra close/reopen from the FCV downgrade does verify, salvages by truncating at the last good record (log_truncate at rd_lsn), then recovery still fails with EINVAL. _log_salvage_message reports bad_offset (leftover in the preallocation); the truncate point is rd_lsn.

      Same NOTICE: WT-4233, WT-14850, WT-17183. Those salvage and continue. This one fails because the extra reopen after backup restore hits EINVAL.

      Need to truncate so a reopen after salvage does not re-read the leftover tail and fail.

            Assignee:
            Sean Watt
            Reporter:
            Sean Watt
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: