ExportXMLWordPrintableJSON

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: 8.0.32, 8.0.34, 8.0.38
    • Component/s: None
    • None
    • Environment:
    • Replication
    • ALL
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      On a standalone mongod 8.0, a write acknowledged with {w:"majority", j:true} can be missing after the mongod process is killed with SIGKILL and restarted. Only the process is killed. The host keeps running, so the kernel page cache survives, and anything mongod handed to the kernel should survive too. The same probe loses nothing on 6.0.29 or 7.0.45. It also loses nothing on any of these versions when run as a three-member replica set.

      Repro (script below)

      1. Start a standalone: mongod --dbpath D --port 28001 --bind_ip 127.0.0.1 --fork --logpath D/mongod.log.
      2. Insert {_id:1, v:0} into t.c with {w:"majority", j:true}.
      3. Start a client loop that runs findAndModify({query:{_id:1}, update:{$inc:{v:1, new:true, writeConcern:{w:"majority", j:true, wtimeout:5000}})}}. The client prints value.v only after a reply with ok:1 and no writeConcernError.
      4. After a random 0.1–1.0 s, run kill -9 on mongod.
      5. Restart mongod on the same dbpath and read db.c.find({_id:1}).readConcern("local").
      6. If the recovered v is lower than the last value the client printed, an acknowledged write was lost.
      7. Repeat for 60 trials. Trials in which no write was acknowledged before the kill are excluded.

      Results (trials with ≥1 lost acknowledged write / trials with an acknowledgement)

      Version Host 1 (WSL2) Host 2 (Azure VM, native)
      6.0.29 0 / 23 0 / 48
      7.0.45 0 / 16 —
      8.0.32 8 / 23 (max 1 write lost) —
      8.0.34 14 / 34 (max 2) 44 / 47
      8.0.38 — 42 / 44 (mean 8.4 writes lost per trial)
      3-member replica set, every member killed, majority read after recovery: 6.0.29 / 7.0.45 / 8.0.32 / 8.0.34 0/16, 0/14, 0/19, 0/16 —

      Indicative variant: the same probe on host 1 with writeConcern:{w:1, j:true} (8.0.34, 30 trials) lost nothing, 0 of the 10 trials with an acknowledgement. At the w:"majority" loss rate on that host (14/34), seeing 0 of 10 by chance has probability ≈ 0.005. This suggests the problem is specific to w:"majority" on a standalone: the majority path may acknowledge before the journal flush that j:true requires.

      Expected

      A write acknowledged with j:true is in the on-disk journal before the acknowledgement. After a process crash and restart with the host still up, startup recovery should restore it, as on 6.0 and 7.0.

      Actual

      On 8.0, the recovered document is behind the last acknowledged value in most trials on host 2. Example from 8.0.38: last acknowledged 1101, recovered 1093.

      Raw CSVs and the effective configuration dumps can be supplied.

      Repro script (probe_mongo_durability.sh; run with MODE=standalone N=60 MONGOD=/path/to/mongod bash probe_mongo_durability.sh)

      #!/usr/bin/env bash
      # probe_mongo_durability.sh -- does a MongoDB version keep every write it
      # acknowledged with {w:"majority", j:true} across SIGKILL of mongod (process
      # kill only: the kernel page cache survives, so anything mongod had handed to
      # the kernel survives too)?
      #
      # A client increments one counter with findAndModify under an explicit
      # {w:"majority", j:true, wtimeout:5000}, and logs each acknowledged value
      # with the reply's operationTime (replica set) after the acknowledgement
      # arrives. mongod (every member, in replica-set mode) is killed at a random
      # moment and restarted; the counter is then read back from the primary with
      # readConcern "majority" (replica set) or "local" (standalone). A recovered
      # value below the last acknowledged one is a lost acknowledged write.
      #
      #   MODE=standalone|rs3   (default standalone)
      #   MONGOD=/path/to/mongod (default: mongod on PATH)   N=trials (default 60)
      # Output: one CSV row per trial on stdout
      #   mode,version,trial,last_acked,last_acked_optime,recovered,lost,read_concern
      # and the effective configuration (version, storage engine, journal commit
      # interval, default write concern, filesystem of the data directory) on stderr.
      set -u
      MODE=${MODE:-standalone}; MONGOD=${MONGOD:-mongod}; N=${N:-60}
      D=${PROBE_DIR:-/tmp/mrepro2-$MODE}; rm -rf "$D"; mkdir -p "$D"
      VER=$($MONGOD --version | sed -n 's/^db version v//p' | head -1)
      PORTS=(28001); [ "$MODE" = rs3 ] && PORTS=(28001 28002 28003)
      URI="mongodb://127.0.0.1:${PORTS[0]}/t"
      [ "$MODE" = rs3 ] && URI="mongodb://127.0.0.1:28001,127.0.0.1:28002,127.0.0.1:28003/t?replicaSet=dur"
      
      start_one() {  # start_one <port>
        local p=$1; mkdir -p "$D/db$p"
        local rs=(); [ "$MODE" = rs3 ] && rs=(--replSet dur)
        $MONGOD --dbpath "$D/db$p" --port "$p" --bind_ip 127.0.0.1 "${rs[@]}" \
          --fork --logpath "$D/mongod$p.log" --logappend >/dev/null 2>&1
        for i in $(seq 1 300); do
          mongosh --quiet --port "$p" --eval 'db.runCommand({ping:1}).ok' 2>/dev/null | grep -q '^1$' && return 0
          sleep 0.1; done; return 1
      }
      start_all() { for p in "${PORTS[@]}"; do start_one "$p" || return 1; done
        if [ "$MODE" = rs3 ] && [ "${1:-}" != noprimary ]; then   # wait for a primary
          for i in $(seq 1 300); do
            mongosh --quiet "$URI" --eval 'db.hello().isWritablePrimary' 2>/dev/null | grep -q true && return 0
            sleep 0.2; done; return 1; fi; }
      kill_all() { for p in "${PORTS[@]}"; do pkill -9 -f "mongod --dbpath $D/db$p" ; done
        while pgrep -f "mongod --dbpath $D/db" >/dev/null; do sleep 0.02; done; }
      
      start_all noprimary || { echo "no mongod" >&2; exit 1; }
      if [ "$MODE" = rs3 ]; then
        mongosh --quiet --port 28001 --eval 'rs.initiate({_id:"dur",members:[
          {_id:0,host:"127.0.0.1:28001",priority:2},{_id:1,host:"127.0.0.1:28002"},{_id:2,host:"127.0.0.1:28003"}]})' >/dev/null
        start_all || { echo "no primary" >&2; exit 1; }
      fi
      mongosh --quiet "$URI" --eval 'db.c.insertOne({_id:1,v:0},{writeConcern:{w:"majority",j:true}})' >/dev/null
      
      { echo "# version $VER mode $MODE"
        mongosh --quiet "$URI" --eval '
          const s = db.serverStatus();
          print("# storageEngine " + s.storageEngine.name + " persistent=" + s.storageEngine.persistent);
          const p = db.adminCommand({getParameter:1, journalCommitInterval:1});
          print("# journalCommitInterval " + JSON.stringify(p.journalCommitInterval));
          try { print("# defaultRWConcern " + JSON.stringify(db.adminCommand({getDefaultRWConcern:1}))); } catch (e) {}
          try { print("# writeConcernMajorityJournalDefault " + rs.conf().writeConcernMajorityJournalDefault); } catch (e) {}'
        echo "# fs $(df -T "$D" | tail -1 | awk '{print $2, $7}')"; } >&2
      
      RC=local; [ "$MODE" = rs3 ] && RC=majority
      echo "mode,version,trial,last_acked,last_acked_optime,recovered,lost,read_concern"
      for t in $(seq 1 "$N"); do
        mongosh --quiet "$URI" --eval '
          for (;;) {
            const r = db.runCommand({findAndModify:"c", query:{_id:1}, update:{$inc:{v:1}}, new:true,
                                     writeConcern:{w:"majority", j:true, wtimeout:5000}});
            if (r.ok !== 1 || r.writeConcernError) continue;
            print(r.value.v + " " + (r.operationTime ? r.operationTime.toString() : "-"));
          }' > "$D/acked.$t" 2>/dev/null &
        w=$!
        sleep "0.$(( RANDOM % 9 + 1 ))$(( RANDOM % 10 ))"
        kill_all; wait $w 2>/dev/null
        last=$(grep -E '^[0-9]+ ' "$D/acked.$t" | tail -1)
        start_all || { echo "restart failed" >&2; exit 1; }
        rec=$(mongosh --quiet "$URI" --eval "print(db.c.find({_id:1}).readConcern('$RC').next().v)")
        la=${last%% *}; lo=${last#* }; [ -z "$last" ] && { la=; lo=; }
        case "$rec" in ''|*[!0-9]*) echo "trial $t: unreadable recovered value [$rec]" >&2; exit 1 ;; esac
        l=0; [ -n "$la" ] && [ "$rec" -lt "$la" ] && l=$(( la - rec ))
        echo "$MODE,$VER,$t,$la,$lo,$rec,$l,$RC"
      done
      kill_all
      

            Assignee:
            Unassigned
            Reporter:
            Rishabh Kumar (EXT)
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated: