Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-44055

All secondary crashed in SessionUpdateTracker and cannot recovery

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Investigating
    • Priority: Major - P3
    • Resolution: Unresolved
    • Affects Version/s: 4.0.3
    • Fix Version/s: None
    • Component/s: Replication
    • Labels:
      None
    • Operating System:
      ALL
    • Steps To Reproduce:
      Hide

      Still don't how to reproduce

      Show
      Still don't how to reproduce

      Description

      One secondary crashed

      2019-10-16T18:51:49.855+0800 F -        [rsSync-0] Fatal Assertion 50843 at src/mongo/db/repl/session_update_tracker.cpp 69
          2019-10-16T18:51:49.855+0800 F -        [rsSync-0]
       
          ***aborting after fassert() failure
      

      Then another secondary crashed after change sync source to primary 

      2019-10-16T18:52:21.381+0800 I REPL     [rsBackgroundSync] Changed sync source from empty to XX.XX.XX.XX:3097(PrimaryNode IP)
      2019-10-16T18:52:21.381+0800 I ASIO     [RS] Connecting to 11.203.27.243:3097
      2019-10-16T18:52:21.780+0800 F -        [rsSync-0] Fatal Assertion 50843 at src/mongo/db/repl/session_update_tracker.cpp 69
      2019-10-16T18:52:21.780+0800 F -        [rsSync-0]

      The secondary cannot be restarted,and generate very wiered errors,but the _tmp database is created long before, and no one delete the directory.  Then no primary can be elected in the replica-set with three members,cause the service unavailable for a long time.

      2019-10-16T18:54:51.635+0800 I STORAGE  [initandlisten] WiredTiger message [1571223291:635501][41642:0x2b90d4829d00], txn-recover: Recovering log 752 through 753
      2019-10-16T18:54:51.730+0800 I STORAGE  [initandlisten] WiredTiger message [1571223291:730545][41642:0x2b90d4829d00], file:local/collection-2--8933994092597418142.wt, txn-recover: Recovering log 753 through 753
      2019-10-16T18:54:51.796+0800 I STORAGE  [initandlisten] WiredTiger message [1571223291:796255][41642:0x2b90d4829d00], file:local/collection-2--8933994092597418142.wt, txn-recover: Set global recovery timestamp: 5da6f62b00000008
      2019-10-16T18:54:51.814+0800 I RECOVERY [initandlisten] WiredTiger recoveryTimestamp. Ts: Timestamp(1571223083, 8)
      2019-10-16T18:54:51.814+0800 I STORAGE  [initandlisten] Triggering the first stable checkpoint. Initial Data: Timestamp(1571223083, 8) PrevStable: Timestamp(0, 0) CurrStable: Timestamp(1571223083, 8)
      2019-10-16T18:54:52.044+0800 E STORAGE  [initandlisten] WiredTiger error (2) [1571223292:44873][41642:0x2b90d4829d00], file:_tmp/collection-6--6434994499321235876.wt, WT_SESSION.open_cursor: __posix_open_file, 715: /home/mongo/mongo3103/data/_tmp/collection-6--6434994499321235876.wt: handle-open: open: No such file or directory Raw: [1571223292:44873][41642:0x2b90d4829d00], file:_tmp/collection-6--6434994499321235876.wt, WT_SESSION.open_cursor: __posix_open_file, 715: /home/mongo/mongo3103/data/_tmp/collection-6--6434994499321235876.wt: handle-open: open: No such file or directory
      2019-10-16T18:54:52.044+0800 E STORAGE  [initandlisten] Failed to get the cursor for uri: table:_tmp/collection-6--6434994499321235876
      2019-10-16T18:54:52.044+0800 E STORAGE  [initandlisten] This may be due to missing data files. Please read the documentation for starting MongoDB with --repair here: http://dochub.mongodb.org/core/repair
      2019-10-16T18:54:52.044+0800 F -        [initandlisten] Fatal Assertion 50883 at src/mongo/db/storage/wiredtiger/wiredtiger_recovery_unit.cpp 538
      2019-10-16T18:54:52.044+0800 F -        [initandlisten]
      
      

      The oplog cause the error in primary maybe as follows

      Timestamp(1571223107, 5603), "t" : NumberLong(2), "h" : NumberLong("4179556560629599034"), "v" : 2, "op" : "n", "ns" : "db.coll", "ui" : BinData(4,"uKDv6Dw1RBKxq+ewcCqntg=="), "fromMigrate" : true, "o2" : { "lsid" : { "id" : BinData(4,"b9QQ1SZVT2mh4vBoD4u/LA=="), "uid" : BinData(0,"Y5mrDaxi8gv8RmdTsQ+1j7fmkr7JUsabhNmXAheU0fg=") }, "txnNumber" : NumberLong(2), "op" : "i", "ns" : "db.coll", "ui" : BinData(4,"uKDv6Dw1RBKxq+ewcCqntg=="), "o" : { "id" : "xx", "platformUId" : "yy", "players" : { "xx" : { "id" : NumberLong(123456), "sId" : NumberLong(1), "loginAt" : ISODate("2019-xx-xxT10:51:47.522Z") } }, "regTime" : ISODate("2019-10-16T10:51:47.522Z"), "id" : NumberLong(3096698), "platform" : "hortor", "platformId" : NumberLong(778902834), "openId" : "xx", "shareCode" : "yy", "channel" : "zz" }, "ts" : Timestamp(1571223107, 4431), "t" : NumberLong(2), "h" : NumberLong("2385553688009329962"), "v" : NumberLong(2), "wall" : ISODate("2019-10-16T10:51:47.520Z"), "stmtId" : 0, "prevOpTime" : { "ts" : Timestamp(0, 0), "t" : NumberLong(-1) } }, "wall" : ISODate("2019-10-16T10:51:47.520Z"), "lsid" : { "id" : BinData(4,"b9QQ1SZVT2mh4vBoD4u/LA=="), "uid" : BinData(0,"Y5mrDaxi8gv8RmdTsQ+1j7fmkr7JUsabhNmXAheU0fg=") }, "txnNumber" : NumberLong(2), "stmtId" : 0, "prevOpTime" : { "ts" : Timestamp(0, 0), "t" : NumberLong(-1) }, "o" : { "$sessionMigrateInfo" : 1 } }
      

       

        Attachments

          Activity

            People

            • Votes:
              2 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated: