Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-75069

Reapplying an insert donor state doc op (for split, merge and tenant migration) on secondaries due to WCE can cause node crash .

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • Labels:
      None
    • Serverless
    • ALL
    • Server Serverless 2023-04-03

      I found that tenant_migration_donor_op_observer.cpp (shared by MTM protocol and shard merge) doesn’t register onRollback hook  (to release Servereless lock and uninstall mtab) on secondaries with the following argument.

                  // onRollback is not registered on secondaries since secondaries should not fail to
                  // apply the write.

      And,  that’s completely not true. On secondaries, if the WriteUnitOfWork(WUOW) fails with WriteConflictException (WCE), we would retry the WUOW (see here). In case of other failures, secondaries don’t retry the WOUW, instead we would crash the server.

       

      So, if the secondaries  retry applying the insert state doc oplog entry due to WCE,  it  will try to reacquire the Serverless lock and would crash the server due to this invariant failure.  The fix will be to register onRollback() hook even for secondaries. This is a problem for MTM (multi-tenant migration) , merge and split.

            Assignee:
            didier.nadeau@mongodb.com Didier Nadeau
            Reporter:
            suganthi.mani@mongodb.com Suganthi Mani
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: