Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-54968

recoverFromOplogAsStandalone doesn't wait for index builds to finish before switching to read-only mode

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Index Maintenance
    • Labels:
      None
    • Storage Execution
    • ALL
    • v4.4

      When starting up a standalone with recoverFromOplogAsStandalone, the oplog gets replayed and can start background index builds. Once all operations are replayed, the server gets put into read-only mode without waiting for index builds to finish. This can cause index builds to fail and crash.

       

      [js_test:incremental_backup_e2e] 2021-03-04T20:17:43.396+0000 d20027| 2021-03-04T20:17:43.396+00:00 I  REPL     21536   [initandlisten] "Completed oplog application for recovery","attr":{"numOpsApplied":10024,"numBatches":18,"applyThroughOpTime":{"ts":{"$timestamp":{"t":1614889059,"i":10034}},"t":1}}
      [js_test:incremental_backup_e2e] 2021-03-04T20:17:43.397+0000 d20027| 2021-03-04T20:17:43.397+00:00 I  INDEX    20391   [IndexBuildsCoordinatorMongod-1] "Index build: collection scan done","attr":{"buildUUID":{"uuid":{"$uuid":"958fde18-4bfd-47a7-9329-66ba3754adff"}},"collectionUUID":{"uuid":{"$uuid":"2779e905-73b0-4a0c-b537-9110fb1084e4"}},"namespace":"test8_fsmdb0.distinct_projection_fsm_7","totalRecords":1000,"readSource":"kNoTimestamp","durationMillis":0}
      [js_test:incremental_backup_e2e] 2021-03-04T20:17:43.398+0000 d20027| 2021-03-04T20:17:43.397+00:00 W  REPL     21558   [initandlisten] "Setting mongod to readOnly mode as a result of specifying 'recoverFromOplogAsStandalone'"
      [js_test:incremental_backup_e2e] 2021-03-04T20:17:43.398+0000 d20027| 2021-03-04T20:17:43.398+00:00 W  CONTROL  20547   [initandlisten] "Document(s) exist in 'system.replset', but started without --replSet. Database contents may appear inconsistent with the writes that were visible when this node was running as part of a replica set. Restart with --replSet unless you are doing maintenance and no other clients are connected. The TTL collection monitor will not start because of this. For more info see http://dochub.mongodb.org/core/ttlcollections"
      [js_test:incremental_backup_e2e] 2021-03-04T20:17:43.398+0000 d20027| 2021-03-04T20:17:43.398+00:00 I  NETWORK  23015   [listener] "Listening on","attr":{"address":"/tmp/mongodb-20027.sock"}
      [js_test:incremental_backup_e2e] 2021-03-04T20:17:43.398+0000 d20027| 2021-03-04T20:17:43.398+00:00 I  NETWORK  23015   [listener] "Listening on","attr":{"address":"0.0.0.0"}
      [js_test:incremental_backup_e2e] 2021-03-04T20:17:43.398+0000 d20027| 2021-03-04T20:17:43.398+00:00 I  NETWORK  23016   [listener] "Waiting for connections","attr":{"port":20027,"ssl":"off"}
      [js_test:incremental_backup_e2e] 2021-03-04T20:17:43.400+0000 d20027| 2021-03-04T20:17:43.400+00:00 I  STORAGE  20649   [IndexBuildsCoordinatorMongod-1] "Index build: failed","attr":{"buildUUID":{"uuid":{"$uuid":"958fde18-4bfd-47a7-9329-66ba3754adff"}},"collectionUUID":{"uuid":{"$uuid":"2779e905-73b0-4a0c-b537-9110fb1084e4"}},"namespace":"test8_fsmdb0.distinct_projection_fsm_7","error":{"code":20,"codeName":"IllegalOperation","errmsg":"Cannot execute a write operation in read-only mode"}}
      [js_test:incremental_backup_e2e] 2021-03-04T20:17:43.400+0000 d20027| | 2021-03-04T20:17:43.400+00:00 F  -        23081   [IndexBuildsCoordinatorMongod-1] "Invariant failure","attr":{"expr":"status.isA<ErrorCategory::Interruption>() || status.isA<ErrorCategory::ShutdownError>()","msg":"Unexpected error code during index build cleanup: IllegalOperation: Cannot execute a write operation in read-only mode","file":"src/mongo/db/index_builds_coordinator.cpp","line":2160}
      [js_test:incremental_backup_e2e] 2021-03-04T20:17:43.401+0000 d20027| 2021-03-04T20:17:43.400+00:00 F  -        23082   [IndexBuildsCoordinatorMongod-1] "\n\n***aborting after invariant() failure\n\n"
      

       

            Assignee:
            backlog-server-execution [DO NOT USE] Backlog - Storage Execution Team
            Reporter:
            gregory.wlodarek@mongodb.com Gregory Wlodarek
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: