Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-71571

build index fail , DB crash ,can not startup DB : collection-53-3002926054102476948.wt: read checksum error for 36864B block at offset 6178557952: block header checksum of 0x1ee9e8e2 doesn't match expected checksum of 0x907357ff

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 4.4.13
    • Component/s: None
    • Labels:
    • ALL

      Problem Statement/Rationale

      Hello there, can someone help me?  I'm running the build  TTL in the background, but it seems the Mongo replica set crashed. and then I'm trying to start up all replica set members, but the startup fails. and try to start up standalone, but still failing.

      db.rawdata.createIndex({"time":1},{background:true, expireAfterSeconds:1209600});

      Steps to Reproduce

      1. The MongoDB replica set crashed while building the TTL index.
      1. try to restart DB but fail.

      Could you tell me :

      • How to resume the Mongo replica set?

       

      Thanks you inadvance.

      Actual Results

       

      {"t":{"$date":"2022-11-23T11:52:57.003+08:00"},"s":"I",  "c":"-",        "id":51773,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"progress meter","attr":{"name":"Index Build: scanning collection","done":153600,"total":390354,"percent":39}}
      {"t":{"$date":"2022-11-23T11:52:57.249+08:00"},"s":"I",  "c":"REPL_HB",  "id":23974,   "ctx":"ReplCoord-3","msg":"Heartbeat failed after max retries","attr":{"target":"ppnewsmongo02:27017","maxHeartbeatRetries":2,"error":{"code":6,"codeName":"HostUnreachable","errmsg":"Error connecting to ppnewsmongo02:27017 (172.16.25.41:27017) :: caused by :: Connection refused"}}}
      {"t":{"$date":"2022-11-23T11:52:57.504+08:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"WiredTiger error","attr":{"error":0,"message":"[1669175577:504738][10117:0x7fdab21e1700], file:csnews_task/collection-53-3002926054102476948.wt, WT_CURSOR.next: __wt_block_read_off, 284: csnews_task/collection-53-3002926054102476948.wt: read checksum error for 36864B block at offset 6178557952: block header checksum of 0x1ee9e8e2 doesn't match expected checksum of 0x907357ff"}}
      {"t":{"$date":"2022-11-23T11:52:57.504+08:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"WiredTiger error","attr":{"error":0,"message":"[1669175577:504885][10117:0x7fdab21e1700], file:csnews_task/collection-53-3002926054102476948.wt, WT_CURSOR.next: __wt_bm_corrupt_dump, 140: {0: 6178557952, 36864, 0x907357ff}: (chunk 1 of 36): 00 00 00 00 00 00 00 00 ba 1c 02 00 00 00 00 00 d2 e0 00 00 02 00 00 00 07 05 00 00 00 80 00 00 e2 e8 e9 1e 01 00 00 00 11 e3 02 49 14 80 e2 c0 21 a1 e0 00 00 07 5f 69 64 00 5a 14 09 4d 2a 8b 8c 71 00 00 00 00 00 00 92 c1 03 f0 f4 b9 00 0b b1 69 9c 02 74 61 73 6b 69 64 00 25 00 00 00 36 65 34 36 32 61 62 32 2d 64 33 31 61 2d 34 38 30 64 2d 62 35 63 64 2d 66 30 34 63 32 63 66 34 64 37 63 37 00 10 73 65 71 75 65 6e 63 65 00 1e 00 00 00 09 74 69 6d 65 00 94 55 44 de 5f 01 00 00 02 72 61 77 68 74 6d 6c 00 31 e0 00 00 50 43 46 45 54 30 4e 55 57 56 42 46 49 47 68 30 62 57 77 2b 49 44 78 6f 64 47 31 73 49 47 78 68 62 6d 63 39 49 6d 56 75 49 69 42 77 63 6d 56 6d 61 58 67 39 49 6d 39 6e 4f 69 42 6f 64 48 52 77 4f 69 38 76 62 32 64 77 4c 6d 31 6c 4c 32 35 7a 49 79 49 2b 49 44 78 6f 5a 57 46 6b 50 69 41 38 62 57 56 30 59 53 42 6a 61 47 46 79 63 32 56 30 50 53 4a 31 64 47 59 74 4f 43 49 2b 49 44 78 30 61 58 52 73 5a 53 42 70 5a 44 30 69 63 47 46 6e 5a 53 31 30 61 05 14 f0 4a 49 2b 51 32 46 79 5a 32 38 67 63 32 68 70 63 43 42 4c 53 56 52 55 57 53 42 68 5a 33 4a 76 64 57 35 6b 4c 43 42 73 5a 57 46 72 61 57 35 6e 49 47 39 70 62 43 77 67 55 47 68 70 62 47 6c 77 63 47 6c 75 5a 58 4d 67 50 43 39 30 61 01 50 f0 48 54 34 67 50 47 31 6c 64 47 45 67 61 48 52 30 63 43 31 6c 63 58 56 70 64 6a 30 69 57 43 31 56 51 53 31 44 62 32 31 77 59 58 52 70 59 6d 78 6c 49 69 42 6a 62 32 35 30 5a 57 35 30 50 53 4a 4a 52 54 31 6c 5a 47 64 6c 49 6a 19 48 f0 7b 62 6d 46 74 5a 54 30 69 64 6d 6c 6c 64 33 42 76 63 6e 51 69 49 47 4e 76 62 6e 52 6c 62 6e 51 39 49 6e 64 70 5a 48 52 6f 50 57 52 6c 64 6d 6c 6a 5a 53 31 33 61 57 52 30 61 43 77 67 61 57 35 70 64 47 6c 68 62 43 31 7a 59 32 46 73 5a 54 30 78 4c 6a 41 73 49 47 31 68 65 47 6c 74 64 57 30 74 63 32 4e 68 62 47 55 39 4d 53 34 77 4c 43 42 31 63 32 56 79 4c 58 4e 6a 59 57 78 68 01 a4 14 50 54 41 69 50 69 35 64 f0 4c 75 59 57 31 6c 50 53 4a 6b 5a 58 4e 6a 63 6d 6c 77 64 47 6c 76 62 69 49 67 59 32 39 75 64 47 56 75 64 44 30 69 52 32 56 75 5a 58 4a 68 62 43 42 6a 59 58 4a 6e 62 79 42 7a 61 47 6c 77 49 45 74 4a 56 46 52 5a 49 48 4a 68 62 69 42 68 31 64 24 49 47 39 75 49 48 52 6f 5a 53 21 10 f0 40 46 7a 64 43 42 76 5a 69 42 43 59 58 4a 6a 5a 57 78 76 62 6d 45 67 64 47 39 33 62 69 77 67 55 32 39 79 5a 32 39 7a 62 32 34 67 63 48 4a 76 64 6d 6c 75 59 32 55 73 49 48 4e 76 64 58 52 6f 5a 57 46 01 40 10 4d 64 58 70 76 05 30 36 ac 01 a8 73 49 48 4a 6c 63 47 39 79 64 47 56 6b 62 48 6b 67 62 32 34 67 54 6d 39 32 49 44 6b 67 5a 48 56 79 61 57 35 6e 49 43 34 75 4c 69 45 64 60 74 5a 58 52 68 49 47 35 68 62 57 55 39 49 6d 74 6c 65 58 64 76 63 6d 52 7a 3a bc 01 0c 74 59 58 4a 21 6c 2c 74 5a 53 77 67 62 6d 56 33 63 79 77 49 54 1c 48 42 70 62 6d 63 73 49 41 2c 3c 48 56 7a 64 48 4a 35 4c 43 42 68 59 32 4e 70 5a 21 30 00 43 01 28 08 56 6a 64 01 40 08 48 6b 73 2d dc 44 68 61 57 35 6c 63 69 77 67 64 47 46 75 61 32 56 79 4c 21 14 18 6d 5a 7a 61 47 39 79 01 64 4c 59 33 4a 31 61 58 4e 6c 4c 43 42 30 62 33 56 79 61 58 4e 74 4e 38 02 2c 59 32 39 77 65 58 4a 70 5a 32 68 30 3a b4 00 24 4b 51 55 74 50 56 45 45 67 51 0d 4c 40 49 46 4e 35 63 "}}
      {"t":{"$date":"2022-11-23T11:52:57.504+08:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"WiredTiger error","attr":{"error":0,"message":"[1669175577:504985][10117:0x7fdab21e1700], file:csnews_task/collection-53-3002926054102476948.wt, WT_CURSOR.next: __wt_bm_corrupt_dump, 140: {0: 6178557952, 36864, 0x907357ff}: (chunk 2 of 36): 33 52 6c 62 58 4d 67 52 32 31 69 53 41 ec 20 6d 39 7a 64 47 39 6a 61 79 4a 10 01 2c 6e 6c 68 62 6d 52 6c 65 43 31 32 5a 01 64 08 6d 6c 6a 41 dc 08 62 32 34 36 a4 02 54 6a 56 68 4d 57 4a 6c 4d 54 6c 68 5a 6a 64 6b 4e 54 67 7a 4f 54 63 4e 58 02 60 69 59 57 6c 6b 64 53 31 7a 61 58 52 6c 4c 58 5a 6c 63 6d 6c 6d 61 57 4e 68 4e 68 02 50 53 6e 42 51 62 46 41 31 63 55 74 35 4d 6b 51 78 51 57 56 79 52 69 f4 3a b8 01 40 31 7a 64 6d 46 73 61 57 52 68 64 47 55 75 4d 44 45 3a a0 00 b0 55 32 4d 55 4e 46 4e 44 45 34 52 44 41 35 4e 44 4a 43 4d 45 46 44 4d 7a 6b 32 4e 7a 67 32 4d 54 4d 30 51 30 4a 47 4d 6a 68 46 49 6a 34 75 e4 71 9c 14 63 6d 39 69 62 33 42 14 02 00 70 29 1c 1c 78 6d 62 32 78 73 62 33 0d f0 48 47 6c 75 61 79 42 79 5a 57 77 39 49 6d 4e 68 62 6d 39 75 01 e0 2c 62 43 49 67 61 48 4a 6c 5a 6a 30 69 85 3c f0 3e 48 4d 36 4c 79 39 33 64 33 63 75 5a 6d 78 6c 5a 58 52 74 62 32 34 75 59 32 39 74 4c 32 31 68 63 6d 6c 30 61 57 31 6c 4c 57 35 6c 64 33 4d 76 4d 6a 41 78 4e 79 38 79 4d 44 51 30 4d 79 39 6a 69 78 04 31 7a 61 78 90 4c 57 74 70 64 48 52 35 4c 57 46 6e 63 6d 39 31 62 6d 51 74 62 47 56 68 61 32 6c 75 5a 79 31 76 61 57 77 74 63 36 28 03 00 76 2e 4c 02 2c 63 48 4a 76 63 47 56 79 64 48 6b 39 a5 98 10 6e 52 70 64 47 42 c0 04 00 44 09 74 3a ec 03 00 47 15 74 1c 73 49 47 78 6c 59 57 74 61 10 40 67 62 32 6c 73 4c 43 42 51 61 47 6c 73 61 58 42 77 45 f0 00 79 32 6c 04 70 77 63 6d 39 77 5a 58 4a 30 65 54 30 69 62 32 63 36 64 58 42 6b 59 58 52 6c 5a 46 39 30 01 fc 36 d0 02 24 49 79 4d 44 45 33 4c 54 45 78 01 04 4c 56 44 45 30 4f 6a 55 31 4f 6a 51 79 4b 7a 41 77 4f 6a 41 77 72 d4 00 38 6d 52 6c 63 32 4e 79 61 58 42 30 61 57 39 75 36 5c 00 08 4a 48 5a 61 90 04 6d 46 61 a0 60 68 63 6d 64 76 49 48 4e 6f 61 58 41 67 53 30 6c 55 56 46 6b 67 63 6d 46 75 1d ec 85 58 0c 64 47 68 6c 61 d0 54 59 58 4e 30 49 47 39 6d 49 45 4a 68 63 6d 4e 6c 62 47 39 75 59 53 61 b8 3c 64 75 4c 43 42 54 62 33 4a 6e 62 33 4e 76 62 69 05 f4 10 32 61 57 35 6a 61 e4 0c 63 32 39 31 01 44 05 40 14 45 78 31 65 6d 39 01 30 3a 34 01 40 77 67 63 6d 56 77 62 33 4a 30 5a 57 52 73 65 53 42 01 48 20 4f 62 33 59 67 4f 53 42 6b 81 58 1c 62 6d 63 67 4c 69 34 75 76 00 01 10 6c 74 59 57 64 3a 54 01 c8 4a 6f 64 48 52 77 63 7a 6f 76 4c 33 64 33 64 79 35 6d 62 47 56 6c 64 47 31 76 62 69 35 6a 62 32 30 76 62 57 56 6b 61 57 45 76 59 32 46 6a 61 47 55 76 62 81 fc f0 52 33 4a 76 62 32 31 66 59 58 4a 30 61 57 4e 73 5a 56 39 70 62 57 46 6e 5a 58 4d 76 61 32 6c 30 64 48 6b 76 59 6d 49 78 5a 57 45 34 5a 6a 42 6c 59 6a 4e 69 4e 7a 59 31 4e 47 55 31 4f 44 56 6c 4e 44 63 79 4d 57 45 30 4d 32 59 78 59 54 41 75 61 6e 42 6e 66 bc 00 2c 5a 69 4f 6d 46 77 63 46 39 70 5a 43 36 88 06 50 4d 54 59 31 4d 6a 41 31 4d 6a 49 77 4f 44 4d 31 4e 6a 49 33 4d 52 c8 04 28 52 33 61 58 52 30 5a 58 49 36 59 0e 10 08 3a 48 00 24 63 33 56 74 62 57 46 79 65 53 4e 04 05 19 3c 18 64 47 6c 30 62 47 55 36 5c 04 00 6b 56 30 02 2c 59 57 64 79 62 33 56 75 5a 43 77 67 79 8c 00 42 61 8c 20 73 49 46 42 6f 61 57 78 70 c9 3c 08 56 7a 49 4e d4 04 1d 78 28 5a 47 56 7a 59 33 4a 70 63 48 52 46 7c 05 28 6b 64 6c 62 6d 56 79 59 57 77 67 "}}
      .......... 
      
      {"t":{"$date":"2022-11-23T11:52:57.508+08:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"WiredTiger error","attr":{"error":-31802,"message":"[1669175577:508190][10117:0x7fdab21e1700], file:csnews_task/collection-53-3002926054102476948.wt, WT_CURSOR.next: __wt_block_read_off, 293: csnews_task/collection-53-3002926054102476948.wt: fatal read error: WT_ERROR: non-specific WiredTiger error"}}
      {"t":{"$date":"2022-11-23T11:52:57.508+08:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"WiredTiger error","attr":{"error":-31804,"message":"[1669175577:508197][10117:0x7fdab21e1700], file:csnews_task/collection-53-3002926054102476948.wt, WT_CURSOR.next: __wt_block_read_off, 293: the process must exit and restart: WT_PANIC: WiredTiger library panic"}}
      {"t":{"$date":"2022-11-23T11:52:57.508+08:00"},"s":"F",  "c":"-",        "id":23089,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"Fatal assertion","attr":{"msgid":50853,"file":"src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp","line":521}}
      {"t":{"$date":"2022-11-23T11:52:57.508+08:00"},"s":"F",  "c":"-",        "id":23090,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"\n\n***aborting after fassert() failure\n\n"}
      {"t":{"$date":"2022-11-23T11:52:57.508+08:00"},"s":"F",  "c":"CONTROL",  "id":4757800, "ctx":"IndexBuildsCoordinatorMongod-0","msg":"Writing fatal message","attr":{"message":"Got signal: 6 (Aborted).\n"}}
      {"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31431,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"BACKTRACE: {bt}","attr":{"bt":{"backtrace":[{"a":"562D2A6072FA","b":"562D2780E000","o":"2DF92FA","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.606","s+":"1EA"},{"a":"562D2A608D89","b":"562D2780E000","o":"2DFAD89","s":"_ZN5mongo15printStackTraceEv","s+":"29"},{"a":"562D2A606116","b":"562D2780E000","o":"2DF8116","s":"_ZN5mongo12_GLOBAL__N_116abruptQuitActionEiP9siginfo_tPv","s+":"66"},{"a":"7FDABF148630","b":"7FDABF139000","o":"F630","s":"_L_unlock_13","s+":"34"},{"a":"7FDABEDA1387","b":"7FDABED6B000","o":"36387","s":"gsignal","s+":"37"},{"a":"7FDABEDA2A78","b":"7FDABED6B000","o":"37A78","s":"abort","s+":"148"},{"a":"562D2875A864","b":"562D2780E000","o":"F4C864","s":"_ZN5mongo25fassertFailedWithLocationEiPKcj","s+":"12B"},{"a":"562D2842F693","b":"562D2780E000","o":"C21693","s":"_ZN5mongo12_GLOBAL__N_141mdb_handle_error_with_startup_suppressionEP18__wt_event_handlerP12__wt_sessioniPKc.cold.1095","s+":"16"},{"a":"562D289372CB","b":"562D2780E000","o":"11292CB","s":"__eventv","s+":"3FB"},{"a":"562D28441A63","b":"562D2780E000","o":"C33A63","s":"__wt_panic_func","s+":"10C"},{"a":"562D2844F037","b":"562D2780E000","o":"C41037","s":"__wt_block_read_off.cold.5","s+":"85"},{"a":"562D28A5C86E","b":"562D2780E000","o":"124E86E","s":"__wt_bm_read","s+":"16E"},{"a":"562D2898F8D2","b":"562D2780E000","o":"11818D2","s":"__wt_bt_read","s+":"92"},{"a":"562D2899F924","b":"562D2780E000","o":"1191924","s":"__page_read","s+":"164"},{"a":"562D289A0F3E","b":"562D2780E000","o":"1192F3E","s":"__wt_page_in_func","s+":"3FE"},{"a":"562D289D1BA2","b":"562D2780E000","o":"11C3BA2","s":"__tree_walk_internal","s+":"332"},{"a":"562D28972330","b":"562D2780E000","o":"1164330","s":"__wt_btcur_next_prefix","s+":"DB0"},{"a":"562D288C714B","b":"562D2780E000","o":"10B914B","s":"__curfile_next","s+":"18B"},{"a":"562D2887F684","b":"562D2780E000","o":"1071684","s":"_ZN5mongo31WiredTigerRecordStoreCursorBase4nextEv","s+":"344"},{"a":"562D29289B91","b":"562D2780E000","o":"1A7BB91","s":"_ZN5mongo14CollectionScan6doWorkEPm","s+":"71"},{"a":"562D292A9154","b":"562D2780E000","o":"1A9B154","s":"_ZN5mongo9PlanStage4workEPm","s+":"64"},{"a":"562D292F1620","b":"562D2780E000","o":"1AE3620","s":"_ZN5mongo16PlanExecutorImpl12_getNextImplEPNS_11SnapshottedINS_8DocumentEEEPNS_8RecordIdE","s+":"230"},{"a":"562D292F230D","b":"562D2780E000","o":"1AE430D","s":"_ZN5mongo16PlanExecutorImpl18getNextSnapshottedEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE","s+":"5D"},{"a":"562D2924377A","b":"562D2780E000","o":"1A3577A","s":"_ZN5mongo15MultiIndexBlock17_doCollectionScanEPNS_16OperationContextEPNS_10CollectionEPNS_19ProgressMeterHolderE","s+":"3CA"},{"a":"562D292441D7","b":"562D2780E000","o":"1A361D7","s":"_ZN5mongo15MultiIndexBlock30insertAllDocumentsInCollectionEPNS_16OperationContextEPNS_10CollectionE","s+":"227"},{"a":"562D29232FA3","b":"562D2780E000","o":"1A24FA3","s":"_ZN5mongo18IndexBuildsManager18startBuildingIndexEPNS_16OperationContextEPNS_10CollectionERKNS_4UUIDE","s+":"53"},{"a":"562D2921C9B1","b":"562D2780E000","o":"1A0E9B1","s":"_ZN5mongo22IndexBuildsCoordinator38_scanCollectionAndInsertKeysIntoSorterEPNS_16OperationContextESt10shared_ptrINS_19ReplIndexBuildStateEE","s+":"1F1"},{"a":"562D2921CBDA","b":"562D2780E000","o":"1A0EBDA","s":"_ZN5mongo22IndexBuildsCoordinator11_buildIndexEPNS_16OperationContextESt10shared_ptrINS_19ReplIndexBuildStateEERKNS0_17IndexBuildOptionsE","s+":"6A"},{"a":"562D2922A749","b":"562D2780E000","o":"1A1C749","s":"_ZN5mongo22IndexBuildsCoordinator19_runIndexBuildInnerEPNS_16OperationContextESt10shared_ptrINS_19ReplIndexBuildStateEERKNS0_17IndexBuildOptionsE","s+":"89"},{"a":"562D2922AFE3","b":"562D2780E000","o":"1A1CFE3","s":"_ZN5mongo22IndexBuildsCoordinator14_runIndexBuildEPNS_16OperationContextERKNS_4UUIDERKNS0_17IndexBuildOptionsE","s+":"2C3"},{"a":"562D290253A7","b":"562D2780E000","o":"18173A7","s":"_ZZN5mongo28IndexBuildsCoordinatorMongod15startIndexBuildEPNS_16OperationContextENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_4UUIDERKSt6vectorINS_7BSONObjESaISB_EERKS9_NS_18IndexBuildProtocolENS_22IndexBuildsCoordinator17IndexBuildOptionsEENUlT_E3_clINS_6StatusEEEDaSL_","s+":"317"},{"a":"562D290255E2","b":"562D2780E000","o":"18175E2","s":"_ZZN5mongo15unique_functionIFvNS_6StatusEEE8makeImplIZNS_28IndexBuildsCoordinatorMongod15startIndexBuildEPNS_16OperationContextENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_4UUIDERKSt6vectorINS_7BSONObjESaISG_EERKSE_NS_18IndexBuildProtocolENS_22IndexBuildsCoordinator17IndexBuildOptionsEEUlT_E3_EEDaOSQ_EN12SpecificImpl4callEOS1_","s+":"32"},{"a":"562D2A124B62","b":"562D2780E000","o":"2916B62","s":"_ZN5mongo10ThreadPool10_doOneTaskEPSt11unique_lockINS_12latch_detail5LatchEE","s+":"132"},{"a":"562D2A1271A6","b":"562D2780E000","o":"29191A6","s":"_ZN5mongo10ThreadPool13_consumeTasksEv","s+":"86"},{"a":"562D2A127F51","b":"562D2780E000","o":"2919F51","s":"_ZN5mongo10ThreadPool17_workerThreadBodyEPS0_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE","s+":"E1"},{"a":"562D2A128280","b":"562D2780E000","o":"291A280","s":"_ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN5mongo4stdx6threadC4IZNS3_10ThreadPool25_startWorkerThread_inlockEvEUlvE2_JELi0EEET_DpOT0_EUlvE_EEEEE6_M_runEv","s+":"60"},{"a":"562D2A7B1C5F","b":"562D2780E000","o":"2FA3C5F","s":"execute_native_thread_routine","s+":"F"},{"a":"7FDABF140EA5","b":"7FDABF139000","o":"7EA5","s":"start_thread","s+":"C5"},{"a":"7FDABEE69B0D","b":"7FDABED6B000","o":"FEB0D","s":"clone","s+":"6D"}],"processInfo":{"mongodbVersion":"4.4.13","gitVersion":"df25c71b8674a78e17468f48bcda5285decb9246","compiledModules":[],"uname":{"sysname":"Linux","release":"3.10.0-1160.76.1.el7.x86_64","version":"#1 SMP Tue Jul 26 14:15:37 UTC 2022","machine":"x86_64"},"somap":[{"b":"562D2780E000","elfType":3,"buildId":"9131759DC3F0A25B4BEE192B57FA82A6EDACDBD4"},{"b":"7FDABF139000","path":"/lib64/libpthread.so.0","elfType":3,"buildId":"E10CC8F2B932FC3DAEDA22F8DAC5EBB969524E5B"},{"b":"7FDABED6B000","path":"/lib64/libc.so.6","elfType":3,"buildId":"FC4FA58E47A5ACC137EADB7689BCE4357C557A96"}]}}}}
      {"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D2A6072FA","b":"562D2780E000","o":"2DF92FA","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.606","s+":"1EA"}}}
      {"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D2A608D89","b":"562D2780E000","o":"2DFAD89","s":"_ZN5mongo15printStackTraceEv","s+":"29"}}}
      {"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D2A606116","b":"562D2780E000","o":"2DF8116","s":"_ZN5mongo12_GLOBAL__N_116abruptQuitActionEiP9siginfo_tPv","s+":"66"}}}
      {"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"7FDABF148630","b":"7FDABF139000","o":"F630","s":"_L_unlock_13","s+":"34"}}}
      {"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"7FDABEDA1387","b":"7FDABED6B000","o":"36387","s":"gsignal","s+":"37"}}}
      {"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"7FDABEDA2A78","b":"7FDABED6B000","o":"37A78","s":"abort","s+":"148"}}}
      {"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D2875A864","b":"562D2780E000","o":"F4C864","s":"_ZN5mongo25fassertFailedWithLocationEiPKcj","s+":"12B"}}}
      {"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D2842F693","b":"562D2780E000","o":"C21693","s":"_ZN5mongo12_GLOBAL__N_141mdb_handle_error_with_startup_suppressionEP18__wt_event_handlerP12__wt_sessioniPKc.cold.1095","s+":"16"}}}
      {"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D289372CB","b":"562D2780E000","o":"11292CB","s":"__eventv","s+":"3FB"}}}
      {"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D28441A63","b":"562D2780E000","o":"C33A63","s":"__wt_panic_func","s+":"10C"}}}
      {"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D2844F037","b":"562D2780E000","o":"C41037","s":"__wt_block_read_off.cold.5","s+":"85"}}}
      {"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D28A5C86E","b":"562D2780E000","o":"124E86E","s":"__wt_bm_read","s+":"16E"}}}
      {"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D2898F8D2","b":"562D2780E000","o":"11818D2","s":"__wt_bt_read","s+":"92"}}}
      {"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D2899F924","b":"562D2780E000","o":"1191924","s":"__page_read","s+":"164"}}}
      {"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D289A0F3E","b":"562D2780E000","o":"1192F3E","s":"__wt_page_in_func","s+":"3FE"}}}
      {"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D289D1BA2","b":"562D2780E000","o":"11C3BA2","s":"__tree_walk_internal","s+":"332"}}}
      {"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D28972330","b":"562D2780E000","o":"1164330","s":"__wt_btcur_next_prefix","s+":"DB0"}}}
      {"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D288C714B","b":"562D2780E000","o":"10B914B","s":"__curfile_next","s+":"18B"}}}
      {"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D2887F684","b":"562D2780E000","o":"1071684","s":"_ZN5mongo31WiredTigerRecordStoreCursorBase4nextEv","s+":"344"}}}
      {"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D29289B91","b":"562D2780E000","o":"1A7BB91","s":"_ZN5mongo14CollectionScan6doWorkEPm","s+":"71"}}}
      {"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D292A9154","b":"562D2780E000","o":"1A9B154","s":"_ZN5mongo9PlanStage4workEPm","s+":"64"}}}
      {"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D292F1620","b":"562D2780E000","o":"1AE3620","s":"_ZN5mongo16PlanExecutorImpl12_getNextImplEPNS_11SnapshottedINS_8DocumentEEEPNS_8RecordIdE","s+":"230"}}}
      {"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D292F230D","b":"562D2780E000","o":"1AE430D","s":"_ZN5mongo16PlanExecutorImpl18getNextSnapshottedEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE","s+":"5D"}}}
      {"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D2924377A","b":"562D2780E000","o":"1A3577A","s":"_ZN5mongo15MultiIndexBlock17_doCollectionScanEPNS_16OperationContextEPNS_10CollectionEPNS_19ProgressMeterHolderE","s+":"3CA"}}}
      {"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D292441D7","b":"562D2780E000","o":"1A361D7","s":"_ZN5mongo15MultiIndexBlock30insertAllDocumentsInCollectionEPNS_16OperationContextEPNS_10CollectionE","s+":"227"}}}
      {"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D29232FA3","b":"562D2780E000","o":"1A24FA3","s":"_ZN5mongo18IndexBuildsManager18startBuildingIndexEPNS_16OperationContextEPNS_10CollectionERKNS_4UUIDE","s+":"53"}}}
      {"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D2921C9B1","b":"562D2780E000","o":"1A0E9B1","s":"_ZN5mongo22IndexBuildsCoordinator38_scanCollectionAndInsertKeysIntoSorterEPNS_16OperationContextESt10shared_ptrINS_19ReplIndexBuildStateEE","s+":"1F1"}}}
      {"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D2921CBDA","b":"562D2780E000","o":"1A0EBDA","s":"_ZN5mongo22IndexBuildsCoordinator11_buildIndexEPNS_16OperationContextESt10shared_ptrINS_19ReplIndexBuildStateEERKNS0_17IndexBuildOptionsE","s+":"6A"}}}
      {"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D2922A749","b":"562D2780E000","o":"1A1C749","s":"_ZN5mongo22IndexBuildsCoordinator19_runIndexBuildInnerEPNS_16OperationContextESt10shared_ptrINS_19ReplIndexBuildStateEERKNS0_17IndexBuildOptionsE","s+":"89"}}}
      {"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D2922AFE3","b":"562D2780E000","o":"1A1CFE3","s":"_ZN5mongo22IndexBuildsCoordinator14_runIndexBuildEPNS_16OperationContextERKNS_4UUIDERKNS0_17IndexBuildOptionsE","s+":"2C3"}}}
      {"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D290253A7","b":"562D2780E000","o":"18173A7","s":"_ZZN5mongo28IndexBuildsCoordinatorMongod15startIndexBuildEPNS_16OperationContextENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_4UUIDERKSt6vectorINS_7BSONObjESaISB_EERKS9_NS_18IndexBuildProtocolENS_22IndexBuildsCoordinator17IndexBuildOptionsEENUlT_E3_clINS_6StatusEEEDaSL_","s+":"317"}}}
      {"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D290255E2","b":"562D2780E000","o":"18175E2","s":"_ZZN5mongo15unique_functionIFvNS_6StatusEEE8makeImplIZNS_28IndexBuildsCoordinatorMongod15startIndexBuildEPNS_16OperationContextENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_4UUIDERKSt6vectorINS_7BSONObjESaISG_EERKSE_NS_18IndexBuildProtocolENS_22IndexBuildsCoordinator17IndexBuildOptionsEEUlT_E3_EEDaOSQ_EN12SpecificImpl4callEOS1_","s+":"32"}}}
      {"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D2A124B62","b":"562D2780E000","o":"2916B62","s":"_ZN5mongo10ThreadPool10_doOneTaskEPSt11unique_lockINS_12latch_detail5LatchEE","s+":"132"}}}
      {"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D2A1271A6","b":"562D2780E000","o":"29191A6","s":"_ZN5mongo10ThreadPool13_consumeTasksEv","s+":"86"}}}
      {"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D2A127F51","b":"562D2780E000","o":"2919F51","s":"_ZN5mongo10ThreadPool17_workerThreadBodyEPS0_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE","s+":"E1"}}}
      {"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D2A128280","b":"562D2780E000","o":"291A280","s":"_ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN5mongo4stdx6threadC4IZNS3_10ThreadPool25_startWorkerThread_inlockEvEUlvE2_JELi0EEET_DpOT0_EUlvE_EEEEE6_M_runEv","s+":"60"}}}
      {"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D2A7B1C5F","b":"562D2780E000","o":"2FA3C5F","s":"execute_native_thread_routine","s+":"F"}}}
      {"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"7FDABF140EA5","b":"7FDABF139000","o":"7EA5","s":"start_thread","s+":"C5"}}}
      {"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"7FDABEE69B0D","b":"7FDABED6B000","o":"FEB0D","s":"clone","s+":"6D"}}}
      

       

       

       

        1. DB_error.log
          19.33 MB
        2. DB_error2.log
          7.88 MB
        3. DB_error3.log
          600 kB

            Assignee:
            chris.kelly@mongodb.com Chris Kelly
            Reporter:
            1102091578@qq.com harz wang
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: