[SERVER-71571] build index fail , DB crash ,can not startup DB : collection-53-3002926054102476948.wt: read checksum error for 36864B block at offset 6178557952: block header checksum of 0x1ee9e8e2 doesn't match expected checksum of 0x907357ff Created: 23/Nov/22  Updated: 01/Dec/22  Resolved: 01/Dec/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 4.4.13
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: harz wang Assignee: Chris Kelly
Resolution: Done Votes: 0
Labels: Bug
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File DB_error.log     Text File DB_error2.log     Text File DB_error3.log    
Issue Links:
Problem/Incident
is caused by SERVER-62873 Startup repair does not handle orphan... Closed
Related
Operating System: ALL
Participants:

 Description   

Problem Statement/Rationale

Hello there, can someone help me?  I'm running the build  TTL in the background, but it seems the Mongo replica set crashed. and then I'm trying to start up all replica set members, but the startup fails. and try to start up standalone, but still failing.

db.rawdata.createIndex({"time":1},{background:true, expireAfterSeconds:1209600});

Steps to Reproduce

  1. The MongoDB replica set crashed while building the TTL index.
  1. try to restart DB but fail.

Could you tell me :

  • How to resume the Mongo replica set?

 

Thanks you inadvance.

Actual Results

 

{"t":{"$date":"2022-11-23T11:52:57.003+08:00"},"s":"I",  "c":"-",        "id":51773,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"progress meter","attr":{"name":"Index Build: scanning collection","done":153600,"total":390354,"percent":39}}
{"t":{"$date":"2022-11-23T11:52:57.249+08:00"},"s":"I",  "c":"REPL_HB",  "id":23974,   "ctx":"ReplCoord-3","msg":"Heartbeat failed after max retries","attr":{"target":"ppnewsmongo02:27017","maxHeartbeatRetries":2,"error":{"code":6,"codeName":"HostUnreachable","errmsg":"Error connecting to ppnewsmongo02:27017 (172.16.25.41:27017) :: caused by :: Connection refused"}}}
{"t":{"$date":"2022-11-23T11:52:57.504+08:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"WiredTiger error","attr":{"error":0,"message":"[1669175577:504738][10117:0x7fdab21e1700], file:csnews_task/collection-53-3002926054102476948.wt, WT_CURSOR.next: __wt_block_read_off, 284: csnews_task/collection-53-3002926054102476948.wt: read checksum error for 36864B block at offset 6178557952: block header checksum of 0x1ee9e8e2 doesn't match expected checksum of 0x907357ff"}}
{"t":{"$date":"2022-11-23T11:52:57.504+08:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"WiredTiger error","attr":{"error":0,"message":"[1669175577:504885][10117:0x7fdab21e1700], file:csnews_task/collection-53-3002926054102476948.wt, WT_CURSOR.next: __wt_bm_corrupt_dump, 140: {0: 6178557952, 36864, 0x907357ff}: (chunk 1 of 36): 00 00 00 00 00 00 00 00 ba 1c 02 00 00 00 00 00 d2 e0 00 00 02 00 00 00 07 05 00 00 00 80 00 00 e2 e8 e9 1e 01 00 00 00 11 e3 02 49 14 80 e2 c0 21 a1 e0 00 00 07 5f 69 64 00 5a 14 09 4d 2a 8b 8c 71 00 00 00 00 00 00 92 c1 03 f0 f4 b9 00 0b b1 69 9c 02 74 61 73 6b 69 64 00 25 00 00 00 36 65 34 36 32 61 62 32 2d 64 33 31 61 2d 34 38 30 64 2d 62 35 63 64 2d 66 30 34 63 32 63 66 34 64 37 63 37 00 10 73 65 71 75 65 6e 63 65 00 1e 00 00 00 09 74 69 6d 65 00 94 55 44 de 5f 01 00 00 02 72 61 77 68 74 6d 6c 00 31 e0 00 00 50 43 46 45 54 30 4e 55 57 56 42 46 49 47 68 30 62 57 77 2b 49 44 78 6f 64 47 31 73 49 47 78 68 62 6d 63 39 49 6d 56 75 49 69 42 77 63 6d 56 6d 61 58 67 39 49 6d 39 6e 4f 69 42 6f 64 48 52 77 4f 69 38 76 62 32 64 77 4c 6d 31 6c 4c 32 35 7a 49 79 49 2b 49 44 78 6f 5a 57 46 6b 50 69 41 38 62 57 56 30 59 53 42 6a 61 47 46 79 63 32 56 30 50 53 4a 31 64 47 59 74 4f 43 49 2b 49 44 78 30 61 58 52 73 5a 53 42 70 5a 44 30 69 63 47 46 6e 5a 53 31 30 61 05 14 f0 4a 49 2b 51 32 46 79 5a 32 38 67 63 32 68 70 63 43 42 4c 53 56 52 55 57 53 42 68 5a 33 4a 76 64 57 35 6b 4c 43 42 73 5a 57 46 72 61 57 35 6e 49 47 39 70 62 43 77 67 55 47 68 70 62 47 6c 77 63 47 6c 75 5a 58 4d 67 50 43 39 30 61 01 50 f0 48 54 34 67 50 47 31 6c 64 47 45 67 61 48 52 30 63 43 31 6c 63 58 56 70 64 6a 30 69 57 43 31 56 51 53 31 44 62 32 31 77 59 58 52 70 59 6d 78 6c 49 69 42 6a 62 32 35 30 5a 57 35 30 50 53 4a 4a 52 54 31 6c 5a 47 64 6c 49 6a 19 48 f0 7b 62 6d 46 74 5a 54 30 69 64 6d 6c 6c 64 33 42 76 63 6e 51 69 49 47 4e 76 62 6e 52 6c 62 6e 51 39 49 6e 64 70 5a 48 52 6f 50 57 52 6c 64 6d 6c 6a 5a 53 31 33 61 57 52 30 61 43 77 67 61 57 35 70 64 47 6c 68 62 43 31 7a 59 32 46 73 5a 54 30 78 4c 6a 41 73 49 47 31 68 65 47 6c 74 64 57 30 74 63 32 4e 68 62 47 55 39 4d 53 34 77 4c 43 42 31 63 32 56 79 4c 58 4e 6a 59 57 78 68 01 a4 14 50 54 41 69 50 69 35 64 f0 4c 75 59 57 31 6c 50 53 4a 6b 5a 58 4e 6a 63 6d 6c 77 64 47 6c 76 62 69 49 67 59 32 39 75 64 47 56 75 64 44 30 69 52 32 56 75 5a 58 4a 68 62 43 42 6a 59 58 4a 6e 62 79 42 7a 61 47 6c 77 49 45 74 4a 56 46 52 5a 49 48 4a 68 62 69 42 68 31 64 24 49 47 39 75 49 48 52 6f 5a 53 21 10 f0 40 46 7a 64 43 42 76 5a 69 42 43 59 58 4a 6a 5a 57 78 76 62 6d 45 67 64 47 39 33 62 69 77 67 55 32 39 79 5a 32 39 7a 62 32 34 67 63 48 4a 76 64 6d 6c 75 59 32 55 73 49 48 4e 76 64 58 52 6f 5a 57 46 01 40 10 4d 64 58 70 76 05 30 36 ac 01 a8 73 49 48 4a 6c 63 47 39 79 64 47 56 6b 62 48 6b 67 62 32 34 67 54 6d 39 32 49 44 6b 67 5a 48 56 79 61 57 35 6e 49 43 34 75 4c 69 45 64 60 74 5a 58 52 68 49 47 35 68 62 57 55 39 49 6d 74 6c 65 58 64 76 63 6d 52 7a 3a bc 01 0c 74 59 58 4a 21 6c 2c 74 5a 53 77 67 62 6d 56 33 63 79 77 49 54 1c 48 42 70 62 6d 63 73 49 41 2c 3c 48 56 7a 64 48 4a 35 4c 43 42 68 59 32 4e 70 5a 21 30 00 43 01 28 08 56 6a 64 01 40 08 48 6b 73 2d dc 44 68 61 57 35 6c 63 69 77 67 64 47 46 75 61 32 56 79 4c 21 14 18 6d 5a 7a 61 47 39 79 01 64 4c 59 33 4a 31 61 58 4e 6c 4c 43 42 30 62 33 56 79 61 58 4e 74 4e 38 02 2c 59 32 39 77 65 58 4a 70 5a 32 68 30 3a b4 00 24 4b 51 55 74 50 56 45 45 67 51 0d 4c 40 49 46 4e 35 63 "}}
{"t":{"$date":"2022-11-23T11:52:57.504+08:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"WiredTiger error","attr":{"error":0,"message":"[1669175577:504985][10117:0x7fdab21e1700], file:csnews_task/collection-53-3002926054102476948.wt, WT_CURSOR.next: __wt_bm_corrupt_dump, 140: {0: 6178557952, 36864, 0x907357ff}: (chunk 2 of 36): 33 52 6c 62 58 4d 67 52 32 31 69 53 41 ec 20 6d 39 7a 64 47 39 6a 61 79 4a 10 01 2c 6e 6c 68 62 6d 52 6c 65 43 31 32 5a 01 64 08 6d 6c 6a 41 dc 08 62 32 34 36 a4 02 54 6a 56 68 4d 57 4a 6c 4d 54 6c 68 5a 6a 64 6b 4e 54 67 7a 4f 54 63 4e 58 02 60 69 59 57 6c 6b 64 53 31 7a 61 58 52 6c 4c 58 5a 6c 63 6d 6c 6d 61 57 4e 68 4e 68 02 50 53 6e 42 51 62 46 41 31 63 55 74 35 4d 6b 51 78 51 57 56 79 52 69 f4 3a b8 01 40 31 7a 64 6d 46 73 61 57 52 68 64 47 55 75 4d 44 45 3a a0 00 b0 55 32 4d 55 4e 46 4e 44 45 34 52 44 41 35 4e 44 4a 43 4d 45 46 44 4d 7a 6b 32 4e 7a 67 32 4d 54 4d 30 51 30 4a 47 4d 6a 68 46 49 6a 34 75 e4 71 9c 14 63 6d 39 69 62 33 42 14 02 00 70 29 1c 1c 78 6d 62 32 78 73 62 33 0d f0 48 47 6c 75 61 79 42 79 5a 57 77 39 49 6d 4e 68 62 6d 39 75 01 e0 2c 62 43 49 67 61 48 4a 6c 5a 6a 30 69 85 3c f0 3e 48 4d 36 4c 79 39 33 64 33 63 75 5a 6d 78 6c 5a 58 52 74 62 32 34 75 59 32 39 74 4c 32 31 68 63 6d 6c 30 61 57 31 6c 4c 57 35 6c 64 33 4d 76 4d 6a 41 78 4e 79 38 79 4d 44 51 30 4d 79 39 6a 69 78 04 31 7a 61 78 90 4c 57 74 70 64 48 52 35 4c 57 46 6e 63 6d 39 31 62 6d 51 74 62 47 56 68 61 32 6c 75 5a 79 31 76 61 57 77 74 63 36 28 03 00 76 2e 4c 02 2c 63 48 4a 76 63 47 56 79 64 48 6b 39 a5 98 10 6e 52 70 64 47 42 c0 04 00 44 09 74 3a ec 03 00 47 15 74 1c 73 49 47 78 6c 59 57 74 61 10 40 67 62 32 6c 73 4c 43 42 51 61 47 6c 73 61 58 42 77 45 f0 00 79 32 6c 04 70 77 63 6d 39 77 5a 58 4a 30 65 54 30 69 62 32 63 36 64 58 42 6b 59 58 52 6c 5a 46 39 30 01 fc 36 d0 02 24 49 79 4d 44 45 33 4c 54 45 78 01 04 4c 56 44 45 30 4f 6a 55 31 4f 6a 51 79 4b 7a 41 77 4f 6a 41 77 72 d4 00 38 6d 52 6c 63 32 4e 79 61 58 42 30 61 57 39 75 36 5c 00 08 4a 48 5a 61 90 04 6d 46 61 a0 60 68 63 6d 64 76 49 48 4e 6f 61 58 41 67 53 30 6c 55 56 46 6b 67 63 6d 46 75 1d ec 85 58 0c 64 47 68 6c 61 d0 54 59 58 4e 30 49 47 39 6d 49 45 4a 68 63 6d 4e 6c 62 47 39 75 59 53 61 b8 3c 64 75 4c 43 42 54 62 33 4a 6e 62 33 4e 76 62 69 05 f4 10 32 61 57 35 6a 61 e4 0c 63 32 39 31 01 44 05 40 14 45 78 31 65 6d 39 01 30 3a 34 01 40 77 67 63 6d 56 77 62 33 4a 30 5a 57 52 73 65 53 42 01 48 20 4f 62 33 59 67 4f 53 42 6b 81 58 1c 62 6d 63 67 4c 69 34 75 76 00 01 10 6c 74 59 57 64 3a 54 01 c8 4a 6f 64 48 52 77 63 7a 6f 76 4c 33 64 33 64 79 35 6d 62 47 56 6c 64 47 31 76 62 69 35 6a 62 32 30 76 62 57 56 6b 61 57 45 76 59 32 46 6a 61 47 55 76 62 81 fc f0 52 33 4a 76 62 32 31 66 59 58 4a 30 61 57 4e 73 5a 56 39 70 62 57 46 6e 5a 58 4d 76 61 32 6c 30 64 48 6b 76 59 6d 49 78 5a 57 45 34 5a 6a 42 6c 59 6a 4e 69 4e 7a 59 31 4e 47 55 31 4f 44 56 6c 4e 44 63 79 4d 57 45 30 4d 32 59 78 59 54 41 75 61 6e 42 6e 66 bc 00 2c 5a 69 4f 6d 46 77 63 46 39 70 5a 43 36 88 06 50 4d 54 59 31 4d 6a 41 31 4d 6a 49 77 4f 44 4d 31 4e 6a 49 33 4d 52 c8 04 28 52 33 61 58 52 30 5a 58 49 36 59 0e 10 08 3a 48 00 24 63 33 56 74 62 57 46 79 65 53 4e 04 05 19 3c 18 64 47 6c 30 62 47 55 36 5c 04 00 6b 56 30 02 2c 59 57 64 79 62 33 56 75 5a 43 77 67 79 8c 00 42 61 8c 20 73 49 46 42 6f 61 57 78 70 c9 3c 08 56 7a 49 4e d4 04 1d 78 28 5a 47 56 7a 59 33 4a 70 63 48 52 46 7c 05 28 6b 64 6c 62 6d 56 79 59 57 77 67 "}}
.......... 
 
{"t":{"$date":"2022-11-23T11:52:57.508+08:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"WiredTiger error","attr":{"error":-31802,"message":"[1669175577:508190][10117:0x7fdab21e1700], file:csnews_task/collection-53-3002926054102476948.wt, WT_CURSOR.next: __wt_block_read_off, 293: csnews_task/collection-53-3002926054102476948.wt: fatal read error: WT_ERROR: non-specific WiredTiger error"}}
{"t":{"$date":"2022-11-23T11:52:57.508+08:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"WiredTiger error","attr":{"error":-31804,"message":"[1669175577:508197][10117:0x7fdab21e1700], file:csnews_task/collection-53-3002926054102476948.wt, WT_CURSOR.next: __wt_block_read_off, 293: the process must exit and restart: WT_PANIC: WiredTiger library panic"}}
{"t":{"$date":"2022-11-23T11:52:57.508+08:00"},"s":"F",  "c":"-",        "id":23089,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"Fatal assertion","attr":{"msgid":50853,"file":"src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp","line":521}}
{"t":{"$date":"2022-11-23T11:52:57.508+08:00"},"s":"F",  "c":"-",        "id":23090,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"\n\n***aborting after fassert() failure\n\n"}
{"t":{"$date":"2022-11-23T11:52:57.508+08:00"},"s":"F",  "c":"CONTROL",  "id":4757800, "ctx":"IndexBuildsCoordinatorMongod-0","msg":"Writing fatal message","attr":{"message":"Got signal: 6 (Aborted).\n"}}
{"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31431,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"BACKTRACE: {bt}","attr":{"bt":{"backtrace":[{"a":"562D2A6072FA","b":"562D2780E000","o":"2DF92FA","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.606","s+":"1EA"},{"a":"562D2A608D89","b":"562D2780E000","o":"2DFAD89","s":"_ZN5mongo15printStackTraceEv","s+":"29"},{"a":"562D2A606116","b":"562D2780E000","o":"2DF8116","s":"_ZN5mongo12_GLOBAL__N_116abruptQuitActionEiP9siginfo_tPv","s+":"66"},{"a":"7FDABF148630","b":"7FDABF139000","o":"F630","s":"_L_unlock_13","s+":"34"},{"a":"7FDABEDA1387","b":"7FDABED6B000","o":"36387","s":"gsignal","s+":"37"},{"a":"7FDABEDA2A78","b":"7FDABED6B000","o":"37A78","s":"abort","s+":"148"},{"a":"562D2875A864","b":"562D2780E000","o":"F4C864","s":"_ZN5mongo25fassertFailedWithLocationEiPKcj","s+":"12B"},{"a":"562D2842F693","b":"562D2780E000","o":"C21693","s":"_ZN5mongo12_GLOBAL__N_141mdb_handle_error_with_startup_suppressionEP18__wt_event_handlerP12__wt_sessioniPKc.cold.1095","s+":"16"},{"a":"562D289372CB","b":"562D2780E000","o":"11292CB","s":"__eventv","s+":"3FB"},{"a":"562D28441A63","b":"562D2780E000","o":"C33A63","s":"__wt_panic_func","s+":"10C"},{"a":"562D2844F037","b":"562D2780E000","o":"C41037","s":"__wt_block_read_off.cold.5","s+":"85"},{"a":"562D28A5C86E","b":"562D2780E000","o":"124E86E","s":"__wt_bm_read","s+":"16E"},{"a":"562D2898F8D2","b":"562D2780E000","o":"11818D2","s":"__wt_bt_read","s+":"92"},{"a":"562D2899F924","b":"562D2780E000","o":"1191924","s":"__page_read","s+":"164"},{"a":"562D289A0F3E","b":"562D2780E000","o":"1192F3E","s":"__wt_page_in_func","s+":"3FE"},{"a":"562D289D1BA2","b":"562D2780E000","o":"11C3BA2","s":"__tree_walk_internal","s+":"332"},{"a":"562D28972330","b":"562D2780E000","o":"1164330","s":"__wt_btcur_next_prefix","s+":"DB0"},{"a":"562D288C714B","b":"562D2780E000","o":"10B914B","s":"__curfile_next","s+":"18B"},{"a":"562D2887F684","b":"562D2780E000","o":"1071684","s":"_ZN5mongo31WiredTigerRecordStoreCursorBase4nextEv","s+":"344"},{"a":"562D29289B91","b":"562D2780E000","o":"1A7BB91","s":"_ZN5mongo14CollectionScan6doWorkEPm","s+":"71"},{"a":"562D292A9154","b":"562D2780E000","o":"1A9B154","s":"_ZN5mongo9PlanStage4workEPm","s+":"64"},{"a":"562D292F1620","b":"562D2780E000","o":"1AE3620","s":"_ZN5mongo16PlanExecutorImpl12_getNextImplEPNS_11SnapshottedINS_8DocumentEEEPNS_8RecordIdE","s+":"230"},{"a":"562D292F230D","b":"562D2780E000","o":"1AE430D","s":"_ZN5mongo16PlanExecutorImpl18getNextSnapshottedEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE","s+":"5D"},{"a":"562D2924377A","b":"562D2780E000","o":"1A3577A","s":"_ZN5mongo15MultiIndexBlock17_doCollectionScanEPNS_16OperationContextEPNS_10CollectionEPNS_19ProgressMeterHolderE","s+":"3CA"},{"a":"562D292441D7","b":"562D2780E000","o":"1A361D7","s":"_ZN5mongo15MultiIndexBlock30insertAllDocumentsInCollectionEPNS_16OperationContextEPNS_10CollectionE","s+":"227"},{"a":"562D29232FA3","b":"562D2780E000","o":"1A24FA3","s":"_ZN5mongo18IndexBuildsManager18startBuildingIndexEPNS_16OperationContextEPNS_10CollectionERKNS_4UUIDE","s+":"53"},{"a":"562D2921C9B1","b":"562D2780E000","o":"1A0E9B1","s":"_ZN5mongo22IndexBuildsCoordinator38_scanCollectionAndInsertKeysIntoSorterEPNS_16OperationContextESt10shared_ptrINS_19ReplIndexBuildStateEE","s+":"1F1"},{"a":"562D2921CBDA","b":"562D2780E000","o":"1A0EBDA","s":"_ZN5mongo22IndexBuildsCoordinator11_buildIndexEPNS_16OperationContextESt10shared_ptrINS_19ReplIndexBuildStateEERKNS0_17IndexBuildOptionsE","s+":"6A"},{"a":"562D2922A749","b":"562D2780E000","o":"1A1C749","s":"_ZN5mongo22IndexBuildsCoordinator19_runIndexBuildInnerEPNS_16OperationContextESt10shared_ptrINS_19ReplIndexBuildStateEERKNS0_17IndexBuildOptionsE","s+":"89"},{"a":"562D2922AFE3","b":"562D2780E000","o":"1A1CFE3","s":"_ZN5mongo22IndexBuildsCoordinator14_runIndexBuildEPNS_16OperationContextERKNS_4UUIDERKNS0_17IndexBuildOptionsE","s+":"2C3"},{"a":"562D290253A7","b":"562D2780E000","o":"18173A7","s":"_ZZN5mongo28IndexBuildsCoordinatorMongod15startIndexBuildEPNS_16OperationContextENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_4UUIDERKSt6vectorINS_7BSONObjESaISB_EERKS9_NS_18IndexBuildProtocolENS_22IndexBuildsCoordinator17IndexBuildOptionsEENUlT_E3_clINS_6StatusEEEDaSL_","s+":"317"},{"a":"562D290255E2","b":"562D2780E000","o":"18175E2","s":"_ZZN5mongo15unique_functionIFvNS_6StatusEEE8makeImplIZNS_28IndexBuildsCoordinatorMongod15startIndexBuildEPNS_16OperationContextENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_4UUIDERKSt6vectorINS_7BSONObjESaISG_EERKSE_NS_18IndexBuildProtocolENS_22IndexBuildsCoordinator17IndexBuildOptionsEEUlT_E3_EEDaOSQ_EN12SpecificImpl4callEOS1_","s+":"32"},{"a":"562D2A124B62","b":"562D2780E000","o":"2916B62","s":"_ZN5mongo10ThreadPool10_doOneTaskEPSt11unique_lockINS_12latch_detail5LatchEE","s+":"132"},{"a":"562D2A1271A6","b":"562D2780E000","o":"29191A6","s":"_ZN5mongo10ThreadPool13_consumeTasksEv","s+":"86"},{"a":"562D2A127F51","b":"562D2780E000","o":"2919F51","s":"_ZN5mongo10ThreadPool17_workerThreadBodyEPS0_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE","s+":"E1"},{"a":"562D2A128280","b":"562D2780E000","o":"291A280","s":"_ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN5mongo4stdx6threadC4IZNS3_10ThreadPool25_startWorkerThread_inlockEvEUlvE2_JELi0EEET_DpOT0_EUlvE_EEEEE6_M_runEv","s+":"60"},{"a":"562D2A7B1C5F","b":"562D2780E000","o":"2FA3C5F","s":"execute_native_thread_routine","s+":"F"},{"a":"7FDABF140EA5","b":"7FDABF139000","o":"7EA5","s":"start_thread","s+":"C5"},{"a":"7FDABEE69B0D","b":"7FDABED6B000","o":"FEB0D","s":"clone","s+":"6D"}],"processInfo":{"mongodbVersion":"4.4.13","gitVersion":"df25c71b8674a78e17468f48bcda5285decb9246","compiledModules":[],"uname":{"sysname":"Linux","release":"3.10.0-1160.76.1.el7.x86_64","version":"#1 SMP Tue Jul 26 14:15:37 UTC 2022","machine":"x86_64"},"somap":[{"b":"562D2780E000","elfType":3,"buildId":"9131759DC3F0A25B4BEE192B57FA82A6EDACDBD4"},{"b":"7FDABF139000","path":"/lib64/libpthread.so.0","elfType":3,"buildId":"E10CC8F2B932FC3DAEDA22F8DAC5EBB969524E5B"},{"b":"7FDABED6B000","path":"/lib64/libc.so.6","elfType":3,"buildId":"FC4FA58E47A5ACC137EADB7689BCE4357C557A96"}]}}}}
{"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D2A6072FA","b":"562D2780E000","o":"2DF92FA","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.606","s+":"1EA"}}}
{"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D2A608D89","b":"562D2780E000","o":"2DFAD89","s":"_ZN5mongo15printStackTraceEv","s+":"29"}}}
{"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D2A606116","b":"562D2780E000","o":"2DF8116","s":"_ZN5mongo12_GLOBAL__N_116abruptQuitActionEiP9siginfo_tPv","s+":"66"}}}
{"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"7FDABF148630","b":"7FDABF139000","o":"F630","s":"_L_unlock_13","s+":"34"}}}
{"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"7FDABEDA1387","b":"7FDABED6B000","o":"36387","s":"gsignal","s+":"37"}}}
{"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"7FDABEDA2A78","b":"7FDABED6B000","o":"37A78","s":"abort","s+":"148"}}}
{"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D2875A864","b":"562D2780E000","o":"F4C864","s":"_ZN5mongo25fassertFailedWithLocationEiPKcj","s+":"12B"}}}
{"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D2842F693","b":"562D2780E000","o":"C21693","s":"_ZN5mongo12_GLOBAL__N_141mdb_handle_error_with_startup_suppressionEP18__wt_event_handlerP12__wt_sessioniPKc.cold.1095","s+":"16"}}}
{"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D289372CB","b":"562D2780E000","o":"11292CB","s":"__eventv","s+":"3FB"}}}
{"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D28441A63","b":"562D2780E000","o":"C33A63","s":"__wt_panic_func","s+":"10C"}}}
{"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D2844F037","b":"562D2780E000","o":"C41037","s":"__wt_block_read_off.cold.5","s+":"85"}}}
{"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D28A5C86E","b":"562D2780E000","o":"124E86E","s":"__wt_bm_read","s+":"16E"}}}
{"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D2898F8D2","b":"562D2780E000","o":"11818D2","s":"__wt_bt_read","s+":"92"}}}
{"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D2899F924","b":"562D2780E000","o":"1191924","s":"__page_read","s+":"164"}}}
{"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D289A0F3E","b":"562D2780E000","o":"1192F3E","s":"__wt_page_in_func","s+":"3FE"}}}
{"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D289D1BA2","b":"562D2780E000","o":"11C3BA2","s":"__tree_walk_internal","s+":"332"}}}
{"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D28972330","b":"562D2780E000","o":"1164330","s":"__wt_btcur_next_prefix","s+":"DB0"}}}
{"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D288C714B","b":"562D2780E000","o":"10B914B","s":"__curfile_next","s+":"18B"}}}
{"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D2887F684","b":"562D2780E000","o":"1071684","s":"_ZN5mongo31WiredTigerRecordStoreCursorBase4nextEv","s+":"344"}}}
{"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D29289B91","b":"562D2780E000","o":"1A7BB91","s":"_ZN5mongo14CollectionScan6doWorkEPm","s+":"71"}}}
{"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D292A9154","b":"562D2780E000","o":"1A9B154","s":"_ZN5mongo9PlanStage4workEPm","s+":"64"}}}
{"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D292F1620","b":"562D2780E000","o":"1AE3620","s":"_ZN5mongo16PlanExecutorImpl12_getNextImplEPNS_11SnapshottedINS_8DocumentEEEPNS_8RecordIdE","s+":"230"}}}
{"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D292F230D","b":"562D2780E000","o":"1AE430D","s":"_ZN5mongo16PlanExecutorImpl18getNextSnapshottedEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE","s+":"5D"}}}
{"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D2924377A","b":"562D2780E000","o":"1A3577A","s":"_ZN5mongo15MultiIndexBlock17_doCollectionScanEPNS_16OperationContextEPNS_10CollectionEPNS_19ProgressMeterHolderE","s+":"3CA"}}}
{"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D292441D7","b":"562D2780E000","o":"1A361D7","s":"_ZN5mongo15MultiIndexBlock30insertAllDocumentsInCollectionEPNS_16OperationContextEPNS_10CollectionE","s+":"227"}}}
{"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D29232FA3","b":"562D2780E000","o":"1A24FA3","s":"_ZN5mongo18IndexBuildsManager18startBuildingIndexEPNS_16OperationContextEPNS_10CollectionERKNS_4UUIDE","s+":"53"}}}
{"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D2921C9B1","b":"562D2780E000","o":"1A0E9B1","s":"_ZN5mongo22IndexBuildsCoordinator38_scanCollectionAndInsertKeysIntoSorterEPNS_16OperationContextESt10shared_ptrINS_19ReplIndexBuildStateEE","s+":"1F1"}}}
{"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D2921CBDA","b":"562D2780E000","o":"1A0EBDA","s":"_ZN5mongo22IndexBuildsCoordinator11_buildIndexEPNS_16OperationContextESt10shared_ptrINS_19ReplIndexBuildStateEERKNS0_17IndexBuildOptionsE","s+":"6A"}}}
{"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D2922A749","b":"562D2780E000","o":"1A1C749","s":"_ZN5mongo22IndexBuildsCoordinator19_runIndexBuildInnerEPNS_16OperationContextESt10shared_ptrINS_19ReplIndexBuildStateEERKNS0_17IndexBuildOptionsE","s+":"89"}}}
{"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D2922AFE3","b":"562D2780E000","o":"1A1CFE3","s":"_ZN5mongo22IndexBuildsCoordinator14_runIndexBuildEPNS_16OperationContextERKNS_4UUIDERKNS0_17IndexBuildOptionsE","s+":"2C3"}}}
{"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D290253A7","b":"562D2780E000","o":"18173A7","s":"_ZZN5mongo28IndexBuildsCoordinatorMongod15startIndexBuildEPNS_16OperationContextENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_4UUIDERKSt6vectorINS_7BSONObjESaISB_EERKS9_NS_18IndexBuildProtocolENS_22IndexBuildsCoordinator17IndexBuildOptionsEENUlT_E3_clINS_6StatusEEEDaSL_","s+":"317"}}}
{"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D290255E2","b":"562D2780E000","o":"18175E2","s":"_ZZN5mongo15unique_functionIFvNS_6StatusEEE8makeImplIZNS_28IndexBuildsCoordinatorMongod15startIndexBuildEPNS_16OperationContextENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_4UUIDERKSt6vectorINS_7BSONObjESaISG_EERKSE_NS_18IndexBuildProtocolENS_22IndexBuildsCoordinator17IndexBuildOptionsEEUlT_E3_EEDaOSQ_EN12SpecificImpl4callEOS1_","s+":"32"}}}
{"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D2A124B62","b":"562D2780E000","o":"2916B62","s":"_ZN5mongo10ThreadPool10_doOneTaskEPSt11unique_lockINS_12latch_detail5LatchEE","s+":"132"}}}
{"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D2A1271A6","b":"562D2780E000","o":"29191A6","s":"_ZN5mongo10ThreadPool13_consumeTasksEv","s+":"86"}}}
{"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D2A127F51","b":"562D2780E000","o":"2919F51","s":"_ZN5mongo10ThreadPool17_workerThreadBodyEPS0_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE","s+":"E1"}}}
{"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D2A128280","b":"562D2780E000","o":"291A280","s":"_ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN5mongo4stdx6threadC4IZNS3_10ThreadPool25_startWorkerThread_inlockEvEUlvE2_JELi0EEET_DpOT0_EUlvE_EEEEE6_M_runEv","s+":"60"}}}
{"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"562D2A7B1C5F","b":"562D2780E000","o":"2FA3C5F","s":"execute_native_thread_routine","s+":"F"}}}
{"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"7FDABF140EA5","b":"7FDABF139000","o":"7EA5","s":"start_thread","s+":"C5"}}}
{"t":{"$date":"2022-11-23T11:52:57.675+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"  Frame: {frame}","attr":{"frame":{"a":"7FDABEE69B0D","b":"7FDABED6B000","o":"FEB0D","s":"clone","s+":"6D"}}}

 

 

 



 Comments   
Comment by Chris Kelly [ 01/Dec/22 ]

I'm glad you were able to resolve the issue! We will continue to look into this independently as well.

I'll go ahead and close this ticket, but feel free to comment if it happens again.

Christopher

Comment by harz wang [ 30/Nov/22 ]

Hi @chris.kelly@mongodb.com ,

Thank you support and advice.

I'm trying to delete the "_repair_incomplete" file after the repair DB fails.  and then start again, but seems fail. (attach the log, if u want to check.)

Anyway, I use the backup has resumed the replica set now.

Thank you again and your help  & have a nice day.

Comment by Chris Kelly [ 28/Nov/22 ]

Harz,

Thanks for pointing that out. Try deleting the "_repair_incomplete" file in your dbpath so that startup can proceed without hitting the "incomplete repair detected". This is left over when a repair fails.

Currently, there is a deficiency in --repair that prevents correcting orphaned index catalog entries which you may be hitting (SERVER-62873). In your case this is exhibited by:

{"t":{"$date":"2022-11-23T17:05:32.190+08:00"},"s":"F", "c":"-", "id":23095, "ctx":"initandlisten","msg":"Fatal assertion","attr":{"msgid":28579,"error":"UnsupportedFormat: Unable to find metadata for table:csnews_task/index-0-857877165683864317 Index: {name: time_1, ns: csnews_task.rawdata} - version either too old or too new for this mongod.","file":"src/mongo/db/storage/wiredtiger/wiredtiger_index.cpp","line":526}}

However, once you start up again normally, this is something that should get corrected automatically on MongoDB 4.4.

Let me know if this works for you.

Regards,

Christopher

Comment by harz wang [ 28/Nov/22 ]

Hi @Chris Kelly

I'm trying to use --repair operation, but unsuccessful. 
For more information, please see the attachment.

and I am using backup to try to resume first.

Thank you & have a nice day.
 

Comment by Chris Kelly [ 25/Nov/22 ]

1102091578@qq.com,

This error message leads us to suspect some form of physical corruption. Please make a complete copy of the database's $dbpath directory to safeguard so that you can work off of the current $dbpath.

The ideal resolution is to perform a clean resync from an unaffected node.

You can also try mongod --repair using the latest patch release of your version (currently 4.4.18) of MongoDB.

In the event that a --repair operation is unsuccessful, then please also provide:

  • The logs leading up to the first occurrence of any issue
  • The logs of the repair operation.
  • The logs of any attempt to start mongod after the repair operation completed.

 

 

Generated at Thu Feb 08 06:19:23 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.