[SERVER-28331] Not able to recover Mongo DB Created: 16/Mar/17  Updated: 27/Jul/18  Resolved: 24/Mar/17

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.4.0
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: hareesh Assignee: Mark Agarunov
Resolution: Done Votes: 0
Labels: envns, rpo, rpu, trcf, wtc
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File SERVER-28331-repair.tar.gz     File WiredTiger.turtle     File WiredTiger.wt    
Operating System: Windows
Participants:

 Description   

2017-03-16T15:04:58.520+0530 I CONTROL  [main] ***** SERVER RESTARTED *****
2017-03-16T15:04:58.853+0530 I CONTROL  [initandlisten] MongoDB starting : pid=6704 port=27018 dbpath=z:\mongodb_win\data 64-bit host=DESKTOP-7E6EI2Q
2017-03-16T15:04:58.861+0530 I CONTROL  [initandlisten] targetMinOS: Windows 7/Windows Server 2008 R2
2017-03-16T15:04:58.872+0530 I CONTROL  [initandlisten] db version v3.4.0
2017-03-16T15:04:58.882+0530 I CONTROL  [initandlisten] git version: f4240c60f005be757399042dc12f6addbc3170c1
2017-03-16T15:04:58.893+0530 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1t-fips  3 May 2016
2017-03-16T15:04:58.902+0530 I CONTROL  [initandlisten] allocator: tcmalloc
2017-03-16T15:04:58.910+0530 I CONTROL  [initandlisten] modules: none
2017-03-16T15:04:58.929+0530 I CONTROL  [initandlisten] build environment:
2017-03-16T15:04:58.947+0530 I CONTROL  [initandlisten]     distmod: 2008plus-ssl
2017-03-16T15:04:58.966+0530 I CONTROL  [initandlisten]     distarch: x86_64
2017-03-16T15:04:58.985+0530 I CONTROL  [initandlisten]     target_arch: x86_64
2017-03-16T15:04:59.004+0530 I CONTROL  [initandlisten] options: { config: "z:\MongoDB_win\mongod.conf", net: { port: 27018 }, repair: true, storage: { dbPath: "z:\mongodb_win\data", journal: { enabled: false } }, systemLog: { destination: "file", logAppend: true, path: "z:\mongodb_win\mongod.log" } }
2017-03-16T15:04:59.032+0530 W -        [initandlisten] Detected unclean shutdown - z:\mongodb_win\data\mongod.lock is not empty.
2017-03-16T15:04:59.059+0530 I -        [initandlisten] Detected data files in z:\mongodb_win\data created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2017-03-16T15:04:59.077+0530 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2017-03-16T15:04:59.098+0530 I STORAGE  [initandlisten] Detected WT journal files.  Running recovery from last checkpoint.
2017-03-16T15:04:59.116+0530 I STORAGE  [initandlisten] journal to nojournal transition config: create,cache_size=1495M,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-03-16T15:04:59.227+0530 E STORAGE  [initandlisten] WiredTiger error (0) [1489656899:226693][6704:140733334836944], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 12288: block header checksum of 1735288098 doesn't match expected checksum of 3704693169
2017-03-16T15:04:59.245+0530 E STORAGE  [initandlisten] WiredTiger error (0) [1489656899:244329][6704:140733334836944], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
2017-03-16T15:04:59.263+0530 E STORAGE  [initandlisten] WiredTiger error (-31804) [1489656899:263646][6704:140733334836944], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2017-03-16T15:04:59.282+0530 I -        [initandlisten] Fatal Assertion 28558 at src\mongo\db\storage\wiredtiger\wiredtiger_util.cpp 361
2017-03-16T15:04:59.301+0530 I -        [initandlisten] 
 
***aborting after fassert() failure
 
 
2017-03-16T15:04:59.449+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\util\stacktrace_windows.cpp(239)                     mongo::printStackTrace+0x43
2017-03-16T15:04:59.459+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\util\signal_handlers_synchronous.cpp(180)            mongo::`anonymous namespace'::printSignalAndBacktrace+0x74
2017-03-16T15:04:59.468+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\util\signal_handlers_synchronous.cpp(236)            mongo::`anonymous namespace'::abruptQuit+0x85
2017-03-16T15:04:59.477+0530 I CONTROL  [initandlisten] ucrtbase.dll                                                                     raise+0x1e7
2017-03-16T15:04:59.486+0530 I CONTROL  [initandlisten] ucrtbase.dll                                                                     abort+0x31
2017-03-16T15:04:59.498+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\util\assert_util.cpp(172)                            mongo::fassertFailedWithLocation+0x181
2017-03-16T15:04:59.517+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\storage\wiredtiger\wiredtiger_util.cpp(361)       mongo::`anonymous namespace'::mdb_handle_error+0x187
2017-03-16T15:04:59.535+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\third_party\wiredtiger\src\support\err.c(291)              __wt_eventv+0x3b8
2017-03-16T15:04:59.553+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\third_party\wiredtiger\src\support\err.c(317)              __wt_err+0x32
2017-03-16T15:04:59.572+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\third_party\wiredtiger\src\support\err.c(527)              __wt_illegal_value+0x5e
2017-03-16T15:04:59.591+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\third_party\wiredtiger\src\block\block_read.c(225)         __wt_block_read_off+0x1c2
2017-03-16T15:04:59.610+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\third_party\wiredtiger\src\block\block_ext.c(1178)         __wt_block_extlist_read+0x85
2017-03-16T15:04:59.629+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\third_party\wiredtiger\src\block\block_ext.c(1141)         __wt_block_extlist_read_avail+0x24
2017-03-16T15:04:59.648+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\third_party\wiredtiger\src\block\block_ckpt.c(128)         __wt_block_checkpoint_load+0x1b1
2017-03-16T15:04:59.665+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\third_party\wiredtiger\src\block\block_mgr.c(97)           __bm_checkpoint_load+0x51
2017-03-16T15:04:59.685+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\third_party\wiredtiger\src\btree\bt_handle.c(100)          __wt_btree_open+0x1cc
2017-03-16T15:04:59.705+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\third_party\wiredtiger\src\conn\conn_dhandle.c(343)        __wt_conn_btree_open+0x90
2017-03-16T15:04:59.724+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\third_party\wiredtiger\src\session\session_dhandle.c(523)  __wt_session_get_btree+0xa2
2017-03-16T15:04:59.737+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\third_party\wiredtiger\src\session\session_dhandle.c(514)  __wt_session_get_btree+0x267
2017-03-16T15:04:59.748+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\third_party\wiredtiger\src\session\session_dhandle.c(336)  __wt_session_get_btree_ckpt+0xc4
2017-03-16T15:04:59.760+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\third_party\wiredtiger\src\cursor\cur_file.c(566)          __wt_curfile_open+0x216
2017-03-16T15:04:59.773+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\third_party\wiredtiger\src\session\session_api.c(354)      __session_open_cursor_int+0x2f7
2017-03-16T15:04:59.789+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\third_party\wiredtiger\src\session\session_api.c(409)      __wt_open_cursor+0x1b
2017-03-16T15:04:59.805+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\third_party\wiredtiger\src\meta\meta_table.c(94)           __wt_metadata_cursor+0x99
2017-03-16T15:04:59.822+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\third_party\wiredtiger\src\conn\conn_api.c(2446)           wiredtiger_open+0xb49
2017-03-16T15:04:59.839+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\storage\wiredtiger\wiredtiger_kv_engine.cpp(247)  mongo::WiredTigerKVEngine::WiredTigerKVEngine+0x75e
2017-03-16T15:04:59.854+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\storage\wiredtiger\wiredtiger_init.cpp(89)        mongo::`anonymous namespace'::WiredTigerFactory::create+0x12a
2017-03-16T15:04:59.871+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\service_context_d.cpp(202)                        mongo::ServiceContextMongoD::initializeGlobalStorageEngine+0x5cb
2017-03-16T15:04:59.888+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\db.cpp(587)                                       mongo::`anonymous namespace'::_initAndListen+0x775
2017-03-16T15:04:59.905+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\db.cpp(823)                                       mongo::`anonymous namespace'::initAndListen+0x27
2017-03-16T15:04:59.922+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\db.cpp(1173)                                      mongoDbMain+0x25d
2017-03-16T15:04:59.941+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\db.cpp(861)                                       wmain+0x35
2017-03-16T15:04:59.960+0530 I CONTROL  [initandlisten] mongod.exe    f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl(255)        __scrt_common_main_seh+0x124
2017-03-16T15:04:59.979+0530 I CONTROL  [initandlisten] KERNEL32.DLL                                                                     BaseThreadInitThunk+0x14
2017-03-16T15:04:59.998+0530 F -        [initandlisten] Got signal: 22 (SIGABRT).
2017-03-16T15:05:00.015+0530 I CONTROL  [initandlisten] *** unhandled exception 0x0000000E at 0x00007FFF053D7788, terminating
2017-03-16T15:05:00.034+0530 I CONTROL  [initandlisten] *** stack trace for unhandled exception:
2017-03-16T15:05:00.132+0530 I CONTROL  [initandlisten] KERNELBASE.dll                                                                     RaiseException+0x68
2017-03-16T15:05:00.147+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\util\signal_handlers_synchronous.cpp(237)            mongo::`anonymous namespace'::abruptQuit+0x9d
2017-03-16T15:05:00.163+0530 I CONTROL  [initandlisten] ucrtbase.dll                                                                       raise+0x1e7
2017-03-16T15:05:00.179+0530 I CONTROL  [initandlisten] ucrtbase.dll                                                                       abort+0x31
2017-03-16T15:05:00.194+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\util\assert_util.cpp(172)                            mongo::fassertFailedWithLocation+0x181
2017-03-16T15:05:00.214+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\db\storage\wiredtiger\wiredtiger_util.cpp(361)       mongo::`anonymous namespace'::mdb_handle_error+0x187
2017-03-16T15:05:00.231+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\third_party\wiredtiger\src\support\err.c(291)              __wt_eventv+0x3b8
2017-03-16T15:05:00.247+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\third_party\wiredtiger\src\support\err.c(317)              __wt_err+0x32
2017-03-16T15:05:00.266+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\third_party\wiredtiger\src\support\err.c(527)              __wt_illegal_value+0x5e
2017-03-16T15:05:00.284+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\third_party\wiredtiger\src\block\block_read.c(225)         __wt_block_read_off+0x1c2
2017-03-16T15:05:00.301+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\third_party\wiredtiger\src\block\block_ext.c(1178)         __wt_block_extlist_read+0x85
2017-03-16T15:05:00.320+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\third_party\wiredtiger\src\block\block_ext.c(1141)         __wt_block_extlist_read_avail+0x24
2017-03-16T15:05:00.339+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\third_party\wiredtiger\src\block\block_ckpt.c(128)         __wt_block_checkpoint_load+0x1b1
2017-03-16T15:05:00.357+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\third_party\wiredtiger\src\block\block_mgr.c(97)           __bm_checkpoint_load+0x51
2017-03-16T15:05:00.376+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\third_party\wiredtiger\src\btree\bt_handle.c(100)          __wt_btree_open+0x1cc
2017-03-16T15:05:00.394+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\third_party\wiredtiger\src\conn\conn_dhandle.c(343)        __wt_conn_btree_open+0x90
2017-03-16T15:05:00.413+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\third_party\wiredtiger\src\session\session_dhandle.c(523)  __wt_session_get_btree+0xa2
2017-03-16T15:05:00.432+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\third_party\wiredtiger\src\session\session_dhandle.c(514)  __wt_session_get_btree+0x267
2017-03-16T15:05:00.451+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\third_party\wiredtiger\src\session\session_dhandle.c(336)  __wt_session_get_btree_ckpt+0xc4
2017-03-16T15:05:00.469+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\third_party\wiredtiger\src\cursor\cur_file.c(566)          __wt_curfile_open+0x216
2017-03-16T15:05:00.488+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\third_party\wiredtiger\src\session\session_api.c(354)      __session_open_cursor_int+0x2f7
2017-03-16T15:05:00.506+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\third_party\wiredtiger\src\session\session_api.c(409)      __wt_open_cursor+0x1b
2017-03-16T15:05:00.525+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\third_party\wiredtiger\src\meta\meta_table.c(94)           __wt_metadata_cursor+0x99
2017-03-16T15:05:00.543+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\third_party\wiredtiger\src\conn\conn_api.c(2446)           wiredtiger_open+0xb49
2017-03-16T15:05:00.562+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\db\storage\wiredtiger\wiredtiger_kv_engine.cpp(247)  mongo::WiredTigerKVEngine::WiredTigerKVEngine+0x75e
2017-03-16T15:05:00.581+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\db\storage\wiredtiger\wiredtiger_init.cpp(89)        mongo::`anonymous namespace'::WiredTigerFactory::create+0x12a
2017-03-16T15:05:00.599+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\db\service_context_d.cpp(202)                        mongo::ServiceContextMongoD::initializeGlobalStorageEngine+0x5cb
2017-03-16T15:05:00.615+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\db\db.cpp(587)                                       mongo::`anonymous namespace'::_initAndListen+0x775
2017-03-16T15:05:00.633+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\db\db.cpp(823)                                       mongo::`anonymous namespace'::initAndListen+0x27
2017-03-16T15:05:00.652+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\db\db.cpp(1173)                                      mongoDbMain+0x25d
2017-03-16T15:05:00.670+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\db\db.cpp(861)                                       wmain+0x35
2017-03-16T15:05:00.689+0530 I CONTROL  [initandlisten] mongod.exe      f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl(255)        __scrt_common_main_seh+0x124
2017-03-16T15:05:00.708+0530 I CONTROL  [initandlisten] KERNEL32.DLL                                                                       BaseThreadInitThunk+0x14
2017-03-16T15:05:00.727+0530 I -        [initandlisten] 
2017-03-16T15:05:00.746+0530 I CONTROL  [initandlisten] writing minidump diagnostic file C:\Program Files\MongoDB\Server\3.2017-03-16T09-35-00.mdmp
2017-03-16T15:05:00.834+0530 I CONTROL  [initandlisten] *** immediate exit due to unhandled exception



 Comments   
Comment by Mark Agarunov [ 24/Mar/17 ]

Hello har33sh,

Unfortunately, this error indicates there was other corruption of the database. Beyond the previous repair attempt, my recommendation would be to preform a clean resync from a node that is not affected or to restore the data from a backup, if possible.

Thanks,
Mark

Comment by hareesh [ 16/Mar/17 ]

Facing the same issue when I killed all the active mongo processes,

2017-03-17T00:44:32.560+0530 I CONTROL  [initandlisten] MongoDB starting : pid=3380 port=27018 dbpath=z:\mongodb_win\data 64-bit host=DESKTOP-7E6EI2Q
2017-03-17T00:44:42.528+0530 I CONTROL  [initandlisten] targetMinOS: Windows 7/Windows Server 2008 R2
2017-03-17T00:44:53.351+0530 I CONTROL  [initandlisten] db version v3.4.0
2017-03-17T00:45:20.251+0530 I CONTROL  [initandlisten] git version: f4240c60f005be757399042dc12f6addbc3170c1
2017-03-17T00:45:37.699+0530 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1t-fips  3 May 2016
2017-03-17T00:45:52.479+0530 I CONTROL  [initandlisten] allocator: tcmalloc
2017-03-17T00:46:05.652+0530 I CONTROL  [initandlisten] modules: none
2017-03-17T00:46:17.860+0530 I CONTROL  [initandlisten] build environment:
2017-03-17T00:46:28.058+0530 I CONTROL  [initandlisten]     distmod: 2008plus-ssl
2017-03-17T00:46:38.580+0530 I CONTROL  [initandlisten]     distarch: x86_64
2017-03-17T00:46:49.059+0530 I CONTROL  [initandlisten]     target_arch: x86_64
2017-03-17T00:46:59.927+0530 I CONTROL  [initandlisten] options: { config: "z:\MongoDB_win\mongod.conf", net: { port: 27018 }, repair: true, storage: { dbPath: "z:\mongodb_win\data", journal: { enabled: false } }, systemLog: { destination: "file", logAppend: true, path: "z:\mongodb_win\mongod.log" } }
2017-03-17T00:47:16.520+0530 W -        [initandlisten] Detected unclean shutdown - z:\mongodb_win\data\mongod.lock is not empty.
2017-03-17T00:47:46.668+0530 I -        [initandlisten] Detected data files in z:\mongodb_win\data created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2017-03-17T00:48:01.591+0530 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2017-03-17T00:48:17.803+0530 I STORAGE  [initandlisten] Detected WT journal files.  Running recovery from last checkpoint.
2017-03-17T00:48:39.012+0530 I STORAGE  [initandlisten] journal to nojournal transition config: create,cache_size=1495M,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-03-17T00:51:30.369+0530 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1495M,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),,log=(enabled=false),
2017-03-17T00:53:39.156+0530 I STORAGE  [initandlisten] Repairing size cache
2017-03-17T00:53:56.195+0530 I STORAGE  [initandlisten] Verify succeeded on uri table:sizeStorer. Not salvaging.
2017-03-17T00:54:16.016+0530 I STORAGE  [initandlisten] Repairing catalog metadata
2017-03-17T00:54:34.883+0530 I STORAGE  [initandlisten] Verify succeeded on uri table:_mdb_catalog. Not salvaging.
2017-03-17T00:54:58.875+0530 E STORAGE  [initandlisten] WiredTiger error (0) [1489692298:875373][3380:140733334836944], file:WiredTiger.wt, WT_CURSOR.next: z:\mongodb_win\data\WiredTiger.turtle: handle-open: CreateFileW: The process cannot access the file because it is being used by another process.
2017-03-17T00:55:13.792+0530 I CONTROL  [initandlisten] 
2017-03-17T00:55:24.162+0530 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2017-03-17T00:55:31.829+0530 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2017-03-17T00:55:42.600+0530 I CONTROL  [initandlisten] 
2017-03-17T00:55:56.289+0530 I STORAGE  [initandlisten] repairDatabase ans_test
2017-03-17T00:56:06.967+0530 I STORAGE  [initandlisten] Repairing collection ans_test.power_k_seil_l
2017-03-17T00:56:18.652+0530 I STORAGE  [initandlisten] Verify failed on uri table:collection-11--4590358338625035404. Running a salvage operation.
2017-03-17T00:57:17.011+0530 I -        [initandlisten] Invariant failure rs.get() src\mongo\db\catalog\database.cpp 195
2017-03-17T00:57:27.240+0530 I -        [initandlisten] 
 
***aborting after invariant() failure
 
 
2017-03-17T00:57:37.646+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\util\stacktrace_windows.cpp(239)               mongo::printStackTrace+0x43
2017-03-17T00:57:49.337+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\util\signal_handlers_synchronous.cpp(180)      mongo::`anonymous namespace'::printSignalAndBacktrace+0x74
2017-03-17T00:58:00.654+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\util\signal_handlers_synchronous.cpp(236)      mongo::`anonymous namespace'::abruptQuit+0x85
2017-03-17T00:58:12.204+0530 I CONTROL  [initandlisten] ucrtbase.dll                                                               raise+0x1e7
2017-03-17T00:58:24.008+0530 I CONTROL  [initandlisten] ucrtbase.dll                                                               abort+0x31
2017-03-17T00:58:34.086+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\util\assert_util.cpp(154)                      mongo::invariantFailed+0x19a
2017-03-17T00:58:44.550+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\catalog\database.cpp(195)                   mongo::Database::_getOrCreateCollectionInstance+0xbf
2017-03-17T00:58:55.487+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\catalog\database.cpp(222)                   mongo::Database::Database+0x342
2017-03-17T00:59:06.963+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\catalog\database_holder.cpp(136)            mongo::DatabaseHolder::openDb+0x3cc
2017-03-17T00:59:18.248+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\repair_database.cpp(249)                    <lambda_47b2694c3219602f6dfbd244ccf4b795>::operator()+0x73
2017-03-17T00:59:30.170+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\util\scopeguard.h(99)                          mongo::ScopeGuardImplBase::SafeExecute<mongo::ScopeGuardImpl0<<lambda_47b2694c3219602f6dfbd244ccf4b795> > >+0x1b
2017-03-17T00:59:41.386+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\repair_database.cpp(291)                    mongo::repairDatabase+0x482
2017-03-17T00:59:52.787+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\db.cpp(384)                                 mongo::`anonymous namespace'::repairDatabasesAndCheckVersion+0x2d4
2017-03-17T01:00:03.990+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\db.cpp(670)                                 mongo::`anonymous namespace'::_initAndListen+0xf75
2017-03-17T01:00:14.565+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\db.cpp(823)                                 mongo::`anonymous namespace'::initAndListen+0x27
2017-03-17T01:00:25.738+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\db.cpp(1173)                                mongoDbMain+0x25d
2017-03-17T01:00:36.174+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\db.cpp(861)                                 wmain+0x35
2017-03-17T01:00:47.728+0530 I CONTROL  [initandlisten] mongod.exe    f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl(255)  __scrt_common_main_seh+0x124
2017-03-17T01:00:58.888+0530 I CONTROL  [initandlisten] KERNEL32.DLL                                                               BaseThreadInitThunk+0x14
2017-03-17T01:01:10.487+0530 F -        [initandlisten] Got signal: 22 (SIGABRT).
2017-03-17T01:01:20.957+0530 I CONTROL  [initandlisten] *** unhandled exception 0x0000000E at 0x00007FFF053D7788, terminating
2017-03-17T01:01:32.654+0530 I CONTROL  [initandlisten] *** stack trace for unhandled exception:
2017-03-17T01:01:44.126+0530 I CONTROL  [initandlisten] KERNELBASE.dll                                                               RaiseException+0x68
2017-03-17T01:01:55.114+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\util\signal_handlers_synchronous.cpp(237)      mongo::`anonymous namespace'::abruptQuit+0x9d
2017-03-17T01:02:05.529+0530 I CONTROL  [initandlisten] ucrtbase.dll                                                                 raise+0x1e7
2017-03-17T01:02:14.289+0530 I CONTROL  [initandlisten] ucrtbase.dll                                                                 abort+0x31
2017-03-17T01:02:24.876+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\util\assert_util.cpp(154)                      mongo::invariantFailed+0x19a
2017-03-17T01:02:36.378+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\db\catalog\database.cpp(195)                   mongo::Database::_getOrCreateCollectionInstance+0xbf
2017-03-17T01:02:47.877+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\db\catalog\database.cpp(222)                   mongo::Database::Database+0x342
2017-03-17T01:02:55.457+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\db\catalog\database_holder.cpp(136)            mongo::DatabaseHolder::openDb+0x3cc
2017-03-17T01:03:02.992+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\db\repair_database.cpp(249)                    <lambda_47b2694c3219602f6dfbd244ccf4b795>::operator()+0x73
2017-03-17T01:03:12.187+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\util\scopeguard.h(99)                          mongo::ScopeGuardImplBase::SafeExecute<mongo::ScopeGuardImpl0<<lambda_47b2694c3219602f6dfbd244ccf4b795> > >+0x1b
2017-03-17T01:03:23.807+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\db\repair_database.cpp(291)                    mongo::repairDatabase+0x482
2017-03-17T01:03:35.082+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\db\db.cpp(384)                                 mongo::`anonymous namespace'::repairDatabasesAndCheckVersion+0x2d4
2017-03-17T01:03:45.566+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\db\db.cpp(670)                                 mongo::`anonymous namespace'::_initAndListen+0xf75
2017-03-17T01:03:56.290+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\db\db.cpp(823)                                 mongo::`anonymous namespace'::initAndListen+0x27
2017-03-17T01:04:07.632+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\db\db.cpp(1173)                                mongoDbMain+0x25d
2017-03-17T01:04:19.104+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\db\db.cpp(861)                                 wmain+0x35
2017-03-17T01:04:30.619+0530 I CONTROL  [initandlisten] mongod.exe      f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl(255)  __scrt_common_main_seh+0x124
2017-03-17T01:04:41.845+0530 I CONTROL  [initandlisten] KERNEL32.DLL                                                                 BaseThreadInitThunk+0x14
2017-03-17T01:04:53.052+0530 I -        [initandlisten] 
2017-03-17T01:05:03.250+0530 I CONTROL  [initandlisten] writing minidump diagnostic file C:\Program Files\MongoDB\Server\3.2017-03-16T19-35-03.mdmp
2017-03-17T01:05:17.273+0530 I CONTROL  [initandlisten] *** immediate exit due to unhandled exception
2017-03-17T01:43:49.967+0530 I CONTROL  [main] ***** SERVER RESTARTED *****
2017-03-17T01:44:07.074+0530 I CONTROL  [initandlisten] MongoDB starting : pid=6628 port=27018 dbpath=z:\mongodb_win\data 64-bit host=DESKTOP-7E6EI2Q
2017-03-17T01:44:09.117+0530 I CONTROL  [initandlisten] targetMinOS: Windows 7/Windows Server 2008 R2
2017-03-17T01:44:09.133+0530 I CONTROL  [initandlisten] db version v3.4.0
2017-03-17T01:44:09.204+0530 I CONTROL  [initandlisten] git version: f4240c60f005be757399042dc12f6addbc3170c1
2017-03-17T01:44:09.218+0530 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1t-fips  3 May 2016
2017-03-17T01:44:09.226+0530 I CONTROL  [initandlisten] allocator: tcmalloc
2017-03-17T01:44:15.737+0530 I CONTROL  [initandlisten] modules: none
2017-03-17T01:44:15.752+0530 I CONTROL  [initandlisten] build environment:
2017-03-17T01:44:15.835+0530 I CONTROL  [initandlisten]     distmod: 2008plus-ssl
2017-03-17T01:44:17.686+0530 I CONTROL  [initandlisten]     distarch: x86_64
2017-03-17T01:44:17.751+0530 I CONTROL  [initandlisten]     target_arch: x86_64
2017-03-17T01:44:17.768+0530 I CONTROL  [initandlisten] options: { config: "z:\MongoDB_win\mongod.conf", net: { port: 27018 }, repair: true, storage: { dbPath: "z:\mongodb_win\data", journal: { enabled: false } }, systemLog: { destination: "file", logAppend: true, path: "z:\mongodb_win\mongod.log" } }
2017-03-17T01:44:17.801+0530 W -        [initandlisten] Detected unclean shutdown - z:\mongodb_win\data\mongod.lock is not empty.
2017-03-17T01:44:17.838+0530 I -        [initandlisten] Detected data files in z:\mongodb_win\data created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2017-03-17T01:44:18.857+0530 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2017-03-17T01:44:31.281+0530 I STORAGE  [initandlisten] Detected WT journal files.  Running recovery from last checkpoint.
2017-03-17T01:44:41.432+0530 I STORAGE  [initandlisten] journal to nojournal transition config: create,cache_size=1495M,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-03-17T01:46:08.394+0530 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1495M,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),,log=(enabled=false),
2017-03-17T01:48:00.350+0530 I STORAGE  [initandlisten] Repairing size cache
2017-03-17T01:48:00.443+0530 I STORAGE  [initandlisten] Verify succeeded on uri table:sizeStorer. Not salvaging.
2017-03-17T01:48:04.840+0530 I STORAGE  [initandlisten] Repairing catalog metadata
2017-03-17T01:48:23.718+0530 I STORAGE  [initandlisten] Verify succeeded on uri table:_mdb_catalog. Not salvaging.
2017-03-17T01:48:29.327+0530 I CONTROL  [initandlisten] 
2017-03-17T01:48:29.373+0530 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2017-03-17T01:48:29.385+0530 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2017-03-17T01:48:31.729+0530 I CONTROL  [initandlisten] 
2017-03-17T01:48:32.510+0530 I STORAGE  [initandlisten] repairDatabase ans_test
2017-03-17T01:48:32.527+0530 I STORAGE  [initandlisten] Repairing collection ans_test.power_k_seil_l
2017-03-17T01:48:32.540+0530 I STORAGE  [initandlisten] Verify failed on uri table:collection-11--4590358338625035404. Running a salvage operation.
2017-03-17T01:48:32.553+0530 I -        [initandlisten] Invariant failure rs.get() src\mongo\db\catalog\database.cpp 195
2017-03-17T01:48:32.566+0530 I -        [initandlisten] 
 
***aborting after invariant() failure
 
 
2017-03-17T01:48:32.665+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\util\stacktrace_windows.cpp(239)               mongo::printStackTrace+0x43
2017-03-17T01:48:32.675+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\util\signal_handlers_synchronous.cpp(180)      mongo::`anonymous namespace'::printSignalAndBacktrace+0x74
2017-03-17T01:48:32.687+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\util\signal_handlers_synchronous.cpp(236)      mongo::`anonymous namespace'::abruptQuit+0x85
2017-03-17T01:48:32.698+0530 I CONTROL  [initandlisten] ucrtbase.dll                                                               raise+0x1e7
2017-03-17T01:48:32.706+0530 I CONTROL  [initandlisten] ucrtbase.dll                                                               abort+0x31
2017-03-17T01:48:32.715+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\util\assert_util.cpp(154)                      mongo::invariantFailed+0x19a
2017-03-17T01:48:32.724+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\catalog\database.cpp(195)                   mongo::Database::_getOrCreateCollectionInstance+0xbf
2017-03-17T01:48:32.735+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\catalog\database.cpp(222)                   mongo::Database::Database+0x342
2017-03-17T01:48:32.754+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\catalog\database_holder.cpp(136)            mongo::DatabaseHolder::openDb+0x3cc
2017-03-17T01:48:32.763+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\repair_database.cpp(249)                    <lambda_47b2694c3219602f6dfbd244ccf4b795>::operator()+0x73
2017-03-17T01:48:32.772+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\util\scopeguard.h(99)                          mongo::ScopeGuardImplBase::SafeExecute<mongo::ScopeGuardImpl0<<lambda_47b2694c3219602f6dfbd244ccf4b795> > >+0x1b
2017-03-17T01:48:32.785+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\repair_database.cpp(291)                    mongo::repairDatabase+0x482
2017-03-17T01:48:32.801+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\db.cpp(384)                                 mongo::`anonymous namespace'::repairDatabasesAndCheckVersion+0x2d4
2017-03-17T01:48:32.814+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\db.cpp(670)                                 mongo::`anonymous namespace'::_initAndListen+0xf75
2017-03-17T01:48:32.831+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\db.cpp(823)                                 mongo::`anonymous namespace'::initAndListen+0x27
2017-03-17T01:48:32.845+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\db.cpp(1173)                                mongoDbMain+0x25d
2017-03-17T01:48:32.862+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\db.cpp(861)                                 wmain+0x35
2017-03-17T01:48:32.884+0530 I CONTROL  [initandlisten] mongod.exe    f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl(255)  __scrt_common_main_seh+0x124
2017-03-17T01:48:32.901+0530 I CONTROL  [initandlisten] KERNEL32.DLL                                                               BaseThreadInitThunk+0x14
2017-03-17T01:48:32.919+0530 F -        [initandlisten] Got signal: 22 (SIGABRT).
2017-03-17T01:48:32.940+0530 I CONTROL  [initandlisten] *** unhandled exception 0x0000000E at 0x00007FFF053D7788, terminating
2017-03-17T01:48:32.962+0530 I CONTROL  [initandlisten] *** stack trace for unhandled exception:
2017-03-17T01:48:39.384+0530 I CONTROL  [initandlisten] KERNELBASE.dll                                                               RaiseException+0x68
2017-03-17T01:48:50.905+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\util\signal_handlers_synchronous.cpp(237)      mongo::`anonymous namespace'::abruptQuit+0x9d
2017-03-17T01:49:02.071+0530 I CONTROL  [initandlisten] ucrtbase.dll                                                                 raise+0x1e7
2017-03-17T01:49:13.344+0530 I CONTROL  [initandlisten] ucrtbase.dll                                                                 abort+0x31
2017-03-17T01:49:24.763+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\util\assert_util.cpp(154)                      mongo::invariantFailed+0x19a
2017-03-17T01:49:36.170+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\db\catalog\database.cpp(195)                   mongo::Database::_getOrCreateCollectionInstance+0xbf
2017-03-17T01:49:46.928+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\db\catalog\database.cpp(222)                   mongo::Database::Database+0x342
2017-03-17T01:49:58.314+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\db\catalog\database_holder.cpp(136)            mongo::DatabaseHolder::openDb+0x3cc
2017-03-17T01:50:09.793+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\db\repair_database.cpp(249)                    <lambda_47b2694c3219602f6dfbd244ccf4b795>::operator()+0x73
2017-03-17T01:50:21.347+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\util\scopeguard.h(99)                          mongo::ScopeGuardImplBase::SafeExecute<mongo::ScopeGuardImpl0<<lambda_47b2694c3219602f6dfbd244ccf4b795> > >+0x1b
2017-03-17T01:50:32.835+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\db\repair_database.cpp(291)                    mongo::repairDatabase+0x482
2017-03-17T01:50:33.311+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\db\db.cpp(384)                                 mongo::`anonymous namespace'::repairDatabasesAndCheckVersion+0x2d4
2017-03-17T01:50:33.345+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\db\db.cpp(670)                                 mongo::`anonymous namespace'::_initAndListen+0xf75
2017-03-17T01:50:33.357+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\db\db.cpp(823)                                 mongo::`anonymous namespace'::initAndListen+0x27
2017-03-17T01:50:33.402+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\db\db.cpp(1173)                                mongoDbMain+0x25d
2017-03-17T01:50:33.437+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\db\db.cpp(861)                                 wmain+0x35
2017-03-17T01:50:33.454+0530 I CONTROL  [initandlisten] mongod.exe      f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl(255)  __scrt_common_main_seh+0x124
2017-03-17T01:50:33.475+0530 I CONTROL  [initandlisten] KERNEL32.DLL                                                                 BaseThreadInitThunk+0x14
2017-03-17T01:50:33.491+0530 I -        [initandlisten] 
2017-03-17T01:50:33.508+0530 I CONTROL  [initandlisten] writing minidump diagnostic file C:\Program Files\MongoDB\Server\3.2017-03-16T20-20-33.mdmp
2017-03-17T01:50:34.637+0530 I CONTROL  [initandlisten] *** immediate exit due to unhandled exception

Comment by Mark Agarunov [ 16/Mar/17 ]

Hello har33sh,

Thank you for the response. Looking over the output you've provided, it looks like you may have another mongod process running simultaneously:

2017-03-17T00:54:58.875+0530 E STORAGE [initandlisten] WiredTiger error (0) [1489692298:875373][3380:140733334836944], file:WiredTiger.wt, WT_CURSOR.next: z:\mongodb_win\data\WiredTiger.turtle: handle-open: CreateFileW: The process cannot access the file because it is being used by another process.

Please verify that there are no other processes running that may be holding this file open.

Unfortunately, beyond the previous repair attempt, my recommendation would be to preform a clean resync from a node that is not affected or to restore the data from a backup, if possible.

Thanks,
Mark

Comment by hareesh [ 16/Mar/17 ]

@Mark I did what you asked me to do, I'm facing problem with collection 11. Adding the logs

2017-03-17T00:44:32.560+0530 I CONTROL  [initandlisten] MongoDB starting : pid=3380 port=27018 dbpath=z:\mongodb_win\data 64-bit host=DESKTOP-7E6EI2Q
2017-03-17T00:44:42.528+0530 I CONTROL  [initandlisten] targetMinOS: Windows 7/Windows Server 2008 R2
2017-03-17T00:44:53.351+0530 I CONTROL  [initandlisten] db version v3.4.0
2017-03-17T00:45:20.251+0530 I CONTROL  [initandlisten] git version: f4240c60f005be757399042dc12f6addbc3170c1
2017-03-17T00:45:37.699+0530 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1t-fips  3 May 2016
2017-03-17T00:45:52.479+0530 I CONTROL  [initandlisten] allocator: tcmalloc
2017-03-17T00:46:05.652+0530 I CONTROL  [initandlisten] modules: none
2017-03-17T00:46:17.860+0530 I CONTROL  [initandlisten] build environment:
2017-03-17T00:46:28.058+0530 I CONTROL  [initandlisten]     distmod: 2008plus-ssl
2017-03-17T00:46:38.580+0530 I CONTROL  [initandlisten]     distarch: x86_64
2017-03-17T00:46:49.059+0530 I CONTROL  [initandlisten]     target_arch: x86_64
2017-03-17T00:46:59.927+0530 I CONTROL  [initandlisten] options: { config: "z:\MongoDB_win\mongod.conf", net: { port: 27018 }, repair: true, storage: { dbPath: "z:\mongodb_win\data", journal: { enabled: false } }, systemLog: { destination: "file", logAppend: true, path: "z:\mongodb_win\mongod.log" } }
2017-03-17T00:47:16.520+0530 W -        [initandlisten] Detected unclean shutdown - z:\mongodb_win\data\mongod.lock is not empty.
2017-03-17T00:47:46.668+0530 I -        [initandlisten] Detected data files in z:\mongodb_win\data created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2017-03-17T00:48:01.591+0530 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2017-03-17T00:48:17.803+0530 I STORAGE  [initandlisten] Detected WT journal files.  Running recovery from last checkpoint.
2017-03-17T00:48:39.012+0530 I STORAGE  [initandlisten] journal to nojournal transition config: create,cache_size=1495M,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-03-17T00:51:30.369+0530 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1495M,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),,log=(enabled=false),
2017-03-17T00:53:39.156+0530 I STORAGE  [initandlisten] Repairing size cache
2017-03-17T00:53:56.195+0530 I STORAGE  [initandlisten] Verify succeeded on uri table:sizeStorer. Not salvaging.
2017-03-17T00:54:16.016+0530 I STORAGE  [initandlisten] Repairing catalog metadata
2017-03-17T00:54:34.883+0530 I STORAGE  [initandlisten] Verify succeeded on uri table:_mdb_catalog. Not salvaging.
2017-03-17T00:54:58.875+0530 E STORAGE  [initandlisten] WiredTiger error (0) [1489692298:875373][3380:140733334836944], file:WiredTiger.wt, WT_CURSOR.next: z:\mongodb_win\data\WiredTiger.turtle: handle-open: CreateFileW: The process cannot access the file because it is being used by another process.
2017-03-17T00:55:13.792+0530 I CONTROL  [initandlisten] 
2017-03-17T00:55:24.162+0530 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2017-03-17T00:55:31.829+0530 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2017-03-17T00:55:42.600+0530 I CONTROL  [initandlisten] 
2017-03-17T00:55:56.289+0530 I STORAGE  [initandlisten] repairDatabase ans_test
2017-03-17T00:56:06.967+0530 I STORAGE  [initandlisten] Repairing collection ans_test.power_k_seil_l
2017-03-17T00:56:18.652+0530 I STORAGE  [initandlisten] Verify failed on uri table:collection-11--4590358338625035404. Running a salvage operation.
2017-03-17T00:57:17.011+0530 I -        [initandlisten] Invariant failure rs.get() src\mongo\db\catalog\database.cpp 195
2017-03-17T00:57:27.240+0530 I -        [initandlisten] 
 
***aborting after invariant() failure
 
 
2017-03-17T00:57:37.646+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\util\stacktrace_windows.cpp(239)               mongo::printStackTrace+0x43
2017-03-17T00:57:49.337+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\util\signal_handlers_synchronous.cpp(180)      mongo::`anonymous namespace'::printSignalAndBacktrace+0x74
2017-03-17T00:58:00.654+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\util\signal_handlers_synchronous.cpp(236)      mongo::`anonymous namespace'::abruptQuit+0x85
2017-03-17T00:58:12.204+0530 I CONTROL  [initandlisten] ucrtbase.dll                                                               raise+0x1e7
2017-03-17T00:58:24.008+0530 I CONTROL  [initandlisten] ucrtbase.dll                                                               abort+0x31
2017-03-17T00:58:34.086+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\util\assert_util.cpp(154)                      mongo::invariantFailed+0x19a
2017-03-17T00:58:44.550+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\catalog\database.cpp(195)                   mongo::Database::_getOrCreateCollectionInstance+0xbf
2017-03-17T00:58:55.487+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\catalog\database.cpp(222)                   mongo::Database::Database+0x342
2017-03-17T00:59:06.963+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\catalog\database_holder.cpp(136)            mongo::DatabaseHolder::openDb+0x3cc
2017-03-17T00:59:18.248+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\repair_database.cpp(249)                    <lambda_47b2694c3219602f6dfbd244ccf4b795>::operator()+0x73
2017-03-17T00:59:30.170+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\util\scopeguard.h(99)                          mongo::ScopeGuardImplBase::SafeExecute<mongo::ScopeGuardImpl0<<lambda_47b2694c3219602f6dfbd244ccf4b795> > >+0x1b
2017-03-17T00:59:41.386+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\repair_database.cpp(291)                    mongo::repairDatabase+0x482
2017-03-17T00:59:52.787+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\db.cpp(384)                                 mongo::`anonymous namespace'::repairDatabasesAndCheckVersion+0x2d4
2017-03-17T01:00:03.990+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\db.cpp(670)                                 mongo::`anonymous namespace'::_initAndListen+0xf75
2017-03-17T01:00:14.565+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\db.cpp(823)                                 mongo::`anonymous namespace'::initAndListen+0x27
2017-03-17T01:00:25.738+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\db.cpp(1173)                                mongoDbMain+0x25d
2017-03-17T01:00:36.174+0530 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\db.cpp(861)                                 wmain+0x35
2017-03-17T01:00:47.728+0530 I CONTROL  [initandlisten] mongod.exe    f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl(255)  __scrt_common_main_seh+0x124
2017-03-17T01:00:58.888+0530 I CONTROL  [initandlisten] KERNEL32.DLL                                                               BaseThreadInitThunk+0x14
2017-03-17T01:01:10.487+0530 F -        [initandlisten] Got signal: 22 (SIGABRT).
2017-03-17T01:01:20.957+0530 I CONTROL  [initandlisten] *** unhandled exception 0x0000000E at 0x00007FFF053D7788, terminating
2017-03-17T01:01:32.654+0530 I CONTROL  [initandlisten] *** stack trace for unhandled exception:
2017-03-17T01:01:44.126+0530 I CONTROL  [initandlisten] KERNELBASE.dll                                                               RaiseException+0x68
2017-03-17T01:01:55.114+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\util\signal_handlers_synchronous.cpp(237)      mongo::`anonymous namespace'::abruptQuit+0x9d
2017-03-17T01:02:05.529+0530 I CONTROL  [initandlisten] ucrtbase.dll                                                                 raise+0x1e7
2017-03-17T01:02:14.289+0530 I CONTROL  [initandlisten] ucrtbase.dll                                                                 abort+0x31
2017-03-17T01:02:24.876+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\util\assert_util.cpp(154)                      mongo::invariantFailed+0x19a
2017-03-17T01:02:36.378+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\db\catalog\database.cpp(195)                   mongo::Database::_getOrCreateCollectionInstance+0xbf
2017-03-17T01:02:47.877+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\db\catalog\database.cpp(222)                   mongo::Database::Database+0x342
2017-03-17T01:02:55.457+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\db\catalog\database_holder.cpp(136)            mongo::DatabaseHolder::openDb+0x3cc
2017-03-17T01:03:02.992+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\db\repair_database.cpp(249)                    <lambda_47b2694c3219602f6dfbd244ccf4b795>::operator()+0x73
2017-03-17T01:03:12.187+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\util\scopeguard.h(99)                          mongo::ScopeGuardImplBase::SafeExecute<mongo::ScopeGuardImpl0<<lambda_47b2694c3219602f6dfbd244ccf4b795> > >+0x1b
2017-03-17T01:03:23.807+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\db\repair_database.cpp(291)                    mongo::repairDatabase+0x482
2017-03-17T01:03:35.082+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\db\db.cpp(384)                                 mongo::`anonymous namespace'::repairDatabasesAndCheckVersion+0x2d4
2017-03-17T01:03:45.566+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\db\db.cpp(670)                                 mongo::`anonymous namespace'::_initAndListen+0xf75
2017-03-17T01:03:56.290+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\db\db.cpp(823)                                 mongo::`anonymous namespace'::initAndListen+0x27
2017-03-17T01:04:07.632+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\db\db.cpp(1173)                                mongoDbMain+0x25d
2017-03-17T01:04:19.104+0530 I CONTROL  [initandlisten] mongod.exe      ...\src\mongo\db\db.cpp(861)                                 wmain+0x35
2017-03-17T01:04:30.619+0530 I CONTROL  [initandlisten] mongod.exe      f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl(255)  __scrt_common_main_seh+0x124
2017-03-17T01:04:41.845+0530 I CONTROL  [initandlisten] KERNEL32.DLL                                                                 BaseThreadInitThunk+0x14
2017-03-17T01:04:53.052+0530 I -        [initandlisten] 
2017-03-17T01:05:03.250+0530 I CONTROL  [initandlisten] writing minidump diagnostic file C:\Program Files\MongoDB\Server\3.2017-03-16T19-35-03.mdmp
2017-03-17T01:05:17.273+0530 I CONTROL  [initandlisten] *** immediate exit due to unhandled exception

Comment by Mark Agarunov [ 16/Mar/17 ]

Hello har33sh,

Thank you for your response. I've attached a repair attempt of the files you've provided. Would you please extract these files and replace them in your $dbpath and let us know if it resolves the issue?

Thanks,
Mark

Comment by hareesh [ 16/Mar/17 ]

1. Mapped network drive
2. Yes, there was, it was due to power cut during indexing of the data

Comment by Kelsey Schubert [ 16/Mar/17 ]

Hi har33sh,

I have a few questions to get to help us get a better understanding of what has occurred.

  1. What kind of underlying storage mechanism are you using? Are the storage devices attached locally or over the network? Are the disks SSDs or HDDs? What kind of RAID and/or volume management system are you using?
  2. Was there an unclean shutdown? If so, what was the root cause?
  3. Would you please provide the WiredTiger.turtle file so we can attempt to repair?

Thank you,
Thomas

Generated at Thu Feb 08 04:17:49 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.