[SERVER-26245] Mongodb crashed with unhandled exception Created: 22/Sep/16  Updated: 06/Apr/23  Resolved: 28/Sep/16

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 3.2.6
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Kris Budde Assignee: Kelsey Schubert
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Windows Server 2012R2 in Vmware virtual Machine
SAN


Attachments: File 3.2016-04-11T13-59-59.mdmp    
Operating System: ALL
Sprint: Platforms 2016-10-10
Participants:

 Description   

We had huge delays on our storage system affecting the mongo performance.
We are still investigating the root cause of our problems. It is possible that mongodb was not able to write the data to the disk for around 5 minutes.
The mongodb process crashed with an unhandled exception.

I've attached the minidump and the exception:

06:23:41 I CONTROL  [conn216298] *** unhandled exception 0xC0000006 at 0x00007FF7459403CB, terminating
 
06:23:41I CONTROL  [conn216298] *** stack trace for unhandled exception:
 
06:23:53 I CONTROL  [conn216298] mongod.exe    ...\src\third_party\snappy-1.1.2\snappy.cc(160)                                  snappy::`anonymous namespace'::IncrementalCopyFastPath+0x4b
06:23:53 I CONTROL  [conn216298] mongod.exe    ...\src\third_party\snappy-1.1.2\snappy.cc(1220)                                 snappy::SnappyArrayWriter::AppendFromSelf+0x83
06:23:53 I CONTROL  [conn216298] mongod.exe    ...\src\third_party\snappy-1.1.2\snappy.cc(792)                                  snappy::SnappyDecompressor::DecompressAllTags<snappy::SnappyArrayWriter>+0x198
06:23:53 I CONTROL  [conn216298] mongod.exe    ...\src\third_party\snappy-1.1.2\snappy.cc(868)                                  snappy::InternalUncompress<snappy::SnappyArrayWriter>+0xac
06:23:53 I CONTROL  [conn216298] mongod.exe    ...\src\third_party\snappy-1.1.2\snappy.cc(1235)                                 snappy::RawUncompress+0x38
06:23:53 I CONTROL  [conn216298] mongod.exe    ...\src\third_party\snappy-1.1.2\snappy-c.cc(58)                                 snappy_uncompress+0x54
06:23:53 I CONTROL  [conn216298] mongod.exe    ...\src\third_party\wiredtiger\ext\compressors\snappy\snappy_compress.c(181)     wt_snappy_decompress+0x54
06:23:53 I CONTROL  [conn216298] mongod.exe    ...\src\third_party\wiredtiger\src\btree\bt_io.c(110)                            __wt_bt_read+0x1f1
06:23:53 I CONTROL  [conn216298] mongod.exe    ...\src\third_party\wiredtiger\src\btree\bt_read.c(391)                          __page_read+0x141
06:23:53 I CONTROL  [conn216298] mongod.exe    ...\src\third_party\wiredtiger\src\btree\bt_read.c(486)                          __wt_page_in_func+0x1b0
06:23:53 I CONTROL  [conn216298] mongod.exe    ...\src\third_party\wiredtiger\src\include\btree.i(1320)                         __wt_page_swap_func+0x3e
06:23:53 I CONTROL  [conn216298] mongod.exe    ...\src\third_party\wiredtiger\src\btree\row_srch.c(446)                         __wt_row_search+0x4b5
06:23:53 I CONTROL  [conn216298] mongod.exe    ...\src\third_party\wiredtiger\src\btree\bt_cursor.c(340)                        __wt_btcur_search+0x2e1
06:23:53 I CONTROL  [conn216298] mongod.exe    ...\src\third_party\wiredtiger\src\cursor\cur_file.c(200)                        __curfile_search+0x1d4
06:23:53 I CONTROL  [conn216298] mongod.exe    ...\src\mongo\db\storage\wiredtiger\wiredtiger_record_store.cpp(523)             mongo::WiredTigerRecordStore::Cursor::seekExact+0x7c
06:23:53 I CONTROL  [conn216298] mongod.exe    ...\src\mongo\db\exec\working_set_common.cpp(102)                                mongo::WorkingSetCommon::fetch+0xd4
06:23:53 I CONTROL  [conn216298] mongod.exe    ...\src\mongo\db\exec\fetch.cpp(125)                                             mongo::FetchStage::work+0x350
06:23:53 I CONTROL  [conn216298] mongod.exe    ...\src\mongo\db\exec\near.cpp(178)                                              mongo::NearStage::bufferNext+0x1d5
06:23:53 I CONTROL  [conn216298] mongod.exe    ...\src\mongo\db\exec\near.cpp(101)                                              mongo::NearStage::work+0xc3
06:23:53 I CONTROL  [conn216298] mongod.exe    ...\src\mongo\db\query\plan_executor.cpp(390)                                    mongo::PlanExecutor::getNextImpl+0x2f4
06:23:53 I CONTROL  [conn216298] mongod.exe    ...\src\mongo\db\query\plan_executor.cpp(319)                                    mongo::PlanExecutor::getNext+0x41
06:23:53 I CONTROL  [conn216298] mongod.exe    ...\src\mongo\db\query\find.cpp(589)                                             mongo::runQuery+0x7aa
06:23:53 I CONTROL  [conn216298] mongod.exe    ...\src\mongo\db\instance.cpp(401)                                               mongo::receivedQuery+0x17d
06:23:53 I CONTROL  [conn216298] mongod.exe    ...\src\mongo\db\instance.cpp(523)                                               mongo::assembleResponse+0x927
06:23:53 I CONTROL  [conn216298] mongod.exe    ...\src\mongo\db\db.cpp(177)                                                     mongo::MyMessageHandler::process+0xce
06:23:53 I CONTROL  [conn216298] mongod.exe    ...\src\mongo\util\net\message_server_port.cpp(231)                              mongo::PortMessageServer::handleIncomingMsg+0x4a7
06:23:53 I CONTROL  [conn216298] mongod.exe    c:\program files (x86)\microsoft visual studio 12.0\vc\include\thr\xthread(188)  std::_LaunchPad<std::_Bind<0,void,std::_Bind<1,void,std::_Pmf_wrap<void (__cdecl mongo::FTDCController::*)(void) __ptr64,void,mongo::FTDCController>,mongo::FTDCController * __ptr64 const> > >::_Go+0x1d
06:23:53 I CONTROL  [conn216298] MSVCP120.dll                                                                                   std::_Pad::_Release+0x6c
06:23:53 I CONTROL  [conn216298] MSVCR120.dll                                                                                   beginthreadex+0x107
06:23:53 I CONTROL  [conn216298] MSVCR120.dll                                                                                   endthreadex+0x192
06:23:53 I CONTROL  [conn216298] KERNEL32.DLL                                                                                   BaseThreadInitThunk+0x22
06:23:53 I -        [conn216298]
06:23:53 I CONTROL  [conn216298] writing minidump diagnostic file C:\Program Files\MongoDB\Server\3.2016-09-22T04-23-53.mdmp
06:24:19 I CONTROL  [conn216298] *** immediate exit due to unhandled exception



 Comments   
Comment by Kelsey Schubert [ 28/Sep/16 ]

Hi kbudde,

Thank you for the additional details and alerting us to this issue. Unfortunately, MongoDB cannot gracefully catch this I/O error as it occurs when attempting to read memory from disk. Consequently, a clean shutdown is not possible.

For MongoDB-related support discussion please consider posting on the mongodb-users group or Stack Overflow with the mongodb tag. A question about managing MongoDB on top of a faulty storage layer would involve more discussion would be best posted on the mongodb-users group.

Kind regards,
Thomas

Comment by Kris Budde [ 23/Sep/16 ]

Hi,

in the windows eventlog are some messages about disk: LSI_SAS: "Reset to device, \Device\RaidPort0, was issued."
Every minute; after 5 minutes it crashed. I'm sure it was an issue of our storage system and not of mongodb itself.
But nevertheless it would be nice if mongodb would not crash in this case and shutdown in a "cleaner" way.

> First, according to the dump, this is MongoDB 3.0.6.
I've checked the buildinfo again. It is 3.2.6. strange.

{ "version" : "3.2.6",
  "gitVersion" : "05552b562c7a0b3143a729aaa0838e558dc49b25",
  "targetMinOS" : "Windows 7/Windows Server 2008 R2",
  "modules" : [],
  "allocator" : "tcmalloc",
  "javascriptEngine" : "mozjs",
  "sysInfo" : "deprecated",
  "versionArray" : [ 
    3, 
    2, 
    6, 
    0 ],
  "openssl" : { "running" : "OpenSSL 1.0.1p-fips 9 Jul 2015",
    "compiled" : "OpenSSL 1.0.1p-fips 9 Jul 2015" },
  "buildEnvironment" : { "distmod" : "2008plus-ssl",
    "distarch" : "x86_64",
    "cc" : "cl: Microsoft (R) C/C++ Optimizing Compiler Version 18.00.31101 for x64",
    "ccflags" : "/nologo /EHsc /W3 /wd4355 /wd4800 /wd4267 /wd4244 /wd4290 /wd4068 /wd4351 /we4013 /we4099 /we4930 /Z7 /errorReport:none /MD /O2 /Oy- /Gw /Gy /Zc:inline",
    "cxx" : "cl: Microsoft (R) C/C++ Optimizing Compiler Version 18.00.31101 for x64",
    "cxxflags" : "/TP",
    "linkflags" : "/nologo /DEBUG /INCREMENTAL:NO /LARGEADDRESSAWARE /OPT:REF",
    "target_arch" : "x86_64",
    "target_os" : "windows" },
  "bits" : 64,
  "debug" : false,
  "maxBsonObjectSize" : 16777216,
  "storageEngines" : [ 
    "devnull", 
    "ephemeralForTest", 
    "mmapv1", 
    "wiredTiger" ],
  "ok" : 1 }

Comment by Mark Benvenuto [ 22/Sep/16 ]

First, according to the dump, this is MongoDB 3.0.6.

Second, 0xC0000006 is EXCEPTION_IN_PAGE_ERROR i.e. STATUS_IN_PAGE_ERROR, This is defined as follows in ntstatus.h:

// The instruction at 0x%p referenced memory at 0x%p. The required data was not placed into memory because of an I/O error status of 0x%x.

This means that Windows tried to read memory from disk (an image file, memory swapped out, etc), and the I/O request failed.

Check the System log in Windows Event Viewer for any I/O errors. Look for I/O errors in the guest OS, the host OS, and the finally check the SAN.

Generated at Thu Feb 08 04:11:32 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.