[SERVER-69717] Mongo DB crashing with an error: Not enough storage is available to process this command Created: 15/Sep/22  Updated: 15/Sep/22  Resolved: 15/Sep/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Mamatha K Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File mongodb_LP (2).log     Text File mongodb_LP_2022-08-22T20-46-44.log    
Issue Links:
Duplicate
is duplicated by SERVER-69716 Mongo DB crashing with an error: Not ... Closed
is duplicated by WT-9868 Mongo DB crashing with an error: Not ... Closed
Operating System: ALL
Participants:

 Description   

Mongo DB is crashing which is causing the application to fail. When extracted mongo Db logs found below error 

2022-09-11T00:36:21.977+0000 E STORAGE [WTJournalFlusher] WiredTiger (-28992) [1662856581:400819][7072:1995194080], WT_SESSION.log_flush: journal/WiredTigerLog.0000158373 handle-sync: FlushFileBuffers error: Not enough storage is available to process this command.

2022-09-11T00:36:21.989+0000 I  [WTJournalFlusher] Invariant failure: s>log_flush(s, "sync=on") resulted in status UnknownError: -28992: Not enough storage is available to process this command.

After restarting application servers mongo started working as expected. But this is the second occurrence and we want to know for possible solutions to fix this issue to avoid future occurrences.

Disk has enough space :  800gb in D drive, 40-50gb is C drive. And mongo DB is around 720gb

Mongo DB version used : 3.2.6

Operating system is : windows.

 

2022-09-11T00:36:21.849+0000 I COMMAND  [conn3987] insert meolutdb.rawDetection ninserted:40 keyUpdates:0 writeConflicts:0 numYields:0 locks:{ Global: { acquireCount:

{ r: 1, w: 1 }

}, Database: { acquireCount:

{ w: 1 }

}, Collection: { acquireCount:

{ w: 1 }

} } 113ms
2022-09-11T00:36:21.977+0000 E STORAGE  [WTJournalFlusher] WiredTiger (-28992) [1662856581:400819][7072:1995194080], WT_SESSION.log_flush: journal/WiredTigerLog.0000158373 handle-sync: FlushFileBuffers error: Not enough storage is available to process this command.
2022-09-11T00:36:21.989+0000 I -        [WTJournalFlusher] Invariant failure: s->log_flush(s, "sync=on") resulted in status UnknownError: -28992: Not enough storage is available to process this command.
 at src\mongo\db\storage\wiredtiger\wiredtiger_session_cache.cpp 203
2022-09-11T00:36:25.065+0000 I COMMAND  [conn4275] update meolutdb.eMSStatus query: { _id: "421d0553-e28b-4feb-a48e-7bfb6ed129fc" } update: { _id: "421d0553-e28b-4feb-a48e-7bfb6ed129fc", _class: "com.emsgt.lut.il.dal.api.EMSStatus", subsystem_id: 3, service_id: 10005, service_description: "Motor Temperature", timestamp: 1662856584631000000, status: "OK", errors: " 34.00 celsius", subsystem_type: "Antenna" } keysExamined:1 docsExamined:1 nMatched:1 nModified:1 keyUpdates:6 writeConflicts:0 numYields:1 locks:{ Global: { acquireCount:

{ r: 2, w: 2 }

}, Database: { acquireCount:

{ w: 2 }

}, Collection: { acquireCount:

{ w: 2 }

} } 113ms
2022-09-11T00:36:26.505+0000 I COMMAND  [conn3965] insert meolutdb.rawDetection ninserted:40 keyUpdates:0 writeConflicts:0 numYields:0 locks:{ Global: { acquireCount:

{ r: 1, w: 1 }

}, Database: { acquireCount:

{ w: 1 }

}, Collection: { acquireCount:

{ w: 1 }

} } 119ms
2022-09-11T00:36:26.632+0000 I COMMAND  [conn4001] insert meolutdb.rawDetection ninserted:40 keyUpdates:0 writeConflicts:0 numYields:0 locks:{ Global: { acquireCount:

{ r: 1, w: 1 }

}, Database: { acquireCount:

{ w: 1 }

}, Collection: { acquireCount:

{ w: 1 }

} } 156ms
2022-09-11T00:36:50.126+0000 I CONTROL  [WTJournalFlusher] mongod.exe    ...\src\mongo\util\stacktrace_windows.cpp(174)                                   mongo::printStackTrace+0x43
2022-09-11T00:36:50.126+0000 I CONTROL  [WTJournalFlusher] mongod.exe    ...\src\mongo\util\log.cpp(136)                                                  mongo::logContext+0xa8
2022-09-11T00:36:50.126+0000 I CONTROL  [WTJournalFlusher] mongod.exe    ...\src\mongo\util\assert_util.cpp(164)                                          mongo::invariantOKFailed+0x14c
2022-09-11T00:36:50.126+0000 I CONTROL  [WTJournalFlusher] mongod.exe    ...\src\mongo\db\storage\wiredtiger\wiredtiger_session_cache.cpp(203)            mongo::WiredTigerSessionCache::waitUntilDurable+0x2bd
2022-09-11T00:36:50.126+0000 I CONTROL  [WTJournalFlusher] mongod.exe    ...\src\mongo\db\storage\wiredtiger\wiredtiger_kv_engine.cpp(97)                 mongo::WiredTigerKVEngine::WiredTigerJournalFlusher::run+0x1a8
2022-09-11T00:36:50.126+0000 I CONTROL  [WTJournalFlusher] mongod.exe    ...\src\mongo\util\background.cpp(152)                                           mongo::BackgroundJob::jobBody+0x1b1
2022-09-11T00:36:50.126+0000 I CONTROL  [WTJournalFlusher] mongod.exe    c:\program files (x86)\microsoft visual studio 12.0\vc\include\thr\xthread(188)  std::LaunchPad<std::_Bind<0,void,std::_Bind<1,void,std::_Pmf_wrap<void (_cdecl mongo::dur::JournalWriter::*)(void) __ptr64,void,mongo::dur::JournalWriter>,mongo::dur::JournalWriter * __ptr64 const> > >::_Go+0x1c
2022-09-11T00:36:50.126+0000 I CONTROL  [WTJournalFlusher] mongod.exe    f:\dd\vctools\crt\crtw32\stdcpp\thr\threadcall.cpp(28)                           _Call_func+0x14
2022-09-11T00:36:50.126+0000 I CONTROL  [WTJournalFlusher] mongod.exe    f:\dd\vctools\crt\crtw32\startup\threadex.c(376)                                 _callthreadstartex+0x17
2022-09-11T00:36:50.126+0000 I CONTROL  [WTJournalFlusher] mongod.exe    f:\dd\vctools\crt\crtw32\startup\threadex.c(354)                                 _threadstartex+0x102
2022-09-11T00:36:50.126+0000 I CONTROL  [WTJournalFlusher] kernel32.dll                                                                                   BaseThreadInitThunk+0xd
2022-09-11T00:36:50.126+0000 I CONTROL  [WTJournalFlusher] 
2022-09-11T00:36:50.126+0000 I -        [WTJournalFlusher] 

***aborting after invariant() failure



 Comments   
Comment by Yuan Fang [ 15/Sep/22 ]

I'm closing this ticket as it appears to be a duplicate submission of SERVER-69716.

Generated at Thu Feb 08 06:14:12 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.