[SERVER-62275] mongoDB Bug about FTDC Created: 27/Dec/21  Updated: 27/Oct/23  Resolved: 16/Jan/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: David kim Assignee: Dmitry Agranat
Resolution: Community Answered Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File metrics.2021-12-24T23-02-30Z-00000     Text File mongod.log    
Operating System: ALL
Steps To Reproduce:

mongod.conf

```

  1. Where and how to store data.
    storage:
      dbPath: /var/lib/mongodb
      journal:
        enabled: true
    #  engine:
    #  mmapv1:
    #  wiredTiger:# where to write logging data.
    systemLog:
      destination: file
      logAppend: true
      path: /var/log/mongodb/mongod.log# network interfaces
    net:
      port: 27017
      bindIp: 127.0.0.1# how the process runs
    processManagement:
      timeZoneInfo: /usr/share/zoneinfo```
Participants:

 Description   

Version : 4.4.10

In Ubuntu 18

all of the sudden Mongodb Server stopped working

I found permission of /var/lib/mongodb is 755 and in /var/lib/mongodb/diagnostic.data, there are metrics files generated in every 2 days and a metrics.interim file

```

{"t":\{"$date":"2021-12-25T15:03:55.428+09:00"}

,"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn24","msg":"Connection ended","attr":{"remote":"127.0.0.1:45082","connectionId":24,"connectionCount":7}}
48609 {"t":

{"$date":"2021-12-25T15:04:31.004+09:00"}

,"s":"F", "c":"CONTROL", "id":4757800, "ctx":"ftdc","msg":"Writing fatal message","attr":{"message":"terminate() called. An exception is active; attempting to gather more information"}}
48610 {"t":

{"$date":"2021-12-25T15:04:31.004+09:00"}

,"s":"F", "c":"CONTROL", "id":4757800, "ctx":"ftdc","msg":"Writing fatal message","attr":{"message":"DBException::toString(): FileStreamFailed: Failed to write to interim file buffer for full-time diagnostic data captu re: /var/lib/mongodb/diagnostic.data/metrics.interim.temp\nActual exception type: mongo::error_details::ExceptionForImpl<(mongo::ErrorCodes::Error)39, mongo::AssertionException>\n"}}
48611 {"t":

{"$date":"2021-12-25T15:04:31.045+09:00"}

,"s":"I", "c":"CONTROL", "id":31431, "ctx":"ftdc","msg":"BACKTRACE: {bt}","attr":{"bt":{"backtrace":[

{"a":"5556DA3F484A","b":"5556D7620000","o":"2DD484A","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTr aceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.606","s+":"1EA"}

,{"a":"5556DA3F62D9","b":"5556D7620000","o":"2DD62D9","s":"ZN5mongo15printStackTraceEv","s+":"29"},{"a":"5556DA3F34A6","b":"5556D7620000","o":"2DD34A6","s":"_ZN5mongo12_GLOBAL_N_111myTerminateE v","s+":"A6"},{"a":"5556DA583476","b":"5556D7620000","o":"2F63476","s":"ZN10cxxabiv111terminateEPFvvE","s+":"6"},{"a":"5556DA6174D9","b":"5556D7620000","o":"2FF74D9","s":"cxa_call_terminate","s+":"39"},{"a":"5556DA582E65","b":"5556D7620000","o":"2F62E65","s": "gxx_personality_v0","s+":"275"},{"a":"7F79D45CD573","b":"7F79D45BD000","o":"10573","s":"_Unwind_GetTextRelBase","s+":"1E03"},{"a":"7F79D45CDAD1","b":"7F79D45BD000","o":"10AD1","s":"_Unwind_RaiseException","s+":"2B1"},{"a":"5556DA5835D7","b":"5556D7620000","o":"2F 635D7","s":"cxa_throw","s+":"37"},{"a":"5556D854D900","b":"5556D7620000","o":"F2D900","s":"_ZN5mongo13error_details23throwExceptionForStatusERKNS_6StatusE","s+":"1B72"},{"a":"5556D8561B1D","b":"5556D7620000","o":"F41B1D","s":"_ZN5mongo21uassertedWithLocationERKNS 6StatusEPKcj","s+":"27B"},{"a":"5556D82BE9A5","b":"5556D7620000","o":"C9E9A5","s":"_ZN5mongo14FTDCController6doLoopEv.cold.395","s+":"2D"},{"a":"5556D8AED15C","b":"5556D7620000","o":"14CD15C","s":"_ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN5mongo4stdx6thread C4IZNS3_14FTDCController5startEvEUlvE0_JELi0EEET_DpOT0_EUlvE_EEEEE6_M_runEv","s+":"5C"},{"a":"5556DA59EF3F","b":"5556D7620000","o":"2F7EF3F","s":"execute_native_thread_routine","s+":"F"},{"a":"7F79D43A56DB","b":"7F79D439E000","o":"76DB","s":"start_thread","s+":"DB"} ,{"a":"7F79D40CE71F","b":"7F79D3FAD000","o":"12171F","s":"clone","s+":"3F"}],"processInfo":{"mongodbVersion":"4.4.10","gitVersion":"58971da1ef93435a9f62bf4708a81713def6e88c","compiledModules":[],"uname":

{"sysname":"Linux","release":"5.4.0-81-generic","version":"#91~ 18.04.1-Ubuntu SMP Fri Jul 23 13:36:29 UTC 2021","machine":"x86_64"}

,"somap":[{"b":"5556D7620000","elfType":3,"buildId":"706585359197300EBD464448C618DD26F2A4291F"},{"b":"7F79D45BD000","path":"/lib/x86_64-linux-gnu/libgcc_s.so.1","elfType":3,"buildId":"679F3AE11120EC 7C483BC9295345D836F5C104F7"},{"b":"7F79D439E000","path":"/lib/x86_64-linux-gnu/libpthread.so.0","elfType":3,"buildId":"68F36706EB2E6EEE4046C4FDCA2A19540B2F6113"},{"b":"7F79D3FAD000","path":"/lib/x86_64-linux-gnu/libc.so.6","elfType":3,"buildId":"CE450EB01A5E5ACC7CE7 B8C26

```



 Comments   
Comment by Dmitry Agranat [ 16/Jan/22 ]

Hi kdaeyoub@gmail.com, given this is an indication of either running out of disk space of other disk-related issues (for example, quote limitations), I will go ahead and close this case.

Comment by David kim [ 30/Dec/21 ]

Oh Okay then I will check it. Thank you.

Comment by Dmitry Agranat [ 29/Dec/21 ]

Both backtraces in the log are cut off in the middle, meaning the OS was unable to complete writing even a few more bytes. This is usually an indication of either running out of disk space of other disk-related issues (for example, quote limitations)

Comment by David kim [ 29/Dec/21 ]

I didn't follow up of out of disk space. But I guess it is not. Because I found this error a week ago and that time I just restarted mongod and I went well. From that time, several days later, I found this error again. metrics files were written after when I restarted mongod. If the root cause is out of disk space, I would find the error as soon as I restarted the mongod. But I went well for a few days. But It is only my opinion so let me upload mongod.log file first. Thank you for your service.

Comment by Dmitry Agranat [ 28/Dec/21 ]

kdaeyoub@gmail.com I am getting "404: Not Found" when trying to access your log. Did you check that you didn't run out of disk space?

Comment by David kim [ 28/Dec/21 ]

Mongod.log is (https://raw.githubusercontent.com/DavidKimDY/uipa_as/log/mongod.log?token=AMDUG5NSZ4PU2G6EKML6W2TBZJVZI)

Comment by Dmitry Agranat [ 27/Dec/21 ]

Thanks for the report kdaeyoub@gmail.com. The backtrace seems to be incomplete, could you provide:

  • the actual log containing this backtrace and a few lines before?
  • last metrics. file located under /var/lib/mongodb/diagnostic.data?

Also, could you check that you didn't run out of disk space?

Generated at Thu Feb 08 05:54:40 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.