[SERVER-62275] mongoDB Bug about FTDC Created: 27/Dec/21 Updated: 27/Oct/23 Resolved: 16/Jan/22 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | David kim | Assignee: | Dmitry Agranat |
| Resolution: | Community Answered | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
| Operating System: | ALL |
| Steps To Reproduce: | mongod.conf ```
|
| Participants: |
| Description |
|
Version : 4.4.10 In Ubuntu 18 all of the sudden Mongodb Server stopped working I found permission of /var/lib/mongodb is 755 and in /var/lib/mongodb/diagnostic.data, there are metrics files generated in every 2 days and a metrics.interim file ``` {"t":\{"$date":"2021-12-25T15:03:55.428+09:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn24","msg":"Connection ended","attr":{"remote":"127.0.0.1:45082","connectionId":24,"connectionCount":7}} ,"s":"F", "c":"CONTROL", "id":4757800, "ctx":"ftdc","msg":"Writing fatal message","attr":{"message":"terminate() called. An exception is active; attempting to gather more information"}} ,"s":"F", "c":"CONTROL", "id":4757800, "ctx":"ftdc","msg":"Writing fatal message","attr":{"message":"DBException::toString(): FileStreamFailed: Failed to write to interim file buffer for full-time diagnostic data captu re: /var/lib/mongodb/diagnostic.data/metrics.interim.temp\nActual exception type: mongo::error_details::ExceptionForImpl<(mongo::ErrorCodes::Error)39, mongo::AssertionException>\n"}} ,"s":"I", "c":"CONTROL", "id":31431, "ctx":"ftdc","msg":"BACKTRACE: {bt}","attr":{"bt":{"backtrace":[ {"a":"5556DA3F484A","b":"5556D7620000","o":"2DD484A","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTr aceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.606","s+":"1EA"},{"a":"5556DA3F62D9","b":"5556D7620000","o":"2DD62D9","s":"ZN5mongo15printStackTraceEv","s+":"29"},{"a":"5556DA3F34A6","b":"5556D7620000","o":"2DD34A6","s":"_ZN5mongo12_GLOBAL_N_111myTerminateE v","s+":"A6"},{"a":"5556DA583476","b":"5556D7620000","o":"2F63476","s":"ZN10cxxabiv111terminateEPFvvE","s+":"6"},{"a":"5556DA6174D9","b":"5556D7620000","o":"2FF74D9","s":"cxa_call_terminate","s+":"39"},{"a":"5556DA582E65","b":"5556D7620000","o":"2F62E65","s": "gxx_personality_v0","s+":"275"},{"a":"7F79D45CD573","b":"7F79D45BD000","o":"10573","s":"_Unwind_GetTextRelBase","s+":"1E03"},{"a":"7F79D45CDAD1","b":"7F79D45BD000","o":"10AD1","s":"_Unwind_RaiseException","s+":"2B1"},{"a":"5556DA5835D7","b":"5556D7620000","o":"2F 635D7","s":"cxa_throw","s+":"37"},{"a":"5556D854D900","b":"5556D7620000","o":"F2D900","s":"_ZN5mongo13error_details23throwExceptionForStatusERKNS_6StatusE","s+":"1B72"},{"a":"5556D8561B1D","b":"5556D7620000","o":"F41B1D","s":"_ZN5mongo21uassertedWithLocationERKNS 6StatusEPKcj","s+":"27B"},{"a":"5556D82BE9A5","b":"5556D7620000","o":"C9E9A5","s":"_ZN5mongo14FTDCController6doLoopEv.cold.395","s+":"2D"},{"a":"5556D8AED15C","b":"5556D7620000","o":"14CD15C","s":"_ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN5mongo4stdx6thread C4IZNS3_14FTDCController5startEvEUlvE0_JELi0EEET_DpOT0_EUlvE_EEEEE6_M_runEv","s+":"5C"},{"a":"5556DA59EF3F","b":"5556D7620000","o":"2F7EF3F","s":"execute_native_thread_routine","s+":"F"},{"a":"7F79D43A56DB","b":"7F79D439E000","o":"76DB","s":"start_thread","s+":"DB"} ,{"a":"7F79D40CE71F","b":"7F79D3FAD000","o":"12171F","s":"clone","s+":"3F"}],"processInfo":{"mongodbVersion":"4.4.10","gitVersion":"58971da1ef93435a9f62bf4708a81713def6e88c","compiledModules":[],"uname": {"sysname":"Linux","release":"5.4.0-81-generic","version":"#91~ 18.04.1-Ubuntu SMP Fri Jul 23 13:36:29 UTC 2021","machine":"x86_64"},"somap":[{"b":"5556D7620000","elfType":3,"buildId":"706585359197300EBD464448C618DD26F2A4291F"},{"b":"7F79D45BD000","path":"/lib/x86_64-linux-gnu/libgcc_s.so.1","elfType":3,"buildId":"679F3AE11120EC 7C483BC9295345D836F5C104F7"},{"b":"7F79D439E000","path":"/lib/x86_64-linux-gnu/libpthread.so.0","elfType":3,"buildId":"68F36706EB2E6EEE4046C4FDCA2A19540B2F6113"},{"b":"7F79D3FAD000","path":"/lib/x86_64-linux-gnu/libc.so.6","elfType":3,"buildId":"CE450EB01A5E5ACC7CE7 B8C26 ``` |
| Comments |
| Comment by Dmitry Agranat [ 16/Jan/22 ] |
|
Hi kdaeyoub@gmail.com, given this is an indication of either running out of disk space of other disk-related issues (for example, quote limitations), I will go ahead and close this case. |
| Comment by David kim [ 30/Dec/21 ] |
|
Oh Okay then I will check it. Thank you. |
| Comment by Dmitry Agranat [ 29/Dec/21 ] |
|
Both backtraces in the log are cut off in the middle, meaning the OS was unable to complete writing even a few more bytes. This is usually an indication of either running out of disk space of other disk-related issues (for example, quote limitations) |
| Comment by David kim [ 29/Dec/21 ] |
|
I didn't follow up of out of disk space. But I guess it is not. Because I found this error a week ago and that time I just restarted mongod and I went well. From that time, several days later, I found this error again. metrics files were written after when I restarted mongod. If the root cause is out of disk space, I would find the error as soon as I restarted the mongod. But I went well for a few days. But It is only my opinion so let me upload mongod.log file first. Thank you for your service. |
| Comment by Dmitry Agranat [ 28/Dec/21 ] |
|
kdaeyoub@gmail.com I am getting "404: Not Found" when trying to access your log. Did you check that you didn't run out of disk space? |
| Comment by David kim [ 28/Dec/21 ] |
|
Mongod.log is (https://raw.githubusercontent.com/DavidKimDY/uipa_as/log/mongod.log?token=AMDUG5NSZ4PU2G6EKML6W2TBZJVZI) |
| Comment by Dmitry Agranat [ 27/Dec/21 ] |
|
Thanks for the report kdaeyoub@gmail.com. The backtrace seems to be incomplete, could you provide:
Also, could you check that you didn't run out of disk space? |