[SERVER-73805] mongos in 6.0.4 hangs under load Created: 08/Feb/23  Updated: 15/Feb/23  Resolved: 15/Feb/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 6.0.4
Fix Version/s: None

Type: Bug Priority: Minor - P4
Reporter: Amit Gupta Assignee: Yuan Fang
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-73597 Sporadic seg faults days/weeks after ... Closed
Operating System: ALL
Steps To Reproduce:

start mongos, wait for a while during very busy times..

 

Participants:

 Description   

Environment:

  • Ubuntu 22.04 latest patches
  • 128GB ram machine, AMD Epic, as well as Intel machines

We just upgraded from 5.0.14 to 6.0.4. The upgrade went smoothly last night. Today during peak hours, our app servers that were getting most of our traffic would have mongos randomly hang.    The mongos process looks like it is still running,  however mongosh cannot connect to it.  

 

I managed to attached gdb to the running process, and here is the backtrace:

gdb --pid=458174
GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <[http://gnu.org/licenses/gpl.html]>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<[https://www.gnu.org/software/gdb/bugs/]>.
Find the GDB manual and other documentation resources online at:
    <[http://www.gnu.org/software/gdb/documentation/]>.
 
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 458174
[New LWP 458175]
[New LWP 458176]
[New LWP 458177]
[New LWP 458178]
[New LWP 458179]
[New LWP 458181]
[New LWP 458182]
[New LWP 458183]
[New LWP 458184]
[New LWP 458186]
[New LWP 458187]
[New LWP 458188]
[New LWP 458194]
[New LWP 458196]
[New LWP 458197]
[New LWP 458199]
[New LWP 458200]
[New LWP 458201]
[New LWP 458202]
[New LWP 458203]
[New LWP 458204]
[New LWP 458205]
[New LWP 458206]
[New LWP 458207]
[New LWP 458208]
[New LWP 458209]
[New LWP 458211]
[New LWP 458212]
[New LWP 458213]
[New LWP 458214]
[New LWP 458215]
[New LWP 458216]
[New LWP 458217]
[New LWP 458218]
[New LWP 458219]
[New LWP 458220]
[New LWP 458221]
[New LWP 458222]
[New LWP 458223]
[New LWP 458224]
[New LWP 458225]
[New LWP 458226]
[New LWP 458227]
[New LWP 458229]
[New LWP 458230]
[New LWP 458231]
[New LWP 458232]
[New LWP 458233]
[New LWP 458234]
[New LWP 458235]
[New LWP 458236]
[New LWP 458237]
[New LWP 458238]
[New LWP 458239]
[New LWP 458240]
[New LWP 458241]
[New LWP 458242]
[New LWP 458243]
[New LWP 458244]
[New LWP 458245]
[New LWP 458246]
[New LWP 458247]
[New LWP 458248]
[New LWP 458249]
[New LWP 458250]
[New LWP 458251]
[New LWP 458252]
[New LWP 458275]
[New LWP 458276]
[New LWP 458277]
[New LWP 458278]
[New LWP 458285]
[New LWP 458286]
[New LWP 458287]
[New LWP 458288]
[New LWP 458289]
[New LWP 458290]
[New LWP 458291]
[New LWP 458292]
[New LWP 458293]
[New LWP 458294]
[New LWP 458295]
[New LWP 458296]
[New LWP 458297]
[New LWP 458298]
[New LWP 458299]
[New LWP 458300]
[New LWP 458301]
[New LWP 458302]
[New LWP 458303]
[New LWP 458304]
[New LWP 458305]
[New LWP 458306]
[New LWP 458307]
[New LWP 458308]
[New LWP 458309]
[New LWP 458310]
[New LWP 458311]
[New LWP 458312]
[New LWP 458313]
[New LWP 458314]
[New LWP 458315]
[New LWP 458316]
[New LWP 458317]
[New LWP 458318]
[New LWP 458319]
[New LWP 458320]
[New LWP 458321]
[New LWP 458322]
[New LWP 458323]
[New LWP 458324]
[New LWP 458325]
[New LWP 458326]
[New LWP 458327]
[New LWP 458328]
[New LWP 458329]
[New LWP 458330]
[New LWP 458331]
[New LWP 458332]
[New LWP 458333]
[New LWP 458334]
[New LWP 458335]
[New LWP 458336]
[New LWP 458337]
[New LWP 458338]
[New LWP 458339]
[New LWP 458340]
[New LWP 458341]
[New LWP 458342]
[New LWP 458343]
[New LWP 458344]
[New LWP 458345]
[New LWP 458346]
[New LWP 458347]
[New LWP 458348]
[New LWP 458349]
[New LWP 458350]
[New LWP 458351]
[New LWP 458352]
[New LWP 458353]
[New LWP 458380]
[New LWP 458381]
[New LWP 458382]
[New LWP 458384]
[New LWP 458385]
[New LWP 458386]
[New LWP 458387]
[New LWP 458388]
[New LWP 458389]
[New LWP 458390]
[New LWP 458391]
[New LWP 458392]
[New LWP 458393]
[New LWP 458394]
[New LWP 458395]
[New LWP 458396]
[New LWP 458397]
[New LWP 458398]
[New LWP 458399]
[New LWP 458400]
[New LWP 458401]
[New LWP 458402]
[New LWP 458403]
[New LWP 458404]
[New LWP 458405]
[New LWP 458406]
[New LWP 458407]
[New LWP 458408]
[New LWP 458409]
[New LWP 458410]
[New LWP 458411]
[New LWP 458412]
[New LWP 458413]
[New LWP 458414]
[New LWP 458415]
[New LWP 458416]
[New LWP 458417]
[New LWP 458418]
[New LWP 458419]
[New LWP 458420]
[New LWP 458421]
[New LWP 458422]
[New LWP 458423]
[New LWP 458424]
[New LWP 458425]
[New LWP 458426]
[New LWP 458427]
[New LWP 458428]
[New LWP 458429]
[New LWP 458430]
[New LWP 458431]
[New LWP 458432]
[New LWP 458433]
[New LWP 458434]
[New LWP 458435]
[New LWP 458436]
[New LWP 458437]
[New LWP 458438]
[New LWP 458439]
[New LWP 458440]
[New LWP 458441]
[New LWP 458442]
[New LWP 458443]
[New LWP 458444]
[New LWP 458445]
[New LWP 458446]
[New LWP 458447]
[New LWP 458448]
[New LWP 458449]
[New LWP 458450]
[New LWP 458451]
[New LWP 458452]
[New LWP 458453]
[New LWP 458454]
[New LWP 458455]
[New LWP 458456]
[New LWP 458457]
[New LWP 458458]
[New LWP 458459]
[New LWP 458460]
[New LWP 458461]
[New LWP 458462]
[New LWP 458463]
[New LWP 458464]
[New LWP 458465]
[New LWP 458466]
[New LWP 458467]
[New LWP 458468]
[New LWP 458469]
[New LWP 458470]
[New LWP 458471]
[New LWP 458472]
[New LWP 458473]
[New LWP 458474]
[New LWP 458475]
[New LWP 458476]
[New LWP 458477]
[New LWP 458478]
[New LWP 458479]
[New LWP 458480]
[New LWP 458481]
[New LWP 458482]
[New LWP 458483]
[New LWP 458484]
[New LWP 458485]
[New LWP 458486]
[New LWP 458487]
[New LWP 458488]
[New LWP 458489]
[New LWP 458490]
[New LWP 458491]
[New LWP 458492]
[New LWP 458493]
[New LWP 458494]
[New LWP 458498]
[New LWP 458499]
[New LWP 458500]
[New LWP 458501]
[New LWP 458502]
[New LWP 458503]
[New LWP 458504]
[New LWP 458505]
[New LWP 458506]
[New LWP 458507]
[New LWP 458508]
[New LWP 458509]
[New LWP 458510]
[New LWP 458511]
[New LWP 458512]
[New LWP 458513]
[New LWP 458514]
[New LWP 458515]
[New LWP 458516]
[New LWP 458517]
[New LWP 458518]
[New LWP 458519]
[New LWP 458520]
[New LWP 458521]
[New LWP 458522]
[New LWP 458523]
[New LWP 458524]
[New LWP 458525]
[New LWP 458526]
[New LWP 458527]
[New LWP 458528]
[New LWP 458529]
[New LWP 458530]
[New LWP 458531]
[New LWP 458532]
[New LWP 458533]
[New LWP 458534]
[New LWP 458535]
[New LWP 458536]
[New LWP 458537]
[New LWP 458538]
[New LWP 458539]
[New LWP 458540]
[New LWP 458541]
[New LWP 458542]
[New LWP 458543]
[New LWP 458544]
[New LWP 458545]
[New LWP 458546]
[New LWP 458547]
[New LWP 458548]
[New LWP 458549]
[New LWP 458550]
[New LWP 458551]
[New LWP 458552]
[New LWP 458553]
[New LWP 458554]
[New LWP 458555]
[New LWP 458556]
[New LWP 458557]
[New LWP 458558]
[New LWP 458559]
[New LWP 458560]
[New LWP 458561]
[New LWP 458562]
[New LWP 458563]
[New LWP 458564]
[New LWP 458565]
[New LWP 458566]
[New LWP 458567]
[New LWP 458568]
[New LWP 458569]
[New LWP 458570]
[New LWP 458571]
[New LWP 458572]
[New LWP 458573]
[New LWP 458574]
[New LWP 458575]
[New LWP 458576]
[New LWP 458577]
[New LWP 458578]
[New LWP 458579]
[New LWP 458580]
[New LWP 458581]
[New LWP 458582]
[New LWP 458583]
[New LWP 458584]
[New LWP 458585]
[New LWP 458586]
[New LWP 458587]
[New LWP 458588]
[New LWP 458589]
[New LWP 458619]
[New LWP 458620]
[New LWP 458621]
[New LWP 458622]
[New LWP 458623]
[New LWP 458624]
[New LWP 458625]
[New LWP 458626]
[New LWP 458627]
[New LWP 458628]
[New LWP 458629]
[New LWP 458630]
[New LWP 458631]
[New LWP 458632]
[New LWP 458633]
[New LWP 458634]
[New LWP 458635]
[New LWP 458636]
[New LWP 458637]
[New LWP 458638]
[New LWP 458639]
[New LWP 458640]
[New LWP 458641]
[New LWP 458642]
[New LWP 458646]
[New LWP 458647]
[New LWP 458648]
[New LWP 458649]
[New LWP 458654]
[New LWP 458669]
[New LWP 458676]
[New LWP 458685]
[New LWP 458704]
[New LWP 458705]
[New LWP 458706]
[New LWP 458717]
[New LWP 458729]
[New LWP 458765]
[New LWP 458766]
[New LWP 458767]
[New LWP 458768]
[New LWP 458769]
[New LWP 458770]
[New LWP 458771]
[New LWP 458772]
[New LWP 458773]
[New LWP 458774]
[New LWP 458775]
[New LWP 458776]
[New LWP 458777]
[New LWP 458778]
[New LWP 458779]
[New LWP 458780]
[New LWP 458783]
[New LWP 458784]
[New LWP 458785]
[New LWP 458786]
[New LWP 458787]
[New LWP 458788]
[New LWP 458798]
[New LWP 458799]
[New LWP 458800]
[New LWP 458801]
[New LWP 458802]
[New LWP 458803]
[New LWP 458804]
[New LWP 458805]
[New LWP 458806]
[New LWP 458807]
[New LWP 458816]
[New LWP 460155]
[New LWP 460156]
[New LWP 460157]
[New LWP 460158]
[New LWP 460159]
[New LWP 460160]
[New LWP 460161]
[New LWP 460162]
[New LWP 460179]
[New LWP 460902]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
__futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55a549d8c668 <mongo::(anonymous namespace)::shutdownTasksComplete+40>) at ./nptl/futex-internal.c:57
57    ./nptl/futex-internal.c: No such file or directory.
(gdb) bt
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0,
    futex_word=0x55a549d8c668 <mongo::(anonymous namespace)::shutdownTasksComplete+40>) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0,
    futex_word=0x55a549d8c668 <mongo::(anonymous namespace)::shutdownTasksComplete+40>) at ./nptl/futex-internal.c:87
#2  _{_}GI{_}__futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55a549d8c668 <mongo::(anonymous namespace)::shutdownTasksComplete+40>,
    expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007fd183046ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55a54bb04d10,
    cond=0x55a549d8c640 <mongo::(anonymous namespace)::shutdownTasksComplete>) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x55a549d8c640 <mongo::(anonymous namespace)::shutdownTasksComplete>, mutex=0x55a54bb04d10) at ./nptl/pthread_cond_wait.c:627
#5  0x000055a548c6ac4c in std::condition_variable::wait(std::unique_lock<std::mutex>&) ()
#6  0x000055a548abc487 in mongo::waitForShutdown() ()
#7  0x000055a546cd1939 in mongo::mongos_main(int, char**) ()
#8  0x000055a546b80bfc in main ()

 



 Comments   
Comment by Yuan Fang [ 15/Feb/23 ]

amit.gupta@opensense.com, Glad to hear it's been resolved. I'll close this ticket now as requested.

Comment by Amit Gupta [ 15/Feb/23 ]

Hi, we resynced every node in the cluster including the config servers last friday night.  The cluster seems to be running smoothly since.  I did not notice any dmesg or any other log entries on the server to indicate a disk problem.  So I'm not sure how the data file got corrupted.  

 

The cluster is operating smoothly right now.  Feel free to close out the ticket!  Thanks for looking into this.

Comment by Yuan Fang [ 14/Feb/23 ]

Hi amit.gupta@opensense.com,

Thank you for providing the diagnostic dataset. After reviewing it with the Triage team, the error messages lead us to suspect some form of physical corruption.

The message of the Invariant failure at the first crash in the mongod.log:

__posix_file_read:428:/var/lib/mongo/collection-97--9154580971862015175.wt: handle-read: pread: failed to read 24576 bytes at offset 101808472064 

The failure message after restarting from the crash also indicates the presence of possible data corruption. :

Failed to open WiredTiger cursor. This may be due to data corruption

The ideal resolution is to perform a clean resync from an unaffected node.

You can also try mongod --repair using the latest patch release of your version (v6.0.4) of MongoDB.

In the event that a --repair operation is unsuccessful, then please also provide:

  • The logs of the repair operation.
  • The logs of any attempt to start mongod after the repair operation completed.

Regards,
Yuan

Comment by Amit Gupta [ 11/Feb/23 ]

It looks like the issue is happening in the code here: https://github.com/wiredtiger/wiredtiger/blob/develop/src/os_posix/os_fs.c

 

if (ret != 0)
            WT_RET_MSG(session, nr == 0 ? WT_ERROR : ret,
              "%s: handle-read: pread: failed to read %" WT_SIZET_FMT " bytes at offset %" PRIuMAX,
              file_handle->name, chunk, (uintmax_t)offset);
    }

Comment by Amit Gupta [ 10/Feb/23 ]

I tried restarting the service and it looks like it crashes the same way.  I uploaded the log file after restarting the crash.

Comment by Amit Gupta [ 10/Feb/23 ]

Here is the log snippet in case it's helpful for other people searching for simple issue (there is more but it's too long to put in this thread.  I'm uploading it separately):

 

{"t":\{"$date":"2023-02-10T13:29:52.829-05:00"}

,"s":"E",  "c":"WT",       "id":22435,   "ctx":"conn5691","msg":"WiredTiger error message","attr":{"error":-31802,"message":

{"ts_sec":1676053792,"ts_usec":829607,"thread":"3356:0x7f534576d700","session_dhandle_name":"file:collection-97--9154580971862015175.wt","session_name":"WT_CURSOR.search","category":"WT_VERB_DEFAULT","category_id":9,"verbose_level":"ERROR","verbose_level_id":-3,"msg":"__posix_file_read:428:/var/lib/mongo/collection-97--9154580971862015175.wt: handle-read: pread: failed to read 24576 bytes at offset 101808472064","error_str":"WT_ERROR: non-specific WiredTiger error","error_code":-31802}

}} {"t":

{"$date":"2023-02-10T13:29:52.829-05:00"}

,"s":"F",  "c":"ASSERT",   "id":23083,   "ctx":"conn5691","msg":"Invariant failure","attr":{"expr":"seekRet","error":"UnknownError: -31802: WT_ERROR: non-specific WiredTiger error","file":"src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp","line":2281}} {"t":

{"$date":"2023-02-10T13:29:52.829-05:00"}

,"s":"F",  "c":"ASSERT",   "id":23084,   "ctx":"conn5691","msg":"\n\n***aborting after invariant() failure\n\n"} {"t":

{"$date":"2023-02-10T13:29:52.829-05:00"}

,"s":"F",  "c":"CONTROL",  "id":6384300, "ctx":"conn5691","msg":"Writing fatal message","attr":{"message":"Got signal: 6 (Aborted).\n"}} {"t":

{"$date":"2023-02-10T13:29:53.035-05:00"}

,"s":"I",  "c":"CONTROL",  "id":31380,   "ctx":"conn5691","msg":"BACKTRACE","attr":{"bt":{"backtrace":[

{"a":"55DE5BC4A364","b":"55DE56E6D000","o":"4DDD364","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.362","C":"mongo::stack_trace_detail::(anonymous namespace)::printStackTraceImpl(mongo::stack_trace_detail::(anonymous namespace)::Options const&, mongo::StackTraceSink*) [clone .constprop.362]","s+":"1F4"}

,{"a":"55DE5BC4C8A9","b":"55DE56E6D000","o":"4DDF8A9","s":"ZN5mongo15printStackTraceEv","C":"mongo::printStackTrace()","s+":"29"},{"a":"55DE5BC46586","b":"55DE56E6D000","o":"4DD9586","s":"abruptQuit","s+":"66"},{"a":"7F5367C90420","b":"7F5367C7C000","o":"14420","s":"funlockfile","s+":"60"},{"a":"7F5367ACD00B","b":"7F5367A8A000","o":"4300B","s":"gsignal","s+":"CB"},{"a":"7F5367AAC859","b":"7F5367A8A000","o":"22859","s":"abort","s+":"12B"},{"a":"55DE58D6C851","b":"55DE56E6D000","o":"1EFF851","s":"_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j","C":"mongo::invariantOKFailed(char const*, mongo::Status const&, char const*, unsigned int)","s+":"145"},{"a":"55DE5980BB51","b":"55DE56E6D000","o":"299EB51","s":"_ZN5mongo31WiredTigerRecordStoreCursorBase9seekExactERKNS_8RecordIdE","C":"mongo::WiredTigerRecordStoreCursorBase::seekExact(mongo::RecordId const&)","s+":"2D1"},{"a":"55DE5A12A6E8","b":"55DE56E6D000","o":"32BD6E8","s":"_ZN5mongo16WorkingSetCommon5fetchEPNS_16OperationContextEPNS_10WorkingSetEmPNS_20SeekableRecordCursorERKNS_13CollectionPtrERKNS_15NamespaceStringE","C":"mongo::WorkingSetCommon::fetch(mongo::OperationContext*, mongo::WorkingSet*, unsigned long, mongo::SeekableRecordCursor*, mongo::CollectionPtr const&, mongo::NamespaceString const&)","s+":"78"},{"a":"55DE5A0DDE84","b":"55DE56E6D000","o":"3270E84","s":"_ZN5mongo10FetchStage6doWorkEPm","C":"mongo::FetchStage::doWork(unsigned long*)","s+":"174"},{"a":"55DE5A110ABA","b":"55DE56E6D000","o":"32A3ABA","s":"_ZN5mongo16ShardFilterStage6doWorkEPm","C":"mongo::ShardFilterStage::doWork(unsigned long*)","s+":"9A"},{"a":"55DE5A0EB605","b":"55DE56E6D000","o":"327E605","s":"_ZN5mongo10LimitStage6doWorkEPm","C":"mongo::LimitStage::doWork(unsigned long*)","s+":"85"},{"a":"55DE5A0CD52D","b":"55DE56E6D000","o":"326052D","s":"_ZN5mongo9PlanStage4workEPm","C":"mongo::PlanStage::work(unsigned long*)","s+":"3D"},{"a":"55DE5A0CC44D","b":"55DE56E6D000","o":"325F44D","s":"_ZN5mongo15CachedPlanStage12pickBestPlanEPNS_15PlanYieldPolicyE","C":"mongo::CachedPlanStage::pickBestPlan(mongo::PlanYieldPolicy*)","s+":"14D"},{"a":"55DE5A17084A","b":"55DE56E6D000","o":"330384A","s":"_ZN5mongo16PlanExecutorImpl13_pickBestPlanEv","C":"mongo::PlanExecutorImpl::_pickBestPlan()","s+":"BA"},{"a":"55DE5A171C65","b":"55DE56E6D000","o":"3304C65","s":"_ZN5mongo16PlanExecutorImplC1EPNS_16OperationContextESt10unique_ptrINS_10WorkingSetESt14default_deleteIS4_EES3_INS_9PlanStageES5_IS8_EES3_INS_13QuerySolutionES5_ISB_EES3_INS_14CanonicalQueryES5_ISE_EERKN5boost13intrusive_ptrINS_17ExpressionContextEEERKNS_13CollectionPtrEbNS_15NamespaceStringENS_15PlanYieldPolicy11YieldPolicyE","C":"mongo::PlanExecutorImpl::PlanExecutorImpl(mongo::OperationContext*, std::unique_ptr<mongo::WorkingSet, std::default_delete<mongo::WorkingSet> >, std::unique_ptr<mongo::PlanStage, std::default_delete<mongo::PlanStage> >, std::unique_ptr<mongo::QuerySolution, std::default_delete<mongo::QuerySolution> >, std::unique_ptr<mongo::CanonicalQuery, std::default_delete<mongo::CanonicalQuery> >, boost::intrusive_ptr<mongo::ExpressionContext> const&, mongo::CollectionPtr const&, bool, mongo::NamespaceString, mongo::PlanYieldPolicy::YieldPolicy)","s+":"445"},{"a":"55DE5A16C9C6","b":"55DE56E6D000","o":"32FF9C6","s":"_ZN5mongo21plan_executor_factory4makeEPNS_16OperationContextESt10unique_ptrINS_10WorkingSetESt14default_deleteIS4_EES3_INS_9PlanStageES5_IS8_EES3_INS_13QuerySolutionES5_ISB_EES3_INS_14CanonicalQueryES5_ISE_EERKN5boost13intrusive_ptrINS_17ExpressionContextEEEPKNS_13CollectionPtrEmNS_15NamespaceStringENS_15PlanYieldPolicy11YieldPolicyE","C":"mongo::plan_executor_factory::make(mongo::OperationContext*, std::unique_ptr<mongo::WorkingSet, std::default_delete<mongo::WorkingSet> >, std::unique_ptr<mongo::PlanStage, std::default_delete<mongo::PlanStage> >, std::unique_ptr<mongo::QuerySolution, std::default_delete<mongo::QuerySolution> >, std::unique_ptr<mongo::CanonicalQuery, std::default_delete<mongo::CanonicalQuery> >, boost::intrusive_ptr<mongo::ExpressionContext> const&, mongo::CollectionPtr const*, unsigned long, mongo::NamespaceString, mongo::PlanYieldPolicy::YieldPolicy)","s+":"106"},{"a":"55DE5A16CE83","b":"55DE56E6D000","o":"32FFE83","s":"_ZN5mongo21plan_executor_factory4makeESt10unique_ptrINS_14CanonicalQueryESt14default_deleteIS2_EES1_INS_10WorkingSetES3_IS6_EES1_INS_9PlanStageES3_IS9_EEPKNS_13CollectionPtrENS_15PlanYieldPolicy11YieldPolicyEmNS_15NamespaceStringES1_INS_13QuerySolutionES3_ISI_EE","C":"mongo::plan_executor_factory::make(std::unique_ptr<mongo::CanonicalQuery, std::default_delete<mongo::CanonicalQuery> >, std::unique_ptr<mongo::WorkingSet, std::default_delete<mongo::WorkingSet> >, std::unique_ptr<mongo::PlanStage, std::default_delete<mongo::PlanStage> >, mongo::CollectionPtr const*, mongo::PlanYieldPolicy::YieldPolicy, unsigned long, mongo::NamespaceString, std::unique_ptr<mongo::QuerySolution, std::default_delete<mongo::QuerySolution> >)","s+":"113"},{"a":"55DE5A15AE5C","b":"55DE56E6D000","o":"32EDE5C","s":"_ZN5mongo12_GLOBAL_N_118getClassicExecutorEPNS_16OperationContextERKNS_13CollectionPtrESt10unique_ptrINS_14CanonicalQueryESt14default_deleteIS7_EENS_15PlanYieldPolicy11YieldPolicyERKNS_18QueryPlannerParamsE","C":"mongo::(anonymous namespace)::getClassicExecutor(mongo::OperationContext*, mongo::CollectionPtr const&, std::unique_ptr<mongo::CanonicalQuery, std::default_delete<mongo::CanonicalQuery> >, mongo::PlanYieldPolicy::YieldPolicy, mongo::QueryPlannerParams const&)","s+":"20C"},{"a":"55DE5A15B0F8","b":"55DE56E6D000","o":"32EE0F8","s":"ZN5mongo11getExecutorEPNS_16OperationContextERKNS_26MultipleCollectionAccessorESt10unique_ptrINS_14CanonicalQueryESt14default_deleteIS6_EESt8functionIFvPS6_EENS_15PlanYieldPolicy11YieldPolicyERKNS_18QueryPlannerParamsE","C":"mongo::getExecutor(mongo::OperationContext*, mongo::MultipleCollectionAccessor const&, std::unique_ptr<mongo::CanonicalQuery, std::default_delete<mongo::CanonicalQuery> >, std::function<void (mongo::CanonicalQuery*)>, mongo::PlanYieldPolicy::YieldPolicy, mongo::QueryPlannerParams const&)","s+":"148"},{"a":"55DE5A15B1FA","b":"55DE56E6D000","o":"32EE1FA","s":"_ZN5mongo15getExecutorFindEPNS_16OperationContextERKNS_26MultipleCollectionAccessorESt10unique_ptrINS_14CanonicalQueryESt14default_deleteIS6_EESt8functionIFvPS6_EEbNS_18QueryPlannerParamsE","C":"mongo::getExecutorFind(mongo::OperationContext*, mongo::MultipleCollectionAccessor const&, std::unique_ptr<mongo::CanonicalQuery, std::default_delete<mongo::CanonicalQuery> >, std::function<void (mongo::CanonicalQuery*)>, bool, mongo::QueryPlannerParams)","s+":"AA"},{"a":"55DE5A1603B1","b":"55DE56E6D000","o":"32F33B1","s":"_ZN5mongo15getExecutorFindEPNS_16OperationContextEPKNS_13CollectionPtrESt10unique_ptrINS_14CanonicalQueryESt14default_deleteIS6_EESt8functionIFvPS6_EEbm","C":"mongo::getExecutorFind(mongo::OperationContext*, mongo::CollectionPtr const*, std::unique_ptr<mongo::CanonicalQuery, std::default_delete<mongo::CanonicalQuery> >, std::function<void (mongo::CanonicalQuery*)>, bool, unsigned long)","s+":"F1"},{"a":"55DE59B8781C","b":"55DE56E6D000","o":"2D1A81C","s":"_ZN5mongo12_GLOBAL_N_17FindCmd10Invocation3runEPNS_16OperationContextEPNS_3rpc21ReplyBuilderInterfaceE","C":"mongo::(anonymous namespace)::FindCmd::Invocation::run(mongo::OperationContext*, mongo::rpc::ReplyBuilderInterface*)","s+":"BFC"},{"a":"55DE5A9D6591","b":"55DE56E6D000","o":"3B69591","s":"ZN5mongo14CommandHelpers20runCommandInvocationEPNS_16OperationContextERKNS_12OpMsgRequestEPNS_17CommandInvocationEPNS_3rpc21ReplyBuilderInterfaceE","C":"mongo::CommandHelpers::runCommandInvocation(mongo::OperationContext*, mongo::OpMsgRequest const&, mongo::CommandInvocation*, mongo::rpc::ReplyBuilderInterface*)","s+":"61"},{"a":"55DE5A9DA609","b":"55DE56E6D000","o":"3B6D609","s":"_ZN5mongo14CommandHelpers20runCommandInvocationESt10shared_ptrINS_23RequestExecutionContextEES1_INS_17CommandInvocationEENS_9transport15ServiceExecutor14ThreadingModelE","C":"mongo::CommandHelpers::runCommandInvocation(std::shared_ptr<mongo::RequestExecutionContext>, std::shared_ptr<mongo::CommandInvocation>, mongo::transport::ServiceExecutor::ThreadingModel)","s+":"1C9"},{"a":"55DE58FD4A47","b":"55DE56E6D000","o":"2167A47","s":"_ZN5mongo12_GLOBAL_N_120runCommandInvocationESt10shared_ptrINS_23RequestExecutionContextEES1_INS_17CommandInvocationEE","C":"mongo::(anonymous namespace)::runCommandInvocation(std::shared_ptr<mongo::RequestExecutionContext>, std::shared_ptr<mongo::CommandInvocation>)","s+":"97"},{"a":"55DE58FE4B46","b":"55DE56E6D000","o":"2177B46","s":"ZN5mongo12_GLOBAL_N_114RunCommandImpl11_runCommandEv","C":"mongo::(anonymous namespace)::RunCommandImpl::runCommand()","s+":"146"},{"a":"55DE58FE5B1B","b":"55DE56E6D000","o":"2178B1B","s":"_ZN5mongo12_GLOBAL_N_114RunCommandImpl8_runImplEv","C":"mongo::(anonymous namespace)::RunCommandImpl::runImpl()","s+":"8B"},{"a":"55DE58FD5AFC","b":"55DE56E6D000","o":"2168AFC","s":"_ZN5mongo12_GLOBAL_N_114RunCommandImpl3runEv","C":"mongo::(anonymous namespace)::RunCommandImpl::run()","s+":"13C"},{"a":"55DE58FDCC3E","b":"55DE56E6D000","o":"216FC3E","s":"ZN5mongo12_GLOBAL_N_119ExecCommandDatabase12_commandExecEv","C":"mongo::(anonymous namespace)::ExecCommandDatabase::commandExec()","s+":"34E"},{"a":"55DE58FE177F","b":"55DE56E6D000","o":"217477F","s":"_ZZN5mongo12_GLOBAL_N_114executeCommandESt10shared_ptrINS0_13HandleRequest16ExecutionContextEEENUlvE0_clEv","C":"mongo::(anonymous namespace)::executeCommand(std::shared_ptr<mongo::(anonymous namespace)::HandleRequest::ExecutionContext>)::

{lambda()#2}::operator()()","s+":"43F"},{"a":"55DE58FE3370","b":"55DE56E6D000","o":"2176370","s":"ZZN5mongo15unique_functionIFvPNS_14future_details15SharedStateBaseEEE8makeImplIZNS1_10FutureImplINS1_8FakeVoidEE16makeContinuationIvZZNOS9_4thenIZNS_12_GLOBAL_N_114executeCommandESt10shared_ptrINSC_13HandleRequest16ExecutionContextEEEUlvE0_EEDaOT_ENKUlvE1_clEvEUlPNS1_15SharedStateImplIS8_EESN_E_EENS7_ISI_EEOT0_EUlS3_E_EEDaSJ_EN12SpecificImpl4callEOS3_","C":"mongo::unique_function<void (mongo::future_details::SharedStateBase*)>::makeImpl<mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::makeContinuation<void, mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::then<mongo::(anonymous namespace)::executeCommand(std::shared_ptr<mongo::(anonymous namespace)::HandleRequest::ExecutionContext>)::{lambda()#2}

>(mongo::(anonymous namespace)::executeCommand(std::shared_ptr<mongo::(anonymous namespace)::HandleRequest::ExecutionContext>)::{lambda()#2}&&) &&::{lambda()#3}::operator()() const::{lambda(mongo::future_details::SharedStateImpl<mongo::future_details::FakeVoid>, mongo::future_details::SharedStateImpl)#1}>(mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::then<mongo::(anonymous namespace)::executeCommand(std::shared_ptr<mongo::(anonymous namespace)::HandleRequest::ExecutionContext>)::{lambda()#2}>(mongo::(anonymous namespace)::executeCommand(std::shared_ptr<mongo::(anonymous namespace)::HandleRequest::ExecutionContext>)::{lambda()#2}&&) &&::{lambda()#3}::operator()() const::{lambda(mongo::future_details::SharedStateImpl<mongo::future_details::FakeVoid>, mongo::future_details::SharedStateImpl)#1}&&)::{lambda(mongo::future_details::SharedStateBase*)#1}>(mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::makeContinuation<void, mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::then<mongo::(anonymous namespace)::executeCommand(std::shared_ptr<mongo::(anonymous namespace)::HandleRequest::ExecutionContext>)::{lambda()#2}>(mongo::(anonymous namespace)::executeCommand(std::shared_ptr<mongo::(anonymous namespace)::HandleRequest::ExecutionContext>)::{lambda()#2}&&) &&::{lambda()#3}::operator()() const::{lambda(mongo::future_details::SharedStateImpl<mongo::future_details::FakeVoid>, mongo::future_details::SharedStateImpl)#1}>(mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::then<mongo::(anonymous namespace)::executeCommand(std::shared_ptr<mongo::(anonymous namespace)::HandleRequest::ExecutionContext>)::{lambda()#2}>(mongo::(anonymous namespace)::executeCommand(std::shared_ptr<mongo::(anonymous namespace)::HandleRequest::ExecutionContext>)::{lambda()#2}&&) &&::{lambda()#3}::operator()() const::{lambda(mongo::future_details::SharedStateImpl<mongo::future_details::FakeVoid>, mongo::future_details::SharedStateImpl)#1}&&)::{lambda(mongo::future_details::SharedStateBase*)#1})::SpecificImpl::call(mongo::future_details::SharedStateBase*&&)","s+":"80"},{"a":"55DE58F93EA7","b":"55DE56E6D000","o":"2126EA7","s":"ZN5mongo14future_details15SharedStateBase20transitionToFinishedEv","C":"mongo::future_details::SharedStateBase::transitionToFinished()","s+":"147"},{"a":"55DE58FE72D2","b":"55DE56E6D000","o":"217A2D2","s":"_ZN5mongo14future_details10FutureImplINS0_8FakeVoidEE11generalImplIZNOS3_17propagateResultToEPNS0_15SharedStateImplIS2_EEEUlOS2_E_ZNOS3_17propagateResultToES7_EUlONS_6StatusEE0_ZNOS3_17propagateResultToES7_EUlvE1_EEDaOT_OT0_OT1","C":"auto mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::generalImpl<mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::propagateResultTo(mongo::future_details::SharedStateImpl<mongo::future_details::FakeVoid>*) &&::

{lambda(mongo::future_details::FakeVoid&&)#1}

, mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::propagateResultTo(mongo::future_details::SharedStateImpl<mongo::future_details::FakeVoid>) &&::{lambda(mongo::Status&&)#2}, mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::propagateResultTo(mongo::future_details::SharedStateImpl<mongo::future_details::FakeVoid>) &&::{lambda()#3}>(mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::propagateResultTo(mongo::future_details::SharedStateImpl<mongo::future_details::FakeVoid>) &&::{lambda(mongo::future_details::FakeVoid&&)#1}&&, mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::propagateResultTo(mongo::future_details::SharedStateImpl<mongo::future_details::FakeVoid>) &&::{lambda(mongo::Status&&)#2}&&, mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::propagateResultTo(mongo::future_details::SharedStateImpl<mongo::future_details::FakeVoid>) &&::{lambda()#3}&&)","s+":"152"},{"a":"55DE58FD7C3C","b":"55DE56E6D000","o":"216AC3C","s":"ZZN5mongo15unique_functionIFvPNS_14future_details15SharedStateBaseEEE8makeImplIZNS1_10FutureImplINS1_8FakeVoidEE16makeContinuationIvZZNOS9_4thenIZNS_12_GLOBAL_N_114executeCommandESt10shared_ptrINSC_13HandleRequest16ExecutionContextEEEUlvE_EEDaOT_ENKUlvE1_clEvEUlPNS1_15SharedStateImplIS8_EESN_E_EENS7_ISI_EEOT0_EUlS3_E_EEDaSJ_EN12SpecificImpl4callEOS3_","C":"mongo::unique_function<void (mongo::future_details::SharedStateBase)>::makeImpl<mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::makeContinuation<void, mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::then<mongo::(anonymous namespace)::executeCommand(std::shared_ptr<mongo::(anonymous namespace)::HandleRequest::ExecutionContext>)::

{lambda()#1}>(mongo::(anonymous namespace)::executeCommand(std::shared_ptr<mongo::(anonymous namespace)::HandleRequest::ExecutionContext>)::{lambda()#1}&&) &&::{lambda()#3}::operator()() const::{lambda(mongo::future_details::SharedStateImpl<mongo::future_details::FakeVoid>, mongo::future_details::SharedStateImpl)#1}>(mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::then<mongo::(anonymous namespace)::executeCommand(std::shared_ptr<mongo::(anonymous namespace)::HandleRequest::ExecutionContext>)::{lambda()#1}>(mongo::(anonymous namespace)::executeCommand(std::shared_ptr<mongo::(anonymous namespace)::HandleRequest::ExecutionContext>)::{lambda()#1}&&) &&::{lambda()#3}::operator()() const::{lambda(mongo::future_details::SharedStateImpl<mongo::future_details::FakeVoid>, mongo::future_details::SharedStateImpl)#1}&&)::{lambda(mongo::future_details::SharedStateBase*)#1}>(mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::makeContinuation<void, mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::then<mongo::(anonymous namespace)::executeCommand(std::shared_ptr<mongo::(anonymous namespace)::HandleRequest::ExecutionContext>)::{lambda()#1}>(mongo::(anonymous namespace)::executeCommand(std::shared_ptr<mongo::(anonymous namespace)::HandleRequest::ExecutionContext>)::{lambda()#1}&&) &&::{lambda()#3}::operator()() const::{lambda(mongo::future_details::SharedStateImpl<mongo::future_details::FakeVoid>, mongo::future_details::SharedStateImpl)#1}>(mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::then<mongo::(anonymous namespace)::executeCommand(std::shared_ptr<mongo::(anonymous namespace)::HandleRequest::ExecutionContext>)::{lambda()#1}>(mongo::(anonymous namespace)::executeCommand(std::shared_ptr<mongo::(anonymous namespace)::HandleRequest::ExecutionContext>)::{lambda()#1}&&) &&::{lambda()#3}::operator()() const::{lambda(mongo::future_details::SharedStateImpl<mongo::future_details::FakeVoid>, mongo::future_details::SharedStateImpl)#1}&&)::{lambda(mongo::future_details::SharedStateBase*)#1})::SpecificImpl::call(mongo::future_details::SharedStateBase*&&)","s+":"AC"},{"a":"55DE58F93EA7","b":"55DE56E6D000","o":"2126EA7","s":"ZN5mongo14future_details15SharedStateBase20transitionToFinishedEv","C":"mongo::future_details::SharedStateBase::transitionToFinished()","s+":"147"},{"a":"55DE58FE2103","b":"55DE56E6D000","o":"2175103","s":"_ZN5mongo12_GLOBAL_N_114executeCommandESt10shared_ptrINS0_13HandleRequest16ExecutionContextEE","C":"mongo::(anonymous namespace)::executeCommand(std::shared_ptr<mongo::(anonymous namespace)::HandleRequest::ExecutionContext>)","s+":"1D3"},{"a":"55DE58FE2975","b":"55DE56E6D000","o":"2175975","s":"ZN5mongo12_GLOBAL_N_116receivedCommandsESt10shared_ptrINS0_13HandleRequest16ExecutionContextEE","C":"mongo::(anonymous namespace)::receivedCommands(std::shared_ptr<mongo::(anonymous namespace)::HandleRequest::ExecutionContext>)","s+":"2D5"},{"a":"55DE58FE32B6","b":"55DE56E6D000","o":"21762B6","s":"ZN5mongo12_GLOBAL_N_115CommandOpRunner3runEv","C":"mongo::(anonymous namespace)::CommandOpRunner::run()","s+":"36"},{"a":"55DE58FDA28F","b":"55DE56E6D000","o":"216D28F","s":"ZN5mongo23ServiceEntryPointCommon13handleRequestEPNS_16OperationContextERKNS_7MessageESt10unique_ptrIKNS0_5HooksESt14default_deleteIS8_EE","C":"mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*, mongo::Message const&, std::unique_ptr<mongo::ServiceEntryPointCommon::Hooks const, std::default_delete<mongo::ServiceEntryPointCommon::Hooks const> >)","s+":"53F"},{"a":"55DE58FCEA4F","b":"55DE56E6D000","o":"2161A4F","s":"_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE","C":"mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*, mongo::Message const&)","s+":"4F"},{"a":"55DE59E34027","b":"55DE56E6D000","o":"2FC7027","s":"_ZN5mongo9transport19ServiceStateMachine4Impl14processMessageEv","C":"mongo::transport::ServiceStateMachine::Impl::processMessage()","s+":"127"},{"a":"55DE59E35D36","b":"55DE56E6D000","o":"2FC8D36","s":"_ZN5mongo9transport19ServiceStateMachine4Impl12startNewLoopERKNS_6StatusE","C":"mongo::transport::ServiceStateMachine::Impl::startNewLoop(mongo::Status const&)","s+":"76"},{"a":"55DE59E361CF","b":"55DE56E6D000","o":"2FC91CF","s":"_ZZN5mongo15unique_functionIFvNS_6StatusEEE8makeImplIZNS_9transport19ServiceStateMachine4Impl15scheduleNewLoopES1_EUlS1_E_EEDaOT_EN12SpecificImpl4callEOS1","C":"mongo::unique_function<void (mongo::Status)>::makeImpl<mongo::transport::ServiceStateMachine::Impl::scheduleNewLoop(mongo::Status)::{lambda(mongo::Status)#1}>(mongo::transport::ServiceStateMachine::Impl::scheduleNewLoop(mongo::Status)::{lambda(mongo::Status)#1}&&)::SpecificImpl::call(mongo::Status&&)","s+":"7F"},{"a":"55DE5B44B2BD","b":"55DE56E6D000","o":"45DE2BD","s":"ZZN5mongo15unique_functionIFvNS_6StatusEEE8makeImplIZNS_9transport26ServiceExecutorSynchronous18runOnDataAvailableERKSt10shared_ptrINS5_7SessionEES3_EUlS1_E_EEDaOT_EN12SpecificImpl4callEOS1","C":"mongo::unique_function<void (mongo::Status)>::makeImpl<mongo::transport::ServiceExecutorSynchronous::runOnDataAvailable(std::shared_ptr<mongo::transport::Session> const&, mongo::unique_function<void (mongo::Status)>)::{lambda(mongo::Status)#1}>(mongo::transport::ServiceExecutorSynchronous::runOnDataAvailable(std::shared_ptr<mongo::transport::Session> const&, mongo::unique_function<void (mongo::Status)>)::{lambda(mongo::Status)#1}&&)::SpecificImpl::call(mongo::Status&&)","s+":"3D"},{"a":"55DE595217D3","b":"55DE56E6D000","o":"26B47D3","s":"_ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_9transport15ServiceExecutor8scheduleENS0_IFvNS_6StatusEEEEEUlvE_EEDaOT_EN12SpecificImpl4callEv","C":"mongo::unique_function<void ()>::makeImpl<mongo::transport::ServiceExecutor::schedule(mongo::unique_function<void (mongo::Status)>)::{lambda()#1}

>(mongo::transport::ServiceExecutor::schedule(mongo::unique_function<void (mongo::Status)>)::{lambda()#1}&&)::SpecificImpl::call()","s+":"33"},{"a":"55DE5B44C34B","b":"55DE56E6D000","o":"45DF34B","s":"_ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_9transport26ServiceExecutorSynchronous12scheduleTaskES2_NS4_15ServiceExecutor13ScheduleFlagsEEUlvE0_EEDaOT_EN12SpecificImpl4callEv","C":"mongo::unique_function<void ()>::makeImpl<mongo::transport::ServiceExecutorSynchronous::scheduleTask(mongo::unique_function<void ()>, mongo::transport::ServiceExecutor::ScheduleFlags)::

{lambda()#2}

>(mongo::transport::ServiceExecutorSynchronous::scheduleTask(mongo::unique_function<void ()>, mongo::transport::ServiceExecutor::ScheduleFlags)::{lambda()#2}&&)::SpecificImpl::call()","s+":"BB"},{"a":"55DE5B44E16C","b":"55DE56E6D000","o":"45E116C","s":"_ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_25launchServiceWorkerThreadES2_EUlvE2_EEDaOT_EN12SpecificImpl4callEv","C":"mongo::unique_function<void ()>::makeImpl<mongo::launchServiceWorkerThread(mongo::unique_function<void ()>)::

{lambda()#4}

>(mongo::launchServiceWorkerThread(mongo::unique_function<void ()>)::{lambda()#4}&&)::SpecificImpl::call()","s+":"5C"},{"a":"55DE5B44E1DC","b":"55DE56E6D000","o":"45E11DC","s":"ZN5mongo12_GLOBAL_N_17runFuncEPv","C":"mongo::(anonymous namespace)::runFunc(void*)","s+":"1C"},{"a":"7F5367C84609","b":"7F5367C7C000","o":"8609","s":"start_thread","s+":"D9"},{"a":"7F5367BA9133","b":"7F5367A8A000","o":"11F133","s":"clone","s+":"43"}],"processInfo":{"mongodbVersion":"6.0.4","gitVersion":"44ff59461c1353638a71e710f385a566bcd2f547","compiledModules":[],"uname":

{"sysname":"Linux","release":"5.4.0-139-generic","version":"#156-Ubuntu SMP Fri Jan 20 17:27:18 UTC 2023","machine":"x86_64"}

,"somap":[\{"b":"55DE56E6D000","elfType":3,"buildId":"C934FA633D74614303D55DB3674B3E4C19C8AAD6"},\{"b":"7F5367C7C000","path":"/lib/x86_64-linux-gnu/libpthread.so.0","elfType":3,"buildId":"7B4536F41CDAA5888408E82D0836E33DCF436466"},\{"b":"7F5367A8A000","path":"/lib/x86_64-linux-gnu/libc.so.6","elfType":3,"buildId":"1878E6B475720C7C51969E69AB2D276FAE6D1DEE"}]}}},"tags":[]} {"t":

{"$date":"2023-02-10T13:29:53.035-05:00"}

,"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn5691","msg":"Frame","attr":{"frame":

{"a":"55DE5BC4A364","b":"55DE56E6D000","o":"4DDD364","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.362","C":"mongo::stack_trace_detail::(anonymous namespace)::printStackTraceImpl(mongo::stack_trace_detail::(anonymous namespace)::Options const&, mongo::StackTraceSink*) [clone .constprop.362]","s+":"1F4"}

}} {"t":

{"$date":"2023-02-10T13:29:53.035-05:00"}

,"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn5691","msg":"Frame","attr":{"frame":

{"a":"55DE5BC4C8A9","b":"55DE56E6D000","o":"4DDF8A9","s":"_ZN5mongo15printStackTraceEv","C":"mongo::printStackTrace()","s+":"29"}

}} {"t":

{"$date":"2023-02-10T13:29:53.035-05:00"}

,"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn5691","msg":"Frame","attr":{"frame":

{"a":"55DE5BC46586","b":"55DE56E6D000","o":"4DD9586","s":"abruptQuit","s+":"66"}

}} {"t":

{"$date":"2023-02-10T13:29:53.035-05:00"}

,"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn5691","msg":"Frame","attr":{"frame":

{"a":"7F5367C90420","b":"7F5367C7C000","o":"14420","s":"funlockfile","s+":"60"}

}} {"t":

{"$date":"2023-02-10T13:29:53.035-05:00"}

,"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn5691","msg":"Frame","attr":{"frame":

{"a":"7F5367ACD00B","b":"7F5367A8A000","o":"4300B","s":"gsignal","s+":"CB"}

}} {"t":

{"$date":"2023-02-10T13:29:53.035-05:00"}

,"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn5691","msg":"Frame","attr":{"frame":

{"a":"7F5367AAC859","b":"7F5367A8A000","o":"22859","s":"abort","s+":"12B"}

}} {"t":

{"$date":"2023-02-10T13:29:53.035-05:00"}

,"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn5691","msg":"Frame","attr":{"frame":

{"a":"55DE58D6C851","b":"55DE56E6D000","o":"1EFF851","s":"_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j","C":"mongo::invariantOKFailed(char const*, mongo::Status const&, char const*, unsigned int)","s+":"145"}

}} {"t":

{"$date":"2023-02-10T13:29:53.035-05:00"}

,"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn5691","msg":"Frame","attr":{"frame":

{"a":"55DE5980BB51","b":"55DE56E6D000","o":"299EB51","s":"_ZN5mongo31WiredTigerRecordStoreCursorBase9seekExactERKNS_8RecordIdE","C":"mongo::WiredTigerRecordStoreCursorBase::seekExact(mongo::RecordId const&)","s+":"2D1"}

}} {"t":

{"$date":"2023-02-10T13:29:53.035-05:00"}

,"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn5691","msg":"Frame","attr":{"frame":

{"a":"55DE5A12A6E8","b":"55DE56E6D000","o":"32BD6E8","s":"_ZN5mongo16WorkingSetCommon5fetchEPNS_16OperationContextEPNS_10WorkingSetEmPNS_20SeekableRecordCursorERKNS_13CollectionPtrERKNS_15NamespaceStringE","C":"mongo::WorkingSetCommon::fetch(mongo::OperationContext*, mongo::WorkingSet*, unsigned long, mongo::SeekableRecordCursor*, mongo::CollectionPtr const&, mongo::NamespaceString const&)","s+":"78"}

}} {"t":

{"$date":"2023-02-10T13:29:53.035-05:00"}

,"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn5691","msg":"Frame","attr":{"frame":

{"a":"55DE5A0DDE84","b":"55DE56E6D000","o":"3270E84","s":"_ZN5mongo10FetchStage6doWorkEPm","C":"mongo::FetchStage::doWork(unsigned long*)","s+":"174"}

}} {"t":

{"$date":"2023-02-10T13:29:53.035-05:00"}

,"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn5691","msg":"Frame","attr":{"frame":

{"a":"55DE5A110ABA","b":"55DE56E6D000","o":"32A3ABA","s":"_ZN5mongo16ShardFilterStage6doWorkEPm","C":"mongo::ShardFilterStage::doWork(unsigned long*)","s+":"9A"}

}} {"t":

{"$date":"2023-02-10T13:29:53.035-05:00"}

,"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn5691","msg":"Frame","attr":{"frame":

{"a":"55DE5A0EB605","b":"55DE56E6D000","o":"327E605","s":"_ZN5mongo10LimitStage6doWorkEPm","C":"mongo::LimitStage::doWork(unsigned long*)","s+":"85"}

}} {"t":

{"$date":"2023-02-10T13:29:53.035-05:00"}

,"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn5691","msg":"Frame","attr":{"frame":

{"a":"55DE5A0CD52D","b":"55DE56E6D000","o":"326052D","s":"_ZN5mongo9PlanStage4workEPm","C":"mongo::PlanStage::work(unsigned long*)","s+":"3D"}

}} {"t":

{"$date":"2023-02-10T13:29:53.035-05:00"}

,"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn5691","msg":"Frame","attr":{"frame":

{"a":"55DE5A0CC44D","b":"55DE56E6D000","o":"325F44D","s":"_ZN5mongo15CachedPlanStage12pickBestPlanEPNS_15PlanYieldPolicyE","C":"mongo::CachedPlanStage::pickBestPlan(mongo::PlanYieldPolicy*)","s+":"14D"}

}} {"t":

{"$date":"2023-02-10T13:29:53.035-05:00"}

,"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn5691","msg":"Frame","attr":{"frame":

{"a":"55DE5A17084A","b":"55DE56E6D000","o":"330384A","s":"_ZN5mongo16PlanExecutorImpl13_pickBestPlanEv","C":"mongo::PlanExecutorImpl::_pickBestPlan()","s+":"BA"}

}} {"t":

{"$date":"2023-02-10T13:29:53.035-05:00"}

,"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn5691","msg":"Frame","attr":{"frame":

{"a":"55DE5A171C65","b":"55DE56E6D000","o":"3304C65","s":"_ZN5mongo16PlanExecutorImplC1EPNS_16OperationContextESt10unique_ptrINS_10WorkingSetESt14default_deleteIS4_EES3_INS_9PlanStageES5_IS8_EES3_INS_13QuerySolutionES5_ISB_EES3_INS_14CanonicalQueryES5_ISE_EERKN5boost13intrusive_ptrINS_17ExpressionContextEEERKNS_13CollectionPtrEbNS_15NamespaceStringENS_15PlanYieldPolicy11YieldPolicyE","C":"mongo::PlanExecutorImpl::PlanExecutorImpl(mongo::OperationContext*, std::unique_ptr<mongo::WorkingSet, std::default_delete<mongo::WorkingSet> >, std::unique_ptr<mongo::PlanStage, std::default_delete<mongo::PlanStage> >, std::unique_ptr<mongo::QuerySolution, std::default_delete<mongo::QuerySolution> >, std::unique_ptr<mongo::CanonicalQuery, std::default_delete<mongo::CanonicalQuery> >, boost::intrusive_ptr<mongo::ExpressionContext> const&, mongo::CollectionPtr const&, bool, mongo::NamespaceString, mongo::PlanYieldPolicy::YieldPolicy)","s+":"445"}

}} 

Comment by Amit Gupta [ 10/Feb/23 ]

So we had a different crash today on one node in the 3rd shared of our cluster.

 

I'm attaching the diagnostic data as well as the full mongod log.  Here is the output of systemctl status mongod:

 

● mongod.service - MongoDB Database Server
     Loaded: loaded (/lib/systemd/system/mongod.service; disabled; vendor preset: enabled)
     Active: failed (Result: signal) since Fri 2023-02-10 13:29:57 EST; 16min ago
       Docs: https://docs.mongodb.org/manual
    Process: 3356 ExecStart=/usr/bin/mongod --config /etc/mongod.conf (code=killed, signal=ABRT)
   Main PID: 3356 (code=killed, signal=ABRT)

Feb 08 18:18:34 pndb3.sendergen.com systemd[1]: Started MongoDB Database Server.
Feb 08 18:18:34 pndb3.sendergen.com mongod[3340]: {"t":

{"$date":"2023-02-08T23:18:34.510Z"}

,"s":"I",  "c":"CONTROL",  "id":5760901, "ctx":"-","msg":"Applied --setParameter options","attr":{"serverParameters":{"cursorTimeoutMillis":{"default":600000,"value":60000000}}}}
Feb 08 18:18:34 pndb3.sendergen.com mongod[3340]: about to fork child process, waiting until server is ready for connections.
Feb 08 18:18:34 pndb3.sendergen.com mongod[3356]: forked process: 3356
Feb 08 18:18:35 pndb3.sendergen.com mongod[3340]: child process started successfully, parent exiting
Feb 10 13:29:57 pndb3.sendergen.com systemd[1]: mongod.service: Main process exited, code=killed, status=6/ABRT
Feb 10 13:29:57 pndb3.sendergen.com systemd[1]: mongod.service: Failed with result 'signal'.

 

 

Comment by Yuan Fang [ 09/Feb/23 ]

Hi amit.gupta@opensense.com,

Thank you for getting us the diagnostic data!

In the meantime, we looked at the backtrace you provided in this ticket (and SERVER-73597), unfortunately, the backtrace (i.e. gdb output) you provided does not provide us with enough information to diagnose the problem. If possible, could you please provide the thread apply all bt (thread apply all backtrace) output from GDB? the reason we are asking for it is that it provides a more complete picture of the state of the program at the time of the issue.

Regards,
Yuan

Comment by Amit Gupta [ 09/Feb/23 ]

Hi Yuan, this issue is probably unrelated to SERVER-73597.  We upgraded from 5.0.14 to 6.0.4.

 

Will get you the logs requested soon.  

Comment by Yuan Fang [ 09/Feb/23 ]

Hi amit.gupta@opensense.com,

Thank you for your report. We need additional information to diagnose the problem. I've created a secure upload portal for you. Files uploaded to this portal are hosted on Box, are visible only to MongoDB employees, and are routinely deleted after some time.

For each node in the replica set spanning a time period that includes the incident, would you please archive (tar or zip) and upload to that link:

  • the mongod logs
  • the $dbpath/diagnostic.data directory (the contents are described here)

Regards,
Yuan
 

Generated at Thu Feb 08 06:25:40 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.