Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-73805

mongos in 6.0.4 hangs under load

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Minor - P4 Minor - P4
    • None
    • Affects Version/s: 6.0.4
    • Component/s: None
    • None
    • ALL
    • Hide

      start mongos, wait for a while during very busy times..

       

      Show
      start mongos, wait for a while during very busy times..  

      Environment:

      • Ubuntu 22.04 latest patches
      • 128GB ram machine, AMD Epic, as well as Intel machines

      We just upgraded from 5.0.14 to 6.0.4. The upgrade went smoothly last night. Today during peak hours, our app servers that were getting most of our traffic would have mongos randomly hang.    The mongos process looks like it is still running,  however mongosh cannot connect to it.  

       

      I managed to attached gdb to the running process, and here is the backtrace:

      gdb --pid=458174
      GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1
      Copyright (C) 2022 Free Software Foundation, Inc.
      License GPLv3+: GNU GPL version 3 or later <[http://gnu.org/licenses/gpl.html]>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
      Type "show copying" and "show warranty" for details.
      This GDB was configured as "x86_64-linux-gnu".
      Type "show configuration" for configuration details.
      For bug reporting instructions, please see:
      <[https://www.gnu.org/software/gdb/bugs/]>.
      Find the GDB manual and other documentation resources online at:
          <[http://www.gnu.org/software/gdb/documentation/]>.
      
      For help, type "help".
      Type "apropos word" to search for commands related to "word".
      Attaching to process 458174
      [New LWP 458175]
      [New LWP 458176]
      [New LWP 458177]
      [New LWP 458178]
      [New LWP 458179]
      [New LWP 458181]
      [New LWP 458182]
      [New LWP 458183]
      [New LWP 458184]
      [New LWP 458186]
      [New LWP 458187]
      [New LWP 458188]
      [New LWP 458194]
      [New LWP 458196]
      [New LWP 458197]
      [New LWP 458199]
      [New LWP 458200]
      [New LWP 458201]
      [New LWP 458202]
      [New LWP 458203]
      [New LWP 458204]
      [New LWP 458205]
      [New LWP 458206]
      [New LWP 458207]
      [New LWP 458208]
      [New LWP 458209]
      [New LWP 458211]
      [New LWP 458212]
      [New LWP 458213]
      [New LWP 458214]
      [New LWP 458215]
      [New LWP 458216]
      [New LWP 458217]
      [New LWP 458218]
      [New LWP 458219]
      [New LWP 458220]
      [New LWP 458221]
      [New LWP 458222]
      [New LWP 458223]
      [New LWP 458224]
      [New LWP 458225]
      [New LWP 458226]
      [New LWP 458227]
      [New LWP 458229]
      [New LWP 458230]
      [New LWP 458231]
      [New LWP 458232]
      [New LWP 458233]
      [New LWP 458234]
      [New LWP 458235]
      [New LWP 458236]
      [New LWP 458237]
      [New LWP 458238]
      [New LWP 458239]
      [New LWP 458240]
      [New LWP 458241]
      [New LWP 458242]
      [New LWP 458243]
      [New LWP 458244]
      [New LWP 458245]
      [New LWP 458246]
      [New LWP 458247]
      [New LWP 458248]
      [New LWP 458249]
      [New LWP 458250]
      [New LWP 458251]
      [New LWP 458252]
      [New LWP 458275]
      [New LWP 458276]
      [New LWP 458277]
      [New LWP 458278]
      [New LWP 458285]
      [New LWP 458286]
      [New LWP 458287]
      [New LWP 458288]
      [New LWP 458289]
      [New LWP 458290]
      [New LWP 458291]
      [New LWP 458292]
      [New LWP 458293]
      [New LWP 458294]
      [New LWP 458295]
      [New LWP 458296]
      [New LWP 458297]
      [New LWP 458298]
      [New LWP 458299]
      [New LWP 458300]
      [New LWP 458301]
      [New LWP 458302]
      [New LWP 458303]
      [New LWP 458304]
      [New LWP 458305]
      [New LWP 458306]
      [New LWP 458307]
      [New LWP 458308]
      [New LWP 458309]
      [New LWP 458310]
      [New LWP 458311]
      [New LWP 458312]
      [New LWP 458313]
      [New LWP 458314]
      [New LWP 458315]
      [New LWP 458316]
      [New LWP 458317]
      [New LWP 458318]
      [New LWP 458319]
      [New LWP 458320]
      [New LWP 458321]
      [New LWP 458322]
      [New LWP 458323]
      [New LWP 458324]
      [New LWP 458325]
      [New LWP 458326]
      [New LWP 458327]
      [New LWP 458328]
      [New LWP 458329]
      [New LWP 458330]
      [New LWP 458331]
      [New LWP 458332]
      [New LWP 458333]
      [New LWP 458334]
      [New LWP 458335]
      [New LWP 458336]
      [New LWP 458337]
      [New LWP 458338]
      [New LWP 458339]
      [New LWP 458340]
      [New LWP 458341]
      [New LWP 458342]
      [New LWP 458343]
      [New LWP 458344]
      [New LWP 458345]
      [New LWP 458346]
      [New LWP 458347]
      [New LWP 458348]
      [New LWP 458349]
      [New LWP 458350]
      [New LWP 458351]
      [New LWP 458352]
      [New LWP 458353]
      [New LWP 458380]
      [New LWP 458381]
      [New LWP 458382]
      [New LWP 458384]
      [New LWP 458385]
      [New LWP 458386]
      [New LWP 458387]
      [New LWP 458388]
      [New LWP 458389]
      [New LWP 458390]
      [New LWP 458391]
      [New LWP 458392]
      [New LWP 458393]
      [New LWP 458394]
      [New LWP 458395]
      [New LWP 458396]
      [New LWP 458397]
      [New LWP 458398]
      [New LWP 458399]
      [New LWP 458400]
      [New LWP 458401]
      [New LWP 458402]
      [New LWP 458403]
      [New LWP 458404]
      [New LWP 458405]
      [New LWP 458406]
      [New LWP 458407]
      [New LWP 458408]
      [New LWP 458409]
      [New LWP 458410]
      [New LWP 458411]
      [New LWP 458412]
      [New LWP 458413]
      [New LWP 458414]
      [New LWP 458415]
      [New LWP 458416]
      [New LWP 458417]
      [New LWP 458418]
      [New LWP 458419]
      [New LWP 458420]
      [New LWP 458421]
      [New LWP 458422]
      [New LWP 458423]
      [New LWP 458424]
      [New LWP 458425]
      [New LWP 458426]
      [New LWP 458427]
      [New LWP 458428]
      [New LWP 458429]
      [New LWP 458430]
      [New LWP 458431]
      [New LWP 458432]
      [New LWP 458433]
      [New LWP 458434]
      [New LWP 458435]
      [New LWP 458436]
      [New LWP 458437]
      [New LWP 458438]
      [New LWP 458439]
      [New LWP 458440]
      [New LWP 458441]
      [New LWP 458442]
      [New LWP 458443]
      [New LWP 458444]
      [New LWP 458445]
      [New LWP 458446]
      [New LWP 458447]
      [New LWP 458448]
      [New LWP 458449]
      [New LWP 458450]
      [New LWP 458451]
      [New LWP 458452]
      [New LWP 458453]
      [New LWP 458454]
      [New LWP 458455]
      [New LWP 458456]
      [New LWP 458457]
      [New LWP 458458]
      [New LWP 458459]
      [New LWP 458460]
      [New LWP 458461]
      [New LWP 458462]
      [New LWP 458463]
      [New LWP 458464]
      [New LWP 458465]
      [New LWP 458466]
      [New LWP 458467]
      [New LWP 458468]
      [New LWP 458469]
      [New LWP 458470]
      [New LWP 458471]
      [New LWP 458472]
      [New LWP 458473]
      [New LWP 458474]
      [New LWP 458475]
      [New LWP 458476]
      [New LWP 458477]
      [New LWP 458478]
      [New LWP 458479]
      [New LWP 458480]
      [New LWP 458481]
      [New LWP 458482]
      [New LWP 458483]
      [New LWP 458484]
      [New LWP 458485]
      [New LWP 458486]
      [New LWP 458487]
      [New LWP 458488]
      [New LWP 458489]
      [New LWP 458490]
      [New LWP 458491]
      [New LWP 458492]
      [New LWP 458493]
      [New LWP 458494]
      [New LWP 458498]
      [New LWP 458499]
      [New LWP 458500]
      [New LWP 458501]
      [New LWP 458502]
      [New LWP 458503]
      [New LWP 458504]
      [New LWP 458505]
      [New LWP 458506]
      [New LWP 458507]
      [New LWP 458508]
      [New LWP 458509]
      [New LWP 458510]
      [New LWP 458511]
      [New LWP 458512]
      [New LWP 458513]
      [New LWP 458514]
      [New LWP 458515]
      [New LWP 458516]
      [New LWP 458517]
      [New LWP 458518]
      [New LWP 458519]
      [New LWP 458520]
      [New LWP 458521]
      [New LWP 458522]
      [New LWP 458523]
      [New LWP 458524]
      [New LWP 458525]
      [New LWP 458526]
      [New LWP 458527]
      [New LWP 458528]
      [New LWP 458529]
      [New LWP 458530]
      [New LWP 458531]
      [New LWP 458532]
      [New LWP 458533]
      [New LWP 458534]
      [New LWP 458535]
      [New LWP 458536]
      [New LWP 458537]
      [New LWP 458538]
      [New LWP 458539]
      [New LWP 458540]
      [New LWP 458541]
      [New LWP 458542]
      [New LWP 458543]
      [New LWP 458544]
      [New LWP 458545]
      [New LWP 458546]
      [New LWP 458547]
      [New LWP 458548]
      [New LWP 458549]
      [New LWP 458550]
      [New LWP 458551]
      [New LWP 458552]
      [New LWP 458553]
      [New LWP 458554]
      [New LWP 458555]
      [New LWP 458556]
      [New LWP 458557]
      [New LWP 458558]
      [New LWP 458559]
      [New LWP 458560]
      [New LWP 458561]
      [New LWP 458562]
      [New LWP 458563]
      [New LWP 458564]
      [New LWP 458565]
      [New LWP 458566]
      [New LWP 458567]
      [New LWP 458568]
      [New LWP 458569]
      [New LWP 458570]
      [New LWP 458571]
      [New LWP 458572]
      [New LWP 458573]
      [New LWP 458574]
      [New LWP 458575]
      [New LWP 458576]
      [New LWP 458577]
      [New LWP 458578]
      [New LWP 458579]
      [New LWP 458580]
      [New LWP 458581]
      [New LWP 458582]
      [New LWP 458583]
      [New LWP 458584]
      [New LWP 458585]
      [New LWP 458586]
      [New LWP 458587]
      [New LWP 458588]
      [New LWP 458589]
      [New LWP 458619]
      [New LWP 458620]
      [New LWP 458621]
      [New LWP 458622]
      [New LWP 458623]
      [New LWP 458624]
      [New LWP 458625]
      [New LWP 458626]
      [New LWP 458627]
      [New LWP 458628]
      [New LWP 458629]
      [New LWP 458630]
      [New LWP 458631]
      [New LWP 458632]
      [New LWP 458633]
      [New LWP 458634]
      [New LWP 458635]
      [New LWP 458636]
      [New LWP 458637]
      [New LWP 458638]
      [New LWP 458639]
      [New LWP 458640]
      [New LWP 458641]
      [New LWP 458642]
      [New LWP 458646]
      [New LWP 458647]
      [New LWP 458648]
      [New LWP 458649]
      [New LWP 458654]
      [New LWP 458669]
      [New LWP 458676]
      [New LWP 458685]
      [New LWP 458704]
      [New LWP 458705]
      [New LWP 458706]
      [New LWP 458717]
      [New LWP 458729]
      [New LWP 458765]
      [New LWP 458766]
      [New LWP 458767]
      [New LWP 458768]
      [New LWP 458769]
      [New LWP 458770]
      [New LWP 458771]
      [New LWP 458772]
      [New LWP 458773]
      [New LWP 458774]
      [New LWP 458775]
      [New LWP 458776]
      [New LWP 458777]
      [New LWP 458778]
      [New LWP 458779]
      [New LWP 458780]
      [New LWP 458783]
      [New LWP 458784]
      [New LWP 458785]
      [New LWP 458786]
      [New LWP 458787]
      [New LWP 458788]
      [New LWP 458798]
      [New LWP 458799]
      [New LWP 458800]
      [New LWP 458801]
      [New LWP 458802]
      [New LWP 458803]
      [New LWP 458804]
      [New LWP 458805]
      [New LWP 458806]
      [New LWP 458807]
      [New LWP 458816]
      [New LWP 460155]
      [New LWP 460156]
      [New LWP 460157]
      [New LWP 460158]
      [New LWP 460159]
      [New LWP 460160]
      [New LWP 460161]
      [New LWP 460162]
      [New LWP 460179]
      [New LWP 460902]
      [Thread debugging using libthread_db enabled]
      Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
      __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55a549d8c668 <mongo::(anonymous namespace)::shutdownTasksComplete+40>) at ./nptl/futex-internal.c:57
      57    ./nptl/futex-internal.c: No such file or directory.
      (gdb) bt
      #0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0,
          futex_word=0x55a549d8c668 <mongo::(anonymous namespace)::shutdownTasksComplete+40>) at ./nptl/futex-internal.c:57
      #1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0,
          futex_word=0x55a549d8c668 <mongo::(anonymous namespace)::shutdownTasksComplete+40>) at ./nptl/futex-internal.c:87
      #2  _{_}GI{_}__futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55a549d8c668 <mongo::(anonymous namespace)::shutdownTasksComplete+40>,
          expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
      #3  0x00007fd183046ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55a54bb04d10,
          cond=0x55a549d8c640 <mongo::(anonymous namespace)::shutdownTasksComplete>) at ./nptl/pthread_cond_wait.c:503
      #4  ___pthread_cond_wait (cond=0x55a549d8c640 <mongo::(anonymous namespace)::shutdownTasksComplete>, mutex=0x55a54bb04d10) at ./nptl/pthread_cond_wait.c:627
      #5  0x000055a548c6ac4c in std::condition_variable::wait(std::unique_lock<std::mutex>&) ()
      #6  0x000055a548abc487 in mongo::waitForShutdown() ()
      #7  0x000055a546cd1939 in mongo::mongos_main(int, char**) ()
      #8  0x000055a546b80bfc in main ()
      

       

            Assignee:
            yuan.fang@mongodb.com Yuan Fang
            Reporter:
            amit.gupta@opensense.com Amit Gupta
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: