[SERVER-44874] Windows Mongo shell not being included by hang analyzer Created: 27/Nov/19  Updated: 27/Oct/23  Resolved: 06/Feb/20

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Eric Milkie Assignee: Robert Guo (Inactive)
Resolution: Works as Designed Votes: 0
Labels: tig-hanganalyzer
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
Operating System: ALL
Sprint: STM 2020-02-10, STM 2020-02-24
Participants:
Linked BF Score: 15
Story Points: 0

 Description   

For a particular BF I'm investigating, which has failed on all branches 3.6, 4.0, 4.2, and master, the hang analyzer finds interesting processes for python.exe, mongobridge.exe, and mongod.exe, but not mongo.exe. For this particular BF, we believe that the hanging process is actually the shell, so for this case it would have been very helpful to have the shell process stack traces. See linked BF for examples. Note that the first BF does actually have a mongo.exe in the Hang Analysis output, but we believe that it's incorrectly linked as a dup; the other BFG's are the ones to look at here.



 Comments   
Comment by Robert Guo (Inactive) [ 06/Feb/20 ]

Mark and I went over the BF. There are two types of unrelated issues. One is where we deadlock, in which case mongo.exe does show up in the hang analyzer. The other, more common case, is where the shell consumes too much memory and gets killed silently, which causes it to not show up in the hang analyzer. We verified the second case by matching the time the logs disappeared from the spawned processes with the window mongo.exe stopped showing up in the system logs and they always matched up.

These two types of issues should have had separate BFs, but it was hard to distinguish them without knowing exactly what to look for.

Comment by Robert Guo (Inactive) [ 06/Feb/20 ]

The root cause is that tasklist.exe is not returning mongo.exe for some reason.

The integration code on our side seems to work fine and I was able to get a non-deadlocked mongo.exe to show up.

Administrator@EC2AMAZ-8RT7GAT /cygdrive/z/data/mci
$ get_pids $proc_list
 
Administrator@EC2AMAZ-8RT7GAT /cygdrive/z/data/mci
$ echo $proc_pids
5584
 
Administrator@EC2AMAZ-8RT7GAT /cygdrive/z/data/mci
$ tasklist /fo:csv | grep mongo.exe
"mongo.exe","5584","RDP-Tcp#0","2","101,504 K"

I'll check with mark.benvenuto@mongodb.com next and see if we can gain more insight into how tasklist works.

Generated at Thu Feb 08 05:07:14 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.