Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-100054

Investigate filling up disk space with core dumps

    • Type: Icon: Bug Bug
    • Resolution: Unresolved
    • Priority: Icon: Minor - P4 Minor - P4
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Correctness
    • ALL
    • 200

      Original slack thread: https://mongodb.slack.com/archives/C07T35KELLC/p1738125749813369

      Max:

      color: Color value is invalid

      looks like the hang analyzer taking core dumps is filling up 100GB in

      https://jira.mongodb.org/browse/BF-36363

      color: Color value is invalid

      . From the system logs in 

      https://parsley.mongodb.com/evergreen/mongodb_mongo_master_enterprise_amazon_linux2_ar[…]_16_41_34/0/system?bookmarks=0,15107&filters=100mongod%2520

      color: Color value is invalid

      , I see the total virtual memory used by all 24 mongod processes is 99GB. The resident memory is less than 5GB. I feel like one avenue would be to investigate what allocated memory within the mongod process because the WT cache is still restricted to 1GB. This is maybe better performed by the Server team so I'll tag 

      color: Color value is invalid

      @Abdul Qadeer

      color: Color value is invalid

       as well

      color: Color value is invalid

      For the Correctness team, I think 

      https://github.com/10gen/mongo/blob/6614f6533ef5932a95dd1b9b2eda8ff280a7a548/buildscripts/resmokelib/hang_analyzer/hang_analyzer.py#L174

      color: Color value is invalid

       needs to be looked into for why it is ineffective at preventing the disk from filling up with core dumpsFor the future, I think the Correctness team can also consider using a named pipe so the contents of the 

      gcore

      color: Color value is invalid

       command can be piped into gzip/pgzip since the compressed size of the core dumps will be much smaller

            Assignee:
            Unassigned Unassigned
            Reporter:
            trevor.guidry@mongodb.com Trevor Guidry
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: