Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-35133

non-"raw" stack traces full of cannot access memory

    • Sharding 2018-12-31
    • 17

      link to task: https://evergreen.mongodb.com/task/mongodb_mongo_master_enterprise_rhel_72_s390x_inmem_concurrency_sharded_replication_with_balancer_cf339b8a8d8708e8b28747fe0cafee7cc79fe9a6_18_05_16_00_03_26/0

      Example output:

      Writing raw stacks to debugger_mongod_7400_raw_stacks.log.
      Redirecting output to debugger_mongod_7400_raw_stacks.log.
      Done logging to debugger_mongod_7400_raw_stacks.log.
      MongoDB GDB commands loaded, run 'mongodb-help' for list of commands
      MongoDB GDB pretty-printers loaded
      MongoDB Lock analysis commands loaded
      Thread 70: "signalP.gThread" (Thread 0x3ff8eeff910 (LWP 7410)) Error occurred in Python command: Cannot access memory at address 0x6a9aa67f880
      
      ...
      
      Thread 1: "mongod" (Thread 0x3ff92a62aa0 (LWP 7400)) Error occurred in Python command: Cannot access memory at address 0x6a9ae1e2a10
      Thread 70: "signalP.gThread" (Thread 0x3ff8eeff910 (LWP 7410))
      Traceback (most recent call last):
        File "/data/mci/1fcc8e1e1b801a663ff93ffc102ac6ea/src/buildscripts/gdb/mongo.py", line 174, in invoke
          self._dump_unique_stacks(stacks)
        File "/data/mci/1fcc8e1e1b801a663ff93ffc102ac6ea/src/buildscripts/gdb/mongo.py", line 233, in _dump_unique_stacks
          print(stack['output'])
      KeyError: 'output'
      Error occurred in Python command: ('output',)
      warning: target file /proc/7400/cmdline contained unexpected null characters
      Saved corefile dump_mongod.7400.core
      Running Hang Analyzer Supplement - MongoDBDumpLocks
      Traceback (most recent call last):
        File "/data/mci/1fcc8e1e1b801a663ff93ffc102ac6ea/src/buildscripts/gdb/mongo_lock.py", line 387, in invoke
          self.mongodb_show_locks()
        File "/data/mci/1fcc8e1e1b801a663ff93ffc102ac6ea/src/buildscripts/gdb/mongo_lock.py", line 394, in mongodb_show_locks
          get_locks(graph=None, thread_dict=thread_dict, show=True)
        File "/data/mci/1fcc8e1e1b801a663ff93ffc102ac6ea/src/buildscripts/gdb/mongo_lock.py", line 348, in get_locks
          find_mutex_holder(graph, thread_dict, show)
        File "/data/mci/1fcc8e1e1b801a663ff93ffc102ac6ea/src/buildscripts/gdb/mongo_lock.py", line 272, in find_mutex_holder
          frame = find_frame(r'std::mutex::lock\(\)')
        File "/data/mci/1fcc8e1e1b801a663ff93ffc102ac6ea/src/buildscripts/gdb/mongo_lock.py", line 254, in find_frame
          block = frame.block()
      RuntimeError: Cannot locate object file for block.
      Error occurred in Python command: Cannot locate object file for block.
      Traceback (most recent call last):
        File "/data/mci/1fcc8e1e1b801a663ff93ffc102ac6ea/src/buildscripts/gdb/mongo_lock.py", line 412, in invoke
          self.mongodb_waitsfor_graph(arg)
        File "/data/mci/1fcc8e1e1b801a663ff93ffc102ac6ea/src/buildscripts/gdb/mongo_lock.py", line 421, in mongodb_waitsfor_graph
          get_locks(graph=graph, thread_dict=thread_dict, show=False)
        File "/data/mci/1fcc8e1e1b801a663ff93ffc102ac6ea/src/buildscripts/gdb/mongo_lock.py", line 348, in get_locks
          find_mutex_holder(graph, thread_dict, show)
        File "/data/mci/1fcc8e1e1b801a663ff93ffc102ac6ea/src/buildscripts/gdb/mongo_lock.py", line 272, in find_mutex_holder
          frame = find_frame(r'std::mutex::lock\(\)')
        File "/data/mci/1fcc8e1e1b801a663ff93ffc102ac6ea/src/buildscripts/gdb/mongo_lock.py", line 254, in find_frame
          block = frame.block()
      RuntimeError: Cannot locate object file for block.
      Error occurred in Python command: Cannot locate object file for block.
      Running Print JavaScript Stack Supplement
      

            Assignee:
            randolph@mongodb.com Randolph Tan
            Reporter:
            randolph@mongodb.com Randolph Tan
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: