Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-10764

Invalid access at address: 0x7fa087553e30 from thread: repl writer worker x

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 2.4.6
    • Component/s: Replication
    • Labels:
    • Environment:
      CentOS 6 64bit
      Dual Quad-Core Xeon
      72GB Ram
    • Linux
    • Hide

      Unsure why it happens.

      Show
      Unsure why it happens.

      Running the following setup:

      • Primary server: 72GB Ram, dual quad-core xeon.
      • Replica 1: 72GB Ram, dual quad-core xeon.
      • Replica 2: 16GB Ram, quad-core xeon.
      • Replica 3: 16GB Ram, quad-core xeon. (Remote location)
      • Arbiter

      All servers are on CentOS 6 64bit. Primary, backup and one of the remote

      Primary server crashed a few weeks ago, backup became primary. I didn't really look into it, just removed the member, deleted the files, and reset it since it was down for a long time i figured it'd be faster.

      Today, it did the same thing again. When I restart it, it gives the same error output.

      Fri Sep 13 07:03:44.555 Invalid access at address: 0x7fad73cd0e30 from thread: repl writer worker 10
      Fri Sep 13 07:03:44.536 [conn74] end connection A.B.C.D:57698 (87 connections now open)
      Fri Sep 13 07:03:44.669 Got signal: 7 (Bus error).
      Fri Sep 13 07:03:45.000 Backtrace:
      0xdddd81 0x6d0d29 0x6d12b2 0x36c860f500 0xa61281 0xa61a32 0xac44f3 0xac58df 0xa903fa 0xa924c7 0xa72449 0xc273d3 0xc26b18 0xdab721 0xe26609 0x36c8607851 0x36c82e890d
      /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdddd81]
      /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x6d0d29]
      /usr/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x262) [0x6d12b2]
      /lib64/libpthread.so.0() [0x36c860f500]
      /usr/bin/mongod(ZN5mongo16NamespaceDetails10_stdAllocEib+0x641) [0xa61281]
      /usr/bin/mongod(_ZN5mongo16NamespaceDetails13allocWillBeAtEPKci+0x32) [0xa61a32]
      /usr/bin/mongod(_ZN5mongo11DataFileMgr6insertEPKcPKvibbbPb+0x1153) [0xac44f3]
      /usr/bin/mongod(_ZN5mongo11DataFileMgr16insertWithObjModEPKcRNS_7BSONObjEbb+0x4f) [0xac58df]
      /usr/bin/mongod(_ZN5mongo14_updateObjectsEbPKcRKNS_7BSONObjES4_bbbRNS_7OpDebugEPNS_11RemoveSaverEbRKNS_24QueryPlanSelectionPolicyEb+0x2eda) [0xa903fa]
      /usr/bin/mongod(_ZN5mongo27updateObjectsForReplicationEPKcRKNS_7BSONObjES4_bbbRNS_7OpDebugEbRKNS_24QueryPlanSelectionPolicyE+0xb7) [0xa924c7]
      /usr/bin/mongod(_ZN5mongo21applyOperation_inlockERKNS_7BSONObjEbb+0xb39) [0xa72449]
      /usr/bin/mongod(_ZN5mongo7replset8SyncTail9syncApplyERKNS_7BSONObjEb+0x713) [0xc273d3]
      /usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x48) [0xc26b18]
      /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) [0xdab721]
      /usr/bin/mongod() [0xe26609]
      /lib64/libpthread.so.0() [0x36c8607851]
      /lib64/libc.so.6(clone+0x6d) [0x36c82e890d]

      Fri Sep 13 07:33:20.945 Invalid access at address: 0x7fa087553e30 from thread: repl writer worker 2
      Fri Sep 13 07:33:20.972 Got signal: 7 (Bus error).
      Fri Sep 13 07:33:20.979 [initandlisten] connection accepted from 192.168.1.5:57728 #169 (109 connections now open)
      Fri Sep 13 07:33:21.023 [rsHealthPoll] replset info A.B.C.D:27017 thinks that we are down
      Fri Sep 13 07:33:20.980 Backtrace:
      0xdddd81 0x6d0d29 0x6d12b2 0x36c860f500 0xa61281 0xa61a32 0xac44f3 0xac58df 0xa903fa 0xa924c7 0xa72449 0xc273d3 0xc26b18 0xdab721 0xe26609 0x36c8607851 0x36c82e890d
      /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdddd81]
      /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x6d0d29]
      /usr/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x262) [0x6d12b2]
      /lib64/libpthread.so.0() [0x36c860f500]
      /usr/bin/mongod(ZN5mongo16NamespaceDetails10_stdAllocEib+0x641) [0xa61281]
      /usr/bin/mongod(_ZN5mongo16NamespaceDetails13allocWillBeAtEPKci+0x32) [0xa61a32]
      /usr/bin/mongod(_ZN5mongo11DataFileMgr6insertEPKcPKvibbbPb+0x1153) [0xac44f3]
      /usr/bin/mongod(_ZN5mongo11DataFileMgr16insertWithObjModEPKcRNS_7BSONObjEbb+0x4f) [0xac58df]
      /usr/bin/mongod(_ZN5mongo14_updateObjectsEbPKcRKNS_7BSONObjES4_bbbRNS_7OpDebugEPNS_11RemoveSaverEbRKNS_24QueryPlanSelectionPolicyEb+0x2eda) [0xa903fa]
      /usr/bin/mongod(_ZN5mongo27updateObjectsForReplicationEPKcRKNS_7BSONObjES4_bbbRNS_7OpDebugEbRKNS_24QueryPlanSelectionPolicyE+0xb7) [0xa924c7]
      /usr/bin/mongod(_ZN5mongo21applyOperation_inlockERKNS_7BSONObjEbb+0xb39) [0xa72449]
      /usr/bin/mongod(_ZN5mongo7replset8SyncTail9syncApplyERKNS_7BSONObjEb+0x713) [0xc273d3]
      /usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x48) [0xc26b18]
      /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) [0xdab721]
      /usr/bin/mongod() [0xe26609]
      /lib64/libpthread.so.0() [0x36c8607851]
      /lib64/libc.so.6(clone+0x6d) [0x36c82e890d]

      DB stats:
      {
      "collections" : 72,
      "objects" : 44619073,
      "avgObjSize" : 2480.755786297936,
      "dataSize" : 110689023524,
      "storageSize" : 124300971392,
      "numExtents" : 494,
      "indexes" : 106,
      "indexSize" : 5254527152,
      "fileSize" : 135159349248,
      "nsSizeMB" : 16,
      "dataFileVersion" :

      { "major" : 4, "minor" : 5 }

      ,
      "ok" : 1
      }

            Assignee:
            Unassigned Unassigned
            Reporter:
            srabouin@gmail.com Steve Rabouin
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: