Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-9922

Mongod instance (2.4.4) crashed with the following errors with segmentation fault

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker - P1
    • Resolution: Duplicate
    • Affects Version/s: 2.4.4
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:
      Mongo instances are running in replication mode.
      O/S: RHEL 6.3 64 bit
    • Operating System:
      Linux
    • Steps To Reproduce:
      Hide

      Crashed at midnight. Log attached for analysis

      Show
      Crashed at midnight. Log attached for analysis

      Description

      Server log
      ==========

      Thu Jun 13 00:05:38.083 [initandlisten] connection accepted from 172.16.223.24:59235 #127832 (5 connections now open)
      Thu Jun 13 00:05:38.083 [conn127832] authenticate db: wse { authenticate: 1, user: "uuapp", nonce: "xxxxxxxxx", key: "yyyyyyyyyyyyyy" }
      Thu Jun 13 00:05:38.086 [conn127832] build index wse.msd20130613 { _id: 1 }
      Thu Jun 13 00:05:38.086 [conn3566] getMore: cursorid not found local.oplog.rs 1958166279006819049
      Thu Jun 13 00:05:38.098 [conn127832] build index done. scanned 0 total records. 0.011 secs
      Thu Jun 13 00:05:38.098 [conn127832] local.oplog.rs Assertion failure firstEmpty >= 0 src/mongo/util/unordered_fast_key_table_internal.h 94
      0xdd2331 0xd9350d 0xc40318 0x8b68f8 0x822216 0xa69cb2 0xa66849 0x9f0923 0x9f147c 0x9f7cae 0x6e7978 0xdbea9e 0x381ea07851 0x381e6e767d 
      /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdd2331]
      /usr/bin/mongod(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) [0xd9350d]
      /usr/bin/mongod(_ZN5mongo3Top6recordERKNS_10StringDataEiixb+0x9e8) [0xc40318]
      /usr/bin/mongod(_ZNK5mongo5CurOp16recordGlobalTimeEx+0x68) [0x8b68f8]
      /usr/bin/mongod(_ZN5mongo6Client7ContextD1Ev+0x46) [0x822216]
      /usr/bin/mongod() [0xa69cb2]
      /usr/bin/mongod(_ZN5mongo5logOpEPKcS1_RKNS_7BSONObjEPS2_Pbb+0x49) [0xa66849]
      /usr/bin/mongod(_ZN5mongo14checkAndInsertEPKcRNS_7BSONObjE+0x133) [0x9f0923]
      /usr/bin/mongod(_ZN5mongo14receivedInsertERNS_7MessageERNS_5CurOpE+0xa2c) [0x9f147c]
      /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0xabe) [0x9f7cae]
      /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x98) [0x6e7978]
      /usr/bin/mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x42e) [0xdbea9e]
      /lib64/libpthread.so.0() [0x381ea07851]
      /lib64/libc.so.6(clone+0x6d) [0x381e6e767d]
      Thu Jun 13 00:05:38.105 [conn4] Assertion: 10334:BSONObj size: -286331154 (0xEEEEEEEE) is invalid. Size must be between 0 and 16793600(16MB) First element: EOO
      Thu Jun 13 00:05:38.105 Invalid access at address: 0x6900 from thread: conn127832
      0xdd2331 0xd93c6b 0xd941ac 0x6ebf2f 0xa51ebd 0xa5289b 0x810411 0xa7c9a6 0x9f47f3 0x9f79f7 0x6e7978 0xdbea9e 0x381ea07851 0x381e6e767d 
      Thu Jun 13 00:05:38.105 Got signal: 11 (Segmentation fault).
      /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdd2331]
      /usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x9b) [0xd93c6b]
      /usr/bin/mongod() [0xd941ac]
      /usr/bin/mongod(_ZNK5mongo7BSONObj14_assertInvalidEv+0x5bf) [0x6ebf2f]
      /usr/bin/mongod(_ZNK5mongo19CoveredIndexMatcher7matchesERKNS_7BSONObjERKNS_7DiskLocEPNS_12MatchDetailsEb+0x20d) [0xa51ebd]
      /usr/bin/mongod(_ZNK5mongo19CoveredIndexMatcher14matchesCurrentEPNS_6CursorEPNS_12MatchDetailsE+0xeb) [0xa5289b]
      /usr/bin/mongod(_ZN5mongo6Cursor14currentMatchesEPNS_12MatchDetailsE+0x41) [0x810411]
      /usr/bin/mongod(_ZN5mongo14processGetMoreEPKcixRNS_5CurOpEiRbPb+0x406) [0xa7c9a6]
      /usr/bin/mongod(_ZN5mongo15receivedGetMoreERNS_10DbResponseERNS_7MessageERNS_5CurOpE+0xec3) [0x9f47f3]
      /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x807) [0x9f79f7]
      /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x98) [0x6e7978]
      /usr/bin/mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x42e) [0xdbea9e]
      /lib64/libpthread.so.0() [0x381ea07851]
      /lib64/libc.so.6(clone+0x6d) [0x381e6e767d]
      Thu Jun 13 00:05:38.143 Backtrace:
      0xdd2331 0x6cfb19 0x6d00a2 0x381ea0f500 0x381e731aff 0x382a292b81 0xdacead 0xd93482 0xc40318 0x8b68f8 0x822216 0x9f14e8 0x9f7cae 0x6e7978 0xdbea9e 0x381ea07851 0x381e6e767d 
      /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdd2331]
      /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x6cfb19]
      /usr/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x262) [0x6d00a2]
      /lib64/libpthread.so.0() [0x381ea0f500]
      /lib64/libc.so.6() [0x381e731aff]
      /usr/lib64/libstdc++.so.6(_ZStlsISt11char_traitsIcEERSt13basic_ostreamIcT_ES5_PKc+0x21) [0x382a292b81]
      /usr/bin/mongod(_ZN5mongo9LogstreamlsEPKc+0xd) [0xdacead]
      /usr/bin/mongod(_ZN5mongo12verifyFailedEPKcS1_j+0x72) [0xd93482]
      /usr/bin/mongod(_ZN5mongo3Top6recordERKNS_10StringDataEiixb+0x9e8) [0xc40318]
      /usr/bin/mongod(_ZNK5mongo5CurOp16recordGlobalTimeEx+0x68) [0x8b68f8]
      /usr/bin/mongod(_ZN5mongo6Client7ContextD1Ev+0x46) [0x822216]
      /usr/bin/mongod(_ZN5mongo14receivedInsertERNS_7MessageERNS_5CurOpE+0xa98) [0x9f14e8]
      /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0xabe) [0x9f7cae]
      /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x98) [0x6e7978]
      /usr/bin/mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x42e) [0xdbea9e]
      /lib64/libpthread.so.0() [0x381ea07851]
      /lib64/libc.so.6(clone+0x6d) [0x381e6e767d]

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Unassigned
              Reporter:
              npunmiya Neeraj Punmiya
              Participants:
              Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: