Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-3869

mongod 2.0.0 crash

    • Linux

      The mongodb primary server in a replicaset crashed after running for hours,

      Here is the log:

      Thu Sep 15 19:33:57 Invalid access at address: 0x7fd8469db000

      Thu Sep 15 19:33:57 Got signal: 11 (Segmentation fault).

      Thu Sep 15 19:33:57 Backtrace:
      0xa83fc9 0xa845a0 0x7fe935a718f0 0x7fe934fb80ba 0x909ee2 0x95e73c 0x880494 0x88526a 0xa96a46 0x635dd7 0x7fe935a689ca 0x7fe93501770d
      /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0xa83fc9]
      /usr/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x220) [0xa845a0]
      /lib/libpthread.so.0(+0xf8f0) [0x7fe935a718f0]
      /lib/libc.so.6(memcpy+0x42a) [0x7fe934fb80ba]
      /usr/bin/mongod(_ZN5mongo22fillQueryResultFromObjERNS_11_BufBuilderINS_16TrivialAllocatorEEEPNS_10ProjectionERKNS_7BSONObjEPNS_7DiskLocE+0x8c2) [0x909ee2]
      /usr/bin/mongod(_ZN5mongo14processGetMoreEPKcixRNS_5CurOpEiRb+0x86c) [0x95e73c]
      /usr/bin/mongod(_ZN5mongo15receivedGetMoreERNS_10DbResponseERNS_7MessageERNS_5CurOpE+0x1b4) [0x880494]
      /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0xf6a) [0x88526a]
      /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x76) [0xa96a46]
      /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) [0x635dd7]
      /lib/libpthread.so.0(+0x69ca) [0x7fe935a689ca]
      /lib/libc.so.6(clone+0x6d) [0x7fe93501770d]

      Thu Sep 15 19:33:57 dbexit:
      Thu Sep 15 19:33:57 [conn36194] shutdown: going to close listening sockets...
      Thu Sep 15 19:33:57 [conn36194] closing listening socket: 5
      Thu Sep 15 19:33:57 [conn36194] closing listening socket: 6
      Thu Sep 15 19:33:57 [conn36194] closing listening socket: 7
      Thu Sep 15 19:33:57 [conn36194] removing socket file: /tmp/mongodb-27017.sock
      Thu Sep 15 19:33:57 [conn36194] shutdown: going to flush diaglog...
      Thu Sep 15 19:33:57 [conn36194] shutdown: going to close sockets...
      Thu Sep 15 19:33:57 [conn36194] shutdown: waiting for fs preallocator...
      Thu Sep 15 19:33:57 [conn36194] shutdown: closing all files...
      Thu Sep 15 19:33:57 [conn1] end connection 127.0.0.1:59305
      Thu Sep 15 19:33:57 [conn36224] got request after shutdown()
      Thu Sep 15 19:33:57 [conn38857] got request after shutdown()
      Thu Sep 15 19:33:57 [conn32693] got request after shutdown()
      Thu Sep 15 19:33:57 [conn36192] got request after shutdown()
      Thu Sep 15 19:33:57 [conn36202] got request after shutdown()
      Thu Sep 15 19:33:57 [conn36198] got request after shutdown()
      Thu Sep 15 19:33:57 Invalid access at address: 0

      Thu Sep 15 19:33:57 Got signal: 11 (Segmentation fault).

      Thu Sep 15 19:33:57 [conn32527] got request after shutdown()
      Thu Sep 15 19:33:58 [conn39067] got request after shutdown()
      Thu Sep 15 19:33:58 [conn36218] got request after shutdown()
      Thu Sep 15 19:33:58 [conn32643] got request after shutdown()
      Thu Sep 15 19:33:58 [conn32668] got request after shutdown()
      Thu Sep 15 19:33:58 [conn39312] got request after shutdown()
      Thu Sep 15 19:33:58 [conn38971] got request after shutdown()
      Thu Sep 15 19:33:58 [conn33274] got request after shutdown()
      Thu Sep 15 19:33:58 [conn36200] got request after shutdown()
      Thu Sep 15 19:33:58 [conn33230] got request after shutdown()
      Thu Sep 15 19:33:58 ERROR: Client::shutdown not called: slaveTracking
      Thu Sep 15 19:33:58 [conn36210] got request after shutdown()
      Thu Sep 15 19:33:58 Backtrace:
      0xa83fc9 0xa845a0 0x7fe935a718f0 0x7fe934fab392 0x7fe934faee83 0xa9d8e1 0x7fe935a68a61 0x7fe93501770d
      /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0xa83fc9]
      /usr/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x220) [0xa845a0]
      /lib/libpthread.so.0(+0xf8f0) [0x7fe935a718f0]
      /lib/libc.so.6(+0x7a392) [0x7fe934fab392]
      /lib/libc.so.6(cfree+0x73) [0x7fe934faee83]
      /usr/bin/mongod(tls_destructor+0xb1) [0xa9d8e1]
      /lib/libpthread.so.0(+0x6a61) [0x7fe935a68a61]
      /lib/libc.so.6(clone+0x6d) [0x7fe93501770d]

      Thu Sep 15 19:33:58 dbexit: ; exiting immediately

          • glibc detected *** /usr/bin/mongod: double free or corruption (fasttop): 0x0000000003688700 ***
            ======= Backtrace: =========
            /lib/libc.so.6(+0x775b6)[0x7fe934fa85b6]
            /lib/libc.so.6(cfree+0x73)[0x7fe934faee83]
            /usr/bin/mongod(ZNSt8_Rb_treeIPN5mongo9MongoFileES2_St9_IdentityIS2_ESt4lessIS2_ESaIS2_EE5eraseERKS2+0x144)[0x67b7b4]
            /usr/bin/mongod(_ZN5mongo9MongoFile9destroyedEv+0x2c)[0x67a59c]
            /usr/bin/mongod(_ZN5mongo8MongoMMF5closeEv+0x59)[0x759149]
            /usr/bin/mongod(_ZN5mongo9MongoFile13closeAllFilesERSt18basic_stringstreamIcSt11char_traitsIcESaIcEE+0x151)[0x67b1c1]
            /usr/bin/mongod(_ZN5mongo14shutdownServerEv+0x6d1)[0x881071]
            /usr/bin/mongod(_ZN5mongo6dbexitENS_8ExitCodeEPKcb+0x2a0)[0x881710]
            /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x4f6)[0xa84126]
            /usr/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x220)[0xa845a0]
            /lib/libpthread.so.0(+0xf8f0)[0x7fe935a718f0]
            /lib/libc.so.6(memcpy+0x42a)[0x7fe934fb80ba]
            /usr/bin/mongod(_ZN5mongo22fillQueryResultFromObjERNS_11_BufBuilderINS_16TrivialAllocatorEEEPNS_10ProjectionERKNS_7BSONObjEPNS_7DiskLocE+0x8c2)[0x909ee2]
            /usr/bin/mongod(_ZN5mongo14processGetMoreEPKcixRNS_5CurOpEiRb+0x86c)[0x95e73c]
            /usr/bin/mongod(_ZN5mongo15receivedGetMoreERNS_10DbResponseERNS_7MessageERNS_5CurOpE+0x1b4)[0x880494]
            /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0xf6a)[0x88526a]
            /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x76)[0xa96a46]
            /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287)[0x635dd7]
            /lib/libpthread.so.0(+0x69ca)[0x7fe935a689ca]
            /lib/libc.so.6(clone+0x6d)[0x7fe93501770d]

            Assignee:
            aaron Aaron Staple
            Reporter:
            lqs Liu Qishuai
            Votes:
            4 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: