Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-11119

mongod connections hang because of assertion failure

XMLWordPrintableJSON

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 2.4.5
    • Component/s: Replication
    • Labels:
    • Environment:
      Ubuntu 12.04
    • ALL
    • Hide

      Happens randomly but seems to be more likely when you do repairDatabase or fsync lock in one of the OTHER nodes of a replica set.

      Show
      Happens randomly but seems to be more likely when you do repairDatabase or fsync lock in one of the OTHER nodes of a replica set.

      I got into a situation where all new connections with the mongo shell hang and never reach the prompt. The log says that an assertion failed but the mongod process does not terminate so that this problem prevails until I manually restart mongod.

      Log messages:

      Thu Oct 10 08:34:10.016 [conn44524] end connection 10.240.29.54:56207 (100 connections now open)
Thu Oct 10 08:34:10.017 [initandlisten] connection accepted from 10.240.29.54:56211 #44528 (101 connections now open)
Thu Oct 10 08:34:10.065 [conn44528]  authenticate db: local { authenticate: 1, nonce: "c055bc811fe10892", user: "__system", key: "deefa05ea4f51cd6cfdf43219f339c3b" }
Thu Oct 10 08:34:14.059 [rsHealthPoll]   warning assertion failure d.size() < 1024 src/mongo/util/concurrency/task.cpp 121
0xb942f3 0xb57191 0xb5f652 0xa355ce 0xa3676c 0xb5f29c 0xb5af25 0xb5b89b 0xbddc5b 0x7fabf8fa5e9a 0x7fabf7e87ccd
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x23) [0xb942f3]
 /usr/bin/mongod(_ZN5mongo9wassertedEPKcS1_j+0x121) [0xb57191]
 /usr/bin/mongod(_ZN5mongo4task6Server4sendEN5boost8functionIFvvEEE+0x182) [0xb5f652]
 /usr/bin/mongod(_ZN5mongo21ReplSetHealthPollTask2upERKNS_7BSONObjERNS_13HeartbeatInfoE+0x88e) [0xa355ce]
 /usr/bin/mongod(_ZN5mongo21ReplSetHealthPollTask6doWorkEv+0xec) [0xa3676c]
 /usr/bin/mongod(_ZN5mongo4task4Task3runEv+0x2c) [0xb5f29c]
 /usr/bin/mongod(_ZN5mongo13BackgroundJob7jobBodyEN5boost10shared_ptrINS0_9JobStatusEEE+0xb5) [0xb5af25]
 /usr/bin/mongod(_ZN5boost6detail11thread_dataINS_3_bi6bind_tIvNS_4_mfi3mf1IvN5mongo13BackgroundJobENS_10shared_ptrINS7_9JobStatusEEEEENS2_5list2INS2_5valueIPS7_EENSD_ISA_EEEEEEE3runEv+0x7b) [0xb5b89b]
 /usr/bin/mongod() [0xbddc5b]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fabf8fa5e9a]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fabf7e87ccd]
Thu Oct 10 08:34:15.592 [initandlisten] connection accepted from 127.0.0.1:43664 #44529 (102 connections now open)

        1. mongo-logs.tar.gz
          7.22 MB

            Assignee:
            victor.hooi Victor Hooi
            Reporter:
            aheck Andreas Heck
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: