Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-16629

primary server fassert() with fatal assertion 16967

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Incomplete
    • Affects Version/s: 2.6.6
    • Fix Version/s: None
    • Component/s: Stability
    • Environment:
      Debian 7.7 / MongoDB 2.6.6 from 10gen repository
    • Operating System:
      Linux

      Description

      We have a replicat set of 3 servers MongoDB 2.6.6.

      Yesterday the primary server has crashed with this assertion :

      Dec 21 21:27:21 localhost mongod.27017[2702]: [TTLMonitor] prod_front.sessions Fatal Assertion 16967
      Dec 21 21:27:21 localhost mongod.27017[2702]: [TTLMonitor] prod_front.sessions 0x11fca91 0x119e889 0x11813bd 0xefd5ed 0xefd754 0xf3c972 0x8b8566 0xc1ccfa 0xc1bcce 0xf3f772 0xf41781 0x1184572 0x1241429 0x7f880b19db50 0x7f880a5407bd
      /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0x11fca91]
      /usr/bin/mongod(_ZN5mongo10logContextEPKc+0x159) [0x119e889]
      /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xcd) [0x11813bd]
      /usr/bin/mongod() [0xefd5ed]
      /usr/bin/mongod(_ZNK5mongo13ExtentManager13getNextRecordERKNS_7DiskLocE+0x24) [0xefd754]
      /usr/bin/mongod(_ZN5mongo17RecordStoreV1Base12deleteRecordERKNS_7DiskLocE+0xb2) [0xf3c972]
      /usr/bin/mongod(_ZN5mongo10Collection14deleteDocumentERKNS_7DiskLocEbbPNS_7BSONObjE+0x636) [0x8b8566]
      /usr/bin/mongod(_ZN5mongo14DeleteExecutor7executeEv+0x9da) [0xc1ccfa]
      /usr/bin/mongod(_ZN5mongo13deleteObjectsERKNS_10StringDataENS_7BSONObjEbbb+0x16e) [0xc1bcce]
      /usr/bin/mongod(_ZN5mongo10TTLMonitor10doTTLForDBERKSs+0xd72) [0xf3f772]
      /usr/bin/mongod(_ZN5mongo10TTLMonitor3runEv+0x4a1) [0xf41781]
      /usr/bin/mongod(_ZN5mongo13BackgroundJob7jobBodyEv+0xd2) [0x1184572]
      /usr/bin/mongod() [0x1241429]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50) [0x7f880b19db50]
      /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f880a5407bd]
      Dec 21 21:27:21 localhost mongod.27017[2702]: [TTLMonitor] 
       
      ***aborting after fassert() failure
       
       
      Dec 21 21:27:21 localhost mongod.27017[2702]: [TTLMonitor] Got signal: 6 (Aborted).
      Backtrace:0x11fca91 0x11fbe6e 0x7f880a4961e0 0x7f880a496165 0x7f880a4993e0 0x118142a 0xefd5ed 0xefd754 0xf3c972 0x8b8566 0xc1ccfa 0xc1bcce 0xf3f772 0xf41781 0x1184572 0x1241429 0x7f880b19db50 0x7f880a5407bd 
      /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0x11fca91]
      /usr/bin/mongod() [0x11fbe6e]
      /lib/x86_64-linux-gnu/libc.so.6(+0x321e0) [0x7f880a4961e0]
      /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7f880a496165]
      /lib/x86_64-linux-gnu/libc.so.6(abort+0x180) [0x7f880a4993e0]
      /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0x13a) [0x118142a]
      /usr/bin/mongod() [0xefd5ed]
      /usr/bin/mongod(_ZNK5mongo13ExtentManager13getNextRecordERKNS_7DiskLocE+0x24) [0xefd754]
      /usr/bin/mongod(_ZN5mongo17RecordStoreV1Base12deleteRecordERKNS_7DiskLocE+0xb2) [0xf3c972]
      /usr/bin/mongod(_ZN5mongo10Collection14deleteDocumentERKNS_7DiskLocEbbPNS_7BSONObjE+0x636) [0x8b8566]
      /usr/bin/mongod(_ZN5mongo14DeleteExecutor7executeEv+0x9da) [0xc1ccfa]
      /usr/bin/mongod(_ZN5mongo13deleteObjectsERKNS_10StringDataENS_7BSONObjEbbb+0x16e) [0xc1bcce]
      /usr/bin/mongod(_ZN5mongo10TTLMonitor10doTTLForDBERKSs+0xd72) [0xf3f772]
      /usr/bin/mongod(_ZN5mongo10TTLMonitor3runEv+0x4a1) [0xf41781]
      /usr/bin/mongod(_ZN5mongo13BackgroundJob7jobBodyEv+0xd2) [0x1184572]
      /usr/bin/mongod() [0x1241429]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50) [0x7f880b19db50]
      /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f880a5407bd]

      Server won't restart (same assertion after 10/15 seconds).

      Is it a data / indexes corruption ? I'm already on production, any help will be appreciate ...

        Attachments

          Activity

            People

            Assignee:
            ramon.fernandez Ramon Fernandez Marina
            Reporter:
            bhuisgen Boris HUISGEN
            Participants:
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: