Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-9059

Fatal Assertion on replication

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Replication
    • Environment:
      Ubuntu 64-bit 12.04, 32GB on Digital Ocean
    • Steps To Reproduce:
      Hide

      rs.add('db03:27017')

      ...

      <wait 6 hours for 100GB of data to replicate>

      Show
      rs.add('db03:27017') ... <wait 6 hours for 100GB of data to replicate>

      Description

      Hi, I'm having a hell of a time getting a new replica set member added.

      Brand new machine with mongodb 2.4 and tried to add a new member to my replica set. Got a SEGFAULT.

      I have no idea what's causing this issue, but I got the exact same segfault a few months ago when 2.2 came out. I repaired the db and STILL got this problem. And it only happens after the entire replication is finished (6 hours later).

      I can't make any new copies of the data! And I need to start sharding because my service is exploding in growth. What am I supposed to do if I can't replicate?

      Sam

      Thu Mar 21 02:01:21.926 [repl writer worker 1] build index done. scanned 9 total records. 0.001 secs
      Thu Mar 21 02:01:22.040 [rsSync] warning replset primary is an older version than we are; upgrade recommended
      Thu Mar 21 02:01:25.229 [repl writer worker 1] ERROR: exception: E11000 duplicate key error index: newsblur.stories.$story_hash_1 dup key: { : "980568:65f04f" } on: { ts: Timestamp 1363810091000|32, h: -7663483269809030053, op: "i", ns: "newsblur.stories", o: { _id: ObjectId('514a172a0f55a7137913adea'), share_user_ids: {}, story_author_name: "", story_date: new Date(1363809963000), story_tags: {}, story_hash: "980568:65f04f", story_feed_id: 980568, story_permalink: "http://feedproxy.google.com/~r/DrudgeReportFeed/~3/Kn0_WSFPhAI/us-syria-crisis-lebanon-idUSBRE92J0AY20130320", story_title: "More Syrian shells land in Lebanon...", comment_user_ids: {}, story_content_z: BinData, story_guid: "http://www.reuters.com/article/2013/03/20/us-syria-crisis-lebanon-idUSBRE92J0AY20130320" } }
      Thu Mar 21 02:01:25.230 [repl writer worker 1] Fatal Assertion 16361
      0xdcae01 0xd8ab83 0xc1ecb6 0xd986b1 0xe13709 0x7f1fac48fe9a 0x7f1fab7a2cbd
      /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdcae01]
      /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xa3) [0xd8ab83]
      /usr/bin/mongod(_ZN5mongo7replset21multiInitialSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x166) [0xc1ecb6]
      /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) [0xd986b1]
      /usr/bin/mongod() [0xe13709]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7f1fac48fe9a]
      /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f1fab7a2cbd]
      Thu Mar 21 02:01:25.238 [repl writer worker 1]

      ***aborting after fassert() failure

      Thu Mar 21 02:01:25.238 Got signal: 6 (Aborted).

      Thu Mar 21 02:01:25.242 Backtrace:
      0xdcae01 0x6ce879 0x7f1fab6e54a0 0x7f1fab6e5425 0x7f1fab6e8b8b 0xd8abbe 0xc1ecb6 0xd986b1 0xe13709 0x7f1fac48fe9a 0x7f1fab7a2cbd
      /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdcae01]
      /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x6ce879]
      /lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x7f1fab6e54a0]
      /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7f1fab6e5425]
      /lib/x86_64-linux-gnu/libc.so.6(abort+0x17b) [0x7f1fab6e8b8b]
      /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xde) [0xd8abbe]
      /usr/bin/mongod(_ZN5mongo7replset21multiInitialSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x166) [0xc1ecb6]
      /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) [0xd986b1]
      /usr/bin/mongod() [0xe13709]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7f1fac48fe9a]
      /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f1fab7a2cbd]

        Attachments

        1. mongodb_2nd_try.txt
          91 kB
        2. mongodb_3rd_try.txt
          300 kB
        3. mongodb.log
          160 kB

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                13 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: