Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-12080

Hard crash - Assertion failure a <= 512*1024*1024 (alignedbuilder.cpp)

    XMLWordPrintableJSON

Details

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major - P3 Major - P3
    • None
    • 2.4.8
    • Stability
    • None
    • Ubuntu 10.04
    • ALL

    Description

      Running a mongo replicaset with two members + one arbiter. Both members will crash hard at the same time with the following stack trace in the log. I have not identified the action that caused it.

      Thu Dec 12 11:56:36.602 [conn56967] end connection 10.16.8.150:45967 (1 connection now open)
      Thu Dec 12 11:56:36.602 [initandlisten] connection accepted from 10.16.8.150:45969 #56969 (2 connections now open)
      Thu Dec 12 11:56:36.604 [conn56969]  authenticate db: local { authenticate: 1, nonce: "817d28460fefee45", user: "__system", key: "7bd865435a880088115abd43c93c4d87" }
      Thu Dec 12 11:56:37.785 [conn56968] end connection 10.16.8.152:43140 (1 connection now open)
      Thu Dec 12 11:56:37.788 [initandlisten] connection accepted from 10.16.8.152:43142 #56970 (2 connections now open)
      Thu Dec 12 11:56:37.790 [conn56970]  authenticate db: local { authenticate: 1, nonce: "11b7719a8c9c8ba2", user: "__system", key: "4cc99b17ea292d63a56f3169f1c24368" }
      Thu Dec 12 11:57:00.872 [initandlisten] connection accepted from 10.128.13.11:53665 #56971 (3 connections now open)
      Thu Dec 12 11:57:00.874 [conn56971] end connection 10.128.13.11:53665 (2 connections now open)
      Thu Dec 12 11:57:06.625 [conn56969] end connection 10.16.8.150:45969 (1 connection now open)
      Thu Dec 12 11:57:06.626 [initandlisten] connection accepted from 10.16.8.150:45971 #56972 (2 connections now open)
      Thu Dec 12 11:57:06.626 [conn56972]  authenticate db: local { authenticate: 1, nonce: "7549836124c801bc", user: "__system", key: "85964ea4bd4ec8f7939da6ce53eef16e" }
      Thu Dec 12 11:57:07.803 [conn56970] end connection 10.16.8.152:43142 (1 connection now open)
      Thu Dec 12 11:57:07.804 [initandlisten] connection accepted from 10.16.8.152:43144 #56973 (2 connections now open)
      Thu Dec 12 11:57:07.805 [conn56973]  authenticate db: local { authenticate: 1, nonce: "53b274b71898941c", user: "__system", key: "6fdd3676517ff5cdf4cab3b425775557" }
      Thu Dec 12 11:57:30.687 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.577MB
      0xde05e1 0x9243b1 0x92448b 0x924502 0x924633 0x9246a0 0x91e0da 0x7eec70 0x7fcc9e 0x7fdefd 0x7fc8c3 0x7fd00f 0x7fdefd 0x7fc8c3 0x7fc9ab 
       /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
       /usr/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x9243b1]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4b) [0x92448b]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x924502]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x83) [0x924633]
       /usr/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x60) [0x9246a0]
       /usr/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0x91e0da]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13fixParentPtrsENS_7DiskLocEii+0xc0) [0x7eec70]
       /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x1ee) [0x7fcc9e]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
       /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x55f) [0x7fd00f]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x44b) [0x7fc9ab]
      Thu Dec 12 11:57:30.697 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.581MB
      0xde05e1 0x9243b1 0x92448b 0x924502 0x924633 0x9246a0 0x91e0da 0x7eec70 0x7fcc9e 0x7fdefd 0x7fc8c3 0x7fd00f 0x7fdefd 0x7fc8c3 0x7fc9ab 
       /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
       /usr/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x9243b1]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4b) [0x92448b]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x924502]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x83) [0x924633]
       /usr/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x60) [0x9246a0]
       /usr/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0x91e0da]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13fixParentPtrsENS_7DiskLocEii+0xc0) [0x7eec70]
       /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x1ee) [0x7fcc9e]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
       /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x55f) [0x7fd00f]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x44b) [0x7fc9ab]
      Thu Dec 12 11:57:30.702 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.585MB
      0xde05e1 0x9243b1 0x92448b 0x924502 0x924633 0x9246a0 0x91e0da 0x7eec70 0x7fcc9e 0x7fdefd 0x7fc8c3 0x7fd00f 0x7fdefd 0x7fc8c3 0x7fc9ab 
       /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
       /usr/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x9243b1]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4b) [0x92448b]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x924502]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x83) [0x924633]
       /usr/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x60) [0x9246a0]
       /usr/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0x91e0da]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13fixParentPtrsENS_7DiskLocEii+0xc0) [0x7eec70]
       /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x1ee) [0x7fcc9e]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
       /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x55f) [0x7fd00f]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x44b) [0x7fc9ab]
      Thu Dec 12 11:57:30.709 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.589MB
      0xde05e1 0x9243b1 0x92448b 0x924502 0x924633 0x9246a0 0x91e0da 0x7eec70 0x7fcc9e 0x7fdefd 0x7fc8c3 0x7fd00f 0x7fdefd 0x7fc8c3 0x7fc9ab 
       /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
       /usr/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x9243b1]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4b) [0x92448b]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x924502]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x83) [0x924633]
       /usr/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x60) [0x9246a0]
       /usr/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0x91e0da]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13fixParentPtrsENS_7DiskLocEii+0xc0) [0x7eec70]
       /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x1ee) [0x7fcc9e]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
       /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x55f) [0x7fd00f]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x44b) [0x7fc9ab]
      Thu Dec 12 11:57:30.715 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.593MB
      0xde05e1 0x9243b1 0x92448b 0x924502 0x924633 0x9246a0 0x91e0da 0x7eec70 0x7fcc9e 0x7fdefd 0x7fc8c3 0x7fd00f 0x7fdefd 0x7fc8c3 0x7fc9ab 
       /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
       /usr/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x9243b1]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4b) [0x92448b]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x924502]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x83) [0x924633]
       /usr/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x60) [0x9246a0]
       /usr/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0x91e0da]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13fixParentPtrsENS_7DiskLocEii+0xc0) [0x7eec70]
       /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x1ee) [0x7fcc9e]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
       /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x55f) [0x7fd00f]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x44b) [0x7fc9ab]
      Thu Dec 12 11:57:30.720 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.597MB
      0xde05e1 0x9243b1 0x92448b 0x924502 0x924633 0x9246a0 0x91e0da 0x7eec70 0x7fcc9e 0x7fdefd 0x7fc8c3 0x7fd00f 0x7fdefd 0x7fc8c3 0x7fc9ab 
       /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
       /usr/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x9243b1]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4b) [0x92448b]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x924502]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x83) [0x924633]
       /usr/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x60) [0x9246a0]
       /usr/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0x91e0da]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13fixParentPtrsENS_7DiskLocEii+0xc0) [0x7eec70]
       /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x1ee) [0x7fcc9e]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
       /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x55f) [0x7fd00f]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x44b) [0x7fc9ab]
      Thu Dec 12 11:57:30.726 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.601MB
      0xde05e1 0x9243b1 0x92448b 0x924502 0x924633 0x9246a0 0x91e0da 0x7eec70 0x7fcc9e 0x7fdefd 0x7fc8c3 0x7fd00f 0x7fdefd 0x7fc8c3 0x7fc9ab 
       /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
       /usr/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x9243b1]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4b) [0x92448b]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x924502]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x83) [0x924633]
       /usr/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x60) [0x9246a0]
       /usr/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0x91e0da]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13fixParentPtrsENS_7DiskLocEii+0xc0) [0x7eec70]
       /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x1ee) [0x7fcc9e]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
       /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x55f) [0x7fd00f]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x44b) [0x7fc9ab]
      Thu Dec 12 11:57:30.732 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.606MB
      0xde05e1 0x9243b1 0x92448b 0x924502 0x924633 0x9246a0 0x91e0da 0x7eec70 0x7fcc9e 0x7fdefd 0x7fc8c3 0x7fd00f 0x7fdefd 0x7fc8c3 0x7fc9ab 
       /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
       /usr/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x9243b1]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4b) [0x92448b]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x924502]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x83) [0x924633]
       /usr/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x60) [0x9246a0]
       /usr/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0x91e0da]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13fixParentPtrsENS_7DiskLocEii+0xc0) [0x7eec70]
       /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x1ee) [0x7fcc9e]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
       /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x55f) [0x7fd00f]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x44b) [0x7fc9ab]
      Thu Dec 12 11:57:30.737 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.61MB
      0xde05e1 0x9243b1 0x92448b 0x924502 0x924633 0x9246a0 0x91e0da 0x7eec70 0x7fcc9e 0x7fdefd 0x7fc8c3 0x7fd00f 0x7fdefd 0x7fc8c3 0x7fc9ab 
       /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
       /usr/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x9243b1]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4b) [0x92448b]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x924502]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x83) [0x924633]
       /usr/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x60) [0x9246a0]
       /usr/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0x91e0da]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13fixParentPtrsENS_7DiskLocEii+0xc0) [0x7eec70]
       /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x1ee) [0x7fcc9e]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
       /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x55f) [0x7fd00f]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x44b) [0x7fc9ab]
      Thu Dec 12 11:57:30.743 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.614MB
      0xde05e1 0x9243b1 0x92448b 0x924502 0x924633 0x9246a0 0x91e0da 0x7eec70 0x7fcc9e 0x7fdefd 0x7fc8c3 0x7fd00f 0x7fdefd 0x7fc8c3 0x7fc9ab 
       /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
       /usr/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x9243b1]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4b) [0x92448b]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x924502]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x83) [0x924633]
       /usr/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x60) [0x9246a0]
       /usr/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0x91e0da]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13fixParentPtrsENS_7DiskLocEii+0xc0) [0x7eec70]
       /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x1ee) [0x7fcc9e]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
       /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x55f) [0x7fd00f]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x44b) [0x7fc9ab]
      Thu Dec 12 11:57:30.748 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.618MB
      Thu Dec 12 11:57:30.748 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.622MB
      Thu Dec 12 11:57:30.749 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.626MB
      Thu Dec 12 11:57:30.749 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.63MB
      Thu Dec 12 11:57:30.750 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.634MB
      Thu Dec 12 11:57:30.750 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.638MB
      Thu Dec 12 11:57:30.750 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.642MB
      Thu Dec 12 11:57:30.750 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.647MB
      Thu Dec 12 11:57:30.750 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.651MB
      Thu Dec 12 11:57:30.750 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.655MB
      0xde05e1 0x9243b1 0x92448b 0x924502 0x924633 0x9246a0 0x91e0da 0x7eec70 0x7fcc9e 0x7fdefd 0x7fc8c3 0x7fd00f 0x7fdefd 0x7fc8c3 0x7fc9ab 
       /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
       /usr/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x9243b1]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4b) [0x92448b]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x924502]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x83) [0x924633]
       /usr/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x60) [0x9246a0]
       /usr/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0x91e0da]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13fixParentPtrsENS_7DiskLocEii+0xc0) [0x7eec70]
       /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x1ee) [0x7fcc9e]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
       /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x55f) [0x7fd00f]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x44b) [0x7fc9ab]
      Thu Dec 12 11:57:30.755 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.659MB
      Thu Dec 12 11:57:30.755 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.663MB
      Thu Dec 12 11:57:30.755 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.667MB
      Thu Dec 12 11:57:30.755 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.671MB
      Thu Dec 12 11:57:30.755 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.675MB
      Thu Dec 12 11:57:30.755 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.679MB
      Thu Dec 12 11:57:30.755 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.683MB
      Thu Dec 12 11:57:30.755 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.687MB
      Thu Dec 12 11:57:30.755 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.692MB
      Thu Dec 12 11:57:30.755 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.696MB
      0xde05e1 0x9243b1 0x92448b 0x924502 0x924633 0x9246a0 0x91e0da 0x7eec70 0x7fcc9e 0x7fdefd 0x7fc8c3 0x7fd00f 0x7fdefd 0x7fc8c3 0x7fc9ab 
       /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
       /usr/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x9243b1]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4b) [0x92448b]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x924502]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x83) [0x924633]
       /usr/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x60) [0x9246a0]
       /usr/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0x91e0da]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13fixParentPtrsENS_7DiskLocEii+0xc0) [0x7eec70]
       /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x1ee) [0x7fcc9e]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
       /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x55f) [0x7fd00f]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x44b) [0x7fc9ab]
      Thu Dec 12 11:57:30.761 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.7MB
      Thu Dec 12 11:57:30.761 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.704MB
      Thu Dec 12 11:57:30.761 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.708MB
      Thu Dec 12 11:57:30.762 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.712MB
      Thu Dec 12 11:57:30.762 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.716MB
      Thu Dec 12 11:57:30.762 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.72MB
      Thu Dec 12 11:57:30.762 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.724MB
      Thu Dec 12 11:57:30.762 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.728MB
      Thu Dec 12 11:57:30.762 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.733MB
      Thu Dec 12 11:57:30.762 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.737MB
      0xde05e1 0x9243b1 0x92448b 0x924502 0x924633 0x9246a0 0x91e0da 0x7eec70 0x7fcc9e 0x7fdefd 0x7fc8c3 0x7fd00f 0x7fdefd 0x7fc8c3 0x7fc9ab 
       /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
       /usr/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x9243b1]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4b) [0x92448b]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x924502]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x83) [0x924633]
       /usr/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x60) [0x9246a0]
       /usr/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0x91e0da]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13fixParentPtrsENS_7DiskLocEii+0xc0) [0x7eec70]
       /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x1ee) [0x7fcc9e]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
       /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x55f) [0x7fd00f]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x44b) [0x7fc9ab]
      Thu Dec 12 11:57:30.768 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.741MB
      Thu Dec 12 11:57:30.768 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.745MB
      Thu Dec 12 11:57:30.768 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.749MB
      Thu Dec 12 11:57:30.768 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.753MB
      Thu Dec 12 11:57:30.768 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.757MB
      Thu Dec 12 11:57:30.768 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.761MB
      Thu Dec 12 11:57:30.768 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.765MB
      Thu Dec 12 11:57:30.768 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.769MB
      Thu Dec 12 11:57:30.768 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.774MB
      Thu Dec 12 11:57:30.768 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.778MB
      0xde05e1 0x9243b1 0x92448b 0x924502 0x924633 0x9246a0 0x91e0da 0x7eec70 0x7fcc9e 0x7fdefd 0x7fc8c3 0x7fd00f 0x7fdefd 0x7fc8c3 0x7fc9ab 
       /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
       /usr/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x9243b1]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4b) [0x92448b]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x924502]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x83) [0x924633]
       /usr/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x60) [0x9246a0]
       /usr/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0x91e0da]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13fixParentPtrsENS_7DiskLocEii+0xc0) [0x7eec70]
       /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x1ee) [0x7fcc9e]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
       /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x55f) [0x7fd00f]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x44b) [0x7fc9ab]
      Thu Dec 12 11:57:30.774 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.782MB
      Thu Dec 12 11:57:30.774 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.786MB
      Thu Dec 12 11:57:30.774 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.79MB
      Thu Dec 12 11:57:30.774 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.794MB
      Thu Dec 12 11:57:30.774 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.798MB
      Thu Dec 12 11:57:30.774 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.802MB
      Thu Dec 12 11:57:30.775 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.806MB
      Thu Dec 12 11:57:30.775 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.81MB
      Thu Dec 12 11:57:30.775 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.814MB
      Thu Dec 12 11:57:30.775 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.819MB
      0xde05e1 0x9243b1 0x92448b 0x924502 0x924633 0x9246a0 0x91e0da 0x7eec70 0x7fcc9e 0x7fdefd 0x7fc8c3 0x7fd00f 0x7fdefd 0x7fc8c3 0x7fc9ab 
       /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
       /usr/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x9243b1]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4b) [0x92448b]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x924502]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x83) [0x924633]
       /usr/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x60) [0x9246a0]
       /usr/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0x91e0da]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13fixParentPtrsENS_7DiskLocEii+0xc0) [0x7eec70]
       /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x1ee) [0x7fcc9e]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
       /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x55f) [0x7fd00f]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x44b) [0x7fc9ab]
      Thu Dec 12 11:57:30.780 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.823MB
      Thu Dec 12 11:57:30.780 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.827MB
      Thu Dec 12 11:57:30.780 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.831MB
      Thu Dec 12 11:57:30.780 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.835MB
      Thu Dec 12 11:57:30.780 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.839MB
      Thu Dec 12 11:57:30.780 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.843MB
      Thu Dec 12 11:57:30.780 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.847MB
      Thu Dec 12 11:57:30.780 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.851MB
      Thu Dec 12 11:57:30.780 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.855MB
      Thu Dec 12 11:57:30.780 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.86MB
      0xde05e1 0x9243b1 0x92448b 0x924502 0x924633 0x9246a0 0x91e0da 0x7eec70 0x7fcc9e 0x7fdefd 0x7fc8c3 0x7fd00f 0x7fdefd 0x7fc8c3 0x7fc9ab 
       /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
       /usr/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x9243b1]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4b) [0x92448b]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x924502]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x83) [0x924633]
       /usr/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x60) [0x9246a0]
       /usr/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0x91e0da]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13fixParentPtrsENS_7DiskLocEii+0xc0) [0x7eec70]
       /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x1ee) [0x7fcc9e]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
       /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x55f) [0x7fd00f]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x44b) [0x7fc9ab]
      Thu Dec 12 11:57:30.787 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.864MB
      Thu Dec 12 11:57:30.787 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.868MB
      Thu Dec 12 11:57:30.787 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.872MB
      Thu Dec 12 11:57:30.787 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.876MB
      Thu Dec 12 11:57:30.787 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.88MB
      Thu Dec 12 11:57:30.788 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.884MB
      Thu Dec 12 11:57:30.788 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.888MB
      Thu Dec 12 11:57:30.788 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.892MB
      Thu Dec 12 11:57:30.788 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.896MB
      Thu Dec 12 11:57:30.788 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.9MB
      0xde05e1 0x9243b1 0x92448b 0x924502 0x924633 0x9246a0 0x91e0da 0x7eec70 0x7fcc9e 0x7fdefd 0x7fc8c3 0x7fd00f 0x7fdefd 0x7fc8c3 0x7fc9ab 
       /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
       /usr/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x9243b1]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4b) [0x92448b]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x924502]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x83) [0x924633]
       /usr/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x60) [0x9246a0]
       /usr/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0x91e0da]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13fixParentPtrsENS_7DiskLocEii+0xc0) [0x7eec70]
       /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x1ee) [0x7fcc9e]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
       /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x55f) [0x7fd00f]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x44b) [0x7fc9ab]
      Thu Dec 12 11:57:30.794 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.905MB
      Thu Dec 12 11:57:30.794 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.909MB
      Thu Dec 12 11:57:30.794 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.913MB
      Thu Dec 12 11:57:30.794 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.917MB
      Thu Dec 12 11:57:30.794 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.921MB
      Thu Dec 12 11:57:30.794 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.925MB
      Thu Dec 12 11:57:30.794 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.929MB
      Thu Dec 12 11:57:30.794 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.933MB
      Thu Dec 12 11:57:30.794 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.937MB
      Thu Dec 12 11:57:30.794 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.941MB
      0xde05e1 0x9243b1 0x92448b 0x924502 0x924633 0x9246a0 0x91e0da 0x7eec70 0x7fcc9e 0x7fdefd 0x7fc8c3 0x7fd00f 0x7fdefd 0x7fc8c3 0x7fc9ab 
       /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
       /usr/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x9243b1]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4b) [0x92448b]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x924502]
       /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x83) [0x924633]
       /usr/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x60) [0x9246a0]
       /usr/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0x91e0da]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13fixParentPtrsENS_7DiskLocEii+0xc0) [0x7eec70]
       /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x1ee) [0x7fcc9e]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
       /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x55f) [0x7fd00f]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
       /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x44b) [0x7fc9ab]
      Thu Dec 12 11:57:30.799 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.946MB
      Thu Dec 12 11:57:30.799 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.95MB
      Thu Dec 12 11:57:30.799 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.954MB
      Thu Dec 12 11:57:30.799 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.958MB
      Thu Dec 12 11:57:30.799 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.962MB
      Thu Dec 12 11:57:30.799 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.966MB
      Thu Dec 12 11:57:30.800 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.97MB
      Thu Dec 12 11:57:30.800 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.974MB
      Thu Dec 12 11:57:30.800 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.978MB
      Thu Dec 12 11:57:32.622 [conn56972] end connection 10.16.8.150:45971 (1 connection now open)
      Thu Dec 12 11:57:32.626 [rsBackgroundSync] replSet sync source problem: 10278 dbclient error communicating with server: 10.16.8.150:27017
      Thu Dec 12 11:57:33.435 [rsHealthPoll] DBClientCursor::init call() failed
      Thu Dec 12 11:57:33.435 [rsHealthPoll] replset info 10.16.8.150:27017 heartbeat failed, retrying
      Thu Dec 12 11:57:33.435 [rsHealthPoll] replSet info 10.16.8.150:27017 is down (or slow to respond): 
      Thu Dec 12 11:57:33.435 [rsHealthPoll] replSet member 10.16.8.150:27017 is now in state DOWN
      Thu Dec 12 11:57:33.437 [rsMgr] not electing self, 10.16.8.152:27017 would veto with 'I don't think 10.16.8.151:27017 is electable'
      Thu Dec 12 11:57:35.437 [rsHealthPoll] replset info 10.16.8.150:27017 heartbeat failed, retrying
      Thu Dec 12 11:57:37.438 [rsHealthPoll] replset info 10.16.8.150:27017 heartbeat failed, retrying
      Thu Dec 12 11:57:37.822 [conn56973] end connection 10.16.8.152:43144 (0 connections now open)
      Thu Dec 12 11:57:37.822 [initandlisten] connection accepted from 10.16.8.152:43149 #56974 (1 connection now open)
      Thu Dec 12 11:57:37.824 [conn56974]  authenticate db: local { authenticate: 1, nonce: "fccb36712470d89a", user: "__system", key: "24dd8d49037f98bf8ab5605a6e2fe136" }
      Thu Dec 12 11:57:38.649 [rsMgr] replSet info electSelf 1
      Thu Dec 12 11:57:39.440 [rsHealthPoll] replset info 10.16.8.150:27017 heartbeat failed, retrying
      Thu Dec 12 11:57:41.449 [rsHealthPoll] replset info 10.16.8.150:27017 heartbeat failed, retrying
      Thu Dec 12 11:57:42.122 [repl writer worker 2]  dorabella.requests warning assertion failure a <= 256*1024*1024 src/mongo/util/alignedbuilder.cpp 90
      0xde05e1 0xda022a 0xd9fe83 0x92dc2f 0x92dec4 0x921a94 0x921fea 0x91e5ef 0x91e6ac 0xc29738 0xc28e58 0xdadf21 0xe28e69 0x7fb4095449ca 0x7fb4088eb1cd 
       /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
       /usr/bin/mongod(_ZN5mongo9wassertedEPKcS1_j+0x11a) [0xda022a]
       /usr/bin/mongod(_ZN5mongo14AlignedBuilder14growReallocateEj+0x63) [0xd9fe83]
       /usr/bin/mongod() [0x92dc2f]
       /usr/bin/mongod(_ZN5mongo3dur13PREPLOGBUFFERERNS0_11JSectHeaderERNS_14AlignedBuilderE+0x214) [0x92dec4]
       /usr/bin/mongod() [0x921a94]
       /usr/bin/mongod(_ZN5mongo3dur11DurableImpl9commitNowEv+0x1a) [0x921fea]
       /usr/bin/mongod(_ZN5mongo3dur11DurableImpl16_aCommitIsNeededEv+0x17f) [0x91e5ef]
       /usr/bin/mongod(_ZN5mongo3dur11DurableImpl14commitIfNeededEb+0x4c) [0x91e6ac]
       /usr/bin/mongod(_ZN5mongo7replset8SyncTail9syncApplyERKNS_7BSONObjEb+0x738) [0xc29738]
       /usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x48) [0xc28e58]
       /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) [0xdadf21]
       /usr/bin/mongod() [0xe28e69]
       /lib/libpthread.so.0(+0x69ca) [0x7fb4095449ca]
       /lib/libc.so.6(clone+0x6d) [0x7fb4088eb1cd]
      Thu Dec 12 11:57:42.734 [repl writer worker 2] rate limiting wassert
      Thu Dec 12 11:57:43.458 [rsHealthPoll] replset info 10.16.8.150:27017 heartbeat failed, retrying
      Thu Dec 12 11:57:45.047 [repl writer worker 2]  dorabella.requests Assertion failure a <= 512*1024*1024 src/mongo/util/alignedbuilder.cpp 91
      0xde05e1 0xda15bd 0xd9fe9d 0x92dc2f 0x92dec4 0x921a94 0x921fea 0x91e5ef 0x91e6ac 0xc29738 0xc28e58 0xdadf21 0xe28e69 0x7fb4095449ca 0x7fb4088eb1cd 
       /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
       /usr/bin/mongod(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) [0xda15bd]
       /usr/bin/mongod(_ZN5mongo14AlignedBuilder14growReallocateEj+0x7d) [0xd9fe9d]
       /usr/bin/mongod() [0x92dc2f]
       /usr/bin/mongod(_ZN5mongo3dur13PREPLOGBUFFERERNS0_11JSectHeaderERNS_14AlignedBuilderE+0x214) [0x92dec4]
       /usr/bin/mongod() [0x921a94]
       /usr/bin/mongod(_ZN5mongo3dur11DurableImpl9commitNowEv+0x1a) [0x921fea]
       /usr/bin/mongod(_ZN5mongo3dur11DurableImpl16_aCommitIsNeededEv+0x17f) [0x91e5ef]
       /usr/bin/mongod(_ZN5mongo3dur11DurableImpl14commitIfNeededEb+0x4c) [0x91e6ac]
       /usr/bin/mongod(_ZN5mongo7replset8SyncTail9syncApplyERKNS_7BSONObjEb+0x738) [0xc29738]
       /usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x48) [0xc28e58]
       /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) [0xdadf21]
       /usr/bin/mongod() [0xe28e69]
       /lib/libpthread.so.0(+0x69ca) [0x7fb4095449ca]
       /lib/libc.so.6(clone+0x6d) [0x7fb4088eb1cd]
      Thu Dec 12 11:57:45.058 [repl writer worker 2] dbexception in groupCommit causing immediate shutdown: 0 assertion src/mongo/util/alignedbuilder.cpp:91
      Thu Dec 12 11:57:45.058 gc1
      Thu Dec 12 11:57:45.059 Got signal: 6 (Aborted).
       
      Thu Dec 12 11:57:45.069 Backtrace:
      0xde05e1 0x6d0559 0x7fb408834be0 0x7fb408834b65 0x7fb4088386b0 0x9ef2f7 0x921dca 0x921fea 0x91e5ef 0x91e6ac 0xc29738 0xc28e58 0xdadf21 0xe28e69 0x7fb4095449ca 0x7fb4088eb1cd 
       /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
       /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x6d0559]
       /lib/libc.so.6(+0x33be0) [0x7fb408834be0]
       /lib/libc.so.6(gsignal+0x35) [0x7fb408834b65]
       /lib/libc.so.6(abort+0x180) [0x7fb4088386b0]
       /usr/bin/mongod(_ZN5mongo10mongoAbortEPKc+0x47) [0x9ef2f7]
       /usr/bin/mongod() [0x921dca]
       /usr/bin/mongod(_ZN5mongo3dur11DurableImpl9commitNowEv+0x1a) [0x921fea]
       /usr/bin/mongod(_ZN5mongo3dur11DurableImpl16_aCommitIsNeededEv+0x17f) [0x91e5ef]
       /usr/bin/mongod(_ZN5mongo3dur11DurableImpl14commitIfNeededEb+0x4c) [0x91e6ac]
       /usr/bin/mongod(_ZN5mongo7replset8SyncTail9syncApplyERKNS_7BSONObjEb+0x738) [0xc29738]
       /usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x48) [0xc28e58]
       /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) [0xdadf21]
       /usr/bin/mongod() [0xe28e69]
       /lib/libpthread.so.0(+0x69ca) [0x7fb4095449ca]
       /lib/libc.so.6(clone+0x6d) [0x7fb4088eb1cd]

      Attachments

        Activity

          People

            bruce.lucas@mongodb.com Bruce Lucas (Inactive)
            beeftornado casey@hulu.com
            Votes:
            2 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: