[SERVER-12080] Hard crash - Assertion failure a <= 512*1024*1024 (alignedbuilder.cpp) Created: 13/Dec/13  Updated: 20/Apr/16  Resolved: 21/May/14

Status: Closed
Project: Core Server
Component/s: Stability
Affects Version/s: 2.4.8
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: casey@hulu.com Assignee: Bruce Lucas (Inactive)
Resolution: Done Votes: 2
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Ubuntu 10.04


Issue Links:
Depends
Related
Operating System: ALL
Participants:

 Description   

Running a mongo replicaset with two members + one arbiter. Both members will crash hard at the same time with the following stack trace in the log. I have not identified the action that caused it.

Thu Dec 12 11:56:36.602 [conn56967] end connection 10.16.8.150:45967 (1 connection now open)
Thu Dec 12 11:56:36.602 [initandlisten] connection accepted from 10.16.8.150:45969 #56969 (2 connections now open)
Thu Dec 12 11:56:36.604 [conn56969]  authenticate db: local { authenticate: 1, nonce: "817d28460fefee45", user: "__system", key: "7bd865435a880088115abd43c93c4d87" }
Thu Dec 12 11:56:37.785 [conn56968] end connection 10.16.8.152:43140 (1 connection now open)
Thu Dec 12 11:56:37.788 [initandlisten] connection accepted from 10.16.8.152:43142 #56970 (2 connections now open)
Thu Dec 12 11:56:37.790 [conn56970]  authenticate db: local { authenticate: 1, nonce: "11b7719a8c9c8ba2", user: "__system", key: "4cc99b17ea292d63a56f3169f1c24368" }
Thu Dec 12 11:57:00.872 [initandlisten] connection accepted from 10.128.13.11:53665 #56971 (3 connections now open)
Thu Dec 12 11:57:00.874 [conn56971] end connection 10.128.13.11:53665 (2 connections now open)
Thu Dec 12 11:57:06.625 [conn56969] end connection 10.16.8.150:45969 (1 connection now open)
Thu Dec 12 11:57:06.626 [initandlisten] connection accepted from 10.16.8.150:45971 #56972 (2 connections now open)
Thu Dec 12 11:57:06.626 [conn56972]  authenticate db: local { authenticate: 1, nonce: "7549836124c801bc", user: "__system", key: "85964ea4bd4ec8f7939da6ce53eef16e" }
Thu Dec 12 11:57:07.803 [conn56970] end connection 10.16.8.152:43142 (1 connection now open)
Thu Dec 12 11:57:07.804 [initandlisten] connection accepted from 10.16.8.152:43144 #56973 (2 connections now open)
Thu Dec 12 11:57:07.805 [conn56973]  authenticate db: local { authenticate: 1, nonce: "53b274b71898941c", user: "__system", key: "6fdd3676517ff5cdf4cab3b425775557" }
Thu Dec 12 11:57:30.687 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.577MB
0xde05e1 0x9243b1 0x92448b 0x924502 0x924633 0x9246a0 0x91e0da 0x7eec70 0x7fcc9e 0x7fdefd 0x7fc8c3 0x7fd00f 0x7fdefd 0x7fc8c3 0x7fc9ab 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
 /usr/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x9243b1]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4b) [0x92448b]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x924502]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x83) [0x924633]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x60) [0x9246a0]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0x91e0da]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13fixParentPtrsENS_7DiskLocEii+0xc0) [0x7eec70]
 /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x1ee) [0x7fcc9e]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
 /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x55f) [0x7fd00f]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x44b) [0x7fc9ab]
Thu Dec 12 11:57:30.697 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.581MB
0xde05e1 0x9243b1 0x92448b 0x924502 0x924633 0x9246a0 0x91e0da 0x7eec70 0x7fcc9e 0x7fdefd 0x7fc8c3 0x7fd00f 0x7fdefd 0x7fc8c3 0x7fc9ab 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
 /usr/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x9243b1]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4b) [0x92448b]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x924502]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x83) [0x924633]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x60) [0x9246a0]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0x91e0da]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13fixParentPtrsENS_7DiskLocEii+0xc0) [0x7eec70]
 /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x1ee) [0x7fcc9e]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
 /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x55f) [0x7fd00f]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x44b) [0x7fc9ab]
Thu Dec 12 11:57:30.702 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.585MB
0xde05e1 0x9243b1 0x92448b 0x924502 0x924633 0x9246a0 0x91e0da 0x7eec70 0x7fcc9e 0x7fdefd 0x7fc8c3 0x7fd00f 0x7fdefd 0x7fc8c3 0x7fc9ab 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
 /usr/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x9243b1]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4b) [0x92448b]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x924502]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x83) [0x924633]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x60) [0x9246a0]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0x91e0da]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13fixParentPtrsENS_7DiskLocEii+0xc0) [0x7eec70]
 /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x1ee) [0x7fcc9e]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
 /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x55f) [0x7fd00f]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x44b) [0x7fc9ab]
Thu Dec 12 11:57:30.709 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.589MB
0xde05e1 0x9243b1 0x92448b 0x924502 0x924633 0x9246a0 0x91e0da 0x7eec70 0x7fcc9e 0x7fdefd 0x7fc8c3 0x7fd00f 0x7fdefd 0x7fc8c3 0x7fc9ab 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
 /usr/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x9243b1]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4b) [0x92448b]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x924502]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x83) [0x924633]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x60) [0x9246a0]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0x91e0da]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13fixParentPtrsENS_7DiskLocEii+0xc0) [0x7eec70]
 /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x1ee) [0x7fcc9e]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
 /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x55f) [0x7fd00f]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x44b) [0x7fc9ab]
Thu Dec 12 11:57:30.715 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.593MB
0xde05e1 0x9243b1 0x92448b 0x924502 0x924633 0x9246a0 0x91e0da 0x7eec70 0x7fcc9e 0x7fdefd 0x7fc8c3 0x7fd00f 0x7fdefd 0x7fc8c3 0x7fc9ab 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
 /usr/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x9243b1]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4b) [0x92448b]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x924502]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x83) [0x924633]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x60) [0x9246a0]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0x91e0da]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13fixParentPtrsENS_7DiskLocEii+0xc0) [0x7eec70]
 /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x1ee) [0x7fcc9e]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
 /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x55f) [0x7fd00f]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x44b) [0x7fc9ab]
Thu Dec 12 11:57:30.720 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.597MB
0xde05e1 0x9243b1 0x92448b 0x924502 0x924633 0x9246a0 0x91e0da 0x7eec70 0x7fcc9e 0x7fdefd 0x7fc8c3 0x7fd00f 0x7fdefd 0x7fc8c3 0x7fc9ab 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
 /usr/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x9243b1]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4b) [0x92448b]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x924502]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x83) [0x924633]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x60) [0x9246a0]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0x91e0da]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13fixParentPtrsENS_7DiskLocEii+0xc0) [0x7eec70]
 /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x1ee) [0x7fcc9e]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
 /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x55f) [0x7fd00f]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x44b) [0x7fc9ab]
Thu Dec 12 11:57:30.726 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.601MB
0xde05e1 0x9243b1 0x92448b 0x924502 0x924633 0x9246a0 0x91e0da 0x7eec70 0x7fcc9e 0x7fdefd 0x7fc8c3 0x7fd00f 0x7fdefd 0x7fc8c3 0x7fc9ab 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
 /usr/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x9243b1]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4b) [0x92448b]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x924502]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x83) [0x924633]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x60) [0x9246a0]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0x91e0da]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13fixParentPtrsENS_7DiskLocEii+0xc0) [0x7eec70]
 /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x1ee) [0x7fcc9e]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
 /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x55f) [0x7fd00f]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x44b) [0x7fc9ab]
Thu Dec 12 11:57:30.732 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.606MB
0xde05e1 0x9243b1 0x92448b 0x924502 0x924633 0x9246a0 0x91e0da 0x7eec70 0x7fcc9e 0x7fdefd 0x7fc8c3 0x7fd00f 0x7fdefd 0x7fc8c3 0x7fc9ab 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
 /usr/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x9243b1]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4b) [0x92448b]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x924502]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x83) [0x924633]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x60) [0x9246a0]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0x91e0da]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13fixParentPtrsENS_7DiskLocEii+0xc0) [0x7eec70]
 /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x1ee) [0x7fcc9e]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
 /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x55f) [0x7fd00f]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x44b) [0x7fc9ab]
Thu Dec 12 11:57:30.737 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.61MB
0xde05e1 0x9243b1 0x92448b 0x924502 0x924633 0x9246a0 0x91e0da 0x7eec70 0x7fcc9e 0x7fdefd 0x7fc8c3 0x7fd00f 0x7fdefd 0x7fc8c3 0x7fc9ab 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
 /usr/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x9243b1]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4b) [0x92448b]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x924502]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x83) [0x924633]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x60) [0x9246a0]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0x91e0da]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13fixParentPtrsENS_7DiskLocEii+0xc0) [0x7eec70]
 /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x1ee) [0x7fcc9e]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
 /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x55f) [0x7fd00f]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x44b) [0x7fc9ab]
Thu Dec 12 11:57:30.743 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.614MB
0xde05e1 0x9243b1 0x92448b 0x924502 0x924633 0x9246a0 0x91e0da 0x7eec70 0x7fcc9e 0x7fdefd 0x7fc8c3 0x7fd00f 0x7fdefd 0x7fc8c3 0x7fc9ab 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
 /usr/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x9243b1]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4b) [0x92448b]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x924502]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x83) [0x924633]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x60) [0x9246a0]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0x91e0da]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13fixParentPtrsENS_7DiskLocEii+0xc0) [0x7eec70]
 /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x1ee) [0x7fcc9e]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
 /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x55f) [0x7fd00f]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x44b) [0x7fc9ab]
Thu Dec 12 11:57:30.748 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.618MB
Thu Dec 12 11:57:30.748 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.622MB
Thu Dec 12 11:57:30.749 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.626MB
Thu Dec 12 11:57:30.749 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.63MB
Thu Dec 12 11:57:30.750 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.634MB
Thu Dec 12 11:57:30.750 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.638MB
Thu Dec 12 11:57:30.750 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.642MB
Thu Dec 12 11:57:30.750 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.647MB
Thu Dec 12 11:57:30.750 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.651MB
Thu Dec 12 11:57:30.750 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.655MB
0xde05e1 0x9243b1 0x92448b 0x924502 0x924633 0x9246a0 0x91e0da 0x7eec70 0x7fcc9e 0x7fdefd 0x7fc8c3 0x7fd00f 0x7fdefd 0x7fc8c3 0x7fc9ab 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
 /usr/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x9243b1]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4b) [0x92448b]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x924502]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x83) [0x924633]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x60) [0x9246a0]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0x91e0da]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13fixParentPtrsENS_7DiskLocEii+0xc0) [0x7eec70]
 /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x1ee) [0x7fcc9e]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
 /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x55f) [0x7fd00f]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x44b) [0x7fc9ab]
Thu Dec 12 11:57:30.755 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.659MB
Thu Dec 12 11:57:30.755 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.663MB
Thu Dec 12 11:57:30.755 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.667MB
Thu Dec 12 11:57:30.755 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.671MB
Thu Dec 12 11:57:30.755 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.675MB
Thu Dec 12 11:57:30.755 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.679MB
Thu Dec 12 11:57:30.755 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.683MB
Thu Dec 12 11:57:30.755 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.687MB
Thu Dec 12 11:57:30.755 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.692MB
Thu Dec 12 11:57:30.755 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.696MB
0xde05e1 0x9243b1 0x92448b 0x924502 0x924633 0x9246a0 0x91e0da 0x7eec70 0x7fcc9e 0x7fdefd 0x7fc8c3 0x7fd00f 0x7fdefd 0x7fc8c3 0x7fc9ab 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
 /usr/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x9243b1]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4b) [0x92448b]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x924502]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x83) [0x924633]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x60) [0x9246a0]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0x91e0da]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13fixParentPtrsENS_7DiskLocEii+0xc0) [0x7eec70]
 /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x1ee) [0x7fcc9e]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
 /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x55f) [0x7fd00f]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x44b) [0x7fc9ab]
Thu Dec 12 11:57:30.761 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.7MB
Thu Dec 12 11:57:30.761 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.704MB
Thu Dec 12 11:57:30.761 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.708MB
Thu Dec 12 11:57:30.762 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.712MB
Thu Dec 12 11:57:30.762 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.716MB
Thu Dec 12 11:57:30.762 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.72MB
Thu Dec 12 11:57:30.762 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.724MB
Thu Dec 12 11:57:30.762 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.728MB
Thu Dec 12 11:57:30.762 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.733MB
Thu Dec 12 11:57:30.762 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.737MB
0xde05e1 0x9243b1 0x92448b 0x924502 0x924633 0x9246a0 0x91e0da 0x7eec70 0x7fcc9e 0x7fdefd 0x7fc8c3 0x7fd00f 0x7fdefd 0x7fc8c3 0x7fc9ab 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
 /usr/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x9243b1]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4b) [0x92448b]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x924502]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x83) [0x924633]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x60) [0x9246a0]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0x91e0da]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13fixParentPtrsENS_7DiskLocEii+0xc0) [0x7eec70]
 /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x1ee) [0x7fcc9e]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
 /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x55f) [0x7fd00f]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x44b) [0x7fc9ab]
Thu Dec 12 11:57:30.768 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.741MB
Thu Dec 12 11:57:30.768 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.745MB
Thu Dec 12 11:57:30.768 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.749MB
Thu Dec 12 11:57:30.768 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.753MB
Thu Dec 12 11:57:30.768 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.757MB
Thu Dec 12 11:57:30.768 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.761MB
Thu Dec 12 11:57:30.768 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.765MB
Thu Dec 12 11:57:30.768 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.769MB
Thu Dec 12 11:57:30.768 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.774MB
Thu Dec 12 11:57:30.768 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.778MB
0xde05e1 0x9243b1 0x92448b 0x924502 0x924633 0x9246a0 0x91e0da 0x7eec70 0x7fcc9e 0x7fdefd 0x7fc8c3 0x7fd00f 0x7fdefd 0x7fc8c3 0x7fc9ab 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
 /usr/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x9243b1]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4b) [0x92448b]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x924502]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x83) [0x924633]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x60) [0x9246a0]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0x91e0da]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13fixParentPtrsENS_7DiskLocEii+0xc0) [0x7eec70]
 /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x1ee) [0x7fcc9e]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
 /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x55f) [0x7fd00f]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x44b) [0x7fc9ab]
Thu Dec 12 11:57:30.774 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.782MB
Thu Dec 12 11:57:30.774 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.786MB
Thu Dec 12 11:57:30.774 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.79MB
Thu Dec 12 11:57:30.774 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.794MB
Thu Dec 12 11:57:30.774 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.798MB
Thu Dec 12 11:57:30.774 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.802MB
Thu Dec 12 11:57:30.775 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.806MB
Thu Dec 12 11:57:30.775 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.81MB
Thu Dec 12 11:57:30.775 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.814MB
Thu Dec 12 11:57:30.775 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.819MB
0xde05e1 0x9243b1 0x92448b 0x924502 0x924633 0x9246a0 0x91e0da 0x7eec70 0x7fcc9e 0x7fdefd 0x7fc8c3 0x7fd00f 0x7fdefd 0x7fc8c3 0x7fc9ab 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
 /usr/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x9243b1]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4b) [0x92448b]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x924502]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x83) [0x924633]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x60) [0x9246a0]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0x91e0da]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13fixParentPtrsENS_7DiskLocEii+0xc0) [0x7eec70]
 /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x1ee) [0x7fcc9e]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
 /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x55f) [0x7fd00f]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x44b) [0x7fc9ab]
Thu Dec 12 11:57:30.780 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.823MB
Thu Dec 12 11:57:30.780 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.827MB
Thu Dec 12 11:57:30.780 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.831MB
Thu Dec 12 11:57:30.780 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.835MB
Thu Dec 12 11:57:30.780 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.839MB
Thu Dec 12 11:57:30.780 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.843MB
Thu Dec 12 11:57:30.780 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.847MB
Thu Dec 12 11:57:30.780 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.851MB
Thu Dec 12 11:57:30.780 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.855MB
Thu Dec 12 11:57:30.780 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.86MB
0xde05e1 0x9243b1 0x92448b 0x924502 0x924633 0x9246a0 0x91e0da 0x7eec70 0x7fcc9e 0x7fdefd 0x7fc8c3 0x7fd00f 0x7fdefd 0x7fc8c3 0x7fc9ab 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
 /usr/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x9243b1]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4b) [0x92448b]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x924502]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x83) [0x924633]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x60) [0x9246a0]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0x91e0da]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13fixParentPtrsENS_7DiskLocEii+0xc0) [0x7eec70]
 /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x1ee) [0x7fcc9e]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
 /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x55f) [0x7fd00f]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x44b) [0x7fc9ab]
Thu Dec 12 11:57:30.787 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.864MB
Thu Dec 12 11:57:30.787 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.868MB
Thu Dec 12 11:57:30.787 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.872MB
Thu Dec 12 11:57:30.787 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.876MB
Thu Dec 12 11:57:30.787 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.88MB
Thu Dec 12 11:57:30.788 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.884MB
Thu Dec 12 11:57:30.788 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.888MB
Thu Dec 12 11:57:30.788 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.892MB
Thu Dec 12 11:57:30.788 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.896MB
Thu Dec 12 11:57:30.788 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.9MB
0xde05e1 0x9243b1 0x92448b 0x924502 0x924633 0x9246a0 0x91e0da 0x7eec70 0x7fcc9e 0x7fdefd 0x7fc8c3 0x7fd00f 0x7fdefd 0x7fc8c3 0x7fc9ab 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
 /usr/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x9243b1]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4b) [0x92448b]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x924502]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x83) [0x924633]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x60) [0x9246a0]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0x91e0da]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13fixParentPtrsENS_7DiskLocEii+0xc0) [0x7eec70]
 /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x1ee) [0x7fcc9e]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
 /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x55f) [0x7fd00f]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x44b) [0x7fc9ab]
Thu Dec 12 11:57:30.794 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.905MB
Thu Dec 12 11:57:30.794 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.909MB
Thu Dec 12 11:57:30.794 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.913MB
Thu Dec 12 11:57:30.794 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.917MB
Thu Dec 12 11:57:30.794 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.921MB
Thu Dec 12 11:57:30.794 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.925MB
Thu Dec 12 11:57:30.794 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.929MB
Thu Dec 12 11:57:30.794 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.933MB
Thu Dec 12 11:57:30.794 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.937MB
Thu Dec 12 11:57:30.794 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.941MB
0xde05e1 0x9243b1 0x92448b 0x924502 0x924633 0x9246a0 0x91e0da 0x7eec70 0x7fcc9e 0x7fdefd 0x7fc8c3 0x7fd00f 0x7fdefd 0x7fc8c3 0x7fc9ab 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
 /usr/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x201) [0x9243b1]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4b) [0x92448b]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x924502]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x83) [0x924633]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x60) [0x9246a0]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0x91e0da]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13fixParentPtrsENS_7DiskLocEii+0xc0) [0x7eec70]
 /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x1ee) [0x7fcc9e]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
 /usr/bin/mongod(_ZN5mongo11BtreeBucketINS_12BtreeData_V1EE5splitENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0x55f) [0x7fd00f]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE10insertHereENS_7DiskLocEiS3_RKNS_5KeyV1ERKNS_8OrderingES3_S3_RNS_12IndexDetailsE+0xabd) [0x7fdefd]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x363) [0x7fc8c3]
 /usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE7_insertENS_7DiskLocES3_RKNS_5KeyV1ERKNS_8OrderingEbS3_S3_RNS_12IndexDetailsE+0x44b) [0x7fc9ab]
Thu Dec 12 11:57:30.799 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.946MB
Thu Dec 12 11:57:30.799 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.95MB
Thu Dec 12 11:57:30.799 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.954MB
Thu Dec 12 11:57:30.799 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.958MB
Thu Dec 12 11:57:30.799 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.962MB
Thu Dec 12 11:57:30.799 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.966MB
Thu Dec 12 11:57:30.800 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.97MB
Thu Dec 12 11:57:30.800 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.974MB
Thu Dec 12 11:57:30.800 [repl writer worker 2] warning: DR102 too much data written uncommitted 314.978MB
Thu Dec 12 11:57:32.622 [conn56972] end connection 10.16.8.150:45971 (1 connection now open)
Thu Dec 12 11:57:32.626 [rsBackgroundSync] replSet sync source problem: 10278 dbclient error communicating with server: 10.16.8.150:27017
Thu Dec 12 11:57:33.435 [rsHealthPoll] DBClientCursor::init call() failed
Thu Dec 12 11:57:33.435 [rsHealthPoll] replset info 10.16.8.150:27017 heartbeat failed, retrying
Thu Dec 12 11:57:33.435 [rsHealthPoll] replSet info 10.16.8.150:27017 is down (or slow to respond): 
Thu Dec 12 11:57:33.435 [rsHealthPoll] replSet member 10.16.8.150:27017 is now in state DOWN
Thu Dec 12 11:57:33.437 [rsMgr] not electing self, 10.16.8.152:27017 would veto with 'I don't think 10.16.8.151:27017 is electable'
Thu Dec 12 11:57:35.437 [rsHealthPoll] replset info 10.16.8.150:27017 heartbeat failed, retrying
Thu Dec 12 11:57:37.438 [rsHealthPoll] replset info 10.16.8.150:27017 heartbeat failed, retrying
Thu Dec 12 11:57:37.822 [conn56973] end connection 10.16.8.152:43144 (0 connections now open)
Thu Dec 12 11:57:37.822 [initandlisten] connection accepted from 10.16.8.152:43149 #56974 (1 connection now open)
Thu Dec 12 11:57:37.824 [conn56974]  authenticate db: local { authenticate: 1, nonce: "fccb36712470d89a", user: "__system", key: "24dd8d49037f98bf8ab5605a6e2fe136" }
Thu Dec 12 11:57:38.649 [rsMgr] replSet info electSelf 1
Thu Dec 12 11:57:39.440 [rsHealthPoll] replset info 10.16.8.150:27017 heartbeat failed, retrying
Thu Dec 12 11:57:41.449 [rsHealthPoll] replset info 10.16.8.150:27017 heartbeat failed, retrying
Thu Dec 12 11:57:42.122 [repl writer worker 2]  dorabella.requests warning assertion failure a <= 256*1024*1024 src/mongo/util/alignedbuilder.cpp 90
0xde05e1 0xda022a 0xd9fe83 0x92dc2f 0x92dec4 0x921a94 0x921fea 0x91e5ef 0x91e6ac 0xc29738 0xc28e58 0xdadf21 0xe28e69 0x7fb4095449ca 0x7fb4088eb1cd 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
 /usr/bin/mongod(_ZN5mongo9wassertedEPKcS1_j+0x11a) [0xda022a]
 /usr/bin/mongod(_ZN5mongo14AlignedBuilder14growReallocateEj+0x63) [0xd9fe83]
 /usr/bin/mongod() [0x92dc2f]
 /usr/bin/mongod(_ZN5mongo3dur13PREPLOGBUFFERERNS0_11JSectHeaderERNS_14AlignedBuilderE+0x214) [0x92dec4]
 /usr/bin/mongod() [0x921a94]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl9commitNowEv+0x1a) [0x921fea]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl16_aCommitIsNeededEv+0x17f) [0x91e5ef]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl14commitIfNeededEb+0x4c) [0x91e6ac]
 /usr/bin/mongod(_ZN5mongo7replset8SyncTail9syncApplyERKNS_7BSONObjEb+0x738) [0xc29738]
 /usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x48) [0xc28e58]
 /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) [0xdadf21]
 /usr/bin/mongod() [0xe28e69]
 /lib/libpthread.so.0(+0x69ca) [0x7fb4095449ca]
 /lib/libc.so.6(clone+0x6d) [0x7fb4088eb1cd]
Thu Dec 12 11:57:42.734 [repl writer worker 2] rate limiting wassert
Thu Dec 12 11:57:43.458 [rsHealthPoll] replset info 10.16.8.150:27017 heartbeat failed, retrying
Thu Dec 12 11:57:45.047 [repl writer worker 2]  dorabella.requests Assertion failure a <= 512*1024*1024 src/mongo/util/alignedbuilder.cpp 91
0xde05e1 0xda15bd 0xd9fe9d 0x92dc2f 0x92dec4 0x921a94 0x921fea 0x91e5ef 0x91e6ac 0xc29738 0xc28e58 0xdadf21 0xe28e69 0x7fb4095449ca 0x7fb4088eb1cd 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
 /usr/bin/mongod(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) [0xda15bd]
 /usr/bin/mongod(_ZN5mongo14AlignedBuilder14growReallocateEj+0x7d) [0xd9fe9d]
 /usr/bin/mongod() [0x92dc2f]
 /usr/bin/mongod(_ZN5mongo3dur13PREPLOGBUFFERERNS0_11JSectHeaderERNS_14AlignedBuilderE+0x214) [0x92dec4]
 /usr/bin/mongod() [0x921a94]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl9commitNowEv+0x1a) [0x921fea]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl16_aCommitIsNeededEv+0x17f) [0x91e5ef]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl14commitIfNeededEb+0x4c) [0x91e6ac]
 /usr/bin/mongod(_ZN5mongo7replset8SyncTail9syncApplyERKNS_7BSONObjEb+0x738) [0xc29738]
 /usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x48) [0xc28e58]
 /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) [0xdadf21]
 /usr/bin/mongod() [0xe28e69]
 /lib/libpthread.so.0(+0x69ca) [0x7fb4095449ca]
 /lib/libc.so.6(clone+0x6d) [0x7fb4088eb1cd]
Thu Dec 12 11:57:45.058 [repl writer worker 2] dbexception in groupCommit causing immediate shutdown: 0 assertion src/mongo/util/alignedbuilder.cpp:91
Thu Dec 12 11:57:45.058 gc1
Thu Dec 12 11:57:45.059 Got signal: 6 (Aborted).
 
Thu Dec 12 11:57:45.069 Backtrace:
0xde05e1 0x6d0559 0x7fb408834be0 0x7fb408834b65 0x7fb4088386b0 0x9ef2f7 0x921dca 0x921fea 0x91e5ef 0x91e6ac 0xc29738 0xc28e58 0xdadf21 0xe28e69 0x7fb4095449ca 0x7fb4088eb1cd 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
 /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x6d0559]
 /lib/libc.so.6(+0x33be0) [0x7fb408834be0]
 /lib/libc.so.6(gsignal+0x35) [0x7fb408834b65]
 /lib/libc.so.6(abort+0x180) [0x7fb4088386b0]
 /usr/bin/mongod(_ZN5mongo10mongoAbortEPKc+0x47) [0x9ef2f7]
 /usr/bin/mongod() [0x921dca]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl9commitNowEv+0x1a) [0x921fea]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl16_aCommitIsNeededEv+0x17f) [0x91e5ef]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl14commitIfNeededEb+0x4c) [0x91e6ac]
 /usr/bin/mongod(_ZN5mongo7replset8SyncTail9syncApplyERKNS_7BSONObjEb+0x738) [0xc29738]
 /usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x48) [0xc28e58]
 /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) [0xdadf21]
 /usr/bin/mongod() [0xe28e69]
 /lib/libpthread.so.0(+0x69ca) [0x7fb4095449ca]
 /lib/libc.so.6(clone+0x6d) [0x7fb4088eb1cd]



 Comments   
Comment by Bruce Lucas (Inactive) [ 21/May/14 ]

Hi Casey,

Since we haven't heard back from you in some time, I'll go ahead and close this ticket out. If you'd like to continue pursuing this issue feel free to re-open the ticket with additional information.

Thanks,
Bruce

Comment by Bruce Lucas (Inactive) [ 09/May/14 ]

Michael - My apologies for the delay in responding. We don't currently document the details of the limits on large indexed arrays; in fact it is a bit difficult to document, as it depends on difficult-to-model factors like how many data and index pages any particular update touches. However I have opened DOCS-3219 to request that we document this in some form.

Casey - it looks to me like we've answered your original question; is that the case? Can we close this ticket now?

Thanks,
Bruce

Comment by Bruce Lucas (Inactive) [ 15/Jan/14 ]

Casey, there's nothing unusual in the information you provided. I see you have only one index on the codes array for example. By separate communication (let me know if you do not get it) I have provided a means for you to make some sample documents available if you are interested in continuing the investigation of why you hit the limit immediately after upgrading, although to be clear, with 200k elements you would eventually hit the limit regardless.

Michael, I still owe you an answer on whether and how we document this kind of limit.

Bruce

Comment by casey@hulu.com [ 09/Jan/14 ]

Oh Oh I see. I can do what I need to over here to make the maximum number of codes per batch smaller, which I think you agree should reduce the likelihood of this crash happening to close to zero.

But there's two open points.

Although there is a workaround for my specific case, I really think that all replicaset members shouldn't hard crash when this happens. I'm not a mongo core server developer, so I don't know, but it sounds like it would pretty hard to do since mongo isn't transactional, you would somehow have to know ahead of time by how much the indexes will change. That sounds really fun.

Second thing is we additionally have a shared mongo cluster that is made up of a few instances, but have a lot of databases running on them. It's more of a self service to developers so the administrators of the cluster have no control over the data being inserted. How can they prevent someone like me accidentally taking down their whole cluster? It hasn't happened yet, but would concern them I'm sure.

Here is some of the information you requested. I can privately send you an example document if it helps. Or if there's anything else I can provide that can help, let me know.

> db.requests.stats()
{
        "ns" : "dorabella.requests",
        "count" : 154979,
        "size" : 782676044,
        "avgObjSize" : 5050.207086121345,
        "storageSize" : 1121132544,
        "numExtents" : 9,
        "nindexes" : 5,
        "lastExtentSize" : 342536192,
        "paddingFactor" : 1.0000000000000007,
        "systemFlags" : 0,
        "userFlags" : 0,
        "totalIndexSize" : 1259218464,
        "indexSizes" : {
                "_id_" : 4586736,
                "codes_1" : 1243381552,
                "created_1" : 3957184,
                "created_by_1" : 2992416,
                "count_1" : 4300576
        },
        "ok" : 1
}

> db.requests.getIndexes()
[                                                                                                                                             
        {                                                                                                                                     
                "v" : 1,                                                                                                                      
                "key" : {
                        "_id" : 1
                },
                "ns" : "dorabella.requests",
                "name" : "_id_"
        },
        {
                "v" : 1,
                "key" : {
                        "codes" : 1
                },
                "unique" : true,
                "ns" : "dorabella.requests",
                "name" : "codes_1"
        },
        {
                "v" : 1,
                "key" : {
                        "created" : 1
                },
                "ns" : "dorabella.requests",
                "name" : "created_1"
        },
        {
                "v" : 1,
                "key" : {
                        "created_by" : 1
                },
                "ns" : "dorabella.requests",
                "name" : "created_by_1"
        },
        {
                "v" : 1,
                "key" : {
                        "count" : 1
                },
                "ns" : "dorabella.requests",
                "name" : "count_1"
        }
]
>

Comment by Michael Graff [ 07/Jan/14 ]

Thanks, Bruce.

Is there a place these recommended practices are gathered, or at least mentioned, so I can avoid stumbling on the next issue?

--Michael

Comment by Bruce Lucas (Inactive) [ 07/Jan/14 ]

Hi Casey,

The buckets I referred to are just the nodes of the b-tree that is used to store the index. Each node holds a particular range of key values; when the number of keys in that range gets too large to fit in the 8 kB bucket, the node must be split into two to accomodate the new key. When this happens the parent pointers of the child nodes must be updated, and that is what makes bucket splits somewhat expensive in terms of journal commit size. There are no tunable parameters to control this.

However node splits are just an exacerbating factor, and are probably not in and of themselves your primary problem: unfortunately arrays with 200 k elements are almost certain to encounter the 512 MB journal commit limit eventually, independent of bucket splits. You will be more likely to hit the limit as the index size grows because the keys will be more spread out within the index, increasing the number of buckets that must be updated for each document operation. It is possible that you hit this problem simply because your index size was growing, and the timing with respect to the upgrade to 2.4 was a coincidence. On the other hand there may have been incidental changes between 2.0.2 and 2.4 that changed the exact details of when and how the limit would be hit; this may also be sensitive to the exact distribution of keys in the document, since that will affect in a complicated way how the index data structure will be updated by any particular operation. If you'd like I can investigate this further if you can provide some large sample documents; if you aren't comfortable attaching them to this public ticket we can provide a private, secure location for you to upload the data. Also can please you provide the output of db.c.stats() and db.c.getIndexes(), where c is the collection in question; this will give us some useful information about the size and nature of the collection and the indexes.

It sounds like your application is already structured to limit the number of codes per document, and is set up to accommodate the need to allocate them in batches. Would it be possible to lower the limit per batch to say 10 k? Possibly this could be implemented in a transparent manner to the user by automatically splitting requests that are too large into multiple batches?

Bruce

Comment by casey@hulu.com [ 07/Jan/14 ]

The way our data is being stored is not how I would have done it, it was decided before my time. I'm just trying to keep the database afloat. Please no judgement...

This database is being used to store our promotion codes that users use at a later time for redemption. We have a collection in this database called `code_requests` that represents every time someone here generates these promotion codes to be given out. They can generate as many promotion codes as they need, but we do limit them at 200K codes per request to adhere to the maximum of 16MB per document. Before the insertion we have an additional check to check the document size and verify it is below 12MB before trying to insert.

Each document in this collection has a `codes` field that is an array of 12 character alphanumeric strings representing a promotion code. So the document could potentially have an array with 200K 12 character strings. This field is indexed so that we can search for a particular code upon redemption. In the past, this wasn't an issue when we ran server 2.0.2. Just before the first occurrence of this problem we had upgraded to 2.4.

Again, I hate storing the data this way, but it is what it is. These "buckets" you mentioned. Are there any configuration parameters that can be tweaked that affect how many splits occur? I think I gather from what you are saying that if there are less splits, there are less changes or at least a decreased size of the changed index.

Comment by Bruce Lucas (Inactive) [ 06/Jan/14 ]

Hi Michael,
You are correct, 74 k entries does not generate 512 MB of new data for the index. However it can change a large amount of data due to the need to move a large number of entries (worst case, all 8 kB worth of entries per bucket) in a large number of buckets (worst case, one bucket per array element), and all of those changes are journaled. The recommended practice is to choose your schema to keep indexed arrays small, no more than a few thousand entries.
Bruce

Comment by Michael Graff [ 06/Jan/14 ]

In my case, I am inserting around 75,000 64-bit integers. I would not have expected 74,000 new entries to come out to 512 MB of data in an index.

--Michael

Comment by Bruce Lucas (Inactive) [ 02/Jan/14 ]

Hi Casey and Michael,

The assert that you observed occurs when an operation on a document containing an indexed array requires a large number of updates to the index, resulting in more than 512 MB of index data that must be journaled in a single operation. The index updates can be a result of a document insert, delete, or move. How big an indexed array can be before hitting this bound depends on how extensively the index changes as a result of the operation, which depends on complex factors such as how many buckets are hit by the updates, and whether b-tree bucket splits are required as a result buckets becoming full.

For example, in the limit as the index size grows, each array entry will be in a different bucket and so require a different bucket to be updated, resulting in 8 kB of journaled data per array entry, for a limit of 512 MB / 8 kB = 64 k elements.

However this bound does not take into account the possibility of bucket splits, which are not frequent because buckets are large, but can result in a lot of journaled data when they do occur. Casey's stack trace indicates that bucket splits were in fact occurring when the 512 MB limit was reached. Casey, as Eliot mentioned can you tell us about any collections that use large indexed arrays - how big are the arrays, what type of elements, and what index(es)?

Thanks,
Bruce

Comment by Michael Graff [ 26/Dec/13 ]

Chopping my data up to ensure that no array to be indexed is "too large" (I chose 10,000 entries max to be extremely safe) seems to be a work-around for this issue. For my data, I am using the entry in the ids table to provide a ranking for some particular value for a document from another collection, so replicating the rank, value, idcount, and numbefore values with subsets of the total list of IDs is fine.

Some advice on how large an array may be before this problem is likely to occur would be wonderful, even if it cannot easily be fixed.

Comment by Michael Graff [ 26/Dec/13 ]

I am writing a largish array, and am getting his sort of crash as well.

I have a simple structure which consists of some metadata, and then multiple subdocuments which have an element I am indexing on.

If I disable the index

{'data.ids': 1}

, adding these elements works. With that index in place, I cannot add certain elements. If I create the index after the data has been added succeeds. However, then re-running my adds fails again, but at a slightly different point. Right now, I can reproduce this using my existing database and data set at will.

The largest array I insert for this part of the process is 106524 entries long, where each element in the ids array is a 64-bit integer. However, the add does not fail at this point, it fails on a subsequent insert, currently the one following the insert.

Lowering the length of the insert to 70,000 entries makes this succeed. Adding 75,000 will at some point cause a crash. However, some elements in the data set already have over 106,000 array elements, with the largest currently around 106,660 entries long.

{
worldid: 104,
classid: 6,
level: 'ALL',
name: 'quests.completed',
target: 'characters',
data: [

{ rank: 1, value: 23, ids: [ 1,2,3,4,5,6,7,8,9,...], idcount: X, numbefore: 156003 }

,

{ ... many more }

]
}

Comment by Eliot Horowitz (Inactive) [ 22/Dec/13 ]

Is there a collection on which you index very large arrays?
If so, can you send a sample doc and indexes?

Generated at Thu Feb 08 03:27:33 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.