[SERVER-12876] Server shuts down when compacting collection due to assertion failure in alignedbuilder.cpp Created: 25/Feb/14  Updated: 11/Jul/16  Resolved: 27/Feb/14

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: 2.6.0-rc0
Fix Version/s: 2.6.0-rc1

Type: Bug Priority: Major - P3
Reporter: Kamran K. Assignee: Eliot Horowitz (Inactive)
Resolution: Done Votes: 0
Labels: 26qa
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File server_12876_log.txt     Text File server_12876_v249_log.txt    
Issue Links:
Related
Operating System: ALL
Steps To Reproduce:

function addData(coll, iterations) {
    var arr = [];
 
    for (var i = 1; i <= iterations; i++) {
        arr.push({_id: i, data: new Array(1024 * 500).join('a')});
 
        if (i % 50 === 0) {
            coll.insert(arr);
            arr = [];
        }
    }
}
 
var myDB = db.getSiblingDB('repro');
myDB.dropDatabase();
 
// add some dummy data and compact
addData(myDB.compactme, 5000);
myDB.runCommand({compact: 'compactme'});

Participants:

 Description   

Running compact on a collection with ~2.6 GB of data causes a server shutdown.

[...snip...]
2014-02-24T20:54:35.388-0500 [conn2] warning: DR102 too much data written uncommitted 560.517MB
2014-02-24T20:54:35.388-0500 [conn2] repro.compactme warning assertion failure a <= 256*1024*1024 src/mongo/util/alignedbuilder.cpp 103
2014-02-24T20:54:35.392-0500 [conn2] repro.compactme 0x100770850 0x10071d4cb 0x10070ab07 0x10070a1a3 0x100227bc6 0x10022764b 0x10021a680 0x100219fb5 0x10021ae3b 0x10021af0f 0x100573c95 0x1005759bb 0x100152f84 0x1001f2665 0x1001f3525 0x1001f47db 0x100434da5 0x1002e5b42 0x10000ed87 0x10072ba81 
 0   mongod                              0x0000000100770850 _ZN5mongo15printStackTraceERSo + 64
 1   mongod                              0x000000010071d4cb _ZN5mongo10logContextEPKc + 155
 2   mongod                              0x000000010070ab07 _ZN5mongo9wassertedEPKcS1_j + 615
 3   mongod                              0x000000010070a1a3 _ZN5mongo14AlignedBuilder14growReallocateEj + 99
 4   mongod                              0x0000000100227bc6 _ZN5mongo3durL21prepBasicWrite_inlockERNS_14AlignedBuilderEPKNS0_11WriteIntentERNS_12RelativePathE + 1174
 5   mongod                              0x000000010022764b _ZN5mongo3dur13PREPLOGBUFFERERNS0_11JSectHeaderERNS_14AlignedBuilderE + 443
 6   mongod                              0x000000010021a680 _ZN5mongo3durL11groupCommitEPNS_4Lock11GlobalWriteE + 1680
 7   mongod                              0x0000000100219fb5 _ZN5mongo3dur11DurableImpl9commitNowEv + 21
 8   mongod                              0x000000010021ae3b _ZN5mongo3dur11DurableImpl16_aCommitIsNeededEv + 635
 9   mongod                              0x000000010021af0f _ZN5mongo3dur11DurableImpl14commitIfNeededEb + 79
 10  mongod                              0x0000000100573c95 _ZN5mongo10Collection14_compactExtentENS_7DiskLocEiRNS_15MultiIndexBlockEPKNS_14CompactOptionsEPNS_12CompactStatsE + 1573
 11  mongod                              0x00000001005759bb _ZN5mongo10Collection7compactEPKNS_14CompactOptionsE + 4075
 12  mongod                              0x0000000100152f84 _ZN5mongo10CompactCmd3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb + 2516
 13  mongod                              0x00000001001f2665 _ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb + 37
 14  mongod                              0x00000001001f3525 _ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb + 2629
 15  mongod                              0x00000001001f47db _ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi + 875
 16  mongod                              0x0000000100434da5 _ZN5mongo11newRunQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_ + 741
 17  mongod                              0x00000001002e5b42 _ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE + 1698
 18  mongod                              0x000000010000ed87 _ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE + 183
 19  mongod                              0x000000010072ba81 _ZN5mongo17PortMessageServer17handleIncomingMsgEPv + 913
2014-02-24T20:54:35.392-0500 [conn2] repro.compactme Assertion failure a <= 512*1024*1024 src/mongo/util/alignedbuilder.cpp 104
2014-02-24T20:54:35.396-0500 [conn2] repro.compactme 0x100770850 0x10071d4cb 0x10070ad43 0x10070a1c3 0x100227bc6 0x10022764b 0x10021a680 0x100219fb5 0x10021ae3b 0x10021af0f 0x100573c95 0x1005759bb 0x100152f84 0x1001f2665 0x1001f3525 0x1001f47db 0x100434da5 0x1002e5b42 0x10000ed87 0x10072ba81 
 0   mongod                              0x0000000100770850 _ZN5mongo15printStackTraceERSo + 64
 1   mongod                              0x000000010071d4cb _ZN5mongo10logContextEPKc + 155
 2   mongod                              0x000000010070ad43 _ZN5mongo12verifyFailedEPKcS1_j + 435
 3   mongod                              0x000000010070a1c3 _ZN5mongo14AlignedBuilder14growReallocateEj + 131
 4   mongod                              0x0000000100227bc6 _ZN5mongo3durL21prepBasicWrite_inlockERNS_14AlignedBuilderEPKNS0_11WriteIntentERNS_12RelativePathE + 1174
 5   mongod                              0x000000010022764b _ZN5mongo3dur13PREPLOGBUFFERERNS0_11JSectHeaderERNS_14AlignedBuilderE + 443
 6   mongod                              0x000000010021a680 _ZN5mongo3durL11groupCommitEPNS_4Lock11GlobalWriteE + 1680
 7   mongod                              0x0000000100219fb5 _ZN5mongo3dur11DurableImpl9commitNowEv + 21
 8   mongod                              0x000000010021ae3b _ZN5mongo3dur11DurableImpl16_aCommitIsNeededEv + 635
 9   mongod                              0x000000010021af0f _ZN5mongo3dur11DurableImpl14commitIfNeededEb + 79
 10  mongod                              0x0000000100573c95 _ZN5mongo10Collection14_compactExtentENS_7DiskLocEiRNS_15MultiIndexBlockEPKNS_14CompactOptionsEPNS_12CompactStatsE + 1573
 11  mongod                              0x00000001005759bb _ZN5mongo10Collection7compactEPKNS_14CompactOptionsE + 4075
 12  mongod                              0x0000000100152f84 _ZN5mongo10CompactCmd3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb + 2516
 13  mongod                              0x00000001001f2665 _ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb + 37
 14  mongod                              0x00000001001f3525 _ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb + 2629
 15  mongod                              0x00000001001f47db _ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi + 875
 16  mongod                              0x0000000100434da5 _ZN5mongo11newRunQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_ + 741
 17  mongod                              0x00000001002e5b42 _ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE + 1698
 18  mongod                              0x000000010000ed87 _ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE + 183
 19  mongod                              0x000000010072ba81 _ZN5mongo17PortMessageServer17handleIncomingMsgEPv + 913
2014-02-24T20:54:35.396-0500 [conn2] dbexception in groupCommit causing immediate shutdown: 0 assertion src/mongo/util/alignedbuilder.cpp:104
2014-02-24T20:54:35.396-0500 [conn2] SEVERE: gc1
Got signal: 6 (Abort trap: 6).
Backtrace:
0x100770850 0x100007466 0x7fff906a390a 0 0x7fff906faf61 0x1002e5482 0x10021a1bd 0x100219fb5 0x10021ae3b 0x10021af0f 0x100573c95 0x1005759bb 0x100152f84 0x1001f2665 0x1001f3525 0x1001f47db 0x100434da5 0x1002e5b42 0x10000ed87 0x10072ba81 
 0   mongod                              0x0000000100770850 _ZN5mongo15printStackTraceERSo + 64
 1   mongod                              0x0000000100007466 _ZN5mongo10abruptQuitEi + 422
 2   libsystem_c.dylib                   0x00007fff906a390a _sigtramp + 26
 3   ???                                 0x0000000000000000 0x0 + 0
 4   libsystem_c.dylib                   0x00007fff906faf61 abort + 143
 5   mongod                              0x00000001002e5482 _ZN5mongo10mongoAbortEPKc + 146
 6   mongod                              0x000000010021a1bd _ZN5mongo3durL11groupCommitEPNS_4Lock11GlobalWriteE + 461
 7   mongod                              0x0000000100219fb5 _ZN5mongo3dur11DurableImpl9commitNowEv + 21
 8   mongod                              0x000000010021ae3b _ZN5mongo3dur11DurableImpl16_aCommitIsNeededEv + 635
 9   mongod                              0x000000010021af0f _ZN5mongo3dur11DurableImpl14commitIfNeededEb + 79
 10  mongod                              0x0000000100573c95 _ZN5mongo10Collection14_compactExtentENS_7DiskLocEiRNS_15MultiIndexBlockEPKNS_14CompactOptionsEPNS_12CompactStatsE + 1573
 11  mongod                              0x00000001005759bb _ZN5mongo10Collection7compactEPKNS_14CompactOptionsE + 4075
 12  mongod                              0x0000000100152f84 _ZN5mongo10CompactCmd3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb + 2516
 13  mongod                              0x00000001001f2665 _ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb + 37
 14  mongod                              0x00000001001f3525 _ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb + 2629
 15  mongod                              0x00000001001f47db _ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi + 875
 16  mongod                              0x0000000100434da5 _ZN5mongo11newRunQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_ + 741
 17  mongod                              0x00000001002e5b42 _ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE + 1698
 18  mongod                              0x000000010000ed87 _ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE + 183
 19  mongod                              0x000000010072ba81 _ZN5mongo17PortMessageServer17handleIncomingMsgEPv + 913


Version: ce97325da8b0494b813c2777283fed532adda9db



 Comments   
Comment by Githook User [ 27/Feb/14 ]

Author:

{u'username': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: SERVER-12876: fix durability check for if we need to commit fixing compact
Branch: master
https://github.com/mongodb/mongo/commit/b605f98c68a03fd0e65be72f9a22af4f107dee6b

Comment by Mark Callaghan [ 26/Feb/14 ]

I ran into the same crash. When running compact in 2.4.9 I get messages like this in the mongod log:
Wed Feb 26 13:43:49.492 [conn11] compact finished extent #14 containing 868577 documents (48.6403MB) oldPadding: 1.12 1

But for 2.6.0 I get :
2014-02-26T14:33:25.051-0500 [conn11] compact finished extent #14 containing 976510 documents (109.369MB) oldPadding: 2.24 2

The "1.12" versus "2.24" makes me wonder whether something changed for the padding factor computation? I have not enabled powerOf2Sizes and I do not specify padding factor or bytes for compact.

Comment by Eric Milkie [ 25/Feb/14 ]

It looks like the four calls to commitIfNeeded() in the 2.4 compact code are still there after the compact refactoring. We'll need to audit the new compact to make sure it's committing often enough.

Comment by Kamran K. [ 25/Feb/14 ]

I wasn't able to trigger a shutdown or a warning with 2.4.9. I tried with the same amount of data and 3x the amount of data (i.e., 15000 documents). The attached log shows the server output for the 3x run with 2.4.9.

Comment by Asya Kamsky [ 25/Feb/14 ]

does it compact successfully in 2.4.x? In other words - it doesn't seem to be a version specific issue - the server always will shut down if more than 512MB is written without being journaled...

Comment by Kamran K. [ 25/Feb/14 ]

Full log attached. It looks like it's failing while copying records for extent #15.

Comment by Asya Kamsky [ 25/Feb/14 ]

Do you have the whole log? What step was it in when this happened?

Generated at Thu Feb 08 03:29:54 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.