[SERVER-17784] Crash during index build on Secondary Created: 29/Mar/15  Updated: 01/Apr/15  Resolved: 01/Apr/15

Status: Closed
Project: Core Server
Component/s: Replication, WiredTiger
Affects Version/s: 3.0.1
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Joseph Feser Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

14.04 LTS Windows Azure A1


Operating System: Linux
Participants:

 Description   

The server is a secondary only to ensure we do not loose data in the primary. It is an Ubuntu 14.04 A1 Instance on Azure.

2015-03-26T06:37:21.180+0000 I ACCESS   [conn19895] Successfully authenticated as principal __system on local
2015-03-26T06:37:23.002+0000 I -        [repl writer worker 15]   Index Build: 2086300/3030801 68%
2015-03-26T06:37:26.002+0000 I -        [repl writer worker 15]   Index Build: 2278000/3030801 75%
2015-03-26T06:37:29.002+0000 I -        [repl writer worker 15]   Index Build: 2481800/3030801 81%
2015-03-26T06:37:30.685+0000 I COMMAND  [conn19894] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "op od", pv: 1, v: 103885, from: "opodmongosc.cloudapp.net:27017", fromId: 0, checkEmpty: false } ntoreturn:1 keyUpdates:0 writeConflicts:0 numYields:0 reslen:162 locks:{} 205ms
2015-03-26T06:37:31.888+0000 I COMMAND  [conn19895] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "op od", pv: 1, v: 103885, from: "opodmongosc3:27017", fromId: 2, checkEmpty: false } ntoreturn:1 keyUpdates:0 writeConflicts:0 numYields:0 reslen:162 locks:{} 226ms
2015-03-26T06:37:32.332+0000 I -        [repl writer worker 15]   Index Build: 2521200/3030801 83%
2015-03-26T06:37:32.786+0000 I COMMAND  [conn19789] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:436 locks:{} 222ms
2015-03-26T06:37:33.412+0000 I COMMAND  [conn19894] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "op od", pv: 1, v: 103885, from: "opodmongosc.cloudapp.net:27017", fromId: 0, checkEmpty: false } ntoreturn:1 keyUpdates:0 writeConflicts:0 numYields:0 reslen:162 locks:{} 199ms
2015-03-26T06:37:33.875+0000 E STORAGE  [repl writer worker 15] WiredTiger (12) [1427351853:866266][1691:0x7f59f0f4c700], file:logs/collection/16--7785950990033672546.wt, cursor.next: memory allocation: Cannot allocate memory
2015-03-26T06:37:33.880+0000 I -        [repl writer worker 15] Invariant failure: ret resulted in status UnknownError 12: Cannot allocate memory at src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp 1110
2015-03-26T06:37:34.554+0000 I COMMAND  [conn19895] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "op od", pv: 1, v: 103885, from: "opodmongosc3:27017", fromId: 2, checkEmpty: false } ntoreturn:1 keyUpdates:0 writeConflicts:0 numYields:0 reslen:162 locks:{} 210ms
2015-03-26T06:37:34.580+0000 I CONTROL  [repl writer worker 15]
 0xf4d299 0xeeda71 0xed421a 0xd6687c 0xd668a2 0x9fd8b2 0xbd0264 0xbd0614 0x92cd90 0xaa814e 0xaa866f 0xc3ac3b 0xc914fe 0x
c93ce5 0xee488b 0xf9b304 0x7f5a03bcd182 0x7f5a0269647d
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"B4D299"},{"b":"400000","o":"AEDA71"},{"b":"400000","o":"AD421A"},{"b":"400000","o":"966
87C"},{"b":"400000","o":"9668A2"},{"b":"400000","o":"5FD8B2"},{"b":"400000","o":"7D0264"},{"b":"400000","o":"7D0614"},{"
b":"400000","o":"52CD90"},{"b":"400000","o":"6A814E"},{"b":"400000","o":"6A866F"},{"b":"400000","o":"83AC3B"},{"b":"4000
00","o":"8914FE"},{"b":"400000","o":"893CE5"},{"b":"400000","o":"AE488B"},{"b":"400000","o":"B9B304"},{"b":"7F5A03BC5000
","o":"8182"},{"b":"7F5A0259C000","o":"FA47D"}],"processInfo":{ "mongodbVersion" : "3.0.1", "gitVersion" : "534b5a3f9d10
f00cd27737fbcd951032248b5952", "uname" : { "sysname" : "Linux", "release" : "3.16.0-31-generic", "version" : "#41~14.04.
1-Ubuntu SMP Wed Feb 11 19:30:13 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildI
d" : "C35E766AD226FC0C16CB0C3885EC3B59E288A3F2" }, { "b" : "7FFFFD2ED000", "elfType" : 3, "buildId" : "5552B9335DDE93494
19BA10896C1E75C9432A946" }, { "b" : "7F5A03BC5000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "bu
ildId" : "9318E8AF0BFBE444731BB0461202EF57F7C39542" }, { "b" : "7F5A03967000", "path" : "/lib/x86_64-linux-gnu/libssl.so
.1.0.0", "elfType" : 3, "buildId" : "FF43D0947510134A8A494063A3C1CF3CEBB27791" }, { "b" : "7F5A0358D000", "path" : "/lib
/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "379F80D2768BA6A21F52781895EE9F47B34A0A85" }, { "b" :
"7F5A03385000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "92FCF41EFE012D6186E31A59AD05BDB
B487769AB" }, { "b" : "7F5A03181000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "C1AE4CB71
95D337A77A3C689051DABAA3980CA0C" }, { "b" : "7F5A02E7D000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfTyp
e" : 3, "buildId" : "19EFDDAB11B3BF5C71570078C59F91CF6592CE9E" }, { "b" : "7F5A02B77000", "path" : "/lib/x86_64-linux-gn
u/libm.so.6", "elfType" : 3, "buildId" : "1D76B71E905CB867B27CEF230FCB20F01A3178F5" }, { "b" : "7F5A02961000", "path" :
"/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "8D0AA71411580EE6C08809695C3984769F25725B" }, { "b" :
"7F5A0259C000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "30C94DC66A1FE95180C3D68D2B89E576
D5AE213C" }, { "b" : "7F5A03DE3000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9F00581AB3C73E3
AEA35995A0C50D24D59A01D47" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf4d299]
 mongod(_ZN5mongo10logContextEPKc+0xE1) [0xeeda71]
 mongod(_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j+0xDA) [0xed421a]
 mongod(_ZN5mongo21WiredTigerRecordStore8Iterator8_getNextEv+0xFC) [0xd6687c]
 mongod(_ZN5mongo21WiredTigerRecordStore8Iterator7getNextEv+0x12) [0xd668a2]
 mongod(_ZN5mongo14CollectionScan4workEPm+0x2B2) [0x9fd8b2]
 mongod(_ZN5mongo12PlanExecutor18getNextSnapshottedEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE+0xA4) [0xbd0264]
 mongod(_ZN5mongo12PlanExecutor7getNextEPNS_7BSONObjEPNS_8RecordIdE+0x34) [0xbd0614]
 mongod(_ZN5mongo15MultiIndexBlock30insertAllDocumentsInCollectionEPSt3setINS_8RecordIdESt4lessIS2_ESaIS2_EE+0x130) [0x9
2cd90]
 mongod(_ZNK5mongo12IndexBuilder6_buildEPNS_16OperationContextEPNS_8DatabaseEbPNS_4Lock6DBLockE+0x42E) [0xaa814e]
 mongod(_ZNK5mongo12IndexBuilder17buildInForegroundEPNS_16OperationContextEPNS_8DatabaseE+0xF) [0xaa866f]
 mongod(_ZN5mongo4repl21applyOperation_inlockEPNS_16OperationContextEPNS_8DatabaseERKNS_7BSONObjEbb+0x146B) [0xc3ac3b]
 mongod(_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEb+0x2EE) [0xc914fe]
 mongod(_ZN5mongo4repl14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x65) [0xc93ce5]
 mongod(_ZN5mongo10threadpool6Worker4loopERKSs+0x2FB) [0xee488b]
 mongod(+0xB9B304) [0xf9b304]
 libpthread.so.0(+0x8182) [0x7f5a03bcd182]
 libc.so.6(clone+0x6D) [0x7f5a0269647d]
-----  END BACKTRACE  -----
2015-03-26T06:37:34.582+0000 I -        [repl writer worker 15]
 
***aborting after invariant() failure



 Comments   
Comment by Ramon Fernandez Marina [ 01/Apr/15 ]

Glad to hear you solve the issue joefeser. Closing the ticket as per your request.

Regards,
Ramón.

Comment by Joseph Feser [ 29/Mar/15 ]

By Default Azure did not have a swap set up. I added 5 gigs. If this is an OOM issue, just close it due to operator issue.

Generated at Thu Feb 08 03:45:33 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.