[SERVER-15135] Server crash while building index Created: 04/Sep/14  Updated: 24/Jan/15  Resolved: 23/Jan/15

Status: Closed
Project: Core Server
Component/s: Index Maintenance, Stability
Affects Version/s: 2.6.4
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Roy Assignee: Unassigned
Resolution: Incomplete Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-15138 Mongo segmentation fault during creat... Closed
Operating System: ALL
Participants:

 Description   

2014-09-04T06:02:37.149+0000 [conn850491] Tenant_42.files 0x11e6111 0x1187e49 0x116a97d 0xe577aa 0xe5000e 0x9356cb 0xa2889a 0xa29ce2 0xa2bea6 0xd5dd6d 0xb9fe62 0xba1440 0x770aef 0x119bf3e 0x7fe9b4b9ae9a 0x7fe9b3eae31d
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0x11e6111]
/usr/bin/mongod(_ZN5mongo10logContextEPKc+0x159) [0x1187e49]
/usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xcd) [0x116a97d]
/usr/bin/mongod() [0xe577aa]
/usr/bin/mongod(ZN5mongo5logOpEPKcS1_RKNS_7BSONObjEPS2_PbbPS3+0xee) [0xe5000e]
/usr/bin/mongod(_ZN5mongo14CmdCreateIndex3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x276b) [0x9356cb]
/usr/bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x3a) [0xa2889a]
/usr/bin/mongod(_ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x1042) [0xa29ce2]
/usr/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x6c6) [0xa2bea6]
/usr/bin/mongod(ZN5mongo11newRunQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1+0x22ed) [0xd5dd6d]
/usr/bin/mongod() [0xb9fe62]
/usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x580) [0xba1440]
/usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x9f) [0x770aef]
/usr/bin/mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x4ee) [0x119bf3e]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fe9b4b9ae9a]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fe9b3eae31d]
2014-09-04T06:02:37.149+0000 [conn850491]

***aborting after fassert() failure

2014-09-04T06:02:37.167+0000 [conn850491] SEVERE: Got signal: 6 (Aborted).
Backtrace:0x11e6111 0x11e54ee 0x7fe9b3df0150 0x7fe9b3df00d5 0x7fe9b3df383b 0x116a9ea 0xe577aa 0xe5000e 0x9356cb 0xa2889a 0xa29ce2 0xa2bea6 0xd5dd6d 0xb9fe62 0xba1440 0x770aef 0x119bf3e 0x7fe9b4b9ae9a 0x7fe9b3eae31d
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0x11e6111]
/usr/bin/mongod() [0x11e54ee]
/lib/x86_64-linux-gnu/libc.so.6(+0x36150) [0x7fe9b3df0150]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7fe9b3df00d5]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x17b) [0x7fe9b3df383b]
/usr/bin/mongod(_ZN5mongo13fassertFailedEi+0x13a) [0x116a9ea]
/usr/bin/mongod() [0xe577aa]
/usr/bin/mongod(ZN5mongo5logOpEPKcS1_RKNS_7BSONObjEPS2_PbbPS3+0xee) [0xe5000e]
/usr/bin/mongod(_ZN5mongo14CmdCreateIndex3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x276b) [0x9356cb]
/usr/bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x3a) [0xa2889a]
/usr/bin/mongod(_ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x1042) [0xa29ce2]
/usr/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x6c6) [0xa2bea6]
/usr/bin/mongod(ZN5mongo11newRunQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1+0x22ed) [0xd5dd6d]
/usr/bin/mongod() [0xb9fe62]
/usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x580) [0xba1440]
/usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x9f) [0x770aef]
/usr/bin/mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x4ee) [0x119bf3e]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fe9b4b9ae9a]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fe9b3eae31d]



 Comments   
Comment by J Rassi [ 04/Sep/14 ]

mamoos1,

  • Could you attach the full mongod log to the ticket?
  • Are you able to reproduce this issue?

k.nebel: could you open a new ticket for the issue, and attach the full mongod log to the ticket?

Comment by Konstantin Nebel [ 04/Sep/14 ]

It feels like I have the same prob? But I am not sure. A developer executed a map reduce job and the primary of one shard is keeping crashing if he executes the map-reduce job.

Let me know if what kind of information you need:

2014-09-04T13:06:13.880+0200 [initandlisten] connection accepted from 10.71.22.15:60021 #13039 (555 connections now open)
2014-09-04T13:06:13.880+0200 [conn12935] collection dmp_995.profilingTagCalls was previously unsharded, new metadata loaded with shard version 3|1||540720fad72e54d60986994b
2014-09-04T13:06:13.880+0200 [conn12935] collection version was loaded at version 3|1||540720fad72e54d60986994b, took 0ms
2014-09-04T13:06:13.880+0200 [conn12882] SyncClusterConnection connecting to [10.71.21.121:27020]
2014-09-04T13:06:13.881+0200 [conn12882] SyncClusterConnection connecting to [10.71.21.122:27020]
2014-09-04T13:06:13.884+0200 [conn12882] collection dmp_990.profilingTagCalls was previously unsharded, new metadata loaded with shard version 3|1||540720ebd72e54d60986993c
2014-09-04T13:06:13.884+0200 [conn12882] collection version was loaded at version 3|1||540720ebd72e54d60986993c, took 4ms
2014-09-04T13:06:13.902+0200 [conn12935] CMD: drop dmp_995.tmp.mr.profilingTagCalls_0
2014-09-04T13:06:13.902+0200 [conn12935] CMD: drop dmp_995.tmp.mr.profilingTagCalls_0_inc
2014-09-04T13:06:13.903+0200 [conn12935] build index on: dmp_995.tmp.mr.profilingTagCalls_0_inc properties: { v: 1, key: { 0: 1 }, name: "_temp_0", ns: "dmp_995.tmp.mr.profilingTagCalls_0_inc" }
2014-09-04T13:06:13.903+0200 [conn12935]         added index to empty collection
2014-09-04T13:06:13.903+0200 [conn12935] build index on: dmp_995.tmp.mr.profilingTagCalls_0 properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "dmp_995.tmp.mr.profilingTagCalls_0" }
2014-09-04T13:06:13.903+0200 [conn12935]         added index to empty collection
2014-09-04T13:06:14.057+0200 [conn12917] opening db:  dmp_tmp_998
2014-09-04T13:06:14.064+0200 [conn12917] CMD: drop dmp_tmp_998.tmp.mr.profilingTagCalls_1
2014-09-04T13:06:14.064+0200 [FileAllocator] allocating new datafile /var/lib/mongo/dmp_tmp_998.ns, filling with zeroes...
2014-09-04T13:06:14.065+0200 [FileAllocator] done allocating datafile /var/lib/mongo/dmp_tmp_998.ns, size: 16MB,  took 0 secs
2014-09-04T13:06:14.067+0200 [FileAllocator] allocating new datafile /var/lib/mongo/dmp_tmp_998.0, filling with zeroes...
2014-09-04T13:06:14.067+0200 [FileAllocator] done allocating datafile /var/lib/mongo/dmp_tmp_998.0, size: 64MB,  took 0 secs
2014-09-04T13:06:14.069+0200 [conn12917] build index on: dmp_tmp_998.tmp.mr.profilingTagCalls_1 properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "dmp_tmp_998.tmp.mr.profilingTagCalls_1" }
2014-09-04T13:06:14.069+0200 [conn12917]         added index to empty collection
2014-09-04T13:06:14.075+0200 [conn12917] SEVERE: Got signal: 6 (Aborted).
Backtrace:0x11e6111 0x11e54ee 0x7fabd9f466b0 0x7fabd9f46635 0x7fabd9f47e15 0x7fabd9f3f75e 0x7fabd9f3f820 0x9a263a 0xa2889a 0xa29ce2 0xa2bea6 0xd5dd6d 0xb9fe62 0xba1440 0x770aef 0x119bf3e 0x7fabda73b9d1 0x7fabd9ffc86d 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0x11e6111]
 /usr/bin/mongod() [0x11e54ee]
 /lib64/libc.so.6(+0x326b0) [0x7fabd9f466b0]
 /lib64/libc.so.6(gsignal+0x35) [0x7fabd9f46635]
 /lib64/libc.so.6(abort+0x175) [0x7fabd9f47e15]
 /lib64/libc.so.6(+0x2b75e) [0x7fabd9f3f75e]
 /lib64/libc.so.6(__assert_perror_fail+0) [0x7fabd9f3f820]
 /usr/bin/mongod(_ZN5mongo2mr22MapReduceFinishCommand3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x47fa) [0x9a263a]
 /usr/bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x3a) [0xa2889a]
 /usr/bin/mongod(_ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x1042) [0xa29ce2]
 /usr/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x6c6) [0xa2bea6]
 /usr/bin/mongod(_ZN5mongo11newRunQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0x22ed) [0xd5dd6d]
 /usr/bin/mongod() [0xb9fe62]
 /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x580) [0xba1440]
 /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x9f) [0x770aef]
 /usr/bin/mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x4ee) [0x119bf3e]
 /lib64/libpthread.so.0(+0x79d1) [0x7fabda73b9d1]
 /lib64/libc.so.6(clone+0x6d) [0x7fabd9ffc86d]

Generated at Thu Feb 08 03:37:03 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.