[SERVER-12040] MongoDB crashes Created: 11/Dec/13  Updated: 10/Dec/14  Resolved: 14/Jul/14

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: 2.4.5
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Machiel Groeneveld Assignee: Ramon Fernandez Marina
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: Linux
Participants:

 Description   

Log content just before crash:

Wed Dec 11 07:25:58.959 [initandlisten] connection accepted from 10.224.27.225:48388 #57 (52 connections now open)
Wed Dec 11 07:25:59.436 [repl writer worker 1]  production.hosts Assertion failure x == _nfields src/mongo/db/jsobj.cpp 1291
0xdd9e31 0xd9b04d 0xa052cd 0xa053a6 0xa9e5eb 0xa9e713 0xa8c151 0xa8fd77 0xa6fce9 0xc23c03 0xc23348 0xda7971 0xe225d9 0x7f2a6c11ef8e 0x7f2a6b422e1d 
 /opt/mongodb/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdd9e31]
 /opt/mongodb/bin/mongod(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) [0xd9b04d]
 /opt/mongodb/bin/mongod() [0xa052cd]
 /opt/mongodb/bin/mongod(_ZN5mongo21BSONObjIteratorSortedC1ERKNS_7BSONObjE+0x36) [0xa053a6]
 /opt/mongodb/bin/mongod(_ZN5mongo11ModSetState20createNewObjFromModsERKSsRNS_14BSONObjBuilderERKNS_7BSONObjE+0x3b) [0xa9e5eb]
 /opt/mongodb/bin/mongod(_ZN5mongo11ModSetState17createNewFromModsEv+0x63) [0xa9e713]
 /opt/mongodb/bin/mongod(_ZN5mongo14_updateObjectsEbPKcRKNS_7BSONObjES4_bbbRNS_7OpDebugEPNS_11RemoveSaverEbRKNS_24QueryPlanSelectionPolicyEb+0x1381) [0xa8c151]
 /opt/mongodb/bin/mongod(_ZN5mongo27updateObjectsForReplicationEPKcRKNS_7BSONObjES4_bbbRNS_7OpDebugEbRKNS_24QueryPlanSelectionPolicyE+0xb7) [0xa8fd77]
 /opt/mongodb/bin/mongod(_ZN5mongo21applyOperation_inlockERKNS_7BSONObjEbb+0xb39) [0xa6fce9]
 /opt/mongodb/bin/mongod(_ZN5mongo7replset8SyncTail9syncApplyERKNS_7BSONObjEb+0x713) [0xc23c03]
 /opt/mongodb/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x48) [0xc23348]
 /opt/mongodb/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) [0xda7971]
 /opt/mongodb/bin/mongod() [0xe225d9]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x7f8e) [0x7f2a6c11ef8e]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f2a6b422e1d]
Wed Dec 11 07:25:59.442 [repl writer worker 1] ERROR: writer worker caught exception: assertion src/mongo/db/jsobj.cpp:1291 on: { ts: Timestamp 1386746650000|907, h: 1748946517237623242, v: 2, op: "u", ns: "p
roduction.hosts", o2: { _id: ObjectId('52a7a38cd2e1f2adf10b837b') }, o: { $set: { affiliate_id: 270398, channel_id: 409, host: "nl.search.yahoo.com", shop_id: 962, time_id: 20131211, updated_at: new Dat
e(1386746650387), visits: 6 } } }
Wed Dec 11 07:25:59.442 [repl writer worker 1]   Fatal Assertion 16360
0xdd9e31 0xd99e63 0xc2342c 0xda7971 0xe225d9 0x7f2a6c11ef8e 0x7f2a6b422e1d 
 /opt/mongodb/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdd9e31]
 /opt/mongodb/bin/mongod(_ZN5mongo13fassertFailedEi+0xa3) [0xd99e63]
 /opt/mongodb/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x12c) [0xc2342c]
 /opt/mongodb/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) [0xda7971]
 /opt/mongodb/bin/mongod() [0xe225d9]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x7f8e) [0x7f2a6c11ef8e]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f2a6b422e1d]
Wed Dec 11 07:25:59.445 [repl writer worker 1] 
 
***aborting after fassert() failure
 
 
Wed Dec 11 07:25:59.445 Got signal: 6 (Aborted).
 
Wed Dec 11 07:25:59.446 [initandlisten] connection accepted from 10.226.125.64:60448 #58 (53 connections now open)
Wed Dec 11 07:25:59.448 Backtrace:
0xdd9e31 0x6d0d09 0x7f2a6b3600b0 0x7f2a6b360037 0x7f2a6b363698 0xd99e9e 0xc2342c 0xda7971 0xe225d9 0x7f2a6c11ef8e 0x7f2a6b422e1d 
 /opt/mongodb/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdd9e31]
 /opt/mongodb/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x6d0d09]
 /lib/x86_64-linux-gnu/libc.so.6(+0x370b0) [0x7f2a6b3600b0]
 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37) [0x7f2a6b360037]
 /lib/x86_64-linux-gnu/libc.so.6(abort+0x148) [0x7f2a6b363698]
 /opt/mongodb/bin/mongod(_ZN5mongo13fassertFailedEi+0xde) [0xd99e9e]
 /opt/mongodb/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x12c) [0xc2342c]
 /opt/mongodb/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) [0xda7971]
 /opt/mongodb/bin/mongod() [0xe225d9]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x7f8e) [0x7f2a6c11ef8e]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f2a6b422e1d]

Log when trying to restart mongodb:

***** SERVER RESTARTED *****
 
 
Wed Dec 11 07:33:32.241 [initandlisten] MongoDB starting : pid=9568 port=27017 dbpath=/mnt/mongodb/data 64-bit host=mongodb2-rs12
Wed Dec 11 07:33:32.241 [initandlisten] db version v2.4.5
Wed Dec 11 07:33:32.241 [initandlisten] git version: a2ddc68ba7c9cee17bfe69ed840383ec3506602b
Wed Dec 11 07:33:32.241 [initandlisten] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49
Wed Dec 11 07:33:32.241 [initandlisten] allocator: tcmalloc
Wed Dec 11 07:33:32.241 [initandlisten] options: { dbpath: "/mnt/mongodb/data", logappend: true, logpath: "/mnt/mongodb/log/mongodb.log", replSet: "rs1" }
Wed Dec 11 07:33:32.259 [initandlisten] journal dir=/mnt/mongodb/data/journal
Wed Dec 11 07:33:32.259 [initandlisten] recover begin
Wed Dec 11 07:33:32.260 [initandlisten] recover lsn: 0
Wed Dec 11 07:33:32.260 [initandlisten] recover /mnt/mongodb/data/journal/j._0
Wed Dec 11 07:33:32.311 [initandlisten] recover cleaning up
Wed Dec 11 07:33:32.311 [initandlisten] removeJournalFiles
Wed Dec 11 07:33:32.322 [initandlisten] recover done
Wed Dec 11 07:33:33.656 [initandlisten] waiting for connections on port 27017
Wed Dec 11 07:33:33.657 [websvr] admin web console waiting for connections on port 28017
Wed Dec 11 07:33:33.673 [rsStart] replSet I am mongodb2-rs12:27017
Wed Dec 11 07:33:33.673 [rsStart] replSet STARTUP2
Wed Dec 11 07:33:33.674 [rsMgr] replSet total number of votes is even - add arbiter or give one member an extra vote
Wed Dec 11 07:33:33.677 [rsHealthPoll] replset info mongodb2-rs14:27017 thinks that we are down
Wed Dec 11 07:33:33.677 [rsHealthPoll] replSet member mongodb2-rs14:27017 is up
Wed Dec 11 07:33:33.677 [rsHealthPoll] replSet member mongodb2-rs14:27017 is now in state PRIMARY
Wed Dec 11 07:33:33.679 [rsHealthPoll] replset info mongodb2-rs13:27017 thinks that we are down
Wed Dec 11 07:33:33.679 [rsHealthPoll] replSet member mongodb2-rs13:27017 is up
Wed Dec 11 07:33:33.679 [rsHealthPoll] replSet member mongodb2-rs13:27017 is now in state SECONDARY
Wed Dec 11 07:33:33.680 [rsHealthPoll] replset info mongodb2-rs11:27017 thinks that we are down
Wed Dec 11 07:33:33.680 [rsHealthPoll] replSet member mongodb2-rs11:27017 is up
Wed Dec 11 07:33:33.680 [rsHealthPoll] replSet member mongodb2-rs11:27017 is now in state SECONDARY
Wed Dec 11 07:33:33.943 [initandlisten] connection accepted from 10.34.202.147:57731 #1 (1 connection now open)
Wed Dec 11 07:33:33.945 [initandlisten] connection accepted from 10.34.202.147:57732 #2 (2 connections now open)
Wed Dec 11 07:33:33.969 [initandlisten] connection accepted from 10.209.152.105:45329 #3 (3 connections now open)
Wed Dec 11 07:33:33.972 [initandlisten] connection accepted from 10.209.152.105:45330 #4 (4 connections now open)
Wed Dec 11 07:33:34.165 [initandlisten] connection accepted from 10.209.152.105:45331 #5 (5 connections now open)
Wed Dec 11 07:33:34.170 [conn5] end connection 10.209.152.105:45331 (4 connections now open)
Wed Dec 11 07:33:34.171 [initandlisten] connection accepted from 10.209.152.105:45332 #6 (5 connections now open)
Wed Dec 11 07:33:34.446 [initandlisten] connection accepted from 10.229.115.64:49603 #7 (6 connections now open)
Wed Dec 11 07:33:34.449 [initandlisten] connection accepted from 10.229.115.64:49604 #8 (7 connections now open)
Wed Dec 11 07:33:34.674 [rsSync] replSet still syncing, not yet to minValid optime 52a8131b:43c
Wed Dec 11 07:33:34.915 [initandlisten] connection accepted from 10.34.202.147:57733 #9 (8 connections now open)
Wed Dec 11 07:33:34.916 [conn9] end connection 10.34.202.147:57733 (7 connections now open)
Wed Dec 11 07:33:34.917 [initandlisten] connection accepted from 10.34.202.147:57734 #10 (8 connections now open)
Wed Dec 11 07:33:34.951 [initandlisten] connection accepted from 10.80.4.82:39092 #11 (9 connections now open)
Wed Dec 11 07:33:34.971 [initandlisten] connection accepted from 10.226.125.64:33477 #12 (10 connections now open)
Wed Dec 11 07:33:34.971 [conn12] end connection 10.226.125.64:33477 (9 connections now open)
Wed Dec 11 07:33:34.972 [initandlisten] connection accepted from 10.226.125.64:33478 #13 (10 connections now open)
Wed Dec 11 07:33:35.096 [initandlisten] connection accepted from 10.33.160.109:57997 #14 (11 connections now open)
Wed Dec 11 07:33:35.428 [initandlisten] connection accepted from 10.51.61.53:54712 #15 (12 connections now open)
Wed Dec 11 07:33:35.618 [initandlisten] connection accepted from 10.34.194.223:52811 #16 (13 connections now open)
Wed Dec 11 07:33:35.667 [initandlisten] connection accepted from 10.224.14.210:54448 #17 (14 connections now open)
Wed Dec 11 07:33:36.269 [initandlisten] connection accepted from 10.227.123.31:59216 #18 (15 connections now open)
Wed Dec 11 07:33:36.583 [initandlisten] connection accepted from 10.244.235.88:38794 #19 (16 connections now open)
Wed Dec 11 07:33:36.763 [initandlisten] connection accepted from 10.54.247.195:42787 #20 (17 connections now open)
Wed Dec 11 07:33:37.107 [initandlisten] connection accepted from 10.54.247.195:42788 #21 (18 connections now open)
Wed Dec 11 07:33:37.349 [initandlisten] connection accepted from 10.36.219.211:38167 #22 (19 connections now open)
Wed Dec 11 07:33:37.420 [initandlisten] connection accepted from 10.37.149.95:57997 #23 (20 connections now open)
Wed Dec 11 07:33:37.850 [initandlisten] connection accepted from 10.48.181.52:57373 #24 (21 connections now open)
Wed Dec 11 07:33:37.851 [initandlisten] connection accepted from 10.64.161.38:51910 #25 (22 connections now open)
Wed Dec 11 07:33:38.241 [initandlisten] connection accepted from 10.54.247.195:42789 #26 (23 connections now open)
Wed Dec 11 07:33:38.244 [initandlisten] connection accepted from 10.226.125.64:33479 #27 (24 connections now open)
Wed Dec 11 07:33:38.246 [initandlisten] connection accepted from 10.64.134.113:57578 #28 (25 connections now open)
Wed Dec 11 07:33:38.252 [initandlisten] connection accepted from 10.34.202.147:57735 #29 (26 connections now open)
Wed Dec 11 07:33:38.255 [initandlisten] connection accepted from 10.36.219.211:38171 #30 (27 connections now open)
Wed Dec 11 07:33:38.257 [initandlisten] connection accepted from 10.32.2.209:39784 #31 (28 connections now open)
Wed Dec 11 07:33:38.257 [initandlisten] connection accepted from 10.33.160.109:57998 #32 (29 connections now open)
Wed Dec 11 07:33:38.340 [initandlisten] connection accepted from 10.209.152.105:45333 #33 (30 connections now open)
Wed Dec 11 07:33:38.383 [initandlisten] connection accepted from 10.51.61.53:54713 #34 (31 connections now open)
Wed Dec 11 07:33:38.383 [initandlisten] connection accepted from 10.32.16.211:49374 #35 (32 connections now open)
Wed Dec 11 07:33:38.385 [initandlisten] connection accepted from 10.48.194.134:44270 #36 (33 connections now open)
Wed Dec 11 07:33:38.386 [initandlisten] connection accepted from 10.64.161.38:51911 #37 (34 connections now open)
Wed Dec 11 07:33:38.388 [initandlisten] connection accepted from 10.229.115.64:49605 #38 (35 connections now open)
Wed Dec 11 07:33:38.390 [initandlisten] connection accepted from 10.226.211.225:34984 #39 (36 connections now open)
Wed Dec 11 07:33:38.395 [initandlisten] connection accepted from 10.224.14.210:54449 #40 (37 connections now open)
Wed Dec 11 07:33:38.396 [initandlisten] connection accepted from 10.32.16.211:49375 #41 (38 connections now open)
Wed Dec 11 07:33:38.397 [initandlisten] connection accepted from 10.224.27.225:48528 #42 (39 connections now open)
Wed Dec 11 07:33:38.398 [initandlisten] connection accepted from 10.244.235.88:38795 #43 (40 connections now open)
Wed Dec 11 07:33:38.404 [initandlisten] connection accepted from 10.48.181.52:57374 #44 (41 connections now open)
Wed Dec 11 07:33:38.433 [initandlisten] connection accepted from 10.244.191.53:34244 #45 (42 connections now open)
Wed Dec 11 07:33:38.502 [initandlisten] connection accepted from 10.227.123.31:59228 #46 (43 connections now open)
Wed Dec 11 07:33:38.674 [rsBackgroundSync] replSet syncing to: mongodb2-rs14:27017
Wed Dec 11 07:33:38.675 [rsSync] replSet still syncing, not yet to minValid optime 52a8131b:43c
Wed Dec 11 07:33:40.813 [initandlisten] connection accepted from 10.37.149.95:57999 #47 (44 connections now open)
Wed Dec 11 07:33:40.813 [initandlisten] connection accepted from 10.36.220.149:59541 #48 (45 connections now open)
Wed Dec 11 07:33:40.880 [initandlisten] connection accepted from 10.226.125.64:33480 #49 (46 connections now open)
Wed Dec 11 07:33:40.888 [initandlisten] connection accepted from 10.64.134.113:57579 #50 (47 connections now open)
Wed Dec 11 07:33:40.907 [initandlisten] connection accepted from 10.32.2.209:39785 #51 (48 connections now open)
Wed Dec 11 07:33:40.957 [initandlisten] connection accepted from 10.244.191.53:34245 #52 (49 connections now open)
Wed Dec 11 07:33:40.989 [initandlisten] connection accepted from 10.48.194.134:44271 #53 (50 connections now open)
Wed Dec 11 07:33:41.017 [initandlisten] connection accepted from 10.34.202.147:57736 #54 (51 connections now open)
Wed Dec 11 07:33:41.025 [initandlisten] connection accepted from 10.226.211.225:34985 #55 (52 connections now open)
Wed Dec 11 07:33:41.302 [initandlisten] connection accepted from 10.33.160.109:57999 #56 (53 connections now open)
Wed Dec 11 07:33:41.310 [initandlisten] connection accepted from 10.224.27.225:48529 #57 (54 connections now open)
Wed Dec 11 07:33:41.697 [conn10] command admin.$cmd command: { replSetHeartbeat: "rs1", v: 4, pv: 1, checkEmpty: false, from: "mongodb2-rs14:27017" } ntoreturn:1 keyUpdates:0  reslen:224 2777ms
Wed Dec 11 07:33:41.697 [conn13] command admin.$cmd command: { replSetHeartbeat: "rs1", v: 4, pv: 1, checkEmpty: false, from: "mongodb2-rs11:27017" } ntoreturn:1 keyUpdates:0  reslen:224 2722ms
Wed Dec 11 07:33:41.697 [rsSyncNotifier] replset setting oplog notifier to mongodb2-rs14:27017
Wed Dec 11 07:33:41.700 [conn6] command admin.$cmd command: { replSetHeartbeat: "rs1", v: 4, pv: 1, checkEmpty: false, from: "mongodb2-rs13:27017" } ntoreturn:1 keyUpdates:0  reslen:224 1524ms
Wed Dec 11 07:33:41.700 [conn47] command admin.$cmd command: { ismaster: 1 } ntoreturn:1 keyUpdates:0  reslen:421 886ms
Wed Dec 11 07:33:41.700 [conn48] command admin.$cmd command: { ismaster: 1 } ntoreturn:1 keyUpdates:0  reslen:421 886ms
Wed Dec 11 07:33:41.700 [conn49] command admin.$cmd command: { ismaster: 1 } ntoreturn:1 keyUpdates:0  reslen:421 820ms
Wed Dec 11 07:33:41.701 [conn50] command admin.$cmd command: { ismaster: 1 } ntoreturn:1 keyUpdates:0  reslen:421 812ms
Wed Dec 11 07:33:41.701 [conn51] command admin.$cmd command: { ismaster: 1 } ntoreturn:1 keyUpdates:0  reslen:421 793ms
Wed Dec 11 07:33:41.701 [conn52] command admin.$cmd command: { ismaster: 1 } ntoreturn:1 keyUpdates:0  reslen:421 743ms
Wed Dec 11 07:33:41.701 [conn53] command admin.$cmd command: { ismaster: 1 } ntoreturn:1 keyUpdates:0  reslen:421 711ms
Wed Dec 11 07:33:41.701 [conn54] command admin.$cmd command: { ismaster: 1 } ntoreturn:1 keyUpdates:0  reslen:421 684ms
Wed Dec 11 07:33:41.701 [conn55] command admin.$cmd command: { ismaster: 1 } ntoreturn:1 keyUpdates:0  reslen:421 676ms
Wed Dec 11 07:33:41.701 [conn56] command admin.$cmd command: { ismaster: 1 } ntoreturn:1 keyUpdates:0  reslen:421 398ms
Wed Dec 11 07:33:41.701 [conn57] command admin.$cmd command: { ismaster: 1 } ntoreturn:1 keyUpdates:0  reslen:421 390ms
Wed Dec 11 07:33:41.704 [initandlisten] connection accepted from 10.226.125.64:33481 #58 (55 connections now open)
Wed Dec 11 07:33:41.712 [initandlisten] connection accepted from 10.33.160.109:58000 #59 (56 connections now open)
Wed Dec 11 07:33:41.775 [rsSync] replSet SECONDARY
Wed Dec 11 07:33:42.662 [repl writer worker 1]  production.hosts Assertion failure x == _nfields src/mongo/db/jsobj.cpp 1291
0xdd9e31 0xd9b04d 0xa052cd 0xa053a6 0xa9e5eb 0xa9e713 0xa8c151 0xa8fd77 0xa6fce9 0xc23c03 0xc23348 0xda7971 0xe225d9 0x7fd3441a7f8e 0x7fd3434abe1d 
 /opt/mongodb/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdd9e31]
 /opt/mongodb/bin/mongod(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) [0xd9b04d]
 /opt/mongodb/bin/mongod() [0xa052cd]
 /opt/mongodb/bin/mongod(_ZN5mongo21BSONObjIteratorSortedC1ERKNS_7BSONObjE+0x36) [0xa053a6]
 /opt/mongodb/bin/mongod(_ZN5mongo11ModSetState20createNewObjFromModsERKSsRNS_14BSONObjBuilderERKNS_7BSONObjE+0x3b) [0xa9e5eb]
 /opt/mongodb/bin/mongod(_ZN5mongo11ModSetState17createNewFromModsEv+0x63) [0xa9e713]
 /opt/mongodb/bin/mongod(_ZN5mongo14_updateObjectsEbPKcRKNS_7BSONObjES4_bbbRNS_7OpDebugEPNS_11RemoveSaverEbRKNS_24QueryPlanSelectionPolicyEb+0x1381) [0xa8c151]
 /opt/mongodb/bin/mongod(_ZN5mongo27updateObjectsForReplicationEPKcRKNS_7BSONObjES4_bbbRNS_7OpDebugEbRKNS_24QueryPlanSelectionPolicyE+0xb7) [0xa8fd77]
 /opt/mongodb/bin/mongod(_ZN5mongo21applyOperation_inlockERKNS_7BSONObjEbb+0xb39) [0xa6fce9]
 /opt/mongodb/bin/mongod(_ZN5mongo7replset8SyncTail9syncApplyERKNS_7BSONObjEb+0x713) [0xc23c03]
 /opt/mongodb/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x48) [0xc23348]
 /opt/mongodb/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) [0xda7971]
 /opt/mongodb/bin/mongod() [0xe225d9]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x7f8e) [0x7fd3441a7f8e]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fd3434abe1d]
Wed Dec 11 07:33:42.680 [repl writer worker 1] ERROR: writer worker caught exception: assertion src/mongo/db/jsobj.cpp:1291 on: { ts: Timestamp 1386746650000|907, h: 1748946517237623242, v: 2, op: "u", ns: "production.hosts", o2: { _id: ObjectId('52a7a38cd2e1f2adf10b837b') }, o: { $set: { affiliate_id: 270398, channel_id: 409, host: "nl.search.yahoo.com", shop_id: 962, time_id: 20131211, updated_at: new Date(1386746650387), visits: 6 } } }
Wed Dec 11 07:33:42.680 [repl writer worker 1]   Fatal Assertion 16360
0xdd9e31 0xd99e63 0xc2342c 0xda7971 0xe225d9 0x7fd3441a7f8e 0x7fd3434abe1d 
 /opt/mongodb/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdd9e31]
 /opt/mongodb/bin/mongod(_ZN5mongo13fassertFailedEi+0xa3) [0xd99e63]
 /opt/mongodb/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x12c) [0xc2342c]
 /opt/mongodb/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) [0xda7971]
 /opt/mongodb/bin/mongod() [0xe225d9]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x7f8e) [0x7fd3441a7f8e]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fd3434abe1d]
Wed Dec 11 07:33:42.689 [repl writer worker 1] 
 
***aborting after fassert() failure
 
 
Wed Dec 11 07:33:42.689 Got signal: 6 (Aborted).
 
Wed Dec 11 07:33:42.698 Backtrace:
0xdd9e31 0x6d0d09 0x7fd3433e90b0 0x7fd3433e9037 0x7fd3433ec698 0xd99e9e 0xc2342c 0xda7971 0xe225d9 0x7fd3441a7f8e 0x7fd3434abe1d 
 /opt/mongodb/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdd9e31]
 /opt/mongodb/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x6d0d09]
 /lib/x86_64-linux-gnu/libc.so.6(+0x370b0) [0x7fd3433e90b0]
 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37) [0x7fd3433e9037]
 /lib/x86_64-linux-gnu/libc.so.6(abort+0x148) [0x7fd3433ec698]
 /opt/mongodb/bin/mongod(_ZN5mongo13fassertFailedEi+0xde) [0xd99e9e]
 /opt/mongodb/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x12c) [0xc2342c]
 /opt/mongodb/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) [0xda7971]
 /opt/mongodb/bin/mongod() [0xe225d9]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x7f8e) [0x7fd3441a7f8e]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fd3434abe1d]



 Comments   
Comment by Ramon Fernandez Marina [ 14/Jul/14 ]

machielg, we haven't heard back from you for a while, so we're going to mark this issue as resolved. If this is still an issue for you feel free to re-open the ticket and provide additional information.


Regards,
Ramón.

Comment by Ramon Fernandez Marina [ 03/Jul/14 ]

machielg, are you still having trouble with your replicaset? If you are, have you had a chance to resync the faulty secondary from the primary? Did the resync fix the problem?

Comment by Ramon Fernandez Marina [ 17/Jun/14 ]

Hi machielg, apologies for the late reply. The logs you sent seem to indicate that a secondary is shutting itself down because it is finding data corruption when trying to replicate from the primary. Upon restart voting occurs, the node becomes a secondary, and when it tries to resume replication it finds data corruption and shuts down again. It's hard to pinpoint the source of this corruption, but bad disks/networks are often to blame. At this stage I think the best way forward is to resync the faulty your secondary from the primary to get a healthy replicaset. Can you please investigate this approach and report back?

Two other things: the logs seem to indicate that you do not have an odd number of voting members in the replica set; please consider adding an arbiter. Also, there have been a number of improvements since 2.4.5, so you may want to consider upgrading to a newer version of MongoDB (2.4.10 is the latest version in the 2.4 series).

Generated at Thu Feb 08 03:27:26 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.