***** SERVER RESTARTED *****
|
|
|
Wed Dec 11 07:33:32.241 [initandlisten] MongoDB starting : pid=9568 port=27017 dbpath=/mnt/mongodb/data 64-bit host=mongodb2-rs12
|
Wed Dec 11 07:33:32.241 [initandlisten] db version v2.4.5
|
Wed Dec 11 07:33:32.241 [initandlisten] git version: a2ddc68ba7c9cee17bfe69ed840383ec3506602b
|
Wed Dec 11 07:33:32.241 [initandlisten] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49
|
Wed Dec 11 07:33:32.241 [initandlisten] allocator: tcmalloc
|
Wed Dec 11 07:33:32.241 [initandlisten] options: { dbpath: "/mnt/mongodb/data", logappend: true, logpath: "/mnt/mongodb/log/mongodb.log", replSet: "rs1" }
|
Wed Dec 11 07:33:32.259 [initandlisten] journal dir=/mnt/mongodb/data/journal
|
Wed Dec 11 07:33:32.259 [initandlisten] recover begin
|
Wed Dec 11 07:33:32.260 [initandlisten] recover lsn: 0
|
Wed Dec 11 07:33:32.260 [initandlisten] recover /mnt/mongodb/data/journal/j._0
|
Wed Dec 11 07:33:32.311 [initandlisten] recover cleaning up
|
Wed Dec 11 07:33:32.311 [initandlisten] removeJournalFiles
|
Wed Dec 11 07:33:32.322 [initandlisten] recover done
|
Wed Dec 11 07:33:33.656 [initandlisten] waiting for connections on port 27017
|
Wed Dec 11 07:33:33.657 [websvr] admin web console waiting for connections on port 28017
|
Wed Dec 11 07:33:33.673 [rsStart] replSet I am mongodb2-rs12:27017
|
Wed Dec 11 07:33:33.673 [rsStart] replSet STARTUP2
|
Wed Dec 11 07:33:33.674 [rsMgr] replSet total number of votes is even - add arbiter or give one member an extra vote
|
Wed Dec 11 07:33:33.677 [rsHealthPoll] replset info mongodb2-rs14:27017 thinks that we are down
|
Wed Dec 11 07:33:33.677 [rsHealthPoll] replSet member mongodb2-rs14:27017 is up
|
Wed Dec 11 07:33:33.677 [rsHealthPoll] replSet member mongodb2-rs14:27017 is now in state PRIMARY
|
Wed Dec 11 07:33:33.679 [rsHealthPoll] replset info mongodb2-rs13:27017 thinks that we are down
|
Wed Dec 11 07:33:33.679 [rsHealthPoll] replSet member mongodb2-rs13:27017 is up
|
Wed Dec 11 07:33:33.679 [rsHealthPoll] replSet member mongodb2-rs13:27017 is now in state SECONDARY
|
Wed Dec 11 07:33:33.680 [rsHealthPoll] replset info mongodb2-rs11:27017 thinks that we are down
|
Wed Dec 11 07:33:33.680 [rsHealthPoll] replSet member mongodb2-rs11:27017 is up
|
Wed Dec 11 07:33:33.680 [rsHealthPoll] replSet member mongodb2-rs11:27017 is now in state SECONDARY
|
Wed Dec 11 07:33:33.943 [initandlisten] connection accepted from 10.34.202.147:57731 #1 (1 connection now open)
|
Wed Dec 11 07:33:33.945 [initandlisten] connection accepted from 10.34.202.147:57732 #2 (2 connections now open)
|
Wed Dec 11 07:33:33.969 [initandlisten] connection accepted from 10.209.152.105:45329 #3 (3 connections now open)
|
Wed Dec 11 07:33:33.972 [initandlisten] connection accepted from 10.209.152.105:45330 #4 (4 connections now open)
|
Wed Dec 11 07:33:34.165 [initandlisten] connection accepted from 10.209.152.105:45331 #5 (5 connections now open)
|
Wed Dec 11 07:33:34.170 [conn5] end connection 10.209.152.105:45331 (4 connections now open)
|
Wed Dec 11 07:33:34.171 [initandlisten] connection accepted from 10.209.152.105:45332 #6 (5 connections now open)
|
Wed Dec 11 07:33:34.446 [initandlisten] connection accepted from 10.229.115.64:49603 #7 (6 connections now open)
|
Wed Dec 11 07:33:34.449 [initandlisten] connection accepted from 10.229.115.64:49604 #8 (7 connections now open)
|
Wed Dec 11 07:33:34.674 [rsSync] replSet still syncing, not yet to minValid optime 52a8131b:43c
|
Wed Dec 11 07:33:34.915 [initandlisten] connection accepted from 10.34.202.147:57733 #9 (8 connections now open)
|
Wed Dec 11 07:33:34.916 [conn9] end connection 10.34.202.147:57733 (7 connections now open)
|
Wed Dec 11 07:33:34.917 [initandlisten] connection accepted from 10.34.202.147:57734 #10 (8 connections now open)
|
Wed Dec 11 07:33:34.951 [initandlisten] connection accepted from 10.80.4.82:39092 #11 (9 connections now open)
|
Wed Dec 11 07:33:34.971 [initandlisten] connection accepted from 10.226.125.64:33477 #12 (10 connections now open)
|
Wed Dec 11 07:33:34.971 [conn12] end connection 10.226.125.64:33477 (9 connections now open)
|
Wed Dec 11 07:33:34.972 [initandlisten] connection accepted from 10.226.125.64:33478 #13 (10 connections now open)
|
Wed Dec 11 07:33:35.096 [initandlisten] connection accepted from 10.33.160.109:57997 #14 (11 connections now open)
|
Wed Dec 11 07:33:35.428 [initandlisten] connection accepted from 10.51.61.53:54712 #15 (12 connections now open)
|
Wed Dec 11 07:33:35.618 [initandlisten] connection accepted from 10.34.194.223:52811 #16 (13 connections now open)
|
Wed Dec 11 07:33:35.667 [initandlisten] connection accepted from 10.224.14.210:54448 #17 (14 connections now open)
|
Wed Dec 11 07:33:36.269 [initandlisten] connection accepted from 10.227.123.31:59216 #18 (15 connections now open)
|
Wed Dec 11 07:33:36.583 [initandlisten] connection accepted from 10.244.235.88:38794 #19 (16 connections now open)
|
Wed Dec 11 07:33:36.763 [initandlisten] connection accepted from 10.54.247.195:42787 #20 (17 connections now open)
|
Wed Dec 11 07:33:37.107 [initandlisten] connection accepted from 10.54.247.195:42788 #21 (18 connections now open)
|
Wed Dec 11 07:33:37.349 [initandlisten] connection accepted from 10.36.219.211:38167 #22 (19 connections now open)
|
Wed Dec 11 07:33:37.420 [initandlisten] connection accepted from 10.37.149.95:57997 #23 (20 connections now open)
|
Wed Dec 11 07:33:37.850 [initandlisten] connection accepted from 10.48.181.52:57373 #24 (21 connections now open)
|
Wed Dec 11 07:33:37.851 [initandlisten] connection accepted from 10.64.161.38:51910 #25 (22 connections now open)
|
Wed Dec 11 07:33:38.241 [initandlisten] connection accepted from 10.54.247.195:42789 #26 (23 connections now open)
|
Wed Dec 11 07:33:38.244 [initandlisten] connection accepted from 10.226.125.64:33479 #27 (24 connections now open)
|
Wed Dec 11 07:33:38.246 [initandlisten] connection accepted from 10.64.134.113:57578 #28 (25 connections now open)
|
Wed Dec 11 07:33:38.252 [initandlisten] connection accepted from 10.34.202.147:57735 #29 (26 connections now open)
|
Wed Dec 11 07:33:38.255 [initandlisten] connection accepted from 10.36.219.211:38171 #30 (27 connections now open)
|
Wed Dec 11 07:33:38.257 [initandlisten] connection accepted from 10.32.2.209:39784 #31 (28 connections now open)
|
Wed Dec 11 07:33:38.257 [initandlisten] connection accepted from 10.33.160.109:57998 #32 (29 connections now open)
|
Wed Dec 11 07:33:38.340 [initandlisten] connection accepted from 10.209.152.105:45333 #33 (30 connections now open)
|
Wed Dec 11 07:33:38.383 [initandlisten] connection accepted from 10.51.61.53:54713 #34 (31 connections now open)
|
Wed Dec 11 07:33:38.383 [initandlisten] connection accepted from 10.32.16.211:49374 #35 (32 connections now open)
|
Wed Dec 11 07:33:38.385 [initandlisten] connection accepted from 10.48.194.134:44270 #36 (33 connections now open)
|
Wed Dec 11 07:33:38.386 [initandlisten] connection accepted from 10.64.161.38:51911 #37 (34 connections now open)
|
Wed Dec 11 07:33:38.388 [initandlisten] connection accepted from 10.229.115.64:49605 #38 (35 connections now open)
|
Wed Dec 11 07:33:38.390 [initandlisten] connection accepted from 10.226.211.225:34984 #39 (36 connections now open)
|
Wed Dec 11 07:33:38.395 [initandlisten] connection accepted from 10.224.14.210:54449 #40 (37 connections now open)
|
Wed Dec 11 07:33:38.396 [initandlisten] connection accepted from 10.32.16.211:49375 #41 (38 connections now open)
|
Wed Dec 11 07:33:38.397 [initandlisten] connection accepted from 10.224.27.225:48528 #42 (39 connections now open)
|
Wed Dec 11 07:33:38.398 [initandlisten] connection accepted from 10.244.235.88:38795 #43 (40 connections now open)
|
Wed Dec 11 07:33:38.404 [initandlisten] connection accepted from 10.48.181.52:57374 #44 (41 connections now open)
|
Wed Dec 11 07:33:38.433 [initandlisten] connection accepted from 10.244.191.53:34244 #45 (42 connections now open)
|
Wed Dec 11 07:33:38.502 [initandlisten] connection accepted from 10.227.123.31:59228 #46 (43 connections now open)
|
Wed Dec 11 07:33:38.674 [rsBackgroundSync] replSet syncing to: mongodb2-rs14:27017
|
Wed Dec 11 07:33:38.675 [rsSync] replSet still syncing, not yet to minValid optime 52a8131b:43c
|
Wed Dec 11 07:33:40.813 [initandlisten] connection accepted from 10.37.149.95:57999 #47 (44 connections now open)
|
Wed Dec 11 07:33:40.813 [initandlisten] connection accepted from 10.36.220.149:59541 #48 (45 connections now open)
|
Wed Dec 11 07:33:40.880 [initandlisten] connection accepted from 10.226.125.64:33480 #49 (46 connections now open)
|
Wed Dec 11 07:33:40.888 [initandlisten] connection accepted from 10.64.134.113:57579 #50 (47 connections now open)
|
Wed Dec 11 07:33:40.907 [initandlisten] connection accepted from 10.32.2.209:39785 #51 (48 connections now open)
|
Wed Dec 11 07:33:40.957 [initandlisten] connection accepted from 10.244.191.53:34245 #52 (49 connections now open)
|
Wed Dec 11 07:33:40.989 [initandlisten] connection accepted from 10.48.194.134:44271 #53 (50 connections now open)
|
Wed Dec 11 07:33:41.017 [initandlisten] connection accepted from 10.34.202.147:57736 #54 (51 connections now open)
|
Wed Dec 11 07:33:41.025 [initandlisten] connection accepted from 10.226.211.225:34985 #55 (52 connections now open)
|
Wed Dec 11 07:33:41.302 [initandlisten] connection accepted from 10.33.160.109:57999 #56 (53 connections now open)
|
Wed Dec 11 07:33:41.310 [initandlisten] connection accepted from 10.224.27.225:48529 #57 (54 connections now open)
|
Wed Dec 11 07:33:41.697 [conn10] command admin.$cmd command: { replSetHeartbeat: "rs1", v: 4, pv: 1, checkEmpty: false, from: "mongodb2-rs14:27017" } ntoreturn:1 keyUpdates:0 reslen:224 2777ms
|
Wed Dec 11 07:33:41.697 [conn13] command admin.$cmd command: { replSetHeartbeat: "rs1", v: 4, pv: 1, checkEmpty: false, from: "mongodb2-rs11:27017" } ntoreturn:1 keyUpdates:0 reslen:224 2722ms
|
Wed Dec 11 07:33:41.697 [rsSyncNotifier] replset setting oplog notifier to mongodb2-rs14:27017
|
Wed Dec 11 07:33:41.700 [conn6] command admin.$cmd command: { replSetHeartbeat: "rs1", v: 4, pv: 1, checkEmpty: false, from: "mongodb2-rs13:27017" } ntoreturn:1 keyUpdates:0 reslen:224 1524ms
|
Wed Dec 11 07:33:41.700 [conn47] command admin.$cmd command: { ismaster: 1 } ntoreturn:1 keyUpdates:0 reslen:421 886ms
|
Wed Dec 11 07:33:41.700 [conn48] command admin.$cmd command: { ismaster: 1 } ntoreturn:1 keyUpdates:0 reslen:421 886ms
|
Wed Dec 11 07:33:41.700 [conn49] command admin.$cmd command: { ismaster: 1 } ntoreturn:1 keyUpdates:0 reslen:421 820ms
|
Wed Dec 11 07:33:41.701 [conn50] command admin.$cmd command: { ismaster: 1 } ntoreturn:1 keyUpdates:0 reslen:421 812ms
|
Wed Dec 11 07:33:41.701 [conn51] command admin.$cmd command: { ismaster: 1 } ntoreturn:1 keyUpdates:0 reslen:421 793ms
|
Wed Dec 11 07:33:41.701 [conn52] command admin.$cmd command: { ismaster: 1 } ntoreturn:1 keyUpdates:0 reslen:421 743ms
|
Wed Dec 11 07:33:41.701 [conn53] command admin.$cmd command: { ismaster: 1 } ntoreturn:1 keyUpdates:0 reslen:421 711ms
|
Wed Dec 11 07:33:41.701 [conn54] command admin.$cmd command: { ismaster: 1 } ntoreturn:1 keyUpdates:0 reslen:421 684ms
|
Wed Dec 11 07:33:41.701 [conn55] command admin.$cmd command: { ismaster: 1 } ntoreturn:1 keyUpdates:0 reslen:421 676ms
|
Wed Dec 11 07:33:41.701 [conn56] command admin.$cmd command: { ismaster: 1 } ntoreturn:1 keyUpdates:0 reslen:421 398ms
|
Wed Dec 11 07:33:41.701 [conn57] command admin.$cmd command: { ismaster: 1 } ntoreturn:1 keyUpdates:0 reslen:421 390ms
|
Wed Dec 11 07:33:41.704 [initandlisten] connection accepted from 10.226.125.64:33481 #58 (55 connections now open)
|
Wed Dec 11 07:33:41.712 [initandlisten] connection accepted from 10.33.160.109:58000 #59 (56 connections now open)
|
Wed Dec 11 07:33:41.775 [rsSync] replSet SECONDARY
|
Wed Dec 11 07:33:42.662 [repl writer worker 1] production.hosts Assertion failure x == _nfields src/mongo/db/jsobj.cpp 1291
|
0xdd9e31 0xd9b04d 0xa052cd 0xa053a6 0xa9e5eb 0xa9e713 0xa8c151 0xa8fd77 0xa6fce9 0xc23c03 0xc23348 0xda7971 0xe225d9 0x7fd3441a7f8e 0x7fd3434abe1d
|
/opt/mongodb/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdd9e31]
|
/opt/mongodb/bin/mongod(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) [0xd9b04d]
|
/opt/mongodb/bin/mongod() [0xa052cd]
|
/opt/mongodb/bin/mongod(_ZN5mongo21BSONObjIteratorSortedC1ERKNS_7BSONObjE+0x36) [0xa053a6]
|
/opt/mongodb/bin/mongod(_ZN5mongo11ModSetState20createNewObjFromModsERKSsRNS_14BSONObjBuilderERKNS_7BSONObjE+0x3b) [0xa9e5eb]
|
/opt/mongodb/bin/mongod(_ZN5mongo11ModSetState17createNewFromModsEv+0x63) [0xa9e713]
|
/opt/mongodb/bin/mongod(_ZN5mongo14_updateObjectsEbPKcRKNS_7BSONObjES4_bbbRNS_7OpDebugEPNS_11RemoveSaverEbRKNS_24QueryPlanSelectionPolicyEb+0x1381) [0xa8c151]
|
/opt/mongodb/bin/mongod(_ZN5mongo27updateObjectsForReplicationEPKcRKNS_7BSONObjES4_bbbRNS_7OpDebugEbRKNS_24QueryPlanSelectionPolicyE+0xb7) [0xa8fd77]
|
/opt/mongodb/bin/mongod(_ZN5mongo21applyOperation_inlockERKNS_7BSONObjEbb+0xb39) [0xa6fce9]
|
/opt/mongodb/bin/mongod(_ZN5mongo7replset8SyncTail9syncApplyERKNS_7BSONObjEb+0x713) [0xc23c03]
|
/opt/mongodb/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x48) [0xc23348]
|
/opt/mongodb/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) [0xda7971]
|
/opt/mongodb/bin/mongod() [0xe225d9]
|
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7f8e) [0x7fd3441a7f8e]
|
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fd3434abe1d]
|
Wed Dec 11 07:33:42.680 [repl writer worker 1] ERROR: writer worker caught exception: assertion src/mongo/db/jsobj.cpp:1291 on: { ts: Timestamp 1386746650000|907, h: 1748946517237623242, v: 2, op: "u", ns: "production.hosts", o2: { _id: ObjectId('52a7a38cd2e1f2adf10b837b') }, o: { $set: { affiliate_id: 270398, channel_id: 409, host: "nl.search.yahoo.com", shop_id: 962, time_id: 20131211, updated_at: new Date(1386746650387), visits: 6 } } }
|
Wed Dec 11 07:33:42.680 [repl writer worker 1] Fatal Assertion 16360
|
0xdd9e31 0xd99e63 0xc2342c 0xda7971 0xe225d9 0x7fd3441a7f8e 0x7fd3434abe1d
|
/opt/mongodb/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdd9e31]
|
/opt/mongodb/bin/mongod(_ZN5mongo13fassertFailedEi+0xa3) [0xd99e63]
|
/opt/mongodb/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x12c) [0xc2342c]
|
/opt/mongodb/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) [0xda7971]
|
/opt/mongodb/bin/mongod() [0xe225d9]
|
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7f8e) [0x7fd3441a7f8e]
|
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fd3434abe1d]
|
Wed Dec 11 07:33:42.689 [repl writer worker 1]
|
|
***aborting after fassert() failure
|
|
|
Wed Dec 11 07:33:42.689 Got signal: 6 (Aborted).
|
|
Wed Dec 11 07:33:42.698 Backtrace:
|
0xdd9e31 0x6d0d09 0x7fd3433e90b0 0x7fd3433e9037 0x7fd3433ec698 0xd99e9e 0xc2342c 0xda7971 0xe225d9 0x7fd3441a7f8e 0x7fd3434abe1d
|
/opt/mongodb/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdd9e31]
|
/opt/mongodb/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x6d0d09]
|
/lib/x86_64-linux-gnu/libc.so.6(+0x370b0) [0x7fd3433e90b0]
|
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37) [0x7fd3433e9037]
|
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148) [0x7fd3433ec698]
|
/opt/mongodb/bin/mongod(_ZN5mongo13fassertFailedEi+0xde) [0xd99e9e]
|
/opt/mongodb/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x12c) [0xc2342c]
|
/opt/mongodb/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) [0xda7971]
|
/opt/mongodb/bin/mongod() [0xe225d9]
|
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7f8e) [0x7fd3441a7f8e]
|
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fd3434abe1d]
|