m31101| note: noprealloc may hurt performance in many applications m31101| Tue Nov 27 17:44:17.754 [initandlisten] MongoDB starting : pid=11355 port=31101 dbpath=/data/db/test-rs0-1 64-bit host=ubuntu m31101| Tue Nov 27 17:44:17.754 [initandlisten] m31101| Tue Nov 27 17:44:17.754 [initandlisten] ** NOTE: This is a development version (2.3.2-pre-) of MongoDB. m31101| Tue Nov 27 17:44:17.754 [initandlisten] ** Not recommended for production. m31101| Tue Nov 27 17:44:17.755 [initandlisten] m31101| Tue Nov 27 17:44:17.755 [initandlisten] db version v2.3.2-pre-, pdfile version 4.5 m31101| Tue Nov 27 17:44:17.755 [initandlisten] git version: d8a2d351932c3247e7e0082b4b91c23614718de5 m31101| Tue Nov 27 17:44:17.755 [initandlisten] build info: Linux ubuntu 3.2.0-29-generic #46-Ubuntu SMP Fri Jul 27 17:03:23 UTC 2012 x86_64 BOOST_LIB_VERSION=1_49 m31101| Tue Nov 27 17:44:17.755 [initandlisten] options: { dbpath: "/data/db/test-rs0-1", noprealloc: true, nopreallocj: true, oplogSize: 40, port: 31101, replSet: "test-rs0", rest: true, smallfiles: true } m31101| Tue Nov 27 17:44:17.785 [initandlisten] journal dir=/data/db/test-rs0-1/journal m31101| Tue Nov 27 17:44:17.785 [initandlisten] recover : no journal files present, no recovery needed m31101| Tue Nov 27 17:44:17.829 [websvr] admin web console waiting for connections on port 32101 m31101| Tue Nov 27 17:44:17.829 [initandlisten] waiting for connections on port 31101 m31101| Tue Nov 27 17:44:17.831 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31101| Tue Nov 27 17:44:17.831 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done m31101| Tue Nov 27 17:44:17.932 [initandlisten] connection accepted from 127.0.0.1:34238 #1 (1 connection now open) m31101| Tue Nov 27 17:44:18.140 [initandlisten] connection accepted from 127.0.0.1:34241 #2 (2 connections now open) m31101| Tue Nov 27 17:44:27.831 [rsStart] trying to contact ubuntu:31100 m31101| Tue Nov 27 17:44:27.833 [rsStart] replSet I am ubuntu:31101 m31101| Tue Nov 27 17:44:27.833 [rsStart] replSet got config version 1 from a remote, saving locally m31101| Tue Nov 27 17:44:27.834 [rsStart] replSet info saving a newer config version to local.system.replset m31101| Tue Nov 27 17:44:27.834 [FileAllocator] allocating new datafile /data/db/test-rs0-1/local.ns, filling with zeroes... m31101| Tue Nov 27 17:44:27.834 [FileAllocator] creating directory /data/db/test-rs0-1/_tmp m31101| Tue Nov 27 17:44:28.228 [FileAllocator] done allocating datafile /data/db/test-rs0-1/local.ns, size: 16MB, took 0.283 secs m31101| Tue Nov 27 17:44:28.229 [FileAllocator] allocating new datafile /data/db/test-rs0-1/local.0, filling with zeroes... m31101| Tue Nov 27 17:44:28.612 [FileAllocator] done allocating datafile /data/db/test-rs0-1/local.0, size: 16MB, took 0.383 secs m31101| Tue Nov 27 17:44:28.859 [rsStart] replSet saveConfigLocally done m31101| Tue Nov 27 17:44:28.860 [rsStart] replSet STARTUP2 m31101| Tue Nov 27 17:44:28.860 [rsSync] ****** m31101| Tue Nov 27 17:44:28.860 [rsSync] creating replication oplog of size: 40MB... m31101| Tue Nov 27 17:44:28.913 [FileAllocator] allocating new datafile /data/db/test-rs0-1/local.1, filling with zeroes... m31101| Tue Nov 27 17:44:29.834 [rsHealthPoll] replSet member ubuntu:31100 is up m31101| Tue Nov 27 17:44:29.834 [rsHealthPoll] replSet member ubuntu:31100 is now in state SECONDARY m31101| Tue Nov 27 17:44:29.834 [rsHealthPoll] replset info ubuntu:31102 thinks that we are down m31101| Tue Nov 27 17:44:29.834 [rsHealthPoll] replSet member ubuntu:31102 is up m31101| Tue Nov 27 17:44:29.835 [rsHealthPoll] replSet member ubuntu:31102 is now in state STARTUP2 m31101| Tue Nov 27 17:44:30.043 [initandlisten] connection accepted from 127.0.0.1:34255 #3 (3 connections now open) m31101| Tue Nov 27 17:44:30.538 [FileAllocator] done allocating datafile /data/db/test-rs0-1/local.1, size: 64MB, took 1.624 secs m31101| Tue Nov 27 17:44:30.613 [rsSync] ****** m31101| Tue Nov 27 17:44:30.613 [rsSync] replSet initial sync pending m31101| Tue Nov 27 17:44:30.613 [rsSync] replSet initial sync need a member to be primary or secondary to do our initial sync m31101| Tue Nov 27 17:44:37.642 [conn2] replSet RECOVERING m31101| Tue Nov 27 17:44:37.642 [conn2] replSet info voting yea for ubuntu:31100 (0) m31101| Tue Nov 27 17:44:37.836 [rsHealthPoll] replSet member ubuntu:31100 is now in state PRIMARY m31101| Tue Nov 27 17:44:37.836 [rsHealthPoll] replSet member ubuntu:31102 is now in state RECOVERING m31101| Tue Nov 27 17:44:41.642 [conn2] end connection 127.0.0.1:34241 (2 connections now open) m31101| Tue Nov 27 17:44:41.643 [initandlisten] connection accepted from 127.0.0.1:34262 #4 (3 connections now open) m31101| Tue Nov 27 17:44:46.613 [rsSync] replSet initial sync pending m31101| Tue Nov 27 17:44:46.613 [rsSync] replSet syncing to: ubuntu:31100 m31101| Tue Nov 27 17:44:46.615 [rsSync] build index local.me { _id: 1 } m31101| Tue Nov 27 17:44:46.616 [rsSync] build index done. scanned 0 total records. 0 secs m31101| Tue Nov 27 17:44:46.616 [rsSync] replSet initial sync drop all databases m31101| Tue Nov 27 17:44:46.616 [rsSync] dropAllDatabasesExceptLocal 1 m31101| Tue Nov 27 17:44:46.616 [rsSync] replSet initial sync clone all databases m31101| Tue Nov 27 17:44:46.617 [rsSync] replSet initial sync cloning db: admin m31101| Tue Nov 27 17:44:46.617 [FileAllocator] allocating new datafile /data/db/test-rs0-1/admin.ns, filling with zeroes... m31101| Tue Nov 27 17:44:47.003 [FileAllocator] done allocating datafile /data/db/test-rs0-1/admin.ns, size: 16MB, took 0.385 secs m31101| Tue Nov 27 17:44:47.003 [FileAllocator] allocating new datafile /data/db/test-rs0-1/admin.0, filling with zeroes... m31101| Tue Nov 27 17:44:47.361 [FileAllocator] done allocating datafile /data/db/test-rs0-1/admin.0, size: 16MB, took 0.358 secs m31101| Tue Nov 27 17:44:47.412 [rsSync] build index admin.foo { _id: 1 } m31101| Tue Nov 27 17:44:47.412 [rsSync] fastBuildIndex dupsToDrop:0 m31101| Tue Nov 27 17:44:47.412 [rsSync] build index done. scanned 1 total records. 0 secs m31101| Tue Nov 27 17:44:47.412 [rsSync] replSet initial sync data copy, starting syncup m31101| Tue Nov 27 17:44:47.412 [rsSync] oplog sync 1 of 3 m31101| Tue Nov 27 17:44:47.413 [rsSync] oplog sync 2 of 3 m31101| Tue Nov 27 17:44:47.413 [rsSync] replSet initial sync building indexes m31101| Tue Nov 27 17:44:47.413 [rsSync] replSet initial sync cloning indexes for : admin m31101| Tue Nov 27 17:44:47.414 [rsSync] oplog sync 3 of 3 m31101| Tue Nov 27 17:44:47.414 [rsSync] replSet initial sync finishing up m31101| Tue Nov 27 17:44:47.470 [rsSync] replSet set minValid=50b54258:1 m31101| Tue Nov 27 17:44:47.470 [rsSync] build index local.replset.minvalid { _id: 1 } m31101| Tue Nov 27 17:44:47.471 [rsSync] build index done. scanned 0 total records. 0 secs m31101| Tue Nov 27 17:44:47.471 [rsSync] replSet initial sync done m31101| Tue Nov 27 17:44:47.862 [rsBackgroundSync] replSet syncing to: ubuntu:31100 m31101| Tue Nov 27 17:44:48.413 [rsSyncNotifier] replset setting oplog notifier to ubuntu:31100 m31101| Tue Nov 27 17:44:49.472 [rsSync] replSet SECONDARY m31101| Tue Nov 27 17:44:51.839 [rsHealthPoll] replSet member ubuntu:31102 is now in state SECONDARY m31101| Tue Nov 27 17:44:52.258 [initandlisten] connection accepted from 127.0.0.1:34280 #5 (4 connections now open) m31101| Tue Nov 27 17:44:52.261 [initandlisten] connection accepted from 127.0.0.1:34283 #6 (5 connections now open) m31101| Tue Nov 27 17:44:58.049 [conn3] end connection 127.0.0.1:34255 (4 connections now open) m31101| Tue Nov 27 17:44:58.049 [initandlisten] connection accepted from 127.0.0.1:34312 #7 (5 connections now open) m31101| Tue Nov 27 17:45:11.649 [conn4] end connection 127.0.0.1:34262 (4 connections now open) m31101| Tue Nov 27 17:45:11.649 [initandlisten] connection accepted from 127.0.0.1:34412 #8 (5 connections now open) m31101| Tue Nov 27 17:45:12.819 [initandlisten] connection accepted from 127.0.0.1:34433 #9 (6 connections now open) m31101| Tue Nov 27 17:45:13.113 [initandlisten] connection accepted from 127.0.0.1:34436 #10 (7 connections now open) m31101| Tue Nov 27 17:45:23.117 [initandlisten] connection accepted from 127.0.0.1:34458 #11 (8 connections now open) m31101| Tue Nov 27 17:45:26.143 [FileAllocator] allocating new datafile /data/db/test-rs0-1/foo20.ns, filling with zeroes... m31101| Tue Nov 27 17:45:26.449 [FileAllocator] done allocating datafile /data/db/test-rs0-1/foo20.ns, size: 16MB, took 0.306 secs m31101| Tue Nov 27 17:45:26.449 [FileAllocator] allocating new datafile /data/db/test-rs0-1/foo20.0, filling with zeroes... m31101| Tue Nov 27 17:45:27.108 [FileAllocator] done allocating datafile /data/db/test-rs0-1/foo20.0, size: 16MB, took 0.658 secs m31101| Tue Nov 27 17:45:27.109 [repl writer worker 1] build index foo20.bar20 { _id: 1 } m31101| Tue Nov 27 17:45:27.110 [repl writer worker 1] build index done. scanned 0 total records. 0 secs m31101| Tue Nov 27 17:45:27.110 [repl writer worker 1] info: creating collection foo20.bar20 on add index m31101| Tue Nov 27 17:45:28.056 [conn7] end connection 127.0.0.1:34312 (7 connections now open) m31101| Tue Nov 27 17:45:28.056 [initandlisten] connection accepted from 127.0.0.1:34497 #12 (8 connections now open) m31101| Tue Nov 27 17:45:30.084 [initandlisten] connection accepted from 127.0.0.1:34509 #13 (9 connections now open) m31101| Tue Nov 27 17:45:30.086 [initandlisten] connection accepted from 127.0.0.1:34512 #14 (10 connections now open) m31101| Tue Nov 27 17:45:41.656 [conn8] end connection 127.0.0.1:34412 (9 connections now open) m31101| Tue Nov 27 17:45:41.656 [initandlisten] connection accepted from 127.0.0.1:34533 #15 (10 connections now open) m31101| Tue Nov 27 17:45:48.115 [conn11] end connection 127.0.0.1:34458 (9 connections now open) m31101| Tue Nov 27 17:45:48.116 [conn9] end connection 127.0.0.1:34433 (8 connections now open) m31101| Tue Nov 27 17:45:49.514 [initandlisten] connection accepted from 127.0.0.1:34556 #16 (9 connections now open) m31101| Tue Nov 27 17:45:49.764 [initandlisten] connection accepted from 127.0.0.1:34560 #17 (10 connections now open) m31101| Tue Nov 27 17:45:50.758 [initandlisten] connection accepted from 127.0.0.1:34587 #18 (11 connections now open) m31101| Tue Nov 27 17:45:50.925 got signal 15 (Terminated), will terminate after current cmd ends m31101| Tue Nov 27 17:45:50.925 [interruptThread] now exiting m31101| Tue Nov 27 17:45:50.925 dbexit: m31101| Tue Nov 27 17:45:50.925 [interruptThread] shutdown: going to close listening sockets... m31101| Tue Nov 27 17:45:50.925 [interruptThread] closing listening socket: 14 m31101| Tue Nov 27 17:45:50.925 [interruptThread] closing listening socket: 15 m31101| Tue Nov 27 17:45:50.925 [interruptThread] closing listening socket: 16 m31101| Tue Nov 27 17:45:50.925 [interruptThread] removing socket file: /tmp/mongodb-31101.sock m31101| Tue Nov 27 17:45:50.925 [interruptThread] shutdown: going to flush diaglog... m31101| Tue Nov 27 17:45:50.925 [interruptThread] shutdown: going to close sockets... m31101| Tue Nov 27 17:45:50.925 [interruptThread] shutdown: waiting for fs preallocator... m31101| Tue Nov 27 17:45:50.925 [interruptThread] shutdown: lock for final commit... m31101| Tue Nov 27 17:45:50.925 [interruptThread] shutdown: final commit... m31101| Tue Nov 27 17:45:50.991 [interruptThread] shutdown: closing all files... m31101| Tue Nov 27 17:45:50.991 [interruptThread] closeAllFiles() finished m31101| Tue Nov 27 17:45:50.991 [interruptThread] journalCleanup... m31101| Tue Nov 27 17:45:50.991 [interruptThread] removeJournalFiles m31101| Tue Nov 27 17:45:50.998 [conn18] end connection 127.0.0.1:34587 (10 connections now open) m31101| Tue Nov 27 17:45:51.045 [interruptThread] shutdown: removing fs lock... m31101| Tue Nov 27 17:45:51.045 dbexit: really exiting now m31101| note: noprealloc may hurt performance in many applications m31101| Tue Nov 27 17:45:51 [initandlisten] MongoDB starting : pid=12371 port=31101 dbpath=/data/db/test-rs0-1 64-bit host=ubuntu m31101| Tue Nov 27 17:45:51 [initandlisten] db version v2.0.6, pdfile version 4.5 m31101| Tue Nov 27 17:45:51 [initandlisten] git version: e1c0cbc25863f6356aa4e31375add7bb49fb05bc m31101| Tue Nov 27 17:45:51 [initandlisten] build info: Linux ip-10-110-9-236 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_41 m31101| Tue Nov 27 17:45:51 [initandlisten] options: { dbpath: "/data/db/test-rs0-1", noprealloc: true, nopreallocj: true, oplogSize: 40, port: 31101, replSet: "test-rs0", rest: true, smallfiles: true } m31101| Tue Nov 27 17:45:51 [initandlisten] journal dir=/data/db/test-rs0-1/journal m31101| Tue Nov 27 17:45:51 [initandlisten] recover : no journal files present, no recovery needed m31101| Tue Nov 27 17:45:52 [websvr] admin web console waiting for connections on port 32101 m31101| Tue Nov 27 17:45:52 [initandlisten] waiting for connections on port 31101 m31101| Tue Nov 27 17:45:52 [initandlisten] connection accepted from 127.0.0.1:34606 #1 m31101| Tue Nov 27 17:45:52 [rsStart] replSet STARTUP2 m31101| Tue Nov 27 17:45:52 [rsSync] replSet SECONDARY m31101| Tue Nov 27 17:45:52 [rsHealthPoll] replSet member ubuntu:31100 is up m31101| Tue Nov 27 17:45:52 [rsHealthPoll] replSet member ubuntu:31100 is now in state PRIMARY m31101| Tue Nov 27 17:45:52 [rsHealthPoll] replSet member ubuntu:31102 is up m31101| Tue Nov 27 17:45:52 [rsHealthPoll] replSet member ubuntu:31102 is now in state SECONDARY m31101| Tue Nov 27 17:45:52 [initandlisten] connection accepted from 127.0.0.1:34609 #2 m31101| Tue Nov 27 17:45:52 [initandlisten] connection accepted from 127.0.0.1:34610 #3 m31101| Tue Nov 27 17:45:53 [initandlisten] connection accepted from 127.0.0.1:34611 #4 m31101| Tue Nov 27 17:45:54 [conn2] end connection 127.0.0.1:34609 m31101| Tue Nov 27 17:45:55 [initandlisten] connection accepted from 127.0.0.1:34615 #5 m31101| Tue Nov 27 17:45:55 [conn4] end connection 127.0.0.1:34611 m31101| Tue Nov 27 17:45:55 [initandlisten] connection accepted from 127.0.0.1:34619 #6 m31101| Tue Nov 27 17:45:56 [rsHealthPoll] DBClientCursor::init call() failed m31101| Tue Nov 27 17:45:56 [rsHealthPoll] replSet info ubuntu:31102 is down (or slow to respond): DBClientBase::findN: transport error: ubuntu:31102 query: { replSetHeartbeat: "test-rs0", v: 1, pv: 1, checkEmpty: false, from: "ubuntu:31101" } m31101| Tue Nov 27 17:45:56 [rsHealthPoll] replSet member ubuntu:31102 is now in state DOWN m31101| Tue Nov 27 17:45:56 [rsHealthPoll] replSet member ubuntu:31100 is now in state SECONDARY m31101| Tue Nov 27 17:45:56 [rsMgr] replSet info electSelf 1 m31101| Tue Nov 27 17:45:56 [rsMgr] replSet PRIMARY m31101| Tue Nov 27 17:45:57 [conn6] end connection 127.0.0.1:34619 m31101| Tue Nov 27 17:45:58 [rsHealthPoll] DBClientCursor::init call() failed m31101| Tue Nov 27 17:45:58 [rsHealthPoll] replSet info ubuntu:31100 is down (or slow to respond): DBClientBase::findN: transport error: ubuntu:31100 query: { replSetHeartbeat: "test-rs0", v: 1, pv: 1, checkEmpty: false, from: "ubuntu:31101" } m31101| Tue Nov 27 17:45:58 [rsHealthPoll] replSet member ubuntu:31100 is now in state DOWN m31101| Tue Nov 27 17:45:58 [rsHealthPoll] replSet member ubuntu:31102 is up m31101| Tue Nov 27 17:45:58 [rsHealthPoll] replSet member ubuntu:31102 is now in state SECONDARY m31101| Tue Nov 27 17:45:58 [initandlisten] connection accepted from 127.0.0.1:34623 #7 m31101| Tue Nov 27 17:45:59 [initandlisten] connection accepted from 127.0.0.1:34628 #8 m31101| Tue Nov 27 17:46:00 [rsHealthPoll] replSet member ubuntu:31100 is up m31101| Tue Nov 27 17:46:00 [rsHealthPoll] replSet member ubuntu:31100 is now in state SECONDARY m31101| Tue Nov 27 17:46:00 [initandlisten] connection accepted from 127.0.0.1:34631 #9 m31101| Tue Nov 27 17:46:00 [initandlisten] connection accepted from 127.0.0.1:34634 #10 m31101| Tue Nov 27 17:46:00 [initandlisten] connection accepted from 127.0.0.1:34635 #11 m31101| Tue Nov 27 17:46:02 [initandlisten] connection accepted from 127.0.0.1:34642 #12 m31101| Tue Nov 27 17:46:02 [initandlisten] connection accepted from 127.0.0.1:34644 #13 m31101| Tue Nov 27 17:46:02 [initandlisten] connection accepted from 127.0.0.1:34645 #14 m31101| Tue Nov 27 17:46:03 [slaveTracking] build index local.slaves { _id: 1 } m31101| Tue Nov 27 17:46:03 [slaveTracking] build index done 0 records 0 secs m31101| Tue Nov 27 17:46:03 [initandlisten] connection accepted from 127.0.0.1:34653 #15 m31101| Tue Nov 27 17:46:04 [conn9] end connection 127.0.0.1:34631 m31101| Tue Nov 27 17:46:09 [initandlisten] connection accepted from 127.0.0.1:34660 #16 m31101| Tue Nov 27 17:46:13 [conn16] end connection 127.0.0.1:34660 m31101| Tue Nov 27 17:46:13 [conn8] end connection 127.0.0.1:34628 m31101| Tue Nov 27 17:46:15 [initandlisten] connection accepted from 127.0.0.1:34704 #17 m31101| Tue Nov 27 17:46:15 [conn11] end connection 127.0.0.1:34635 m31101| Tue Nov 27 17:46:15 [conn10] end connection 127.0.0.1:34634 m31101| Tue Nov 27 17:46:15 [initandlisten] connection accepted from 127.0.0.1:34708 #18 m31101| Tue Nov 27 17:46:15 [initandlisten] connection accepted from 127.0.0.1:34710 #19 m31101| Tue Nov 27 17:46:16 [initandlisten] connection accepted from 127.0.0.1:34736 #20 m31101| Tue Nov 27 17:46:16 [initandlisten] connection accepted from 127.0.0.1:34747 #21 m31101| Tue Nov 27 17:46:17 [initandlisten] connection accepted from 127.0.0.1:34768 #22 m31101| Tue Nov 27 17:46:18 [FileAllocator] allocating new datafile /data/db/test-rs0-1/fooMixed20.ns, filling with zeroes... m31101| Tue Nov 27 17:46:18 [FileAllocator] done allocating datafile /data/db/test-rs0-1/fooMixed20.ns, size: 16MB, took 0.354 secs m31101| Tue Nov 27 17:46:18 [FileAllocator] allocating new datafile /data/db/test-rs0-1/fooMixed20.0, filling with zeroes... m31101| Tue Nov 27 17:46:19 [FileAllocator] done allocating datafile /data/db/test-rs0-1/fooMixed20.0, size: 16MB, took 0.32 secs m31101| Tue Nov 27 17:46:19 [conn22] build index fooMixed20.barMixed20 { _id: 1 } m31101| Tue Nov 27 17:46:19 [conn22] build index done 0 records 0 secs m31101| Tue Nov 27 17:46:19 [conn22] info: creating collection fooMixed20.barMixed20 on add index m31101| Tue Nov 27 17:46:19 [conn22] couldn't make room for new record (len: 168) in capped ns local.oplog.rs m31101| Extent 0 (capExtent) m31101| magic: 41424344 extent->ns: local.oplog.rs m31101| fr: null lr: null extent->len: 41943040 m31101| local.oplog.rs Assertion failure len * 5 > lastExtentSize db/namespace.cpp 431 m31101| 0x57a8a6 0x5853eb 0x86043c 0x86b2c6 0x85f1e5 0x85f22a 0x8a65e1 0x832544 0x830182 0x887f9f 0x88c649 0x88e945 0xaa0bc8 0x6389f7 0x7f4e36eb5e9a 0x7f4e363d34bd m31101| mongod-2.0(_ZN5mongo12sayDbContextEPKc+0x96) [0x57a8a6] m31101| mongod-2.0(_ZN5mongo8assertedEPKcS1_j+0xfb) [0x5853eb] m31101| mongod-2.0(_ZNK5mongo16NamespaceDetails13maybeComplainEPKci+0x8ec) [0x86043c] m31101| mongod-2.0(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x256) [0x86b2c6] m31101| mongod-2.0(_ZN5mongo16NamespaceDetails6_allocEPKci+0x25) [0x85f1e5] m31101| mongod-2.0(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x3a) [0x85f22a] m31101| mongod-2.0(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x161) [0x8a65e1] m31101| mongod-2.0() [0x832544] m31101| mongod-2.0(_ZN5mongo5logOpEPKcS1_RKNS_7BSONObjEPS2_Pb+0x42) [0x830182] m31101| mongod-2.0(_ZN5mongo14checkAndInsertEPKcRNS_7BSONObjE+0xbf) [0x887f9f] m31101| mongod-2.0(_ZN5mongo14receivedInsertERNS_7MessageERNS_5CurOpE+0x439) [0x88c649] m31101| mongod-2.0(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x11c5) [0x88e945] m31101| mongod-2.0(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x78) [0xaa0bc8] m31101| mongod-2.0(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) [0x6389f7] m31101| /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7f4e36eb5e9a] m31101| /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f4e363d34bd] m31101| Tue Nov 27 17:46:19 [conn22] insert fooMixed20.system.indexes exception: assertion db/namespace.cpp:431 700ms m31101| Tue Nov 27 17:46:19 [conn15] getMore: cursorid not found local.oplog.rs 4994816232156589526 m31101| Tue Nov 27 17:46:19 [conn12] getMore: cursorid not found local.oplog.rs 3423941959508322590 m31101| Tue Nov 27 17:46:19 [conn15] end connection 127.0.0.1:34653 m31101| Tue Nov 27 17:46:19 [conn12] end connection 127.0.0.1:34642 m31101| Tue Nov 27 17:46:19 [initandlisten] connection accepted from 127.0.0.1:34778 #23 m31101| Tue Nov 27 17:46:19 [conn23] SyncClusterConnection connecting to [ubuntu:29000] m31101| Tue Nov 27 17:46:19 [conn23] SyncClusterConnection connecting to [ubuntu:29001] m31101| Tue Nov 27 17:46:19 [conn23] SyncClusterConnection connecting to [ubuntu:29002] m31101| Tue Nov 27 17:46:19 [conn22] SyncClusterConnection connecting to [ubuntu:29000] m31101| Tue Nov 27 17:46:19 [conn22] SyncClusterConnection connecting to [ubuntu:29001] m31101| Tue Nov 27 17:46:19 [conn22] SyncClusterConnection connecting to [ubuntu:29002] m31101| Tue Nov 27 17:46:19 [conn22] starting new replica set monitor for replica set test-rs0 with seed of ubuntu:31100,ubuntu:31101,ubuntu:31102 m31101| Tue Nov 27 17:46:19 [conn22] successfully connected to seed ubuntu:31100 for replica set test-rs0 m31101| Tue Nov 27 17:46:19 [conn22] changing hosts to { 0: "ubuntu:31100", 1: "ubuntu:31102", 2: "ubuntu:31101" } from test-rs0/ m31101| Tue Nov 27 17:46:19 [conn22] trying to add new host ubuntu:31100 to replica set test-rs0 m31101| Tue Nov 27 17:46:19 [conn22] successfully connected to new host ubuntu:31100 in replica set test-rs0 m31101| Tue Nov 27 17:46:19 [conn22] trying to add new host ubuntu:31101 to replica set test-rs0 m31101| Tue Nov 27 17:46:19 [initandlisten] connection accepted from 127.0.0.1:34788 #24 m31101| Tue Nov 27 17:46:19 [conn22] successfully connected to new host ubuntu:31101 in replica set test-rs0 m31101| Tue Nov 27 17:46:19 [conn22] trying to add new host ubuntu:31102 to replica set test-rs0 m31101| Tue Nov 27 17:46:19 [conn22] successfully connected to new host ubuntu:31102 in replica set test-rs0 m31101| Tue Nov 27 17:46:19 [initandlisten] connection accepted from 127.0.0.1:34791 #25 m31101| Tue Nov 27 17:46:19 [conn22] replica set monitor for replica set test-rs0 started, address is test-rs0/ubuntu:31100,ubuntu:31101,ubuntu:31102 m31101| Tue Nov 27 17:46:19 [conn22] starting new replica set monitor for replica set test-rs1 with seed of ubuntu:31200,ubuntu:31201,ubuntu:31202 m31101| Tue Nov 27 17:46:19 [ReplicaSetMonitorWatcher] starting m31101| Tue Nov 27 17:46:19 [conn22] successfully connected to seed ubuntu:31200 for replica set test-rs1 m31101| Tue Nov 27 17:46:19 [conn22] changing hosts to { 0: "ubuntu:31200", 1: "ubuntu:31202", 2: "ubuntu:31201" } from test-rs1/ m31101| Tue Nov 27 17:46:19 [conn22] trying to add new host ubuntu:31200 to replica set test-rs1 m31101| Tue Nov 27 17:46:19 [conn22] successfully connected to new host ubuntu:31200 in replica set test-rs1 m31101| Tue Nov 27 17:46:19 [conn22] trying to add new host ubuntu:31201 to replica set test-rs1 m31101| Tue Nov 27 17:46:19 [conn22] successfully connected to new host ubuntu:31201 in replica set test-rs1 m31101| Tue Nov 27 17:46:19 [conn22] trying to add new host ubuntu:31202 to replica set test-rs1 m31101| Tue Nov 27 17:46:19 [conn22] successfully connected to new host ubuntu:31202 in replica set test-rs1 m31101| Tue Nov 27 17:46:19 [conn22] replica set monitor for replica set test-rs1 started, address is test-rs1/ubuntu:31200,ubuntu:31201,ubuntu:31202 m31101| Tue Nov 27 17:46:19 [conn22] received splitChunk request: { splitChunk: "fooMixed20.barMixed20", keyPattern: { _id: 1.0 }, min: { _id: MinKey }, max: { _id: MaxKey }, from: "test-rs0/ubuntu:31100,ubuntu:31101,ubuntu:31102", splitKeys: [ { _id: 0.0 } ], shardId: "fooMixed20.barMixed20-_id_MinKey", configdb: "ubuntu:29000,ubuntu:29001,ubuntu:29002" } m31101| Tue Nov 27 17:46:19 [conn22] created new distributed lock for fooMixed20.barMixed20 on ubuntu:29000,ubuntu:29001,ubuntu:29002 ( lock timeout : 900000, ping interval : 30000, process : 0 ) m31101| Tue Nov 27 17:46:19 [LockPinger] creating distributed lock ping thread for ubuntu:29000,ubuntu:29001,ubuntu:29002 and process ubuntu:31101:1354056379:52313605 (sleeping for 30000ms) m31101| Tue Nov 27 17:46:20 [initandlisten] connection accepted from 127.0.0.1:34805 #26 m31101| Tue Nov 27 17:46:20 [initandlisten] connection accepted from 127.0.0.1:34806 #27 m31101| Tue Nov 27 17:46:20 [conn22] distributed lock 'fooMixed20.barMixed20/ubuntu:31101:1354056379:52313605' acquired, ts : 50b542bb53ed56c79f14367d m31101| Tue Nov 27 17:46:20 [conn22] splitChunk accepted at version 1|0 m31101| Tue Nov 27 17:46:20 [conn22] about to log metadata event: { _id: "ubuntu-2012-11-27T22:46:20-0", server: "ubuntu", clientAddr: "127.0.0.1:34768", time: new Date(1354056380853), what: "split", ns: "fooMixed20.barMixed20", details: { before: { min: { _id: MinKey }, max: { _id: MaxKey }, lastmod: Timestamp 1000|0 }, left: { min: { _id: MinKey }, max: { _id: 0.0 }, lastmod: Timestamp 1000|1 }, right: { min: { _id: 0.0 }, max: { _id: MaxKey }, lastmod: Timestamp 1000|2 } } } m31101| Tue Nov 27 17:46:21 [conn22] distributed lock 'fooMixed20.barMixed20/ubuntu:31101:1354056379:52313605' unlocked. m31101| Tue Nov 27 17:46:21 [conn22] command admin.$cmd command: { splitChunk: "fooMixed20.barMixed20", keyPattern: { _id: 1.0 }, min: { _id: MinKey }, max: { _id: MaxKey }, from: "test-rs0/ubuntu:31100,ubuntu:31101,ubuntu:31102", splitKeys: [ { _id: 0.0 } ], shardId: "fooMixed20.barMixed20-_id_MinKey", configdb: "ubuntu:29000,ubuntu:29001,ubuntu:29002" } ntoreturn:1 reslen:37 1790ms m31101| Tue Nov 27 17:46:21 [conn22] received moveChunk request: { moveChunk: "fooMixed20.barMixed20", from: "test-rs0/ubuntu:31100,ubuntu:31101,ubuntu:31102", to: "test-rs1/ubuntu:31200,ubuntu:31201,ubuntu:31202", min: { _id: 0.0 }, max: { _id: MaxKey }, maxChunkSizeBytes: 52428800, shardId: "fooMixed20.barMixed20-_id_0.0", configdb: "ubuntu:29000,ubuntu:29001,ubuntu:29002" } m31101| Tue Nov 27 17:46:21 [conn22] created new distributed lock for fooMixed20.barMixed20 on ubuntu:29000,ubuntu:29001,ubuntu:29002 ( lock timeout : 900000, ping interval : 30000, process : 0 ) m31101| Tue Nov 27 17:46:21 [conn22] distributed lock 'fooMixed20.barMixed20/ubuntu:31101:1354056379:52313605' acquired, ts : 50b542bd53ed56c79f14367e m31101| Tue Nov 27 17:46:21 [conn22] about to log metadata event: { _id: "ubuntu-2012-11-27T22:46:21-1", server: "ubuntu", clientAddr: "127.0.0.1:34768", time: new Date(1354056381990), what: "moveChunk.start", ns: "fooMixed20.barMixed20", details: { min: { _id: 0.0 }, max: { _id: MaxKey }, from: "test-rs0", to: "test-rs1" } } m31101| Tue Nov 27 17:46:22 [conn22] moveChunk request accepted at version 1|2 m31101| Tue Nov 27 17:46:22 [conn22] moveChunk number of documents: 0 m31101| Tue Nov 27 17:46:22 [initandlisten] connection accepted from 127.0.0.1:34810 #28 m31101| Tue Nov 27 17:46:22 [initandlisten] connection accepted from 127.0.0.1:34813 #29 m31101| Tue Nov 27 17:46:22 [initandlisten] connection accepted from 127.0.0.1:34815 #30 m31101| Tue Nov 27 17:46:22 [conn27] end connection 127.0.0.1:34806 m31101| Tue Nov 27 17:46:22 [conn26] end connection 127.0.0.1:34805 m31101| Tue Nov 27 17:46:22 [conn30] end connection 127.0.0.1:34815 m31101| Tue Nov 27 17:46:23 [conn22] moveChunk data transfer progress: { active: true, ns: "fooMixed20.barMixed20", from: "test-rs0/ubuntu:31100,ubuntu:31101,ubuntu:31102", min: { _id: 0.0 }, max: { _id: MaxKey }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 m31101| Tue Nov 27 17:46:23 [conn5] end connection 127.0.0.1:34615 m31101| Tue Nov 27 17:46:23 [initandlisten] connection accepted from 127.0.0.1:34823 #31 m31101| Tue Nov 27 17:46:23 [initandlisten] connection accepted from 127.0.0.1:34824 #32 m31101| Tue Nov 27 17:46:23 [initandlisten] connection accepted from 127.0.0.1:34825 #33 m31101| Tue Nov 27 17:46:24 [conn22] moveChunk data transfer progress: { active: false, ns: "fooMixed20.barMixed20", from: "test-rs0/ubuntu:31100,ubuntu:31101,ubuntu:31102", min: { _id: 0.0 }, max: { _id: MaxKey }, state: "fail", errmsg: "assertion db/namespace.cpp:431", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 m31101| Tue Nov 27 17:46:24 [conn22] warning: moveChunk error transferring data caused migration abort: { active: false, ns: "fooMixed20.barMixed20", from: "test-rs0/ubuntu:31100,ubuntu:31101,ubuntu:31102", min: { _id: 0.0 }, max: { _id: MaxKey }, state: "fail", errmsg: "assertion db/namespace.cpp:431", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } m31101| Tue Nov 27 17:46:24 [conn22] distributed lock 'fooMixed20.barMixed20/ubuntu:31101:1354056379:52313605' unlocked. m31101| Tue Nov 27 17:46:24 [conn22] about to log metadata event: { _id: "ubuntu-2012-11-27T22:46:24-2", server: "ubuntu", clientAddr: "127.0.0.1:34768", time: new Date(1354056384531), what: "moveChunk.from", ns: "fooMixed20.barMixed20", details: { min: { _id: 0.0 }, max: { _id: MaxKey }, step1: 0, step2: 804, step3: 3, note: "aborted" } } m31101| Tue Nov 27 17:46:24 [conn22] command admin.$cmd command: { moveChunk: "fooMixed20.barMixed20", from: "test-rs0/ubuntu:31100,ubuntu:31101,ubuntu:31102", to: "test-rs1/ubuntu:31200,ubuntu:31201,ubuntu:31202", min: { _id: 0.0 }, max: { _id: MaxKey }, maxChunkSizeBytes: 52428800, shardId: "fooMixed20.barMixed20-_id_0.0", configdb: "ubuntu:29000,ubuntu:29001,ubuntu:29002" } ntoreturn:1 reslen:370 3356ms m31101| Tue Nov 27 17:46:24 [conn22] received splitChunk request: { splitChunk: "fooMixed20.barMixed20", keyPattern: { _id: 1.0 }, min: { _id: MinKey }, max: { _id: 0.0 }, from: "test-rs0/ubuntu:31100,ubuntu:31101,ubuntu:31102", splitKeys: [ { _id: -300.0 } ], shardId: "fooMixed20.barMixed20-_id_MinKey", configdb: "ubuntu:29000,ubuntu:29001,ubuntu:29002" } m31101| Tue Nov 27 17:46:24 [conn22] created new distributed lock for fooMixed20.barMixed20 on ubuntu:29000,ubuntu:29001,ubuntu:29002 ( lock timeout : 900000, ping interval : 30000, process : 0 ) m31101| Tue Nov 27 17:46:25 [conn22] distributed lock 'fooMixed20.barMixed20/ubuntu:31101:1354056379:52313605' acquired, ts : 50b542c053ed56c79f14367f m31101| Tue Nov 27 17:46:25 [conn22] splitChunk accepted at version 1|2 m31101| Tue Nov 27 17:46:25 [conn32] end connection 127.0.0.1:34824 m31101| Tue Nov 27 17:46:25 [conn33] end connection 127.0.0.1:34825 m31101| Tue Nov 27 17:46:25 [conn22] about to log metadata event: { _id: "ubuntu-2012-11-27T22:46:25-3", server: "ubuntu", clientAddr: "127.0.0.1:34768", time: new Date(1354056385468), what: "split", ns: "fooMixed20.barMixed20", details: { before: { min: { _id: MinKey }, max: { _id: 0.0 }, lastmod: Timestamp 1000|1 }, left: { min: { _id: MinKey }, max: { _id: -300.0 }, lastmod: Timestamp 1000|3 }, right: { min: { _id: -300.0 }, max: { _id: 0.0 }, lastmod: Timestamp 1000|4 } } } m31101| Tue Nov 27 17:46:25 [conn22] distributed lock 'fooMixed20.barMixed20/ubuntu:31101:1354056379:52313605' unlocked. m31101| Tue Nov 27 17:46:25 [conn22] command admin.$cmd command: { splitChunk: "fooMixed20.barMixed20", keyPattern: { _id: 1.0 }, min: { _id: MinKey }, max: { _id: 0.0 }, from: "test-rs0/ubuntu:31100,ubuntu:31101,ubuntu:31102", splitKeys: [ { _id: -300.0 } ], shardId: "fooMixed20.barMixed20-_id_MinKey", configdb: "ubuntu:29000,ubuntu:29001,ubuntu:29002" } ntoreturn:1 reslen:37 1190ms m31101| Tue Nov 27 17:46:25 [conn22] received splitChunk request: { splitChunk: "fooMixed20.barMixed20", keyPattern: { _id: 1.0 }, min: { _id: -300.0 }, max: { _id: 0.0 }, from: "test-rs0/ubuntu:31100,ubuntu:31101,ubuntu:31102", splitKeys: [ { _id: -200.0 } ], shardId: "fooMixed20.barMixed20-_id_-300.0", configdb: "ubuntu:29000,ubuntu:29001,ubuntu:29002" } m31101| Tue Nov 27 17:46:25 [conn22] created new distributed lock for fooMixed20.barMixed20 on ubuntu:29000,ubuntu:29001,ubuntu:29002 ( lock timeout : 900000, ping interval : 30000, process : 0 ) m31101| Tue Nov 27 17:46:26 [initandlisten] connection accepted from 127.0.0.1:34828 #34 m31101| Tue Nov 27 17:46:26 [initandlisten] connection accepted from 127.0.0.1:34829 #35 m31101| Tue Nov 27 17:46:26 [conn7] end connection 127.0.0.1:34623 m31101| Tue Nov 27 17:46:26 [initandlisten] connection accepted from 127.0.0.1:34831 #36 m31101| Tue Nov 27 17:46:26 [conn22] distributed lock 'fooMixed20.barMixed20/ubuntu:31101:1354056379:52313605' acquired, ts : 50b542c153ed56c79f143680 m31101| Tue Nov 27 17:46:26 [conn22] splitChunk accepted at version 1|4 m31101| Tue Nov 27 17:46:26 [conn22] about to log metadata event: { _id: "ubuntu-2012-11-27T22:46:26-4", server: "ubuntu", clientAddr: "127.0.0.1:34768", time: new Date(1354056386786), what: "split", ns: "fooMixed20.barMixed20", details: { before: { min: { _id: -300.0 }, max: { _id: 0.0 }, lastmod: Timestamp 1000|4 }, left: { min: { _id: -300.0 }, max: { _id: -200.0 }, lastmod: Timestamp 1000|5 }, right: { min: { _id: -200.0 }, max: { _id: 0.0 }, lastmod: Timestamp 1000|6 } } } m31101| Tue Nov 27 17:46:26 [initandlisten] connection accepted from 127.0.0.1:34832 #37 m31101| Tue Nov 27 17:46:27 [conn22] distributed lock 'fooMixed20.barMixed20/ubuntu:31101:1354056379:52313605' unlocked. m31101| Tue Nov 27 17:46:27 [conn22] command admin.$cmd command: { splitChunk: "fooMixed20.barMixed20", keyPattern: { _id: 1.0 }, min: { _id: -300.0 }, max: { _id: 0.0 }, from: "test-rs0/ubuntu:31100,ubuntu:31101,ubuntu:31102", splitKeys: [ { _id: -200.0 } ], shardId: "fooMixed20.barMixed20-_id_-300.0", configdb: "ubuntu:29000,ubuntu:29001,ubuntu:29002" } ntoreturn:1 reslen:37 1214ms m31101| Tue Nov 27 17:46:27 [conn22] received splitChunk request: { splitChunk: "fooMixed20.barMixed20", keyPattern: { _id: 1.0 }, min: { _id: -200.0 }, max: { _id: 0.0 }, from: "test-rs0/ubuntu:31100,ubuntu:31101,ubuntu:31102", splitKeys: [ { _id: -100.0 } ], shardId: "fooMixed20.barMixed20-_id_-200.0", configdb: "ubuntu:29000,ubuntu:29001,ubuntu:29002" } m31101| Tue Nov 27 17:46:27 [conn22] created new distributed lock for fooMixed20.barMixed20 on ubuntu:29000,ubuntu:29001,ubuntu:29002 ( lock timeout : 900000, ping interval : 30000, process : 0 ) m31101| Tue Nov 27 17:46:27 [conn22] distributed lock 'fooMixed20.barMixed20/ubuntu:31101:1354056379:52313605' acquired, ts : 50b542c353ed56c79f143681 m31101| Tue Nov 27 17:46:27 [conn22] splitChunk accepted at version 1|6 m31101| Tue Nov 27 17:46:27 [conn22] about to log metadata event: { _id: "ubuntu-2012-11-27T22:46:27-5", server: "ubuntu", clientAddr: "127.0.0.1:34768", time: new Date(1354056387915), what: "split", ns: "fooMixed20.barMixed20", details: { before: { min: { _id: -200.0 }, max: { _id: 0.0 }, lastmod: Timestamp 1000|6 }, left: { min: { _id: -200.0 }, max: { _id: -100.0 }, lastmod: Timestamp 1000|7 }, right: { min: { _id: -100.0 }, max: { _id: 0.0 }, lastmod: Timestamp 1000|8 } } } m31101| Tue Nov 27 17:46:28 [conn22] distributed lock 'fooMixed20.barMixed20/ubuntu:31101:1354056379:52313605' unlocked. m31101| Tue Nov 27 17:46:28 [conn22] command admin.$cmd command: { splitChunk: "fooMixed20.barMixed20", keyPattern: { _id: 1.0 }, min: { _id: -200.0 }, max: { _id: 0.0 }, from: "test-rs0/ubuntu:31100,ubuntu:31101,ubuntu:31102", splitKeys: [ { _id: -100.0 } ], shardId: "fooMixed20.barMixed20-_id_-200.0", configdb: "ubuntu:29000,ubuntu:29001,ubuntu:29002" } ntoreturn:1 reslen:37 1065ms m31101| Tue Nov 27 17:46:28 [conn22] received splitChunk request: { splitChunk: "fooMixed20.barMixed20", keyPattern: { _id: 1.0 }, min: { _id: 0.0 }, max: { _id: MaxKey }, from: "test-rs0/ubuntu:31100,ubuntu:31101,ubuntu:31102", splitKeys: [ { _id: 100.0 } ], shardId: "fooMixed20.barMixed20-_id_0.0", configdb: "ubuntu:29000,ubuntu:29001,ubuntu:29002" } m31101| Tue Nov 27 17:46:28 [conn22] created new distributed lock for fooMixed20.barMixed20 on ubuntu:29000,ubuntu:29001,ubuntu:29002 ( lock timeout : 900000, ping interval : 30000, process : 0 ) m31101| Tue Nov 27 17:46:28 [conn34] end connection 127.0.0.1:34828 m31101| Tue Nov 27 17:46:28 [conn35] end connection 127.0.0.1:34829 m31101| Tue Nov 27 17:46:28 [conn22] distributed lock 'fooMixed20.barMixed20/ubuntu:31101:1354056379:52313605' acquired, ts : 50b542c453ed56c79f143682 m31101| Tue Nov 27 17:46:28 [conn22] splitChunk accepted at version 1|8 m31101| Tue Nov 27 17:46:29 [conn22] about to log metadata event: { _id: "ubuntu-2012-11-27T22:46:29-6", server: "ubuntu", clientAddr: "127.0.0.1:34768", time: new Date(1354056389075), what: "split", ns: "fooMixed20.barMixed20", details: { before: { min: { _id: 0.0 }, max: { _id: MaxKey }, lastmod: Timestamp 1000|2 }, left: { min: { _id: 0.0 }, max: { _id: 100.0 }, lastmod: Timestamp 1000|9 }, right: { min: { _id: 100.0 }, max: { _id: MaxKey }, lastmod: Timestamp 1000|10 } } } m31101| Tue Nov 27 17:46:29 [initandlisten] connection accepted from 127.0.0.1:34840 #38 m31101| Tue Nov 27 17:46:29 [initandlisten] connection accepted from 127.0.0.1:34841 #39 m31101| Tue Nov 27 17:46:29 [conn22] distributed lock 'fooMixed20.barMixed20/ubuntu:31101:1354056379:52313605' unlocked. m31101| Tue Nov 27 17:46:29 [conn22] command admin.$cmd command: { splitChunk: "fooMixed20.barMixed20", keyPattern: { _id: 1.0 }, min: { _id: 0.0 }, max: { _id: MaxKey }, from: "test-rs0/ubuntu:31100,ubuntu:31101,ubuntu:31102", splitKeys: [ { _id: 100.0 } ], shardId: "fooMixed20.barMixed20-_id_0.0", configdb: "ubuntu:29000,ubuntu:29001,ubuntu:29002" } ntoreturn:1 reslen:37 1250ms m31101| Tue Nov 27 17:46:29 [conn22] received splitChunk request: { splitChunk: "fooMixed20.barMixed20", keyPattern: { _id: 1.0 }, min: { _id: 100.0 }, max: { _id: MaxKey }, from: "test-rs0/ubuntu:31100,ubuntu:31101,ubuntu:31102", splitKeys: [ { _id: 200.0 } ], shardId: "fooMixed20.barMixed20-_id_100.0", configdb: "ubuntu:29000,ubuntu:29001,ubuntu:29002" } m31101| Tue Nov 27 17:46:29 [conn22] created new distributed lock for fooMixed20.barMixed20 on ubuntu:29000,ubuntu:29001,ubuntu:29002 ( lock timeout : 900000, ping interval : 30000, process : 0 ) m31101| Tue Nov 27 17:46:30 [conn22] distributed lock 'fooMixed20.barMixed20/ubuntu:31101:1354056379:52313605' acquired, ts : 50b542c553ed56c79f143683 m31101| Tue Nov 27 17:46:30 [conn22] splitChunk accepted at version 1|10 m31101| Tue Nov 27 17:46:30 [conn22] about to log metadata event: { _id: "ubuntu-2012-11-27T22:46:30-7", server: "ubuntu", clientAddr: "127.0.0.1:34768", time: new Date(1354056390344), what: "split", ns: "fooMixed20.barMixed20", details: { before: { min: { _id: 100.0 }, max: { _id: MaxKey }, lastmod: Timestamp 1000|10 }, left: { min: { _id: 100.0 }, max: { _id: 200.0 }, lastmod: Timestamp 1000|11 }, right: { min: { _id: 200.0 }, max: { _id: MaxKey }, lastmod: Timestamp 1000|12 } } } m31101| Tue Nov 27 17:46:30 [conn22] distributed lock 'fooMixed20.barMixed20/ubuntu:31101:1354056379:52313605' unlocked. m31101| Tue Nov 27 17:46:30 [conn22] command admin.$cmd command: { splitChunk: "fooMixed20.barMixed20", keyPattern: { _id: 1.0 }, min: { _id: 100.0 }, max: { _id: MaxKey }, from: "test-rs0/ubuntu:31100,ubuntu:31101,ubuntu:31102", splitKeys: [ { _id: 200.0 } ], shardId: "fooMixed20.barMixed20-_id_100.0", configdb: "ubuntu:29000,ubuntu:29001,ubuntu:29002" } ntoreturn:1 reslen:37 1401ms m31101| Tue Nov 27 17:46:30 [conn22] received splitChunk request: { splitChunk: "fooMixed20.barMixed20", keyPattern: { _id: 1.0 }, min: { _id: 200.0 }, max: { _id: MaxKey }, from: "test-rs0/ubuntu:31100,ubuntu:31101,ubuntu:31102", splitKeys: [ { _id: 300.0 } ], shardId: "fooMixed20.barMixed20-_id_200.0", configdb: "ubuntu:29000,ubuntu:29001,ubuntu:29002" } m31101| Tue Nov 27 17:46:30 [conn22] created new distributed lock for fooMixed20.barMixed20 on ubuntu:29000,ubuntu:29001,ubuntu:29002 ( lock timeout : 900000, ping interval : 30000, process : 0 ) m31101| Tue Nov 27 17:46:31 [conn38] end connection 127.0.0.1:34840 m31101| Tue Nov 27 17:46:31 [conn39] end connection 127.0.0.1:34841 m31101| Tue Nov 27 17:46:31 [conn22] distributed lock 'fooMixed20.barMixed20/ubuntu:31101:1354056379:52313605' acquired, ts : 50b542c653ed56c79f143684 m31101| Tue Nov 27 17:46:31 [conn22] splitChunk accepted at version 1|12 m31101| Tue Nov 27 17:46:31 [conn22] about to log metadata event: { _id: "ubuntu-2012-11-27T22:46:31-8", server: "ubuntu", clientAddr: "127.0.0.1:34768", time: new Date(1354056391729), what: "split", ns: "fooMixed20.barMixed20", details: { before: { min: { _id: 200.0 }, max: { _id: MaxKey }, lastmod: Timestamp 1000|12 }, left: { min: { _id: 200.0 }, max: { _id: 300.0 }, lastmod: Timestamp 1000|13 }, right: { min: { _id: 300.0 }, max: { _id: MaxKey }, lastmod: Timestamp 1000|14 } } } m31101| Tue Nov 27 17:46:32 [conn22] distributed lock 'fooMixed20.barMixed20/ubuntu:31101:1354056379:52313605' unlocked. m31101| Tue Nov 27 17:46:32 [conn22] command admin.$cmd command: { splitChunk: "fooMixed20.barMixed20", keyPattern: { _id: 1.0 }, min: { _id: 200.0 }, max: { _id: MaxKey }, from: "test-rs0/ubuntu:31100,ubuntu:31101,ubuntu:31102", splitKeys: [ { _id: 300.0 } ], shardId: "fooMixed20.barMixed20-_id_200.0", configdb: "ubuntu:29000,ubuntu:29001,ubuntu:29002" } ntoreturn:1 reslen:37 1308ms m31101| Tue Nov 27 17:46:32 [conn22] received moveChunk request: { moveChunk: "fooMixed20.barMixed20", from: "test-rs0/ubuntu:31100,ubuntu:31101,ubuntu:31102", to: "test-rs1/ubuntu:31200,ubuntu:31201,ubuntu:31202", min: { _id: -200.0 }, max: { _id: -100.0 }, maxChunkSizeBytes: 52428800, shardId: "fooMixed20.barMixed20-_id_-200.0", configdb: "ubuntu:29000,ubuntu:29001,ubuntu:29002" } m31101| Tue Nov 27 17:46:32 [conn22] created new distributed lock for fooMixed20.barMixed20 on ubuntu:29000,ubuntu:29001,ubuntu:29002 ( lock timeout : 900000, ping interval : 30000, process : 0 ) m31101| Tue Nov 27 17:46:32 [initandlisten] connection accepted from 127.0.0.1:34848 #40 m31101| Tue Nov 27 17:46:32 [initandlisten] connection accepted from 127.0.0.1:34849 #41 m31101| Tue Nov 27 17:46:32 [conn22] distributed lock 'fooMixed20.barMixed20/ubuntu:31101:1354056379:52313605' acquired, ts : 50b542c853ed56c79f143685 m31101| Tue Nov 27 17:46:32 [conn22] about to log metadata event: { _id: "ubuntu-2012-11-27T22:46:32-9", server: "ubuntu", clientAddr: "127.0.0.1:34768", time: new Date(1354056392622), what: "moveChunk.start", ns: "fooMixed20.barMixed20", details: { min: { _id: -200.0 }, max: { _id: -100.0 }, from: "test-rs0", to: "test-rs1" } } m31101| Tue Nov 27 17:46:32 [conn22] moveChunk request accepted at version 1|14 m31101| Tue Nov 27 17:46:32 [conn22] moveChunk number of documents: 0 m31101| Tue Nov 27 17:46:32 [initandlisten] connection accepted from 127.0.0.1:34850 #42 m31101| Tue Nov 27 17:46:32 [conn42] end connection 127.0.0.1:34850 m31101| Tue Nov 27 17:46:33 [conn22] moveChunk data transfer progress: { active: false, ns: "fooMixed20.barMixed20", from: "test-rs0/ubuntu:31100,ubuntu:31101,ubuntu:31102", min: { _id: -200.0 }, max: { _id: -100.0 }, state: "fail", errmsg: "assertion db/namespace.cpp:431", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 m31101| Tue Nov 27 17:46:33 [conn22] warning: moveChunk error transferring data caused migration abort: { active: false, ns: "fooMixed20.barMixed20", from: "test-rs0/ubuntu:31100,ubuntu:31101,ubuntu:31102", min: { _id: -200.0 }, max: { _id: -100.0 }, state: "fail", errmsg: "assertion db/namespace.cpp:431", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } m31101| Tue Nov 27 17:46:34 [conn22] distributed lock 'fooMixed20.barMixed20/ubuntu:31101:1354056379:52313605' unlocked. m31101| Tue Nov 27 17:46:34 [conn22] about to log metadata event: { _id: "ubuntu-2012-11-27T22:46:34-10", server: "ubuntu", clientAddr: "127.0.0.1:34768", time: new Date(1354056394055), what: "moveChunk.from", ns: "fooMixed20.barMixed20", details: { min: { _id: -200.0 }, max: { _id: -100.0 }, step1: 0, step2: 631, step3: 0, note: "aborted" } } m31101| Tue Nov 27 17:46:34 [conn22] command admin.$cmd command: { moveChunk: "fooMixed20.barMixed20", from: "test-rs0/ubuntu:31100,ubuntu:31101,ubuntu:31102", to: "test-rs1/ubuntu:31200,ubuntu:31201,ubuntu:31202", min: { _id: -200.0 }, max: { _id: -100.0 }, maxChunkSizeBytes: 52428800, shardId: "fooMixed20.barMixed20-_id_-200.0", configdb: "ubuntu:29000,ubuntu:29001,ubuntu:29002" } ntoreturn:1 reslen:378 1989ms m31101| Tue Nov 27 17:46:34 [conn41] end connection 127.0.0.1:34849 m31101| Tue Nov 27 17:46:34 [conn40] end connection 127.0.0.1:34848 m31101| Tue Nov 27 17:46:34 [FileAllocator] allocating new datafile /data/db/test-rs0-1/fooMixed22.ns, filling with zeroes... m31101| Tue Nov 27 17:46:35 [FileAllocator] done allocating datafile /data/db/test-rs0-1/fooMixed22.ns, size: 16MB, took 0.293 secs m31101| Tue Nov 27 17:46:35 [FileAllocator] allocating new datafile /data/db/test-rs0-1/fooMixed22.0, filling with zeroes... m31101| Tue Nov 27 17:46:35 [initandlisten] connection accepted from 127.0.0.1:34853 #43 m31101| Tue Nov 27 17:46:35 [initandlisten] connection accepted from 127.0.0.1:34854 #44 m31101| Tue Nov 27 17:46:35 [FileAllocator] done allocating datafile /data/db/test-rs0-1/fooMixed22.0, size: 16MB, took 0.362 secs m31101| Tue Nov 27 17:46:35 [conn19] build index fooMixed22.barMixed22 { _id: 1 } m31101| Tue Nov 27 17:46:35 [conn19] build index done 0 records 0 secs m31101| Tue Nov 27 17:46:35 [conn19] info: creating collection fooMixed22.barMixed22 on add index m31101| Tue Nov 27 17:46:35 [conn19] couldn't make room for new record (len: 168) in capped ns local.oplog.rs m31101| Extent 0 (capExtent) m31101| magic: 41424344 extent->ns: local.oplog.rs m31101| fr: null lr: null extent->len: 41943040 m31101| local.oplog.rs Assertion failure len * 5 > lastExtentSize db/namespace.cpp 431 m31101| 0x57a8a6 0x5853eb 0x86043c 0x86b2c6 0x85f1e5 0x85f22a 0x8a65e1 0x832544 0x830182 0x887f9f 0x88c649 0x88e945 0xaa0bc8 0x6389f7 0x7f4e36eb5e9a 0x7f4e363d34bd m31101| mongod-2.0(_ZN5mongo12sayDbContextEPKc+0x96) [0x57a8a6] m31101| mongod-2.0(_ZN5mongo8assertedEPKcS1_j+0xfb) [0x5853eb] m31101| mongod-2.0(_ZNK5mongo16NamespaceDetails13maybeComplainEPKci+0x8ec) [0x86043c] m31101| mongod-2.0(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x256) [0x86b2c6] m31101| mongod-2.0(_ZN5mongo16NamespaceDetails6_allocEPKci+0x25) [0x85f1e5] m31101| mongod-2.0(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x3a) [0x85f22a] m31101| mongod-2.0(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x161) [0x8a65e1] m31101| mongod-2.0() [0x832544] m31101| mongod-2.0(_ZN5mongo5logOpEPKcS1_RKNS_7BSONObjEPS2_Pb+0x42) [0x830182] m31101| mongod-2.0(_ZN5mongo14checkAndInsertEPKcRNS_7BSONObjE+0xbf) [0x887f9f] m31101| mongod-2.0(_ZN5mongo14receivedInsertERNS_7MessageERNS_5CurOpE+0x439) [0x88c649] m31101| mongod-2.0(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x11c5) [0x88e945] m31101| mongod-2.0(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x78) [0xaa0bc8] m31101| mongod-2.0(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) [0x6389f7] m31101| /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7f4e36eb5e9a] m31101| /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f4e363d34bd] m31101| Tue Nov 27 17:46:35 [conn19] insert fooMixed22.system.indexes exception: assertion db/namespace.cpp:431 665ms m31101| Tue Nov 27 17:46:36 [initandlisten] connection accepted from 127.0.0.1:34858 #45 m31101| Tue Nov 27 17:46:36 [conn19] received splitChunk request: { splitChunk: "fooMixed22.barMixed22", keyPattern: { _id: 1.0 }, min: { _id: MinKey }, max: { _id: MaxKey }, from: "test-rs0", splitKeys: [ { _id: 0.0 } ], shardId: "fooMixed22.barMixed22-_id_MinKey", configdb: "ubuntu:29000,ubuntu:29001,ubuntu:29002" } m31101| Tue Nov 27 17:46:36 [conn19] created new distributed lock for fooMixed22.barMixed22 on ubuntu:29000,ubuntu:29001,ubuntu:29002 ( lock timeout : 900000, ping interval : 30000, process : 0 ) m31101| Tue Nov 27 17:46:37 [conn19] distributed lock 'fooMixed22.barMixed22/ubuntu:31101:1354056379:52313605' acquired, ts : 50b542cc53ed56c79f143686 m31101| Tue Nov 27 17:46:37 [conn19] splitChunk accepted at version 1|0 m31101| Tue Nov 27 17:46:37 [conn19] about to log metadata event: { _id: "ubuntu-2012-11-27T22:46:37-11", server: "ubuntu", clientAddr: "127.0.0.1:34710", time: new Date(1354056397370), what: "split", ns: "fooMixed22.barMixed22", details: { before: { min: { _id: MinKey }, max: { _id: MaxKey }, lastmod: Timestamp 1000|0 }, left: { min: { _id: MinKey }, max: { _id: 0.0 }, lastmod: Timestamp 1000|1 }, right: { min: { _id: 0.0 }, max: { _id: MaxKey }, lastmod: Timestamp 1000|2 } } } m31101| Tue Nov 27 17:46:37 [conn44] end connection 127.0.0.1:34853 m31101| Tue Nov 27 17:46:37 [conn43] end connection 127.0.0.1:34854 m31101| Tue Nov 27 17:46:37 [conn19] distributed lock 'fooMixed22.barMixed22/ubuntu:31101:1354056379:52313605' unlocked. m31101| Tue Nov 27 17:46:37 [conn19] command admin.$cmd command: { splitChunk: "fooMixed22.barMixed22", keyPattern: { _id: 1.0 }, min: { _id: MinKey }, max: { _id: MaxKey }, from: "test-rs0", splitKeys: [ { _id: 0.0 } ], shardId: "fooMixed22.barMixed22-_id_MinKey", configdb: "ubuntu:29000,ubuntu:29001,ubuntu:29002" } ntoreturn:1 reslen:37 1616ms m31101| Tue Nov 27 17:46:37 [conn19] received moveChunk request: { moveChunk: "fooMixed22.barMixed22", from: "test-rs0/ubuntu:31100,ubuntu:31101,ubuntu:31102", to: "test-rs1/ubuntu:31200,ubuntu:31201,ubuntu:31202", fromShard: "test-rs0", toShard: "test-rs1", min: { _id: 0.0 }, max: { _id: MaxKey }, maxChunkSizeBytes: 52428800, shardId: "fooMixed22.barMixed22-_id_0.0", configdb: "ubuntu:29000,ubuntu:29001,ubuntu:29002", secondaryThrottle: false } m31101| Tue Nov 27 17:46:37 [conn19] created new distributed lock for fooMixed22.barMixed22 on ubuntu:29000,ubuntu:29001,ubuntu:29002 ( lock timeout : 900000, ping interval : 30000, process : 0 ) m31101| Tue Nov 27 17:46:38 [conn19] distributed lock 'fooMixed22.barMixed22/ubuntu:31101:1354056379:52313605' acquired, ts : 50b542cd53ed56c79f143687 m31101| Tue Nov 27 17:46:38 [conn19] about to log metadata event: { _id: "ubuntu-2012-11-27T22:46:38-12", server: "ubuntu", clientAddr: "127.0.0.1:34710", time: new Date(1354056398448), what: "moveChunk.start", ns: "fooMixed22.barMixed22", details: { min: { _id: 0.0 }, max: { _id: MaxKey }, from: "test-rs0", to: "test-rs1" } } m31101| Tue Nov 27 17:46:38 [initandlisten] connection accepted from 127.0.0.1:34861 #46 m31101| Tue Nov 27 17:46:38 [initandlisten] connection accepted from 127.0.0.1:34862 #47 m31101| Tue Nov 27 17:46:38 [conn19] moveChunk request accepted at version 1|2 m31101| Tue Nov 27 17:46:38 [conn19] moveChunk number of documents: 0 m31101| Tue Nov 27 17:46:38 [initandlisten] connection accepted from 127.0.0.1:34863 #48 m31101| Tue Nov 27 17:46:39 [conn48] end connection 127.0.0.1:34863 m31101| Tue Nov 27 17:46:39 [conn19] moveChunk data transfer progress: { active: false, ns: "fooMixed22.barMixed22", from: "test-rs0/ubuntu:31100,ubuntu:31101,ubuntu:31102", min: { _id: 0.0 }, max: { _id: MaxKey }, state: "fail", errmsg: "assertion db/namespace.cpp:431", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 m31101| Tue Nov 27 17:46:39 [conn19] warning: moveChunk error transferring data caused migration abort: { active: false, ns: "fooMixed22.barMixed22", from: "test-rs0/ubuntu:31100,ubuntu:31101,ubuntu:31102", min: { _id: 0.0 }, max: { _id: MaxKey }, state: "fail", errmsg: "assertion db/namespace.cpp:431", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } m31101| Tue Nov 27 17:46:39 [conn19] distributed lock 'fooMixed22.barMixed22/ubuntu:31101:1354056379:52313605' unlocked. m31101| Tue Nov 27 17:46:39 [conn19] about to log metadata event: { _id: "ubuntu-2012-11-27T22:46:39-13", server: "ubuntu", clientAddr: "127.0.0.1:34710", time: new Date(1354056399841), what: "moveChunk.from", ns: "fooMixed22.barMixed22", details: { min: { _id: 0.0 }, max: { _id: MaxKey }, step1: 0, step2: 639, step3: 0, note: "aborted" } } m31101| Tue Nov 27 17:46:40 [conn19] command admin.$cmd command: { moveChunk: "fooMixed22.barMixed22", from: "test-rs0/ubuntu:31100,ubuntu:31101,ubuntu:31102", to: "test-rs1/ubuntu:31200,ubuntu:31201,ubuntu:31202", fromShard: "test-rs0", toShard: "test-rs1", min: { _id: 0.0 }, max: { _id: MaxKey }, maxChunkSizeBytes: 52428800, shardId: "fooMixed22.barMixed22-_id_0.0", configdb: "ubuntu:29000,ubuntu:29001,ubuntu:29002", secondaryThrottle: false } ntoreturn:1 reslen:370 2075ms m31101| Tue Nov 27 17:46:40 [conn19] received splitChunk request: { splitChunk: "fooMixed22.barMixed22", keyPattern: { _id: 1.0 }, min: { _id: MinKey }, max: { _id: 0.0 }, from: "test-rs0", splitKeys: [ { _id: -300.0 } ], shardId: "fooMixed22.barMixed22-_id_MinKey", configdb: "ubuntu:29000,ubuntu:29001,ubuntu:29002" } m31101| Tue Nov 27 17:46:40 [conn19] created new distributed lock for fooMixed22.barMixed22 on ubuntu:29000,ubuntu:29001,ubuntu:29002 ( lock timeout : 900000, ping interval : 30000, process : 0 ) m31101| Tue Nov 27 17:46:40 [conn46] end connection 127.0.0.1:34861 m31101| Tue Nov 27 17:46:40 [conn47] end connection 127.0.0.1:34862 m31101| Tue Nov 27 17:46:40 [conn19] distributed lock 'fooMixed22.barMixed22/ubuntu:31101:1354056379:52313605' acquired, ts : 50b542d053ed56c79f143688 m31101| Tue Nov 27 17:46:40 [conn19] splitChunk accepted at version 1|2 m31101| Tue Nov 27 17:46:40 [conn19] about to log metadata event: { _id: "ubuntu-2012-11-27T22:46:40-14", server: "ubuntu", clientAddr: "127.0.0.1:34710", time: new Date(1354056400972), what: "split", ns: "fooMixed22.barMixed22", details: { before: { min: { _id: MinKey }, max: { _id: 0.0 }, lastmod: Timestamp 1000|1 }, left: { min: { _id: MinKey }, max: { _id: -300.0 }, lastmod: Timestamp 1000|3 }, right: { min: { _id: -300.0 }, max: { _id: 0.0 }, lastmod: Timestamp 1000|4 } } } m31101| Tue Nov 27 17:46:41 [conn19] distributed lock 'fooMixed22.barMixed22/ubuntu:31101:1354056379:52313605' unlocked. m31101| Tue Nov 27 17:46:41 [conn19] command admin.$cmd command: { splitChunk: "fooMixed22.barMixed22", keyPattern: { _id: 1.0 }, min: { _id: MinKey }, max: { _id: 0.0 }, from: "test-rs0", splitKeys: [ { _id: -300.0 } ], shardId: "fooMixed22.barMixed22-_id_MinKey", configdb: "ubuntu:29000,ubuntu:29001,ubuntu:29002" } ntoreturn:1 reslen:37 1291ms m31101| Tue Nov 27 17:46:41 [conn19] received splitChunk request: { splitChunk: "fooMixed22.barMixed22", keyPattern: { _id: 1.0 }, min: { _id: -300.0 }, max: { _id: 0.0 }, from: "test-rs0", splitKeys: [ { _id: -200.0 } ], shardId: "fooMixed22.barMixed22-_id_-300.0", configdb: "ubuntu:29000,ubuntu:29001,ubuntu:29002" } m31101| Tue Nov 27 17:46:41 [conn19] created new distributed lock for fooMixed22.barMixed22 on ubuntu:29000,ubuntu:29001,ubuntu:29002 ( lock timeout : 900000, ping interval : 30000, process : 0 ) m31101| Tue Nov 27 17:46:41 [initandlisten] connection accepted from 127.0.0.1:34867 #49 m31101| Tue Nov 27 17:46:41 [initandlisten] connection accepted from 127.0.0.1:34868 #50 m31101| Tue Nov 27 17:46:41 [conn19] distributed lock 'fooMixed22.barMixed22/ubuntu:31101:1354056379:52313605' acquired, ts : 50b542d153ed56c79f143689 m31101| Tue Nov 27 17:46:41 [conn19] splitChunk accepted at version 1|4 m31101| Tue Nov 27 17:46:42 [conn19] about to log metadata event: { _id: "ubuntu-2012-11-27T22:46:42-15", server: "ubuntu", clientAddr: "127.0.0.1:34710", time: new Date(1354056402081), what: "split", ns: "fooMixed22.barMixed22", details: { before: { min: { _id: -300.0 }, max: { _id: 0.0 }, lastmod: Timestamp 1000|4 }, left: { min: { _id: -300.0 }, max: { _id: -200.0 }, lastmod: Timestamp 1000|5 }, right: { min: { _id: -200.0 }, max: { _id: 0.0 }, lastmod: Timestamp 1000|6 } } } m31101| Tue Nov 27 17:46:42 [conn19] distributed lock 'fooMixed22.barMixed22/ubuntu:31101:1354056379:52313605' unlocked. m31101| Tue Nov 27 17:46:42 [conn19] command admin.$cmd command: { splitChunk: "fooMixed22.barMixed22", keyPattern: { _id: 1.0 }, min: { _id: -300.0 }, max: { _id: 0.0 }, from: "test-rs0", splitKeys: [ { _id: -200.0 } ], shardId: "fooMixed22.barMixed22-_id_-300.0", configdb: "ubuntu:29000,ubuntu:29001,ubuntu:29002" } ntoreturn:1 reslen:37 1341ms m31101| Tue Nov 27 17:46:42 [conn19] received splitChunk request: { splitChunk: "fooMixed22.barMixed22", keyPattern: { _id: 1.0 }, min: { _id: -200.0 }, max: { _id: 0.0 }, from: "test-rs0", splitKeys: [ { _id: -100.0 } ], shardId: "fooMixed22.barMixed22-_id_-200.0", configdb: "ubuntu:29000,ubuntu:29001,ubuntu:29002" } m31101| Tue Nov 27 17:46:42 [conn19] created new distributed lock for fooMixed22.barMixed22 on ubuntu:29000,ubuntu:29001,ubuntu:29002 ( lock timeout : 900000, ping interval : 30000, process : 0 ) m31101| Tue Nov 27 17:46:43 [conn19] distributed lock 'fooMixed22.barMixed22/ubuntu:31101:1354056379:52313605' acquired, ts : 50b542d253ed56c79f14368a m31101| Tue Nov 27 17:46:43 [conn19] splitChunk accepted at version 1|6 m31101| Tue Nov 27 17:46:43 [conn19] about to log metadata event: { _id: "ubuntu-2012-11-27T22:46:43-16", server: "ubuntu", clientAddr: "127.0.0.1:34710", time: new Date(1354056403484), what: "split", ns: "fooMixed22.barMixed22", details: { before: { min: { _id: -200.0 }, max: { _id: 0.0 }, lastmod: Timestamp 1000|6 }, left: { min: { _id: -200.0 }, max: { _id: -100.0 }, lastmod: Timestamp 1000|7 }, right: { min: { _id: -100.0 }, max: { _id: 0.0 }, lastmod: Timestamp 1000|8 } } } m31101| Tue Nov 27 17:46:43 [conn50] end connection 127.0.0.1:34868 m31101| Tue Nov 27 17:46:43 [conn49] end connection 127.0.0.1:34867 m31101| Tue Nov 27 17:46:43 [conn19] distributed lock 'fooMixed22.barMixed22/ubuntu:31101:1354056379:52313605' unlocked. m31101| Tue Nov 27 17:46:43 [conn19] command admin.$cmd command: { splitChunk: "fooMixed22.barMixed22", keyPattern: { _id: 1.0 }, min: { _id: -200.0 }, max: { _id: 0.0 }, from: "test-rs0", splitKeys: [ { _id: -100.0 } ], shardId: "fooMixed22.barMixed22-_id_-200.0", configdb: "ubuntu:29000,ubuntu:29001,ubuntu:29002" } ntoreturn:1 reslen:37 1165ms m31101| Tue Nov 27 17:46:43 [conn19] received splitChunk request: { splitChunk: "fooMixed22.barMixed22", keyPattern: { _id: 1.0 }, min: { _id: 0.0 }, max: { _id: MaxKey }, from: "test-rs0", splitKeys: [ { _id: 100.0 } ], shardId: "fooMixed22.barMixed22-_id_0.0", configdb: "ubuntu:29000,ubuntu:29001,ubuntu:29002" } m31101| Tue Nov 27 17:46:43 [conn19] created new distributed lock for fooMixed22.barMixed22 on ubuntu:29000,ubuntu:29001,ubuntu:29002 ( lock timeout : 900000, ping interval : 30000, process : 0 ) m31101| Tue Nov 27 17:46:44 [conn19] distributed lock 'fooMixed22.barMixed22/ubuntu:31101:1354056379:52313605' acquired, ts : 50b542d353ed56c79f14368b m31101| Tue Nov 27 17:46:44 [conn19] splitChunk accepted at version 1|8 m31101| Tue Nov 27 17:46:44 [initandlisten] connection accepted from 127.0.0.1:34871 #51 m31101| Tue Nov 27 17:46:44 [initandlisten] connection accepted from 127.0.0.1:34872 #52 m31101| Tue Nov 27 17:46:44 [conn19] about to log metadata event: { _id: "ubuntu-2012-11-27T22:46:44-17", server: "ubuntu", clientAddr: "127.0.0.1:34710", time: new Date(1354056404628), what: "split", ns: "fooMixed22.barMixed22", details: { before: { min: { _id: 0.0 }, max: { _id: MaxKey }, lastmod: Timestamp 1000|2 }, left: { min: { _id: 0.0 }, max: { _id: 100.0 }, lastmod: Timestamp 1000|9 }, right: { min: { _id: 100.0 }, max: { _id: MaxKey }, lastmod: Timestamp 1000|10 } } } m31101| Tue Nov 27 17:46:45 [initandlisten] connection accepted from 127.0.0.1:34873 #53 m31101| Tue Nov 27 17:46:45 [conn19] distributed lock 'fooMixed22.barMixed22/ubuntu:31101:1354056379:52313605' unlocked. m31101| Tue Nov 27 17:46:45 [conn19] command admin.$cmd command: { splitChunk: "fooMixed22.barMixed22", keyPattern: { _id: 1.0 }, min: { _id: 0.0 }, max: { _id: MaxKey }, from: "test-rs0", splitKeys: [ { _id: 100.0 } ], shardId: "fooMixed22.barMixed22-_id_0.0", configdb: "ubuntu:29000,ubuntu:29001,ubuntu:29002" } ntoreturn:1 reslen:37 1918ms m31101| Tue Nov 27 17:46:45 [conn19] received splitChunk request: { splitChunk: "fooMixed22.barMixed22", keyPattern: { _id: 1.0 }, min: { _id: 100.0 }, max: { _id: MaxKey }, from: "test-rs0", splitKeys: [ { _id: 200.0 } ], shardId: "fooMixed22.barMixed22-_id_100.0", configdb: "ubuntu:29000,ubuntu:29001,ubuntu:29002" } m31101| Tue Nov 27 17:46:45 [conn19] created new distributed lock for fooMixed22.barMixed22 on ubuntu:29000,ubuntu:29001,ubuntu:29002 ( lock timeout : 900000, ping interval : 30000, process : 0 ) m31101| Tue Nov 27 17:46:46 [conn19] distributed lock 'fooMixed22.barMixed22/ubuntu:31101:1354056379:52313605' acquired, ts : 50b542d553ed56c79f14368c m31101| Tue Nov 27 17:46:46 [conn19] splitChunk accepted at version 1|10 m31101| Tue Nov 27 17:46:46 [conn19] about to log metadata event: { _id: "ubuntu-2012-11-27T22:46:46-18", server: "ubuntu", clientAddr: "127.0.0.1:34710", time: new Date(1354056406471), what: "split", ns: "fooMixed22.barMixed22", details: { before: { min: { _id: 100.0 }, max: { _id: MaxKey }, lastmod: Timestamp 1000|10 }, left: { min: { _id: 100.0 }, max: { _id: 200.0 }, lastmod: Timestamp 1000|11 }, right: { min: { _id: 200.0 }, max: { _id: MaxKey }, lastmod: Timestamp 1000|12 } } } m31101| Tue Nov 27 17:46:46 [conn51] end connection 127.0.0.1:34871 m31101| Tue Nov 27 17:46:46 [conn52] end connection 127.0.0.1:34872 m31101| Tue Nov 27 17:46:47 [conn19] distributed lock 'fooMixed22.barMixed22/ubuntu:31101:1354056379:52313605' unlocked. m31101| Tue Nov 27 17:46:47 [conn19] command admin.$cmd command: { splitChunk: "fooMixed22.barMixed22", keyPattern: { _id: 1.0 }, min: { _id: 100.0 }, max: { _id: MaxKey }, from: "test-rs0", splitKeys: [ { _id: 200.0 } ], shardId: "fooMixed22.barMixed22-_id_100.0", configdb: "ubuntu:29000,ubuntu:29001,ubuntu:29002" } ntoreturn:1 reslen:37 1483ms m31101| Tue Nov 27 17:46:47 [conn19] received splitChunk request: { splitChunk: "fooMixed22.barMixed22", keyPattern: { _id: 1.0 }, min: { _id: 200.0 }, max: { _id: MaxKey }, from: "test-rs0", splitKeys: [ { _id: 300.0 } ], shardId: "fooMixed22.barMixed22-_id_200.0", configdb: "ubuntu:29000,ubuntu:29001,ubuntu:29002" } m31101| Tue Nov 27 17:46:47 [conn19] created new distributed lock for fooMixed22.barMixed22 on ubuntu:29000,ubuntu:29001,ubuntu:29002 ( lock timeout : 900000, ping interval : 30000, process : 0 ) m31101| Tue Nov 27 17:46:47 [initandlisten] connection accepted from 127.0.0.1:34881 #54 m31101| Tue Nov 27 17:46:47 [initandlisten] connection accepted from 127.0.0.1:34882 #55 m31101| Tue Nov 27 17:46:47 [conn19] distributed lock 'fooMixed22.barMixed22/ubuntu:31101:1354056379:52313605' acquired, ts : 50b542d753ed56c79f14368d m31101| Tue Nov 27 17:46:47 [conn19] splitChunk accepted at version 1|12 m31101| Tue Nov 27 17:46:47 [conn19] about to log metadata event: { _id: "ubuntu-2012-11-27T22:46:47-19", server: "ubuntu", clientAddr: "127.0.0.1:34710", time: new Date(1354056407958), what: "split", ns: "fooMixed22.barMixed22", details: { before: { min: { _id: 200.0 }, max: { _id: MaxKey }, lastmod: Timestamp 1000|12 }, left: { min: { _id: 200.0 }, max: { _id: 300.0 }, lastmod: Timestamp 1000|13 }, right: { min: { _id: 300.0 }, max: { _id: MaxKey }, lastmod: Timestamp 1000|14 } } } m31101| Tue Nov 27 17:46:48 [conn19] distributed lock 'fooMixed22.barMixed22/ubuntu:31101:1354056379:52313605' unlocked. m31101| Tue Nov 27 17:46:48 [conn19] command admin.$cmd command: { splitChunk: "fooMixed22.barMixed22", keyPattern: { _id: 1.0 }, min: { _id: 200.0 }, max: { _id: MaxKey }, from: "test-rs0", splitKeys: [ { _id: 300.0 } ], shardId: "fooMixed22.barMixed22-_id_200.0", configdb: "ubuntu:29000,ubuntu:29001,ubuntu:29002" } ntoreturn:1 reslen:37 1218ms m31101| Tue Nov 27 17:46:48 [conn19] received moveChunk request: { moveChunk: "fooMixed22.barMixed22", from: "test-rs0/ubuntu:31100,ubuntu:31101,ubuntu:31102", to: "test-rs1/ubuntu:31200,ubuntu:31201,ubuntu:31202", fromShard: "test-rs0", toShard: "test-rs1", min: { _id: -200.0 }, max: { _id: -100.0 }, maxChunkSizeBytes: 52428800, shardId: "fooMixed22.barMixed22-_id_-200.0", configdb: "ubuntu:29000,ubuntu:29001,ubuntu:29002", secondaryThrottle: false } m31101| Tue Nov 27 17:46:48 [conn19] created new distributed lock for fooMixed22.barMixed22 on ubuntu:29000,ubuntu:29001,ubuntu:29002 ( lock timeout : 900000, ping interval : 30000, process : 0 ) m31101| Tue Nov 27 17:46:48 [conn19] distributed lock 'fooMixed22.barMixed22/ubuntu:31101:1354056379:52313605' acquired, ts : 50b542d853ed56c79f14368e m31101| Tue Nov 27 17:46:48 [conn19] about to log metadata event: { _id: "ubuntu-2012-11-27T22:46:48-20", server: "ubuntu", clientAddr: "127.0.0.1:34710", time: new Date(1354056408951), what: "moveChunk.start", ns: "fooMixed22.barMixed22", details: { min: { _id: -200.0 }, max: { _id: -100.0 }, from: "test-rs0", to: "test-rs1" } } m31101| Tue Nov 27 17:46:49 [conn19] moveChunk request accepted at version 1|14 m31101| Tue Nov 27 17:46:49 [conn19] moveChunk number of documents: 0 m31101| Tue Nov 27 17:46:49 [initandlisten] connection accepted from 127.0.0.1:34885 #56 m31101| Tue Nov 27 17:46:49 [conn56] end connection 127.0.0.1:34885 m31101| Tue Nov 27 17:46:49 [conn54] end connection 127.0.0.1:34882 m31101| Tue Nov 27 17:46:49 [conn55] end connection 127.0.0.1:34881 m31101| Tue Nov 27 17:46:50 [conn19] moveChunk data transfer progress: { active: false, ns: "fooMixed22.barMixed22", from: "test-rs0/ubuntu:31100,ubuntu:31101,ubuntu:31102", min: { _id: -200.0 }, max: { _id: -100.0 }, state: "fail", errmsg: "assertion db/namespace.cpp:431", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 m31101| Tue Nov 27 17:46:50 [conn19] warning: moveChunk error transferring data caused migration abort: { active: false, ns: "fooMixed22.barMixed22", from: "test-rs0/ubuntu:31100,ubuntu:31101,ubuntu:31102", min: { _id: -200.0 }, max: { _id: -100.0 }, state: "fail", errmsg: "assertion db/namespace.cpp:431", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } m31101| Tue Nov 27 17:46:50 [conn19] distributed lock 'fooMixed22.barMixed22/ubuntu:31101:1354056379:52313605' unlocked. m31101| Tue Nov 27 17:46:50 [conn19] about to log metadata event: { _id: "ubuntu-2012-11-27T22:46:50-21", server: "ubuntu", clientAddr: "127.0.0.1:34710", time: new Date(1354056410429), what: "moveChunk.from", ns: "fooMixed22.barMixed22", details: { min: { _id: -200.0 }, max: { _id: -100.0 }, step1: 0, step2: 706, step3: 0, note: "aborted" } } m31101| Tue Nov 27 17:46:50 [initandlisten] connection accepted from 127.0.0.1:34888 #57 m31101| Tue Nov 27 17:46:50 [initandlisten] connection accepted from 127.0.0.1:34889 #58 m31101| Tue Nov 27 17:46:50 [LockPinger] SyncClusterConnection connecting to [ubuntu:29000] m31101| Tue Nov 27 17:46:50 [LockPinger] SyncClusterConnection connecting to [ubuntu:29001] m31101| Tue Nov 27 17:46:50 [LockPinger] SyncClusterConnection connecting to [ubuntu:29002] m31101| Tue Nov 27 17:46:50 [conn19] command admin.$cmd command: { moveChunk: "fooMixed22.barMixed22", from: "test-rs0/ubuntu:31100,ubuntu:31101,ubuntu:31102", to: "test-rs1/ubuntu:31200,ubuntu:31201,ubuntu:31202", fromShard: "test-rs0", toShard: "test-rs1", min: { _id: -200.0 }, max: { _id: -100.0 }, maxChunkSizeBytes: 52428800, shardId: "fooMixed22.barMixed22-_id_-200.0", configdb: "ubuntu:29000,ubuntu:29001,ubuntu:29002", secondaryThrottle: false } ntoreturn:1 reslen:378 2165ms m31101| Tue Nov 27 17:46:50 [conn17] end connection 127.0.0.1:34704 m31101| Tue Nov 27 17:46:50 [conn19] end connection 127.0.0.1:34710 m31101| Tue Nov 27 17:46:50 [conn45] end connection 127.0.0.1:34858 m31101| Tue Nov 27 17:46:50 [conn53] end connection 127.0.0.1:34873 m31101| Tue Nov 27 17:46:52 [clientcursormon] mem (MB) res:18 virt:3054 mapped:160 m31101| Tue Nov 27 17:46:52 [initandlisten] connection accepted from 127.0.0.1:34911 #59 m31101| Tue Nov 27 17:46:52 [conn37] end connection 127.0.0.1:34832 m31101| Tue Nov 27 17:46:52 [conn22] end connection 127.0.0.1:34768 m31101| Tue Nov 27 17:46:52 [conn20] end connection 127.0.0.1:34736 m31101| Tue Nov 27 17:46:52 [conn23] end connection 127.0.0.1:34778 m31101| Tue Nov 27 17:46:52 [initandlisten] connection accepted from 127.0.0.1:34915 #60 m31101| Tue Nov 27 17:46:52 [initandlisten] connection accepted from 127.0.0.1:34917 #61 m31101| Tue Nov 27 17:46:52 [conn57] end connection 127.0.0.1:34888 m31101| Tue Nov 27 17:46:52 [conn58] end connection 127.0.0.1:34889 m31101| Tue Nov 27 17:46:53 [conn31] end connection 127.0.0.1:34823 m31101| Tue Nov 27 17:46:53 [initandlisten] connection accepted from 127.0.0.1:34936 #62 m31101| Tue Nov 27 17:46:53 [initandlisten] connection accepted from 127.0.0.1:34944 #63 m31101| Tue Nov 27 17:46:53 [conn36] end connection 127.0.0.1:34831 m31101| Tue Nov 27 17:46:53 [initandlisten] connection accepted from 127.0.0.1:34948 #64 m31101| Tue Nov 27 17:46:53 [initandlisten] connection accepted from 127.0.0.1:34949 #65 m31101| Tue Nov 27 17:46:53 [conn65] end connection 127.0.0.1:34949 m31101| Tue Nov 27 17:46:53 [initandlisten] connection accepted from 127.0.0.1:34950 #66 m31101| Tue Nov 27 17:46:53 [initandlisten] connection accepted from 127.0.0.1:34952 #67 m31101| Tue Nov 27 17:46:54 [rsHealthPoll] DBClientCursor::init call() failed m31101| Tue Nov 27 17:46:54 [rsHealthPoll] replSet info ubuntu:31100 is down (or slow to respond): DBClientBase::findN: transport error: ubuntu:31100 query: { replSetHeartbeat: "test-rs0", v: 1, pv: 1, checkEmpty: false, from: "ubuntu:31101" } m31101| Tue Nov 27 17:46:54 [rsHealthPoll] replSet member ubuntu:31100 is now in state DOWN m31101| Tue Nov 27 17:46:54 [initandlisten] connection accepted from 127.0.0.1:34967 #68 m31101| Tue Nov 27 17:46:55 [conn64] end connection 127.0.0.1:34948 m31101| Tue Nov 27 17:46:56 [rsHealthPoll] replSet member ubuntu:31100 is up m31101| Tue Nov 27 17:46:56 [rsHealthPoll] replSet member ubuntu:31100 is now in state SECONDARY m31101| Tue Nov 27 17:46:56 [initandlisten] connection accepted from 127.0.0.1:34971 #69 m31101| Tue Nov 27 17:46:56 [conn62] end connection 127.0.0.1:34936 m31101| Tue Nov 27 17:46:56 [conn69] end connection 127.0.0.1:34971 m31101| Tue Nov 27 17:46:58 [rsHealthPoll] DBClientCursor::init call() failed m31101| Tue Nov 27 17:46:58 [rsHealthPoll] replSet info ubuntu:31102 is down (or slow to respond): DBClientBase::findN: transport error: ubuntu:31102 query: { replSetHeartbeat: "test-rs0", v: 1, pv: 1, checkEmpty: false, from: "ubuntu:31101" } m31101| Tue Nov 27 17:46:58 [rsHealthPoll] replSet member ubuntu:31102 is now in state DOWN m31101| Tue Nov 27 17:46:59 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed m31101| Tue Nov 27 17:46:59 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed m31101| Tue Nov 27 17:46:59 [initandlisten] connection accepted from 127.0.0.1:34979 #70 m31101| Tue Nov 27 17:47:00 [rsHealthPoll] replSet member ubuntu:31102 is up m31101| Tue Nov 27 17:47:00 [rsHealthPoll] replSet member ubuntu:31102 is now in state STARTUP2 m31101| Tue Nov 27 17:47:01 [initandlisten] connection accepted from 127.0.0.1:34987 #71 m31101| Tue Nov 27 17:47:02 [rsHealthPoll] replSet member ubuntu:31102 is now in state SECONDARY m31101| Tue Nov 27 17:47:02 [initandlisten] connection accepted from 127.0.0.1:34988 #72 m31101| Tue Nov 27 17:47:03 [conn71] end connection 127.0.0.1:34987 m31101| Tue Nov 27 17:47:03 [initandlisten] connection accepted from 127.0.0.1:34992 #73 m31101| Tue Nov 27 17:47:03 [conn3] replSet info stepping down as primary secs=50 m31101| Tue Nov 27 17:47:03 [conn3] replSet relinquishing primary state m31101| Tue Nov 27 17:47:03 [conn3] replSet SECONDARY m31101| Tue Nov 27 17:47:03 [conn3] replSet closing client sockets after reqlinquishing primary m31101| Tue Nov 27 17:47:03 [conn24] end connection 127.0.0.1:34788 m31101| Tue Nov 27 17:47:03 [conn25] end connection 127.0.0.1:34791 m31101| Tue Nov 27 17:47:03 [initandlisten] connection accepted from 127.0.0.1:34993 #74 m31101| Tue Nov 27 17:47:03 [conn3] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:34610] m31101| Tue Nov 27 17:47:03 [conn1] end connection 127.0.0.1:34606 m31101| Tue Nov 27 17:47:04 [rsHealthPoll] replSet info ubuntu:31100 is down (or slow to respond): socket exception m31101| Tue Nov 27 17:47:04 [rsHealthPoll] replSet member ubuntu:31100 is now in state DOWN m31101| Tue Nov 27 17:47:04 [rsHealthPoll] replSet info ubuntu:31102 is down (or slow to respond): socket exception m31101| Tue Nov 27 17:47:04 [rsHealthPoll] replSet member ubuntu:31102 is now in state DOWN m31101| Tue Nov 27 17:47:04 [conn72] end connection 127.0.0.1:34988 m31101| Tue Nov 27 17:47:04 [initandlisten] connection accepted from 127.0.0.1:34994 #75 m31101| Tue Nov 27 17:47:05 [conn73] end connection 127.0.0.1:34992 m31101| Tue Nov 27 17:47:05 [initandlisten] connection accepted from 127.0.0.1:34995 #76 m31101| Tue Nov 27 17:47:06 [rsHealthPoll] replSet member ubuntu:31100 is up m31101| Tue Nov 27 17:47:06 [rsHealthPoll] replSet member ubuntu:31100 is now in state SECONDARY m31101| Tue Nov 27 17:47:06 [rsHealthPoll] replSet member ubuntu:31102 is up m31101| Tue Nov 27 17:47:06 [rsHealthPoll] replSet member ubuntu:31102 is now in state SECONDARY m31101| Tue Nov 27 17:47:06 [conn75] end connection 127.0.0.1:34994 m31101| Tue Nov 27 17:47:08 [conn68] end connection 127.0.0.1:34967 m31101| Tue Nov 27 17:47:08 [initandlisten] connection accepted from 127.0.0.1:35001 #77 m31101| Tue Nov 27 17:47:09 [ReplicaSetMonitorWatcher] Socket say send() errno:9 Bad file descriptor 127.0.0.1:31101 m31101| Tue Nov 27 17:47:09 [ReplicaSetMonitorWatcher] trying reconnect to ubuntu:31100 m31101| Tue Nov 27 17:47:09 [ReplicaSetMonitorWatcher] reconnect ubuntu:31100 ok m31101| Tue Nov 27 17:47:09 [ReplicaSetMonitorWatcher] Socket say send() errno:9 Bad file descriptor 127.0.0.1:31100 m31101| Tue Nov 27 17:47:09 [ReplicaSetMonitorWatcher] trying reconnect to ubuntu:31101 m31101| Tue Nov 27 17:47:09 [initandlisten] connection accepted from 127.0.0.1:35005 #78 m31101| Tue Nov 27 17:47:09 [ReplicaSetMonitorWatcher] reconnect ubuntu:31101 ok m31101| Tue Nov 27 17:47:09 [ReplicaSetMonitorWatcher] Socket say send() errno:9 Bad file descriptor 127.0.0.1:31101 m31101| Tue Nov 27 17:47:09 [ReplicaSetMonitorWatcher] trying reconnect to ubuntu:31102 m31101| Tue Nov 27 17:47:09 [ReplicaSetMonitorWatcher] reconnect ubuntu:31102 ok m31101| Tue Nov 27 17:47:09 [ReplicaSetMonitorWatcher] Socket say send() errno:9 Bad file descriptor 127.0.0.1:31102 m31101| Tue Nov 27 17:47:10 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: ubuntu:31100 { setName: "test-rs0", ismaster: false, secondary: true, hosts: [ "ubuntu:31100", "ubuntu:31102", "ubuntu:31101" ], me: "ubuntu:31100", maxBsonObjectSize: 16777216, localTime: new Date(1354056430676), ok: 1.0 } m31101| Tue Nov 27 17:47:10 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: ubuntu:31101 { setName: "test-rs0", ismaster: false, secondary: true, hosts: [ "ubuntu:31101", "ubuntu:31102", "ubuntu:31100" ], me: "ubuntu:31101", maxBsonObjectSize: 16777216, ok: 1.0 } m31101| Tue Nov 27 17:47:10 [initandlisten] connection accepted from 127.0.0.1:35009 #79 m31101| Tue Nov 27 17:47:10 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: ubuntu:31102 { setName: "test-rs0", ismaster: false, secondary: true, hosts: [ "ubuntu:31102", "ubuntu:31101", "ubuntu:31100" ], me: "ubuntu:31102", maxBsonObjectSize: 16777216, localTime: new Date(1354056430678), ok: 1.0 } m31101| Tue Nov 27 17:47:10 [conn77] replSet info voting yea for ubuntu:31100 (0) m31101| Tue Nov 27 17:47:11 [ReplicaSetMonitorWatcher] Socket say send() errno:9 Bad file descriptor 127.0.0.1:31202 m31101| Tue Nov 27 17:47:11 [ReplicaSetMonitorWatcher] Socket say send() errno:9 Bad file descriptor 127.0.0.1:31200 m31101| Tue Nov 27 17:47:11 [ReplicaSetMonitorWatcher] Socket say send() errno:9 Bad file descriptor 127.0.0.1:31201 m31101| Tue Nov 27 17:47:11 [ReplicaSetMonitorWatcher] trying reconnect to ubuntu:31202 m31101| Tue Nov 27 17:47:11 [ReplicaSetMonitorWatcher] reconnect ubuntu:31202 ok m31101| Tue Nov 27 17:47:11 [ReplicaSetMonitorWatcher] Socket say send() errno:9 Bad file descriptor 127.0.0.1:31202 m31101| Tue Nov 27 17:47:11 got kill or ctrl c or hup signal 15 (Terminated), will terminate after current cmd ends m31101| Tue Nov 27 17:47:11 [interruptThread] now exiting m31101| Tue Nov 27 17:47:11 dbexit: m31101| Tue Nov 27 17:47:11 [interruptThread] shutdown: going to close listening sockets... m31101| Tue Nov 27 17:47:11 [interruptThread] closing listening socket: 64 m31101| Tue Nov 27 17:47:11 [interruptThread] closing listening socket: 65 m31101| Tue Nov 27 17:47:11 [interruptThread] closing listening socket: 67 m31101| Tue Nov 27 17:47:11 [interruptThread] removing socket file: /tmp/mongodb-31101.sock m31101| Tue Nov 27 17:47:11 [interruptThread] shutdown: going to flush diaglog... m31101| Tue Nov 27 17:47:11 [interruptThread] shutdown: going to close sockets... m31101| Tue Nov 27 17:47:11 [interruptThread] shutdown: waiting for fs preallocator... m31101| Tue Nov 27 17:47:11 [interruptThread] shutdown: lock for final commit... m31101| Tue Nov 27 17:47:11 [interruptThread] shutdown: final commit... m31101| Tue Nov 27 17:47:11 [conn78] end connection 127.0.0.1:35005 m31101| Tue Nov 27 17:47:11 [conn79] end connection 127.0.0.1:35009 m31101| Tue Nov 27 17:47:11 [interruptThread] shutdown: closing all files... m31101| Tue Nov 27 17:47:11 [interruptThread] closeAllFiles() finished m31101| Tue Nov 27 17:47:11 [interruptThread] journalCleanup... m31101| Tue Nov 27 17:47:11 [interruptThread] removeJournalFiles m31101| Tue Nov 27 17:47:11 [interruptThread] shutdown: removing fs lock... m31101| Tue Nov 27 17:47:11 dbexit: really exiting now m31101| note: noprealloc may hurt performance in many applications m31101| Tue Nov 27 17:47:12 [initandlisten] MongoDB starting : pid=13510 port=31101 dbpath=/data/db/test-rs0-1 64-bit host=ubuntu m31101| Tue Nov 27 17:47:12 [initandlisten] db version v2.2.2-rc1, pdfile version 4.5 m31101| Tue Nov 27 17:47:12 [initandlisten] git version: f3700f4ec30ee51a7b242166b8f6aa487dd13846 m31101| Tue Nov 27 17:47:12 [initandlisten] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49 m31101| Tue Nov 27 17:47:12 [initandlisten] options: { dbpath: "/data/db/test-rs0-1", noprealloc: true, nopreallocj: true, oplogSize: 40, port: 31101, replSet: "test-rs0", rest: true, smallfiles: true } m31101| Tue Nov 27 17:47:13 [initandlisten] journal dir=/data/db/test-rs0-1/journal m31101| Tue Nov 27 17:47:13 [initandlisten] recover : no journal files present, no recovery needed m31101| Tue Nov 27 17:47:13 [websvr] admin web console waiting for connections on port 32101 m31101| Tue Nov 27 17:47:13 [initandlisten] waiting for connections on port 31101 m31101| Tue Nov 27 17:47:13 [rsStart] replSet I am ubuntu:31101 m31101| Tue Nov 27 17:47:13 [rsStart] replSet STARTUP2 m31101| Tue Nov 27 17:47:13 [rsHealthPoll] replSet member ubuntu:31102 is up m31101| Tue Nov 27 17:47:13 [rsHealthPoll] replSet member ubuntu:31102 is now in state SECONDARY m31101| Tue Nov 27 17:47:13 [rsHealthPoll] replSet member ubuntu:31100 is up m31101| Tue Nov 27 17:47:13 [rsHealthPoll] replSet member ubuntu:31100 is now in state PRIMARY m31101| Tue Nov 27 17:47:13 [rsSync] replSet initial sync pending m31101| Tue Nov 27 17:47:13 [rsSync] replSet initial sync need a member to be primary or secondary to do our initial sync m31101| Tue Nov 27 17:47:13 [initandlisten] connection accepted from 127.0.0.1:35031 #1 (1 connection now open) m31101| Tue Nov 27 17:47:13 [initandlisten] connection accepted from 127.0.0.1:35033 #2 (2 connections now open) m31101| Tue Nov 27 17:47:14 [initandlisten] connection accepted from 127.0.0.1:35037 #3 (3 connections now open) m31101| Tue Nov 27 17:47:15 [initandlisten] connection accepted from 127.0.0.1:35040 #4 (4 connections now open) m31101| Tue Nov 27 17:47:22 [initandlisten] connection accepted from 127.0.0.1:35045 #5 (5 connections now open) m31101| Tue Nov 27 17:47:22 [initandlisten] connection accepted from 127.0.0.1:35047 #6 (6 connections now open) m31101| Tue Nov 27 17:47:22 [initandlisten] connection accepted from 127.0.0.1:35049 #7 (7 connections now open) m31101| Tue Nov 27 17:47:23 [initandlisten] connection accepted from 127.0.0.1:35052 #8 (8 connections now open) m31101| Tue Nov 27 17:47:27 [conn4] end connection 127.0.0.1:35040 (7 connections now open) m31101| Tue Nov 27 17:47:27 [initandlisten] connection accepted from 127.0.0.1:35059 #9 (8 connections now open) m31101| Tue Nov 27 17:47:29 [rsSync] replSet initial sync pending m31101| Tue Nov 27 17:47:29 [rsSync] replSet syncing to: ubuntu:31100 m31101| Tue Nov 27 17:47:29 [rsSync] replSet initial sync drop all databases m31101| Tue Nov 27 17:47:29 [rsSync] dropAllDatabasesExceptLocal 5 m31101| Tue Nov 27 17:47:29 [rsSync] removeJournalFiles m31101| Tue Nov 27 17:47:29 [rsSync] removeJournalFiles m31101| Tue Nov 27 17:47:29 [rsSync] removeJournalFiles m31101| Tue Nov 27 17:47:29 [rsSync] removeJournalFiles m31101| Tue Nov 27 17:47:29 [rsSync] replSet initial sync clone all databases m31101| Tue Nov 27 17:47:29 [rsSync] replSet initial sync cloning db: foo20 m31101| Tue Nov 27 17:47:29 [FileAllocator] allocating new datafile /data/db/test-rs0-1/foo20.ns, filling with zeroes... m31101| Tue Nov 27 17:47:29 [FileAllocator] creating directory /data/db/test-rs0-1/_tmp m31101| Tue Nov 27 17:47:29 [FileAllocator] done allocating datafile /data/db/test-rs0-1/foo20.ns, size: 16MB, took 0.409 secs m31101| Tue Nov 27 17:47:29 [FileAllocator] allocating new datafile /data/db/test-rs0-1/foo20.0, filling with zeroes... m31101| Tue Nov 27 17:47:30 [FileAllocator] done allocating datafile /data/db/test-rs0-1/foo20.0, size: 16MB, took 0.327 secs m31101| Tue Nov 27 17:47:30 [rsSync] build index foo20.bar20 { _id: 1 } m31101| Tue Nov 27 17:47:30 [rsSync] fastBuildIndex dupsToDrop:0 m31101| Tue Nov 27 17:47:30 [rsSync] build index done. scanned 0 total records. 0 secs m31101| Tue Nov 27 17:47:30 [rsSync] replSet initial sync cloning db: admin m31101| Tue Nov 27 17:47:30 [FileAllocator] allocating new datafile /data/db/test-rs0-1/admin.ns, filling with zeroes... m31101| Tue Nov 27 17:47:30 [FileAllocator] done allocating datafile /data/db/test-rs0-1/admin.ns, size: 16MB, took 0.323 secs m31101| Tue Nov 27 17:47:30 [FileAllocator] allocating new datafile /data/db/test-rs0-1/admin.0, filling with zeroes... m31101| Tue Nov 27 17:47:30 [FileAllocator] done allocating datafile /data/db/test-rs0-1/admin.0, size: 16MB, took 0.303 secs m31101| Tue Nov 27 17:47:30 [rsSync] build index admin.foo { _id: 1 } m31101| Tue Nov 27 17:47:30 [rsSync] fastBuildIndex dupsToDrop:0 m31101| Tue Nov 27 17:47:30 [rsSync] build index done. scanned 1 total records. 0.01 secs m31101| Tue Nov 27 17:47:30 [rsSync] replSet initial sync data copy, starting syncup m31101| Tue Nov 27 17:47:30 [rsSync] oplog sync 1 of 3 m31101| Tue Nov 27 17:47:30 [rsSync] couldn't make room for new record (len: 160) in capped ns local.oplog.rs m31101| Extent 0 (capExtent) m31101| magic: 41424344 extent->ns: local.oplog.rs m31101| fr: null lr: null extent->len: 41943040 m31101| local.oplog.rs Assertion failure len * 5 > lastExtentSize src/mongo/db/namespace_details.cpp 444 m31101| 0xaffd31 0xac65cd 0x8113e8 0x65862a 0x80fd77 0x810068 0x85cc3e 0x81e23b 0x9a78f6 0x9879c0 0x988ee8 0x98a027 0x9a5518 0x9a55ba 0x9a5878 0xb45ba9 0x7fd6454dce9a 0x7fd6447f24bd m31101| mongod-2.2(_ZN5mongo15printStackTraceERSo+0x21) [0xaffd31] m31101| mongod-2.2(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) [0xac65cd] m31101| mongod-2.2(_ZNK5mongo16NamespaceDetails13maybeComplainEPKci+0xa48) [0x8113e8] m31101| mongod-2.2(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x31a) [0x65862a] m31101| mongod-2.2(_ZN5mongo16NamespaceDetails6_allocEPKci+0x27) [0x80fd77] m31101| mongod-2.2(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x38) [0x810068] m31101| mongod-2.2(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x6e) [0x85cc3e] m31101| mongod-2.2(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27b) [0x81e23b] m31101| mongod-2.2(_ZN5mongo7replset11InitialSync16oplogApplicationERKNS_7BSONObjES4_+0xe6) [0x9a78f6] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl30_syncDoInitialSync_applyToHeadERNS_7replset8SyncTailEPNS_11OplogReaderEPKNS_6MemberERKNS_7BSONObjERS9_+0x370) [0x9879c0] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl18_syncDoInitialSyncEv+0x8d8) [0x988ee8] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl17syncDoInitialSyncEv+0x37) [0x98a027] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl11_syncThreadEv+0x68) [0x9a5518] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x9a55ba] m31101| mongod-2.2(_ZN5mongo15startSyncThreadEv+0xa8) [0x9a5878] m31101| mongod-2.2() [0xb45ba9] m31101| /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fd6454dce9a] m31101| /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fd6447f24bd] m31101| Tue Nov 27 17:47:30 [rsSync] replSet initial sync failed during oplog application phase m31101| Tue Nov 27 17:47:30 [rsSync] replSet cleaning up [1] m31101| Tue Nov 27 17:47:30 [rsSync] replSet cleaning up [2] m31101| Tue Nov 27 17:47:30 [rsSync] replSet initial sync failed will try again m31101| Tue Nov 27 17:47:32 [initandlisten] connection accepted from 127.0.0.1:35069 #10 (9 connections now open) m31101| Tue Nov 27 17:47:32 [initandlisten] connection accepted from 127.0.0.1:35070 #11 (10 connections now open) m31101| Tue Nov 27 17:47:32 [initandlisten] connection accepted from 127.0.0.1:35071 #12 (11 connections now open) m31101| Tue Nov 27 17:47:36 [rsSync] replSet initial sync pending m31101| Tue Nov 27 17:47:36 [rsSync] replSet syncing to: ubuntu:31100 m31101| Tue Nov 27 17:47:36 [rsSync] replSet initial sync drop all databases m31101| Tue Nov 27 17:47:36 [rsSync] dropAllDatabasesExceptLocal 3 m31101| Tue Nov 27 17:47:36 [rsSync] removeJournalFiles m31101| Tue Nov 27 17:47:36 [rsSync] removeJournalFiles m31101| Tue Nov 27 17:47:36 [rsSync] replSet initial sync clone all databases m31101| Tue Nov 27 17:47:36 [rsSync] replSet initial sync cloning db: foo20 m31101| Tue Nov 27 17:47:36 [FileAllocator] allocating new datafile /data/db/test-rs0-1/foo20.ns, filling with zeroes... m31101| Tue Nov 27 17:47:37 [FileAllocator] done allocating datafile /data/db/test-rs0-1/foo20.ns, size: 16MB, took 0.352 secs m31101| Tue Nov 27 17:47:37 [FileAllocator] allocating new datafile /data/db/test-rs0-1/foo20.0, filling with zeroes... m31101| Tue Nov 27 17:47:37 [FileAllocator] done allocating datafile /data/db/test-rs0-1/foo20.0, size: 16MB, took 0.345 secs m31101| Tue Nov 27 17:47:37 [rsSync] build index foo20.bar20 { _id: 1 } m31101| Tue Nov 27 17:47:37 [rsSync] fastBuildIndex dupsToDrop:0 m31101| Tue Nov 27 17:47:37 [rsSync] build index done. scanned 0 total records. 0 secs m31101| Tue Nov 27 17:47:37 [rsSync] replSet initial sync cloning db: admin m31101| Tue Nov 27 17:47:37 [FileAllocator] allocating new datafile /data/db/test-rs0-1/admin.ns, filling with zeroes... m31101| Tue Nov 27 17:47:38 [FileAllocator] done allocating datafile /data/db/test-rs0-1/admin.ns, size: 16MB, took 0.315 secs m31101| Tue Nov 27 17:47:38 [FileAllocator] allocating new datafile /data/db/test-rs0-1/admin.0, filling with zeroes... m31101| Tue Nov 27 17:47:38 [FileAllocator] done allocating datafile /data/db/test-rs0-1/admin.0, size: 16MB, took 0.295 secs m31101| Tue Nov 27 17:47:38 [rsSync] build index admin.foo { _id: 1 } m31101| Tue Nov 27 17:47:38 [rsSync] fastBuildIndex dupsToDrop:0 m31101| Tue Nov 27 17:47:38 [rsSync] build index done. scanned 1 total records. 0 secs m31101| Tue Nov 27 17:47:38 [rsSync] replSet initial sync data copy, starting syncup m31101| Tue Nov 27 17:47:38 [rsSync] oplog sync 1 of 3 m31101| Tue Nov 27 17:47:38 [rsSync] couldn't make room for new record (len: 160) in capped ns local.oplog.rs m31101| Extent 0 (capExtent) m31101| magic: 41424344 extent->ns: local.oplog.rs m31101| fr: null lr: null extent->len: 41943040 m31101| local.oplog.rs Assertion failure len * 5 > lastExtentSize src/mongo/db/namespace_details.cpp 444 m31101| 0xaffd31 0xac65cd 0x8113e8 0x65862a 0x80fd77 0x810068 0x85cc3e 0x81e23b 0x9a78f6 0x9879c0 0x988ee8 0x98a027 0x9a5518 0x9a55ba 0x9a5878 0xb45ba9 0x7fd6454dce9a 0x7fd6447f24bd m31101| mongod-2.2(_ZN5mongo15printStackTraceERSo+0x21) [0xaffd31] m31101| mongod-2.2(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) [0xac65cd] m31101| mongod-2.2(_ZNK5mongo16NamespaceDetails13maybeComplainEPKci+0xa48) [0x8113e8] m31101| mongod-2.2(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x31a) [0x65862a] m31101| mongod-2.2(_ZN5mongo16NamespaceDetails6_allocEPKci+0x27) [0x80fd77] m31101| mongod-2.2(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x38) [0x810068] m31101| mongod-2.2(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x6e) [0x85cc3e] m31101| mongod-2.2(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27b) [0x81e23b] m31101| mongod-2.2(_ZN5mongo7replset11InitialSync16oplogApplicationERKNS_7BSONObjES4_+0xe6) [0x9a78f6] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl30_syncDoInitialSync_applyToHeadERNS_7replset8SyncTailEPNS_11OplogReaderEPKNS_6MemberERKNS_7BSONObjERS9_+0x370) [0x9879c0] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl18_syncDoInitialSyncEv+0x8d8) [0x988ee8] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl17syncDoInitialSyncEv+0x37) [0x98a027] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl11_syncThreadEv+0x68) [0x9a5518] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x9a55ba] m31101| mongod-2.2(_ZN5mongo15startSyncThreadEv+0xa8) [0x9a5878] m31101| mongod-2.2() [0xb45ba9] m31101| /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fd6454dce9a] m31101| /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fd6447f24bd] m31101| Tue Nov 27 17:47:38 [rsSync] replSet initial sync failed during oplog application phase m31101| Tue Nov 27 17:47:38 [rsSync] replSet cleaning up [1] m31101| Tue Nov 27 17:47:38 [rsSync] replSet cleaning up [2] m31101| Tue Nov 27 17:47:38 [rsSync] replSet initial sync failed will try again m31101| Tue Nov 27 17:47:38 [conn3] end connection 127.0.0.1:35037 (10 connections now open) m31101| Tue Nov 27 17:47:38 [initandlisten] connection accepted from 127.0.0.1:35080 #13 (11 connections now open) m31101| Tue Nov 27 17:47:44 [rsSync] replSet initial sync pending m31101| Tue Nov 27 17:47:44 [rsSync] replSet syncing to: ubuntu:31100 m31101| Tue Nov 27 17:47:44 [rsSync] replSet initial sync drop all databases m31101| Tue Nov 27 17:47:44 [rsSync] dropAllDatabasesExceptLocal 3 m31101| Tue Nov 27 17:47:44 [rsSync] removeJournalFiles m31101| Tue Nov 27 17:47:44 [rsSync] removeJournalFiles m31101| Tue Nov 27 17:47:44 [rsSync] replSet initial sync clone all databases m31101| Tue Nov 27 17:47:44 [rsSync] replSet initial sync cloning db: foo20 m31101| Tue Nov 27 17:47:44 [FileAllocator] allocating new datafile /data/db/test-rs0-1/foo20.ns, filling with zeroes... m31101| Tue Nov 27 17:47:44 [FileAllocator] done allocating datafile /data/db/test-rs0-1/foo20.ns, size: 16MB, took 0.365 secs m31101| Tue Nov 27 17:47:44 [FileAllocator] allocating new datafile /data/db/test-rs0-1/foo20.0, filling with zeroes... m31101| Tue Nov 27 17:47:45 [FileAllocator] done allocating datafile /data/db/test-rs0-1/foo20.0, size: 16MB, took 0.303 secs m31101| Tue Nov 27 17:47:45 [rsSync] build index foo20.bar20 { _id: 1 } m31101| Tue Nov 27 17:47:45 [rsSync] fastBuildIndex dupsToDrop:0 m31101| Tue Nov 27 17:47:45 [rsSync] build index done. scanned 0 total records. 0 secs m31101| Tue Nov 27 17:47:45 [rsSync] replSet initial sync cloning db: admin m31101| Tue Nov 27 17:47:45 [FileAllocator] allocating new datafile /data/db/test-rs0-1/admin.ns, filling with zeroes... m31101| Tue Nov 27 17:47:45 [FileAllocator] done allocating datafile /data/db/test-rs0-1/admin.ns, size: 16MB, took 0.315 secs m31101| Tue Nov 27 17:47:45 [FileAllocator] allocating new datafile /data/db/test-rs0-1/admin.0, filling with zeroes... m31101| Tue Nov 27 17:47:45 [FileAllocator] done allocating datafile /data/db/test-rs0-1/admin.0, size: 16MB, took 0.288 secs m31101| Tue Nov 27 17:47:45 [rsSync] build index admin.foo { _id: 1 } m31101| Tue Nov 27 17:47:45 [rsSync] fastBuildIndex dupsToDrop:0 m31101| Tue Nov 27 17:47:45 [rsSync] build index done. scanned 1 total records. 0 secs m31101| Tue Nov 27 17:47:45 [rsSync] replSet initial sync data copy, starting syncup m31101| Tue Nov 27 17:47:45 [rsSync] oplog sync 1 of 3 m31101| Tue Nov 27 17:47:45 [rsSync] couldn't make room for new record (len: 160) in capped ns local.oplog.rs m31101| Extent 0 (capExtent) m31101| magic: 41424344 extent->ns: local.oplog.rs m31101| fr: null lr: null extent->len: 41943040 m31101| local.oplog.rs Assertion failure len * 5 > lastExtentSize src/mongo/db/namespace_details.cpp 444 m31101| 0xaffd31 0xac65cd 0x8113e8 0x65862a 0x80fd77 0x810068 0x85cc3e 0x81e23b 0x9a78f6 0x9879c0 0x988ee8 0x98a027 0x9a5518 0x9a55ba 0x9a5878 0xb45ba9 0x7fd6454dce9a 0x7fd6447f24bd m31101| mongod-2.2(_ZN5mongo15printStackTraceERSo+0x21) [0xaffd31] m31101| mongod-2.2(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) [0xac65cd] m31101| mongod-2.2(_ZNK5mongo16NamespaceDetails13maybeComplainEPKci+0xa48) [0x8113e8] m31101| mongod-2.2(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x31a) [0x65862a] m31101| mongod-2.2(_ZN5mongo16NamespaceDetails6_allocEPKci+0x27) [0x80fd77] m31101| mongod-2.2(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x38) [0x810068] m31101| mongod-2.2(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x6e) [0x85cc3e] m31101| mongod-2.2(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27b) [0x81e23b] m31101| mongod-2.2(_ZN5mongo7replset11InitialSync16oplogApplicationERKNS_7BSONObjES4_+0xe6) [0x9a78f6] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl30_syncDoInitialSync_applyToHeadERNS_7replset8SyncTailEPNS_11OplogReaderEPKNS_6MemberERKNS_7BSONObjERS9_+0x370) [0x9879c0] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl18_syncDoInitialSyncEv+0x8d8) [0x988ee8] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl17syncDoInitialSyncEv+0x37) [0x98a027] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl11_syncThreadEv+0x68) [0x9a5518] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x9a55ba] m31101| mongod-2.2(_ZN5mongo15startSyncThreadEv+0xa8) [0x9a5878] m31101| mongod-2.2() [0xb45ba9] m31101| /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fd6454dce9a] m31101| /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fd6447f24bd] m31101| Tue Nov 27 17:47:45 [rsSync] replSet initial sync failed during oplog application phase m31101| Tue Nov 27 17:47:45 [rsSync] replSet cleaning up [1] m31101| Tue Nov 27 17:47:45 [rsSync] replSet cleaning up [2] m31101| Tue Nov 27 17:47:45 [rsSync] replSet initial sync failed will try again m31101| Tue Nov 27 17:47:51 [rsSync] replSet initial sync pending m31101| Tue Nov 27 17:47:51 [rsSync] replSet syncing to: ubuntu:31100 m31101| Tue Nov 27 17:47:51 [rsSync] replSet initial sync drop all databases m31101| Tue Nov 27 17:47:51 [rsSync] dropAllDatabasesExceptLocal 3 m31101| Tue Nov 27 17:47:52 [rsSync] removeJournalFiles m31101| Tue Nov 27 17:47:52 [rsSync] removeJournalFiles m31101| Tue Nov 27 17:47:52 [rsSync] replSet initial sync clone all databases m31101| Tue Nov 27 17:47:52 [rsSync] replSet initial sync cloning db: foo20 m31101| Tue Nov 27 17:47:52 [FileAllocator] allocating new datafile /data/db/test-rs0-1/foo20.ns, filling with zeroes... m31101| Tue Nov 27 17:47:52 [FileAllocator] done allocating datafile /data/db/test-rs0-1/foo20.ns, size: 16MB, took 0.339 secs m31101| Tue Nov 27 17:47:52 [FileAllocator] allocating new datafile /data/db/test-rs0-1/foo20.0, filling with zeroes... m31101| Tue Nov 27 17:47:52 [FileAllocator] done allocating datafile /data/db/test-rs0-1/foo20.0, size: 16MB, took 0.354 secs m31101| Tue Nov 27 17:47:52 [rsSync] build index foo20.bar20 { _id: 1 } m31101| Tue Nov 27 17:47:52 [rsSync] fastBuildIndex dupsToDrop:0 m31101| Tue Nov 27 17:47:52 [rsSync] build index done. scanned 0 total records. 0 secs m31101| Tue Nov 27 17:47:52 [rsSync] replSet initial sync cloning db: admin m31101| Tue Nov 27 17:47:52 [FileAllocator] allocating new datafile /data/db/test-rs0-1/admin.ns, filling with zeroes... m31101| Tue Nov 27 17:47:53 [FileAllocator] done allocating datafile /data/db/test-rs0-1/admin.ns, size: 16MB, took 0.315 secs m31101| Tue Nov 27 17:47:53 [FileAllocator] allocating new datafile /data/db/test-rs0-1/admin.0, filling with zeroes... m31101| Tue Nov 27 17:47:53 [FileAllocator] done allocating datafile /data/db/test-rs0-1/admin.0, size: 16MB, took 0.295 secs m31101| Tue Nov 27 17:47:53 [rsSync] build index admin.foo { _id: 1 } m31101| Tue Nov 27 17:47:53 [rsSync] fastBuildIndex dupsToDrop:0 m31101| Tue Nov 27 17:47:53 [rsSync] build index done. scanned 1 total records. 0 secs m31101| Tue Nov 27 17:47:53 [rsSync] replSet initial sync data copy, starting syncup m31101| Tue Nov 27 17:47:53 [rsSync] oplog sync 1 of 3 m31101| Tue Nov 27 17:47:53 [rsSync] couldn't make room for new record (len: 160) in capped ns local.oplog.rs m31101| Extent 0 (capExtent) m31101| magic: 41424344 extent->ns: local.oplog.rs m31101| fr: null lr: null extent->len: 41943040 m31101| local.oplog.rs Assertion failure len * 5 > lastExtentSize src/mongo/db/namespace_details.cpp 444 m31101| 0xaffd31 0xac65cd 0x8113e8 0x65862a 0x80fd77 0x810068 0x85cc3e 0x81e23b 0x9a78f6 0x9879c0 0x988ee8 0x98a027 0x9a5518 0x9a55ba 0x9a5878 0xb45ba9 0x7fd6454dce9a 0x7fd6447f24bd m31101| mongod-2.2(_ZN5mongo15printStackTraceERSo+0x21) [0xaffd31] m31101| mongod-2.2(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) [0xac65cd] m31101| mongod-2.2(_ZNK5mongo16NamespaceDetails13maybeComplainEPKci+0xa48) [0x8113e8] m31101| mongod-2.2(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x31a) [0x65862a] m31101| mongod-2.2(_ZN5mongo16NamespaceDetails6_allocEPKci+0x27) [0x80fd77] m31101| mongod-2.2(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x38) [0x810068] m31101| mongod-2.2(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x6e) [0x85cc3e] m31101| mongod-2.2(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27b) [0x81e23b] m31101| mongod-2.2(_ZN5mongo7replset11InitialSync16oplogApplicationERKNS_7BSONObjES4_+0xe6) [0x9a78f6] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl30_syncDoInitialSync_applyToHeadERNS_7replset8SyncTailEPNS_11OplogReaderEPKNS_6MemberERKNS_7BSONObjERS9_+0x370) [0x9879c0] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl18_syncDoInitialSyncEv+0x8d8) [0x988ee8] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl17syncDoInitialSyncEv+0x37) [0x98a027] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl11_syncThreadEv+0x68) [0x9a5518] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x9a55ba] m31101| mongod-2.2(_ZN5mongo15startSyncThreadEv+0xa8) [0x9a5878] m31101| mongod-2.2() [0xb45ba9] m31101| /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fd6454dce9a] m31101| /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fd6447f24bd] m31101| Tue Nov 27 17:47:53 [rsSync] replSet initial sync failed during oplog application phase m31101| Tue Nov 27 17:47:53 [rsSync] replSet cleaning up [1] m31101| Tue Nov 27 17:47:53 [rsSync] replSet cleaning up [2] m31101| Tue Nov 27 17:47:53 [rsSync] replSet initial sync failed will try again m31101| Tue Nov 27 17:47:57 [conn9] end connection 127.0.0.1:35059 (10 connections now open) m31101| Tue Nov 27 17:47:57 [initandlisten] connection accepted from 127.0.0.1:35106 #14 (11 connections now open) m31101| Tue Nov 27 17:47:59 [rsSync] replSet initial sync pending m31101| Tue Nov 27 17:47:59 [rsSync] replSet syncing to: ubuntu:31100 m31101| Tue Nov 27 17:47:59 [rsSync] replSet initial sync drop all databases m31101| Tue Nov 27 17:47:59 [rsSync] dropAllDatabasesExceptLocal 3 m31101| Tue Nov 27 17:47:59 [rsSync] removeJournalFiles m31101| Tue Nov 27 17:47:59 [rsSync] removeJournalFiles m31101| Tue Nov 27 17:47:59 [rsSync] replSet initial sync clone all databases m31101| Tue Nov 27 17:47:59 [rsSync] replSet initial sync cloning db: foo20 m31101| Tue Nov 27 17:47:59 [FileAllocator] allocating new datafile /data/db/test-rs0-1/foo20.ns, filling with zeroes... m31101| Tue Nov 27 17:48:00 [FileAllocator] done allocating datafile /data/db/test-rs0-1/foo20.ns, size: 16MB, took 0.385 secs m31101| Tue Nov 27 17:48:00 [FileAllocator] allocating new datafile /data/db/test-rs0-1/foo20.0, filling with zeroes... m31101| Tue Nov 27 17:48:00 [FileAllocator] done allocating datafile /data/db/test-rs0-1/foo20.0, size: 16MB, took 0.321 secs m31101| Tue Nov 27 17:48:00 [rsSync] build index foo20.bar20 { _id: 1 } m31101| Tue Nov 27 17:48:00 [rsSync] fastBuildIndex dupsToDrop:0 m31101| Tue Nov 27 17:48:00 [rsSync] build index done. scanned 0 total records. 0 secs m31101| Tue Nov 27 17:48:00 [rsSync] replSet initial sync cloning db: admin m31101| Tue Nov 27 17:48:00 [FileAllocator] allocating new datafile /data/db/test-rs0-1/admin.ns, filling with zeroes... m31101| Tue Nov 27 17:48:00 [FileAllocator] done allocating datafile /data/db/test-rs0-1/admin.ns, size: 16MB, took 0.315 secs m31101| Tue Nov 27 17:48:00 [FileAllocator] allocating new datafile /data/db/test-rs0-1/admin.0, filling with zeroes... m31101| Tue Nov 27 17:48:01 [FileAllocator] done allocating datafile /data/db/test-rs0-1/admin.0, size: 16MB, took 0.303 secs m31101| Tue Nov 27 17:48:01 [rsSync] build index admin.foo { _id: 1 } m31101| Tue Nov 27 17:48:01 [rsSync] fastBuildIndex dupsToDrop:0 m31101| Tue Nov 27 17:48:01 [rsSync] build index done. scanned 1 total records. 0 secs m31101| Tue Nov 27 17:48:01 [rsSync] replSet initial sync data copy, starting syncup m31101| Tue Nov 27 17:48:01 [rsSync] oplog sync 1 of 3 m31101| Tue Nov 27 17:48:01 [rsSync] couldn't make room for new record (len: 160) in capped ns local.oplog.rs m31101| Extent 0 (capExtent) m31101| magic: 41424344 extent->ns: local.oplog.rs m31101| fr: null lr: null extent->len: 41943040 m31101| local.oplog.rs Assertion failure len * 5 > lastExtentSize src/mongo/db/namespace_details.cpp 444 m31101| 0xaffd31 0xac65cd 0x8113e8 0x65862a 0x80fd77 0x810068 0x85cc3e 0x81e23b 0x9a78f6 0x9879c0 0x988ee8 0x98a027 0x9a5518 0x9a55ba 0x9a5878 0xb45ba9 0x7fd6454dce9a 0x7fd6447f24bd m31101| mongod-2.2(_ZN5mongo15printStackTraceERSo+0x21) [0xaffd31] m31101| mongod-2.2(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) [0xac65cd] m31101| mongod-2.2(_ZNK5mongo16NamespaceDetails13maybeComplainEPKci+0xa48) [0x8113e8] m31101| mongod-2.2(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x31a) [0x65862a] m31101| mongod-2.2(_ZN5mongo16NamespaceDetails6_allocEPKci+0x27) [0x80fd77] m31101| mongod-2.2(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x38) [0x810068] m31101| mongod-2.2(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x6e) [0x85cc3e] m31101| mongod-2.2(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27b) [0x81e23b] m31101| mongod-2.2(_ZN5mongo7replset11InitialSync16oplogApplicationERKNS_7BSONObjES4_+0xe6) [0x9a78f6] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl30_syncDoInitialSync_applyToHeadERNS_7replset8SyncTailEPNS_11OplogReaderEPKNS_6MemberERKNS_7BSONObjERS9_+0x370) [0x9879c0] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl18_syncDoInitialSyncEv+0x8d8) [0x988ee8] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl17syncDoInitialSyncEv+0x37) [0x98a027] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl11_syncThreadEv+0x68) [0x9a5518] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x9a55ba] m31101| mongod-2.2(_ZN5mongo15startSyncThreadEv+0xa8) [0x9a5878] m31101| mongod-2.2() [0xb45ba9] m31101| /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fd6454dce9a] m31101| /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fd6447f24bd] m31101| Tue Nov 27 17:48:01 [rsSync] replSet initial sync failed during oplog application phase m31101| Tue Nov 27 17:48:01 [rsSync] replSet cleaning up [1] m31101| Tue Nov 27 17:48:01 [rsSync] replSet cleaning up [2] m31101| Tue Nov 27 17:48:01 [rsSync] replSet initial sync failed will try again m31101| Tue Nov 27 17:48:07 [rsSync] replSet initial sync pending m31101| Tue Nov 27 17:48:07 [rsSync] replSet syncing to: ubuntu:31100 m31101| Tue Nov 27 17:48:07 [rsSync] replSet initial sync drop all databases m31101| Tue Nov 27 17:48:07 [rsSync] dropAllDatabasesExceptLocal 3 m31101| Tue Nov 27 17:48:07 [rsSync] removeJournalFiles m31101| Tue Nov 27 17:48:07 [rsSync] removeJournalFiles m31101| Tue Nov 27 17:48:07 [rsSync] replSet initial sync clone all databases m31101| Tue Nov 27 17:48:07 [rsSync] replSet initial sync cloning db: foo20 m31101| Tue Nov 27 17:48:07 [FileAllocator] allocating new datafile /data/db/test-rs0-1/foo20.ns, filling with zeroes... m31101| Tue Nov 27 17:48:07 [FileAllocator] done allocating datafile /data/db/test-rs0-1/foo20.ns, size: 16MB, took 0.322 secs m31101| Tue Nov 27 17:48:07 [FileAllocator] allocating new datafile /data/db/test-rs0-1/foo20.0, filling with zeroes... m31101| Tue Nov 27 17:48:07 [FileAllocator] done allocating datafile /data/db/test-rs0-1/foo20.0, size: 16MB, took 0.32 secs m31101| Tue Nov 27 17:48:08 [rsSync] build index foo20.bar20 { _id: 1 } m31101| Tue Nov 27 17:48:08 [rsSync] fastBuildIndex dupsToDrop:0 m31101| Tue Nov 27 17:48:08 [rsSync] build index done. scanned 0 total records. 0 secs m31101| Tue Nov 27 17:48:08 [rsSync] replSet initial sync cloning db: admin m31101| Tue Nov 27 17:48:08 [FileAllocator] allocating new datafile /data/db/test-rs0-1/admin.ns, filling with zeroes... m31101| Tue Nov 27 17:48:08 [FileAllocator] done allocating datafile /data/db/test-rs0-1/admin.ns, size: 16MB, took 0.307 secs m31101| Tue Nov 27 17:48:08 [FileAllocator] allocating new datafile /data/db/test-rs0-1/admin.0, filling with zeroes... m31101| Tue Nov 27 17:48:08 [FileAllocator] done allocating datafile /data/db/test-rs0-1/admin.0, size: 16MB, took 0.326 secs m31101| Tue Nov 27 17:48:08 [rsSync] build index admin.foo { _id: 1 } m31101| Tue Nov 27 17:48:08 [rsSync] fastBuildIndex dupsToDrop:0 m31101| Tue Nov 27 17:48:08 [rsSync] build index done. scanned 1 total records. 0 secs m31101| Tue Nov 27 17:48:08 [rsSync] replSet initial sync data copy, starting syncup m31101| Tue Nov 27 17:48:08 [rsSync] oplog sync 1 of 3 m31101| Tue Nov 27 17:48:08 [rsSync] couldn't make room for new record (len: 160) in capped ns local.oplog.rs m31101| Extent 0 (capExtent) m31101| magic: 41424344 extent->ns: local.oplog.rs m31101| fr: null lr: null extent->len: 41943040 m31101| local.oplog.rs Assertion failure len * 5 > lastExtentSize src/mongo/db/namespace_details.cpp 444 m31101| 0xaffd31 0xac65cd 0x8113e8 0x65862a 0x80fd77 0x810068 0x85cc3e 0x81e23b 0x9a78f6 0x9879c0 0x988ee8 0x98a027 0x9a5518 0x9a55ba 0x9a5878 0xb45ba9 0x7fd6454dce9a 0x7fd6447f24bd m31101| mongod-2.2(_ZN5mongo15printStackTraceERSo+0x21) [0xaffd31] m31101| mongod-2.2(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) [0xac65cd] m31101| mongod-2.2(_ZNK5mongo16NamespaceDetails13maybeComplainEPKci+0xa48) [0x8113e8] m31101| mongod-2.2(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x31a) [0x65862a] m31101| mongod-2.2(_ZN5mongo16NamespaceDetails6_allocEPKci+0x27) [0x80fd77] m31101| mongod-2.2(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x38) [0x810068] m31101| mongod-2.2(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x6e) [0x85cc3e] m31101| mongod-2.2(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27b) [0x81e23b] m31101| mongod-2.2(_ZN5mongo7replset11InitialSync16oplogApplicationERKNS_7BSONObjES4_+0xe6) [0x9a78f6] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl30_syncDoInitialSync_applyToHeadERNS_7replset8SyncTailEPNS_11OplogReaderEPKNS_6MemberERKNS_7BSONObjERS9_+0x370) [0x9879c0] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl18_syncDoInitialSyncEv+0x8d8) [0x988ee8] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl17syncDoInitialSyncEv+0x37) [0x98a027] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl11_syncThreadEv+0x68) [0x9a5518] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x9a55ba] m31101| mongod-2.2(_ZN5mongo15startSyncThreadEv+0xa8) [0x9a5878] m31101| mongod-2.2() [0xb45ba9] m31101| /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fd6454dce9a] m31101| /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fd6447f24bd] m31101| Tue Nov 27 17:48:08 [rsSync] replSet initial sync failed during oplog application phase m31101| Tue Nov 27 17:48:08 [rsSync] replSet cleaning up [1] m31101| Tue Nov 27 17:48:08 [rsSync] replSet cleaning up [2] m31101| Tue Nov 27 17:48:08 [rsSync] replSet initial sync failed will try again m31101| Tue Nov 27 17:48:08 [conn13] end connection 127.0.0.1:35080 (10 connections now open) m31101| Tue Nov 27 17:48:08 [initandlisten] connection accepted from 127.0.0.1:35124 #15 (11 connections now open) m31101| Tue Nov 27 17:48:14 [rsSync] replSet initial sync pending m31101| Tue Nov 27 17:48:14 [rsSync] replSet syncing to: ubuntu:31100 m31101| Tue Nov 27 17:48:14 [rsSync] replSet initial sync drop all databases m31101| Tue Nov 27 17:48:14 [rsSync] dropAllDatabasesExceptLocal 3 m31101| Tue Nov 27 17:48:14 [rsSync] removeJournalFiles m31101| Tue Nov 27 17:48:14 [rsSync] removeJournalFiles m31101| Tue Nov 27 17:48:14 [rsSync] replSet initial sync clone all databases m31101| Tue Nov 27 17:48:14 [rsSync] replSet initial sync cloning db: foo20 m31101| Tue Nov 27 17:48:14 [FileAllocator] allocating new datafile /data/db/test-rs0-1/foo20.ns, filling with zeroes... m31101| Tue Nov 27 17:48:15 [FileAllocator] done allocating datafile /data/db/test-rs0-1/foo20.ns, size: 16MB, took 0.348 secs m31101| Tue Nov 27 17:48:15 [FileAllocator] allocating new datafile /data/db/test-rs0-1/foo20.0, filling with zeroes... m31101| Tue Nov 27 17:48:15 [FileAllocator] done allocating datafile /data/db/test-rs0-1/foo20.0, size: 16MB, took 0.311 secs m31101| Tue Nov 27 17:48:15 [rsSync] build index foo20.bar20 { _id: 1 } m31101| Tue Nov 27 17:48:15 [rsSync] fastBuildIndex dupsToDrop:0 m31101| Tue Nov 27 17:48:15 [rsSync] build index done. scanned 0 total records. 0 secs m31101| Tue Nov 27 17:48:15 [rsSync] replSet initial sync cloning db: admin m31101| Tue Nov 27 17:48:15 [FileAllocator] allocating new datafile /data/db/test-rs0-1/admin.ns, filling with zeroes... m31101| Tue Nov 27 17:48:15 [FileAllocator] done allocating datafile /data/db/test-rs0-1/admin.ns, size: 16MB, took 0.307 secs m31101| Tue Nov 27 17:48:15 [FileAllocator] allocating new datafile /data/db/test-rs0-1/admin.0, filling with zeroes... m31101| Tue Nov 27 17:48:16 [FileAllocator] done allocating datafile /data/db/test-rs0-1/admin.0, size: 16MB, took 0.303 secs m31101| Tue Nov 27 17:48:16 [rsSync] build index admin.foo { _id: 1 } m31101| Tue Nov 27 17:48:16 [rsSync] fastBuildIndex dupsToDrop:0 m31101| Tue Nov 27 17:48:16 [rsSync] build index done. scanned 1 total records. 0 secs m31101| Tue Nov 27 17:48:16 [rsSync] replSet initial sync data copy, starting syncup m31101| Tue Nov 27 17:48:16 [rsSync] oplog sync 1 of 3 m31101| Tue Nov 27 17:48:16 [rsSync] couldn't make room for new record (len: 160) in capped ns local.oplog.rs m31101| Extent 0 (capExtent) m31101| magic: 41424344 extent->ns: local.oplog.rs m31101| fr: null lr: null extent->len: 41943040 m31101| local.oplog.rs Assertion failure len * 5 > lastExtentSize src/mongo/db/namespace_details.cpp 444 m31101| 0xaffd31 0xac65cd 0x8113e8 0x65862a 0x80fd77 0x810068 0x85cc3e 0x81e23b 0x9a78f6 0x9879c0 0x988ee8 0x98a027 0x9a5518 0x9a55ba 0x9a5878 0xb45ba9 0x7fd6454dce9a 0x7fd6447f24bd m31101| mongod-2.2(_ZN5mongo15printStackTraceERSo+0x21) [0xaffd31] m31101| mongod-2.2(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) [0xac65cd] m31101| mongod-2.2(_ZNK5mongo16NamespaceDetails13maybeComplainEPKci+0xa48) [0x8113e8] m31101| mongod-2.2(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x31a) [0x65862a] m31101| mongod-2.2(_ZN5mongo16NamespaceDetails6_allocEPKci+0x27) [0x80fd77] m31101| mongod-2.2(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x38) [0x810068] m31101| mongod-2.2(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x6e) [0x85cc3e] m31101| mongod-2.2(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27b) [0x81e23b] m31101| mongod-2.2(_ZN5mongo7replset11InitialSync16oplogApplicationERKNS_7BSONObjES4_+0xe6) [0x9a78f6] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl30_syncDoInitialSync_applyToHeadERNS_7replset8SyncTailEPNS_11OplogReaderEPKNS_6MemberERKNS_7BSONObjERS9_+0x370) [0x9879c0] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl18_syncDoInitialSyncEv+0x8d8) [0x988ee8] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl17syncDoInitialSyncEv+0x37) [0x98a027] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl11_syncThreadEv+0x68) [0x9a5518] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x9a55ba] m31101| mongod-2.2(_ZN5mongo15startSyncThreadEv+0xa8) [0x9a5878] m31101| mongod-2.2() [0xb45ba9] m31101| /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fd6454dce9a] m31101| /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fd6447f24bd] m31101| Tue Nov 27 17:48:16 [rsSync] replSet initial sync failed during oplog application phase m31101| Tue Nov 27 17:48:16 [rsSync] replSet cleaning up [1] m31101| Tue Nov 27 17:48:16 [rsSync] replSet cleaning up [2] m31101| Tue Nov 27 17:48:16 [rsSync] replSet initial sync failed will try again m31101| Tue Nov 27 17:48:22 [rsSync] replSet initial sync pending m31101| Tue Nov 27 17:48:22 [rsSync] replSet syncing to: ubuntu:31100 m31101| Tue Nov 27 17:48:22 [rsSync] replSet initial sync drop all databases m31101| Tue Nov 27 17:48:22 [rsSync] dropAllDatabasesExceptLocal 3 m31101| Tue Nov 27 17:48:22 [rsSync] removeJournalFiles m31101| Tue Nov 27 17:48:22 [rsSync] removeJournalFiles m31101| Tue Nov 27 17:48:22 [rsSync] replSet initial sync clone all databases m31101| Tue Nov 27 17:48:22 [rsSync] replSet initial sync cloning db: foo20 m31101| Tue Nov 27 17:48:22 [FileAllocator] allocating new datafile /data/db/test-rs0-1/foo20.ns, filling with zeroes... m31101| Tue Nov 27 17:48:22 [FileAllocator] done allocating datafile /data/db/test-rs0-1/foo20.ns, size: 16MB, took 0.347 secs m31101| Tue Nov 27 17:48:22 [FileAllocator] allocating new datafile /data/db/test-rs0-1/foo20.0, filling with zeroes... m31101| Tue Nov 27 17:48:23 [FileAllocator] done allocating datafile /data/db/test-rs0-1/foo20.0, size: 16MB, took 0.312 secs m31101| Tue Nov 27 17:48:23 [rsSync] build index foo20.bar20 { _id: 1 } m31101| Tue Nov 27 17:48:23 [rsSync] fastBuildIndex dupsToDrop:0 m31101| Tue Nov 27 17:48:23 [rsSync] build index done. scanned 0 total records. 0 secs m31101| Tue Nov 27 17:48:23 [rsSync] replSet initial sync cloning db: admin m31101| Tue Nov 27 17:48:23 [FileAllocator] allocating new datafile /data/db/test-rs0-1/admin.ns, filling with zeroes... m31101| Tue Nov 27 17:48:23 [FileAllocator] done allocating datafile /data/db/test-rs0-1/admin.ns, size: 16MB, took 0.307 secs m31101| Tue Nov 27 17:48:23 [FileAllocator] allocating new datafile /data/db/test-rs0-1/admin.0, filling with zeroes... m31101| Tue Nov 27 17:48:23 [FileAllocator] done allocating datafile /data/db/test-rs0-1/admin.0, size: 16MB, took 0.305 secs m31101| Tue Nov 27 17:48:23 [rsSync] build index admin.foo { _id: 1 } m31101| Tue Nov 27 17:48:23 [rsSync] fastBuildIndex dupsToDrop:0 m31101| Tue Nov 27 17:48:23 [rsSync] build index done. scanned 1 total records. 0 secs m31101| Tue Nov 27 17:48:23 [rsSync] replSet initial sync data copy, starting syncup m31101| Tue Nov 27 17:48:23 [rsSync] oplog sync 1 of 3 m31101| Tue Nov 27 17:48:23 [rsSync] local.oplog.rs Assertion failure !loc.isNull() src/mongo/db/pdfile.cpp 1636 m31101| 0xaffd31 0xac65cd 0x85ce03 0x81e23b 0x9a78f6 0x9879c0 0x988ee8 0x98a027 0x9a5518 0x9a55ba 0x9a5878 0xb45ba9 0x7fd6454dce9a 0x7fd6447f24bd m31101| mongod-2.2(_ZN5mongo15printStackTraceERSo+0x21) [0xaffd31] m31101| mongod-2.2(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) [0xac65cd] m31101| mongod-2.2() [0x85ce03] m31101| mongod-2.2(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27b) [0x81e23b] m31101| mongod-2.2(_ZN5mongo7replset11InitialSync16oplogApplicationERKNS_7BSONObjES4_+0xe6) [0x9a78f6] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl30_syncDoInitialSync_applyToHeadERNS_7replset8SyncTailEPNS_11OplogReaderEPKNS_6MemberERKNS_7BSONObjERS9_+0x370) [0x9879c0] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl18_syncDoInitialSyncEv+0x8d8) [0x988ee8] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl17syncDoInitialSyncEv+0x37) [0x98a027] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl11_syncThreadEv+0x68) [0x9a5518] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x9a55ba] m31101| mongod-2.2(_ZN5mongo15startSyncThreadEv+0xa8) [0x9a5878] m31101| mongod-2.2() [0xb45ba9] m31101| /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fd6454dce9a] m31101| /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fd6447f24bd] m31101| Tue Nov 27 17:48:23 [rsSync] replSet initial sync failed during oplog application phase m31101| Tue Nov 27 17:48:23 [rsSync] replSet cleaning up [1] m31101| Tue Nov 27 17:48:23 [rsSync] replSet cleaning up [2] m31101| Tue Nov 27 17:48:23 [rsSync] replSet initial sync failed will try again m31101| Tue Nov 27 17:48:27 [conn14] end connection 127.0.0.1:35106 (10 connections now open) m31101| Tue Nov 27 17:48:27 [initandlisten] connection accepted from 127.0.0.1:35151 #16 (11 connections now open) m31101| Tue Nov 27 17:48:29 [rsSync] replSet initial sync pending m31101| Tue Nov 27 17:48:29 [rsSync] replSet syncing to: ubuntu:31100 m31101| Tue Nov 27 17:48:29 [rsSync] replSet initial sync drop all databases m31101| Tue Nov 27 17:48:29 [rsSync] dropAllDatabasesExceptLocal 3 m31101| Tue Nov 27 17:48:29 [rsSync] removeJournalFiles m31101| Tue Nov 27 17:48:29 [rsSync] removeJournalFiles m31101| Tue Nov 27 17:48:29 [rsSync] replSet initial sync clone all databases m31101| Tue Nov 27 17:48:29 [rsSync] replSet initial sync cloning db: foo20 m31101| Tue Nov 27 17:48:29 [FileAllocator] allocating new datafile /data/db/test-rs0-1/foo20.ns, filling with zeroes... m31101| Tue Nov 27 17:48:30 [FileAllocator] done allocating datafile /data/db/test-rs0-1/foo20.ns, size: 16MB, took 0.347 secs m31101| Tue Nov 27 17:48:30 [FileAllocator] allocating new datafile /data/db/test-rs0-1/foo20.0, filling with zeroes... m31101| Tue Nov 27 17:48:30 [FileAllocator] done allocating datafile /data/db/test-rs0-1/foo20.0, size: 16MB, took 0.313 secs m31101| Tue Nov 27 17:48:30 [rsSync] build index foo20.bar20 { _id: 1 } m31101| Tue Nov 27 17:48:30 [rsSync] fastBuildIndex dupsToDrop:0 m31101| Tue Nov 27 17:48:30 [rsSync] build index done. scanned 0 total records. 0 secs m31101| Tue Nov 27 17:48:30 [rsSync] replSet initial sync cloning db: admin m31101| Tue Nov 27 17:48:30 [FileAllocator] allocating new datafile /data/db/test-rs0-1/admin.ns, filling with zeroes... m31101| Tue Nov 27 17:48:31 [FileAllocator] done allocating datafile /data/db/test-rs0-1/admin.ns, size: 16MB, took 0.307 secs m31101| Tue Nov 27 17:48:31 [FileAllocator] allocating new datafile /data/db/test-rs0-1/admin.0, filling with zeroes... m31101| Tue Nov 27 17:48:31 [FileAllocator] done allocating datafile /data/db/test-rs0-1/admin.0, size: 16MB, took 0.303 secs m31101| Tue Nov 27 17:48:31 [rsSync] build index admin.foo { _id: 1 } m31101| Tue Nov 27 17:48:31 [rsSync] fastBuildIndex dupsToDrop:0 m31101| Tue Nov 27 17:48:31 [rsSync] build index done. scanned 1 total records. 0 secs m31101| Tue Nov 27 17:48:31 [rsSync] replSet initial sync data copy, starting syncup m31101| Tue Nov 27 17:48:31 [rsSync] oplog sync 1 of 3 m31101| Tue Nov 27 17:48:31 [rsSync] local.oplog.rs Assertion failure !loc.isNull() src/mongo/db/pdfile.cpp 1636 m31101| 0xaffd31 0xac65cd 0x85ce03 0x81e23b 0x9a78f6 0x9879c0 0x988ee8 0x98a027 0x9a5518 0x9a55ba 0x9a5878 0xb45ba9 0x7fd6454dce9a 0x7fd6447f24bd m31101| mongod-2.2(_ZN5mongo15printStackTraceERSo+0x21) [0xaffd31] m31101| mongod-2.2(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) [0xac65cd] m31101| mongod-2.2() [0x85ce03] m31101| mongod-2.2(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27b) [0x81e23b] m31101| mongod-2.2(_ZN5mongo7replset11InitialSync16oplogApplicationERKNS_7BSONObjES4_+0xe6) [0x9a78f6] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl30_syncDoInitialSync_applyToHeadERNS_7replset8SyncTailEPNS_11OplogReaderEPKNS_6MemberERKNS_7BSONObjERS9_+0x370) [0x9879c0] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl18_syncDoInitialSyncEv+0x8d8) [0x988ee8] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl17syncDoInitialSyncEv+0x37) [0x98a027] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl11_syncThreadEv+0x68) [0x9a5518] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x9a55ba] m31101| mongod-2.2(_ZN5mongo15startSyncThreadEv+0xa8) [0x9a5878] m31101| mongod-2.2() [0xb45ba9] m31101| /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fd6454dce9a] m31101| /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fd6447f24bd] m31101| Tue Nov 27 17:48:31 [rsSync] replSet initial sync failed during oplog application phase m31101| Tue Nov 27 17:48:31 [rsSync] replSet cleaning up [1] m31101| Tue Nov 27 17:48:31 [rsSync] replSet cleaning up [2] m31101| Tue Nov 27 17:48:31 [rsSync] replSet initial sync failed will try again m31101| Tue Nov 27 17:48:37 [rsSync] replSet initial sync pending m31101| Tue Nov 27 17:48:37 [rsSync] replSet syncing to: ubuntu:31100 m31101| Tue Nov 27 17:48:37 [rsSync] replSet initial sync drop all databases m31101| Tue Nov 27 17:48:37 [rsSync] dropAllDatabasesExceptLocal 3 m31101| Tue Nov 27 17:48:37 [rsSync] removeJournalFiles m31101| Tue Nov 27 17:48:37 [rsSync] removeJournalFiles m31101| Tue Nov 27 17:48:37 [rsSync] replSet initial sync clone all databases m31101| Tue Nov 27 17:48:37 [rsSync] replSet initial sync cloning db: foo20 m31101| Tue Nov 27 17:48:37 [FileAllocator] allocating new datafile /data/db/test-rs0-1/foo20.ns, filling with zeroes... m31101| Tue Nov 27 17:48:37 [FileAllocator] done allocating datafile /data/db/test-rs0-1/foo20.ns, size: 16MB, took 0.331 secs m31101| Tue Nov 27 17:48:37 [FileAllocator] allocating new datafile /data/db/test-rs0-1/foo20.0, filling with zeroes... m31101| Tue Nov 27 17:48:38 [FileAllocator] done allocating datafile /data/db/test-rs0-1/foo20.0, size: 16MB, took 0.312 secs m31101| Tue Nov 27 17:48:38 [rsSync] build index foo20.bar20 { _id: 1 } m31101| Tue Nov 27 17:48:38 [rsSync] fastBuildIndex dupsToDrop:0 m31101| Tue Nov 27 17:48:38 [rsSync] build index done. scanned 0 total records. 0 secs m31101| Tue Nov 27 17:48:38 [rsSync] replSet initial sync cloning db: admin m31101| Tue Nov 27 17:48:38 [FileAllocator] allocating new datafile /data/db/test-rs0-1/admin.ns, filling with zeroes... m31101| Tue Nov 27 17:48:38 [FileAllocator] done allocating datafile /data/db/test-rs0-1/admin.ns, size: 16MB, took 0.307 secs m31101| Tue Nov 27 17:48:38 [FileAllocator] allocating new datafile /data/db/test-rs0-1/admin.0, filling with zeroes... m31101| Tue Nov 27 17:48:38 [conn15] end connection 127.0.0.1:35124 (10 connections now open) m31101| Tue Nov 27 17:48:38 [initandlisten] connection accepted from 127.0.0.1:35169 #17 (11 connections now open) m31101| Tue Nov 27 17:48:38 [FileAllocator] done allocating datafile /data/db/test-rs0-1/admin.0, size: 16MB, took 0.295 secs m31101| Tue Nov 27 17:48:38 [rsSync] build index admin.foo { _id: 1 } m31101| Tue Nov 27 17:48:38 [rsSync] fastBuildIndex dupsToDrop:0 m31101| Tue Nov 27 17:48:38 [rsSync] build index done. scanned 1 total records. 0 secs m31101| Tue Nov 27 17:48:38 [rsSync] replSet initial sync data copy, starting syncup m31101| Tue Nov 27 17:48:38 [rsSync] oplog sync 1 of 3 m31101| Tue Nov 27 17:48:38 [rsSync] local.oplog.rs Assertion failure !loc.isNull() src/mongo/db/pdfile.cpp 1636 m31101| 0xaffd31 0xac65cd 0x85ce03 0x81e23b 0x9a78f6 0x9879c0 0x988ee8 0x98a027 0x9a5518 0x9a55ba 0x9a5878 0xb45ba9 0x7fd6454dce9a 0x7fd6447f24bd m31101| mongod-2.2(_ZN5mongo15printStackTraceERSo+0x21) [0xaffd31] m31101| mongod-2.2(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) [0xac65cd] m31101| mongod-2.2() [0x85ce03] m31101| mongod-2.2(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27b) [0x81e23b] m31101| mongod-2.2(_ZN5mongo7replset11InitialSync16oplogApplicationERKNS_7BSONObjES4_+0xe6) [0x9a78f6] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl30_syncDoInitialSync_applyToHeadERNS_7replset8SyncTailEPNS_11OplogReaderEPKNS_6MemberERKNS_7BSONObjERS9_+0x370) [0x9879c0] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl18_syncDoInitialSyncEv+0x8d8) [0x988ee8] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl17syncDoInitialSyncEv+0x37) [0x98a027] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl11_syncThreadEv+0x68) [0x9a5518] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x9a55ba] m31101| mongod-2.2(_ZN5mongo15startSyncThreadEv+0xa8) [0x9a5878] m31101| mongod-2.2() [0xb45ba9] m31101| /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fd6454dce9a] m31101| /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fd6447f24bd] m31101| Tue Nov 27 17:48:38 [rsSync] replSet initial sync failed during oplog application phase m31101| Tue Nov 27 17:48:38 [rsSync] replSet cleaning up [1] m31101| Tue Nov 27 17:48:38 [rsSync] replSet cleaning up [2] m31101| Tue Nov 27 17:48:38 [rsSync] replSet initial sync failed will try again m31101| Tue Nov 27 17:48:44 [rsSync] replSet initial sync pending m31101| Tue Nov 27 17:48:44 [rsSync] replSet syncing to: ubuntu:31100 m31101| Tue Nov 27 17:48:44 [rsSync] replSet initial sync drop all databases m31101| Tue Nov 27 17:48:44 [rsSync] dropAllDatabasesExceptLocal 3 m31101| Tue Nov 27 17:48:45 [rsSync] removeJournalFiles m31101| Tue Nov 27 17:48:45 [rsSync] removeJournalFiles m31101| Tue Nov 27 17:48:45 [rsSync] replSet initial sync clone all databases m31101| Tue Nov 27 17:48:45 [rsSync] replSet initial sync cloning db: foo20 m31101| Tue Nov 27 17:48:45 [FileAllocator] allocating new datafile /data/db/test-rs0-1/foo20.ns, filling with zeroes... m31101| Tue Nov 27 17:48:45 [FileAllocator] done allocating datafile /data/db/test-rs0-1/foo20.ns, size: 16MB, took 0.356 secs m31101| Tue Nov 27 17:48:45 [FileAllocator] allocating new datafile /data/db/test-rs0-1/foo20.0, filling with zeroes... m31101| Tue Nov 27 17:48:45 [FileAllocator] done allocating datafile /data/db/test-rs0-1/foo20.0, size: 16MB, took 0.311 secs m31101| Tue Nov 27 17:48:45 [rsSync] build index foo20.bar20 { _id: 1 } m31101| Tue Nov 27 17:48:45 [rsSync] fastBuildIndex dupsToDrop:0 m31101| Tue Nov 27 17:48:45 [rsSync] build index done. scanned 0 total records. 0 secs m31101| Tue Nov 27 17:48:45 [rsSync] replSet initial sync cloning db: admin m31101| Tue Nov 27 17:48:45 [FileAllocator] allocating new datafile /data/db/test-rs0-1/admin.ns, filling with zeroes... m31101| Tue Nov 27 17:48:46 [FileAllocator] done allocating datafile /data/db/test-rs0-1/admin.ns, size: 16MB, took 0.315 secs m31101| Tue Nov 27 17:48:46 [FileAllocator] allocating new datafile /data/db/test-rs0-1/admin.0, filling with zeroes... m31101| Tue Nov 27 17:48:46 [FileAllocator] done allocating datafile /data/db/test-rs0-1/admin.0, size: 16MB, took 0.295 secs m31101| Tue Nov 27 17:48:46 [rsSync] build index admin.foo { _id: 1 } m31101| Tue Nov 27 17:48:46 [rsSync] fastBuildIndex dupsToDrop:0 m31101| Tue Nov 27 17:48:46 [rsSync] build index done. scanned 1 total records. 0 secs m31101| Tue Nov 27 17:48:46 [rsSync] replSet initial sync data copy, starting syncup m31101| Tue Nov 27 17:48:46 [rsSync] oplog sync 1 of 3 m31101| Tue Nov 27 17:48:46 [rsSync] local.oplog.rs Assertion failure !loc.isNull() src/mongo/db/pdfile.cpp 1636 m31101| 0xaffd31 0xac65cd 0x85ce03 0x81e23b 0x9a78f6 0x9879c0 0x988ee8 0x98a027 0x9a5518 0x9a55ba 0x9a5878 0xb45ba9 0x7fd6454dce9a 0x7fd6447f24bd m31101| mongod-2.2(_ZN5mongo15printStackTraceERSo+0x21) [0xaffd31] m31101| mongod-2.2(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) [0xac65cd] m31101| mongod-2.2() [0x85ce03] m31101| mongod-2.2(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27b) [0x81e23b] m31101| mongod-2.2(_ZN5mongo7replset11InitialSync16oplogApplicationERKNS_7BSONObjES4_+0xe6) [0x9a78f6] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl30_syncDoInitialSync_applyToHeadERNS_7replset8SyncTailEPNS_11OplogReaderEPKNS_6MemberERKNS_7BSONObjERS9_+0x370) [0x9879c0] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl18_syncDoInitialSyncEv+0x8d8) [0x988ee8] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl17syncDoInitialSyncEv+0x37) [0x98a027] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl11_syncThreadEv+0x68) [0x9a5518] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x9a55ba] m31101| mongod-2.2(_ZN5mongo15startSyncThreadEv+0xa8) [0x9a5878] m31101| mongod-2.2() [0xb45ba9] m31101| /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fd6454dce9a] m31101| /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fd6447f24bd] m31101| Tue Nov 27 17:48:46 [rsSync] replSet initial sync failed during oplog application phase m31101| Tue Nov 27 17:48:46 [rsSync] replSet cleaning up [1] m31101| Tue Nov 27 17:48:46 [rsSync] replSet cleaning up [2] m31101| Tue Nov 27 17:48:46 [rsSync] replSet initial sync failed will try again m31101| Tue Nov 27 17:48:52 [rsSync] replSet initial sync pending m31101| Tue Nov 27 17:48:52 [rsSync] replSet syncing to: ubuntu:31100 m31101| Tue Nov 27 17:48:52 [rsSync] replSet initial sync drop all databases m31101| Tue Nov 27 17:48:52 [rsSync] dropAllDatabasesExceptLocal 3 m31101| Tue Nov 27 17:48:52 [rsSync] removeJournalFiles m31101| Tue Nov 27 17:48:52 [rsSync] removeJournalFiles m31101| Tue Nov 27 17:48:52 [rsSync] replSet initial sync clone all databases m31101| Tue Nov 27 17:48:52 [rsSync] replSet initial sync cloning db: foo20 m31101| Tue Nov 27 17:48:52 [FileAllocator] allocating new datafile /data/db/test-rs0-1/foo20.ns, filling with zeroes... m31101| Tue Nov 27 17:48:53 [FileAllocator] done allocating datafile /data/db/test-rs0-1/foo20.ns, size: 16MB, took 0.356 secs m31101| Tue Nov 27 17:48:53 [FileAllocator] allocating new datafile /data/db/test-rs0-1/foo20.0, filling with zeroes... m31101| Tue Nov 27 17:48:53 [FileAllocator] done allocating datafile /data/db/test-rs0-1/foo20.0, size: 16MB, took 0.303 secs m31101| Tue Nov 27 17:48:53 [rsSync] build index foo20.bar20 { _id: 1 } m31101| Tue Nov 27 17:48:53 [rsSync] fastBuildIndex dupsToDrop:0 m31101| Tue Nov 27 17:48:53 [rsSync] build index done. scanned 0 total records. 0 secs m31101| Tue Nov 27 17:48:53 [rsSync] replSet initial sync cloning db: admin m31101| Tue Nov 27 17:48:53 [FileAllocator] allocating new datafile /data/db/test-rs0-1/admin.ns, filling with zeroes... m31101| Tue Nov 27 17:48:53 [FileAllocator] done allocating datafile /data/db/test-rs0-1/admin.ns, size: 16MB, took 0.315 secs m31101| Tue Nov 27 17:48:53 [FileAllocator] allocating new datafile /data/db/test-rs0-1/admin.0, filling with zeroes... m31101| Tue Nov 27 17:48:54 [FileAllocator] done allocating datafile /data/db/test-rs0-1/admin.0, size: 16MB, took 0.329 secs m31101| Tue Nov 27 17:48:54 [rsSync] build index admin.foo { _id: 1 } m31101| Tue Nov 27 17:48:54 [rsSync] fastBuildIndex dupsToDrop:0 m31101| Tue Nov 27 17:48:54 [rsSync] build index done. scanned 1 total records. 0 secs m31101| Tue Nov 27 17:48:54 [rsSync] replSet initial sync data copy, starting syncup m31101| Tue Nov 27 17:48:54 [rsSync] oplog sync 1 of 3 m31101| Tue Nov 27 17:48:54 [rsSync] local.oplog.rs Assertion failure !loc.isNull() src/mongo/db/pdfile.cpp 1636 m31101| 0xaffd31 0xac65cd 0x85ce03 0x81e23b 0x9a78f6 0x9879c0 0x988ee8 0x98a027 0x9a5518 0x9a55ba 0x9a5878 0xb45ba9 0x7fd6454dce9a 0x7fd6447f24bd m31101| mongod-2.2(_ZN5mongo15printStackTraceERSo+0x21) [0xaffd31] m31101| mongod-2.2(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) [0xac65cd] m31101| mongod-2.2() [0x85ce03] m31101| mongod-2.2(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27b) [0x81e23b] m31101| mongod-2.2(_ZN5mongo7replset11InitialSync16oplogApplicationERKNS_7BSONObjES4_+0xe6) [0x9a78f6] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl30_syncDoInitialSync_applyToHeadERNS_7replset8SyncTailEPNS_11OplogReaderEPKNS_6MemberERKNS_7BSONObjERS9_+0x370) [0x9879c0] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl18_syncDoInitialSyncEv+0x8d8) [0x988ee8] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl17syncDoInitialSyncEv+0x37) [0x98a027] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl11_syncThreadEv+0x68) [0x9a5518] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x9a55ba] m31101| mongod-2.2(_ZN5mongo15startSyncThreadEv+0xa8) [0x9a5878] m31101| mongod-2.2() [0xb45ba9] m31101| /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fd6454dce9a] m31101| /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fd6447f24bd] m31101| Tue Nov 27 17:48:54 [rsSync] replSet initial sync failed during oplog application phase m31101| Tue Nov 27 17:48:54 [rsSync] replSet cleaning up [1] m31101| Tue Nov 27 17:48:54 [rsSync] replSet cleaning up [2] m31101| Tue Nov 27 17:48:54 [rsSync] replSet initial sync failed will try again m31101| Tue Nov 27 17:48:57 [conn16] end connection 127.0.0.1:35151 (10 connections now open) m31101| Tue Nov 27 17:48:57 [initandlisten] connection accepted from 127.0.0.1:35195 #18 (11 connections now open) m31101| Tue Nov 27 17:49:00 [rsSync] replSet initial sync pending m31101| Tue Nov 27 17:49:00 [rsSync] replSet syncing to: ubuntu:31100 m31101| Tue Nov 27 17:49:00 [rsSync] replSet initial sync drop all databases m31101| Tue Nov 27 17:49:00 [rsSync] dropAllDatabasesExceptLocal 3 m31101| Tue Nov 27 17:49:00 [rsSync] removeJournalFiles m31101| Tue Nov 27 17:49:00 [rsSync] removeJournalFiles m31101| Tue Nov 27 17:49:00 [rsSync] replSet initial sync clone all databases m31101| Tue Nov 27 17:49:00 [rsSync] replSet initial sync cloning db: foo20 m31101| Tue Nov 27 17:49:00 [FileAllocator] allocating new datafile /data/db/test-rs0-1/foo20.ns, filling with zeroes... m31101| Tue Nov 27 17:49:00 [FileAllocator] done allocating datafile /data/db/test-rs0-1/foo20.ns, size: 16MB, took 0.331 secs m31101| Tue Nov 27 17:49:00 [FileAllocator] allocating new datafile /data/db/test-rs0-1/foo20.0, filling with zeroes... m31101| Tue Nov 27 17:49:00 [FileAllocator] done allocating datafile /data/db/test-rs0-1/foo20.0, size: 16MB, took 0.295 secs m31101| Tue Nov 27 17:49:01 [rsSync] build index foo20.bar20 { _id: 1 } m31101| Tue Nov 27 17:49:01 [rsSync] fastBuildIndex dupsToDrop:0 m31101| Tue Nov 27 17:49:01 [rsSync] build index done. scanned 0 total records. 0 secs m31101| Tue Nov 27 17:49:01 [rsSync] replSet initial sync cloning db: admin m31101| Tue Nov 27 17:49:01 [FileAllocator] allocating new datafile /data/db/test-rs0-1/admin.ns, filling with zeroes... m31101| Tue Nov 27 17:49:01 [FileAllocator] done allocating datafile /data/db/test-rs0-1/admin.ns, size: 16MB, took 0.307 secs m31101| Tue Nov 27 17:49:01 [FileAllocator] allocating new datafile /data/db/test-rs0-1/admin.0, filling with zeroes... m31101| Tue Nov 27 17:49:01 [FileAllocator] done allocating datafile /data/db/test-rs0-1/admin.0, size: 16MB, took 0.304 secs m31101| Tue Nov 27 17:49:01 [rsSync] build index admin.foo { _id: 1 } m31101| Tue Nov 27 17:49:01 [rsSync] fastBuildIndex dupsToDrop:0 m31101| Tue Nov 27 17:49:01 [rsSync] build index done. scanned 1 total records. 0 secs m31101| Tue Nov 27 17:49:01 [rsSync] replSet initial sync data copy, starting syncup m31101| Tue Nov 27 17:49:01 [rsSync] oplog sync 1 of 3 m31101| Tue Nov 27 17:49:01 [rsSync] local.oplog.rs Assertion failure !loc.isNull() src/mongo/db/pdfile.cpp 1636 m31101| 0xaffd31 0xac65cd 0x85ce03 0x81e23b 0x9a78f6 0x9879c0 0x988ee8 0x98a027 0x9a5518 0x9a55ba 0x9a5878 0xb45ba9 0x7fd6454dce9a 0x7fd6447f24bd m31101| mongod-2.2(_ZN5mongo15printStackTraceERSo+0x21) [0xaffd31] m31101| mongod-2.2(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) [0xac65cd] m31101| mongod-2.2() [0x85ce03] m31101| mongod-2.2(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27b) [0x81e23b] m31101| mongod-2.2(_ZN5mongo7replset11InitialSync16oplogApplicationERKNS_7BSONObjES4_+0xe6) [0x9a78f6] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl30_syncDoInitialSync_applyToHeadERNS_7replset8SyncTailEPNS_11OplogReaderEPKNS_6MemberERKNS_7BSONObjERS9_+0x370) [0x9879c0] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl18_syncDoInitialSyncEv+0x8d8) [0x988ee8] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl17syncDoInitialSyncEv+0x37) [0x98a027] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl11_syncThreadEv+0x68) [0x9a5518] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x9a55ba] m31101| mongod-2.2(_ZN5mongo15startSyncThreadEv+0xa8) [0x9a5878] m31101| mongod-2.2() [0xb45ba9] m31101| /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fd6454dce9a] m31101| /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fd6447f24bd] m31101| Tue Nov 27 17:49:01 [rsSync] replSet initial sync failed during oplog application phase m31101| Tue Nov 27 17:49:01 [rsSync] replSet cleaning up [1] m31101| Tue Nov 27 17:49:01 [rsSync] replSet cleaning up [2] m31101| Tue Nov 27 17:49:01 [rsSync] replSet initial sync failed will try again m31101| Tue Nov 27 17:49:07 [rsSync] replSet initial sync pending m31101| Tue Nov 27 17:49:07 [rsSync] replSet syncing to: ubuntu:31100 m31101| Tue Nov 27 17:49:07 [rsSync] replSet initial sync drop all databases m31101| Tue Nov 27 17:49:07 [rsSync] dropAllDatabasesExceptLocal 3 m31101| Tue Nov 27 17:49:07 [rsSync] removeJournalFiles m31101| Tue Nov 27 17:49:07 [rsSync] removeJournalFiles m31101| Tue Nov 27 17:49:07 [rsSync] replSet initial sync clone all databases m31101| Tue Nov 27 17:49:07 [rsSync] replSet initial sync cloning db: foo20 m31101| Tue Nov 27 17:49:07 [FileAllocator] allocating new datafile /data/db/test-rs0-1/foo20.ns, filling with zeroes... m31101| Tue Nov 27 17:49:08 [FileAllocator] done allocating datafile /data/db/test-rs0-1/foo20.ns, size: 16MB, took 0.356 secs m31101| Tue Nov 27 17:49:08 [FileAllocator] allocating new datafile /data/db/test-rs0-1/foo20.0, filling with zeroes... m31101| Tue Nov 27 17:49:08 [FileAllocator] done allocating datafile /data/db/test-rs0-1/foo20.0, size: 16MB, took 0.388 secs m31101| Tue Nov 27 17:49:08 [conn17] end connection 127.0.0.1:35169 (10 connections now open) m31101| Tue Nov 27 17:49:08 [initandlisten] connection accepted from 127.0.0.1:35212 #19 (11 connections now open) m31101| Tue Nov 27 17:49:08 [rsSync] build index foo20.bar20 { _id: 1 } m31101| Tue Nov 27 17:49:08 [rsSync] fastBuildIndex dupsToDrop:0 m31101| Tue Nov 27 17:49:08 [rsSync] build index done. scanned 0 total records. 0 secs m31101| Tue Nov 27 17:49:08 [rsSync] replSet initial sync cloning db: admin m31101| Tue Nov 27 17:49:08 [FileAllocator] allocating new datafile /data/db/test-rs0-1/admin.ns, filling with zeroes... m31101| Tue Nov 27 17:49:09 [FileAllocator] done allocating datafile /data/db/test-rs0-1/admin.ns, size: 16MB, took 0.323 secs m31101| Tue Nov 27 17:49:09 [FileAllocator] allocating new datafile /data/db/test-rs0-1/admin.0, filling with zeroes... m31101| Tue Nov 27 17:49:09 [FileAllocator] done allocating datafile /data/db/test-rs0-1/admin.0, size: 16MB, took 0.286 secs m31101| Tue Nov 27 17:49:09 [rsSync] build index admin.foo { _id: 1 } m31101| Tue Nov 27 17:49:09 [rsSync] fastBuildIndex dupsToDrop:0 m31101| Tue Nov 27 17:49:09 [rsSync] build index done. scanned 1 total records. 0 secs m31101| Tue Nov 27 17:49:09 [rsSync] replSet initial sync data copy, starting syncup m31101| Tue Nov 27 17:49:09 [rsSync] oplog sync 1 of 3 m31101| Tue Nov 27 17:49:09 [rsSync] local.oplog.rs Assertion failure !loc.isNull() src/mongo/db/pdfile.cpp 1636 m31101| 0xaffd31 0xac65cd 0x85ce03 0x81e23b 0x9a78f6 0x9879c0 0x988ee8 0x98a027 0x9a5518 0x9a55ba 0x9a5878 0xb45ba9 0x7fd6454dce9a 0x7fd6447f24bd m31101| mongod-2.2(_ZN5mongo15printStackTraceERSo+0x21) [0xaffd31] m31101| mongod-2.2(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) [0xac65cd] m31101| mongod-2.2() [0x85ce03] m31101| mongod-2.2(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27b) [0x81e23b] m31101| mongod-2.2(_ZN5mongo7replset11InitialSync16oplogApplicationERKNS_7BSONObjES4_+0xe6) [0x9a78f6] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl30_syncDoInitialSync_applyToHeadERNS_7replset8SyncTailEPNS_11OplogReaderEPKNS_6MemberERKNS_7BSONObjERS9_+0x370) [0x9879c0] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl18_syncDoInitialSyncEv+0x8d8) [0x988ee8] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl17syncDoInitialSyncEv+0x37) [0x98a027] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl11_syncThreadEv+0x68) [0x9a5518] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x9a55ba] m31101| mongod-2.2(_ZN5mongo15startSyncThreadEv+0xa8) [0x9a5878] m31101| mongod-2.2() [0xb45ba9] m31101| /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fd6454dce9a] m31101| /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fd6447f24bd] m31101| Tue Nov 27 17:49:09 [rsSync] replSet initial sync failed during oplog application phase m31101| Tue Nov 27 17:49:09 [rsSync] replSet cleaning up [1] m31101| Tue Nov 27 17:49:09 [rsSync] replSet cleaning up [2] m31101| Tue Nov 27 17:49:09 [rsSync] replSet initial sync failed will try again m31101| Tue Nov 27 17:49:15 [rsSync] replSet initial sync pending m31101| Tue Nov 27 17:49:15 [rsSync] replSet syncing to: ubuntu:31100 m31101| Tue Nov 27 17:49:15 [rsSync] replSet initial sync drop all databases m31101| Tue Nov 27 17:49:15 [rsSync] dropAllDatabasesExceptLocal 3 m31101| Tue Nov 27 17:49:15 [rsSync] removeJournalFiles m31101| Tue Nov 27 17:49:15 [rsSync] removeJournalFiles m31101| Tue Nov 27 17:49:15 [rsSync] replSet initial sync clone all databases m31101| Tue Nov 27 17:49:15 [rsSync] replSet initial sync cloning db: foo20 m31101| Tue Nov 27 17:49:15 [FileAllocator] allocating new datafile /data/db/test-rs0-1/foo20.ns, filling with zeroes... m31101| Tue Nov 27 17:49:15 [FileAllocator] done allocating datafile /data/db/test-rs0-1/foo20.ns, size: 16MB, took 0.339 secs m31101| Tue Nov 27 17:49:15 [FileAllocator] allocating new datafile /data/db/test-rs0-1/foo20.0, filling with zeroes... m31101| Tue Nov 27 17:49:16 [FileAllocator] done allocating datafile /data/db/test-rs0-1/foo20.0, size: 16MB, took 0.322 secs m31101| Tue Nov 27 17:49:16 [rsSync] build index foo20.bar20 { _id: 1 } m31101| Tue Nov 27 17:49:16 [rsSync] fastBuildIndex dupsToDrop:0 m31101| Tue Nov 27 17:49:16 [rsSync] build index done. scanned 0 total records. 0 secs m31101| Tue Nov 27 17:49:16 [rsSync] replSet initial sync cloning db: admin m31101| Tue Nov 27 17:49:16 [FileAllocator] allocating new datafile /data/db/test-rs0-1/admin.ns, filling with zeroes... m31101| Tue Nov 27 17:49:16 [FileAllocator] done allocating datafile /data/db/test-rs0-1/admin.ns, size: 16MB, took 0.315 secs m31101| Tue Nov 27 17:49:16 [FileAllocator] allocating new datafile /data/db/test-rs0-1/admin.0, filling with zeroes... m31101| Tue Nov 27 17:49:16 [FileAllocator] done allocating datafile /data/db/test-rs0-1/admin.0, size: 16MB, took 0.296 secs m31101| Tue Nov 27 17:49:16 [rsSync] build index admin.foo { _id: 1 } m31101| Tue Nov 27 17:49:16 [rsSync] fastBuildIndex dupsToDrop:0 m31101| Tue Nov 27 17:49:16 [rsSync] build index done. scanned 1 total records. 0 secs m31101| Tue Nov 27 17:49:16 [rsSync] replSet initial sync data copy, starting syncup m31101| Tue Nov 27 17:49:16 [rsSync] oplog sync 1 of 3 m31101| Tue Nov 27 17:49:16 [rsSync] local.oplog.rs Assertion failure !loc.isNull() src/mongo/db/pdfile.cpp 1636 m31101| 0xaffd31 0xac65cd 0x85ce03 0x81e23b 0x9a78f6 0x9879c0 0x988ee8 0x98a027 0x9a5518 0x9a55ba 0x9a5878 0xb45ba9 0x7fd6454dce9a 0x7fd6447f24bd m31101| mongod-2.2(_ZN5mongo15printStackTraceERSo+0x21) [0xaffd31] m31101| mongod-2.2(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) [0xac65cd] m31101| mongod-2.2() [0x85ce03] m31101| mongod-2.2(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27b) [0x81e23b] m31101| mongod-2.2(_ZN5mongo7replset11InitialSync16oplogApplicationERKNS_7BSONObjES4_+0xe6) [0x9a78f6] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl30_syncDoInitialSync_applyToHeadERNS_7replset8SyncTailEPNS_11OplogReaderEPKNS_6MemberERKNS_7BSONObjERS9_+0x370) [0x9879c0] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl18_syncDoInitialSyncEv+0x8d8) [0x988ee8] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl17syncDoInitialSyncEv+0x37) [0x98a027] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl11_syncThreadEv+0x68) [0x9a5518] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x9a55ba] m31101| mongod-2.2(_ZN5mongo15startSyncThreadEv+0xa8) [0x9a5878] m31101| mongod-2.2() [0xb45ba9] m31101| /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fd6454dce9a] m31101| /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fd6447f24bd] m31101| Tue Nov 27 17:49:16 [rsSync] replSet initial sync failed during oplog application phase m31101| Tue Nov 27 17:49:16 [rsSync] replSet cleaning up [1] m31101| Tue Nov 27 17:49:16 [rsSync] replSet cleaning up [2] m31101| Tue Nov 27 17:49:16 [rsSync] replSet initial sync failed will try again m31101| Tue Nov 27 17:49:22 [rsSync] replSet initial sync pending m31101| Tue Nov 27 17:49:22 [rsSync] replSet syncing to: ubuntu:31100 m31101| Tue Nov 27 17:49:22 [rsSync] replSet initial sync drop all databases m31101| Tue Nov 27 17:49:22 [rsSync] dropAllDatabasesExceptLocal 3 m31101| Tue Nov 27 17:49:23 [rsSync] removeJournalFiles m31101| Tue Nov 27 17:49:23 [rsSync] removeJournalFiles m31101| Tue Nov 27 17:49:23 [rsSync] replSet initial sync clone all databases m31101| Tue Nov 27 17:49:23 [rsSync] replSet initial sync cloning db: foo20 m31101| Tue Nov 27 17:49:23 [FileAllocator] allocating new datafile /data/db/test-rs0-1/foo20.ns, filling with zeroes... m31101| Tue Nov 27 17:49:23 [FileAllocator] done allocating datafile /data/db/test-rs0-1/foo20.ns, size: 16MB, took 0.323 secs m31101| Tue Nov 27 17:49:23 [FileAllocator] allocating new datafile /data/db/test-rs0-1/foo20.0, filling with zeroes... m31101| Tue Nov 27 17:49:23 [FileAllocator] done allocating datafile /data/db/test-rs0-1/foo20.0, size: 16MB, took 0.312 secs m31101| Tue Nov 27 17:49:23 [rsSync] build index foo20.bar20 { _id: 1 } m31101| Tue Nov 27 17:49:23 [rsSync] fastBuildIndex dupsToDrop:0 m31101| Tue Nov 27 17:49:23 [rsSync] build index done. scanned 0 total records. 0 secs m31101| Tue Nov 27 17:49:23 [rsSync] replSet initial sync cloning db: admin m31101| Tue Nov 27 17:49:23 [FileAllocator] allocating new datafile /data/db/test-rs0-1/admin.ns, filling with zeroes... m31101| Tue Nov 27 17:49:24 [FileAllocator] done allocating datafile /data/db/test-rs0-1/admin.ns, size: 16MB, took 0.325 secs m31101| Tue Nov 27 17:49:24 [FileAllocator] allocating new datafile /data/db/test-rs0-1/admin.0, filling with zeroes... m31101| Tue Nov 27 17:49:24 [FileAllocator] done allocating datafile /data/db/test-rs0-1/admin.0, size: 16MB, took 0.304 secs m31101| Tue Nov 27 17:49:24 [rsSync] build index admin.foo { _id: 1 } m31101| Tue Nov 27 17:49:24 [rsSync] fastBuildIndex dupsToDrop:0 m31101| Tue Nov 27 17:49:24 [rsSync] build index done. scanned 1 total records. 0 secs m31101| Tue Nov 27 17:49:24 [rsSync] replSet initial sync data copy, starting syncup m31101| Tue Nov 27 17:49:24 [rsSync] oplog sync 1 of 3 m31101| Tue Nov 27 17:49:24 [rsSync] local.oplog.rs Assertion failure !loc.isNull() src/mongo/db/pdfile.cpp 1636 m31101| 0xaffd31 0xac65cd 0x85ce03 0x81e23b 0x9a78f6 0x9879c0 0x988ee8 0x98a027 0x9a5518 0x9a55ba 0x9a5878 0xb45ba9 0x7fd6454dce9a 0x7fd6447f24bd m31101| mongod-2.2(_ZN5mongo15printStackTraceERSo+0x21) [0xaffd31] m31101| mongod-2.2(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) [0xac65cd] m31101| mongod-2.2() [0x85ce03] m31101| mongod-2.2(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27b) [0x81e23b] m31101| mongod-2.2(_ZN5mongo7replset11InitialSync16oplogApplicationERKNS_7BSONObjES4_+0xe6) [0x9a78f6] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl30_syncDoInitialSync_applyToHeadERNS_7replset8SyncTailEPNS_11OplogReaderEPKNS_6MemberERKNS_7BSONObjERS9_+0x370) [0x9879c0] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl18_syncDoInitialSyncEv+0x8d8) [0x988ee8] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl17syncDoInitialSyncEv+0x37) [0x98a027] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl11_syncThreadEv+0x68) [0x9a5518] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x9a55ba] m31101| mongod-2.2(_ZN5mongo15startSyncThreadEv+0xa8) [0x9a5878] m31101| mongod-2.2() [0xb45ba9] m31101| /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fd6454dce9a] m31101| /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fd6447f24bd] m31101| Tue Nov 27 17:49:24 [rsSync] replSet initial sync failed during oplog application phase m31101| Tue Nov 27 17:49:24 [rsSync] replSet cleaning up [1] m31101| Tue Nov 27 17:49:24 [rsSync] replSet cleaning up [2] m31101| Tue Nov 27 17:49:24 [rsSync] replSet initial sync failed will try again m31101| Tue Nov 27 17:49:27 [conn18] end connection 127.0.0.1:35195 (10 connections now open) m31101| Tue Nov 27 17:49:27 [initandlisten] connection accepted from 127.0.0.1:35240 #20 (11 connections now open) m31101| Tue Nov 27 17:49:30 [rsSync] replSet initial sync pending m31101| Tue Nov 27 17:49:30 [rsSync] replSet syncing to: ubuntu:31100 m31101| Tue Nov 27 17:49:30 [rsSync] replSet initial sync drop all databases m31101| Tue Nov 27 17:49:30 [rsSync] dropAllDatabasesExceptLocal 3 m31101| Tue Nov 27 17:49:30 [rsSync] removeJournalFiles m31101| Tue Nov 27 17:49:30 [rsSync] removeJournalFiles m31101| Tue Nov 27 17:49:30 [rsSync] replSet initial sync clone all databases m31101| Tue Nov 27 17:49:30 [rsSync] replSet initial sync cloning db: foo20 m31101| Tue Nov 27 17:49:30 [FileAllocator] allocating new datafile /data/db/test-rs0-1/foo20.ns, filling with zeroes... m31101| Tue Nov 27 17:49:31 [FileAllocator] done allocating datafile /data/db/test-rs0-1/foo20.ns, size: 16MB, took 0.347 secs m31101| Tue Nov 27 17:49:31 [FileAllocator] allocating new datafile /data/db/test-rs0-1/foo20.0, filling with zeroes... m31101| Tue Nov 27 17:49:31 [FileAllocator] done allocating datafile /data/db/test-rs0-1/foo20.0, size: 16MB, took 0.295 secs m31101| Tue Nov 27 17:49:31 [rsSync] build index foo20.bar20 { _id: 1 } m31101| Tue Nov 27 17:49:31 [rsSync] fastBuildIndex dupsToDrop:0 m31101| Tue Nov 27 17:49:31 [rsSync] build index done. scanned 0 total records. 0 secs m31101| Tue Nov 27 17:49:31 [rsSync] replSet initial sync cloning db: admin m31101| Tue Nov 27 17:49:31 [FileAllocator] allocating new datafile /data/db/test-rs0-1/admin.ns, filling with zeroes... m31101| Tue Nov 27 17:49:31 [FileAllocator] done allocating datafile /data/db/test-rs0-1/admin.ns, size: 16MB, took 0.307 secs m31101| Tue Nov 27 17:49:31 [FileAllocator] allocating new datafile /data/db/test-rs0-1/admin.0, filling with zeroes... m31101| Tue Nov 27 17:49:32 [FileAllocator] done allocating datafile /data/db/test-rs0-1/admin.0, size: 16MB, took 0.311 secs m31101| Tue Nov 27 17:49:32 [rsSync] build index admin.foo { _id: 1 } m31101| Tue Nov 27 17:49:32 [rsSync] fastBuildIndex dupsToDrop:0 m31101| Tue Nov 27 17:49:32 [rsSync] build index done. scanned 1 total records. 0 secs m31101| Tue Nov 27 17:49:32 [rsSync] replSet initial sync data copy, starting syncup m31101| Tue Nov 27 17:49:32 [rsSync] oplog sync 1 of 3 m31101| Tue Nov 27 17:49:32 [rsSync] local.oplog.rs Assertion failure !loc.isNull() src/mongo/db/pdfile.cpp 1636 m31101| 0xaffd31 0xac65cd 0x85ce03 0x81e23b 0x9a78f6 0x9879c0 0x988ee8 0x98a027 0x9a5518 0x9a55ba 0x9a5878 0xb45ba9 0x7fd6454dce9a 0x7fd6447f24bd m31101| mongod-2.2(_ZN5mongo15printStackTraceERSo+0x21) [0xaffd31] m31101| mongod-2.2(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) [0xac65cd] m31101| mongod-2.2() [0x85ce03] m31101| mongod-2.2(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27b) [0x81e23b] m31101| mongod-2.2(_ZN5mongo7replset11InitialSync16oplogApplicationERKNS_7BSONObjES4_+0xe6) [0x9a78f6] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl30_syncDoInitialSync_applyToHeadERNS_7replset8SyncTailEPNS_11OplogReaderEPKNS_6MemberERKNS_7BSONObjERS9_+0x370) [0x9879c0] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl18_syncDoInitialSyncEv+0x8d8) [0x988ee8] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl17syncDoInitialSyncEv+0x37) [0x98a027] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl11_syncThreadEv+0x68) [0x9a5518] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x9a55ba] m31101| mongod-2.2(_ZN5mongo15startSyncThreadEv+0xa8) [0x9a5878] m31101| mongod-2.2() [0xb45ba9] m31101| /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fd6454dce9a] m31101| /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fd6447f24bd] m31101| Tue Nov 27 17:49:32 [rsSync] replSet initial sync failed during oplog application phase m31101| Tue Nov 27 17:49:32 [rsSync] replSet cleaning up [1] m31101| Tue Nov 27 17:49:32 [rsSync] replSet cleaning up [2] m31101| Tue Nov 27 17:49:32 [rsSync] replSet initial sync failed will try again m31101| Tue Nov 27 17:49:38 [rsSync] replSet initial sync pending m31101| Tue Nov 27 17:49:38 [rsSync] replSet syncing to: ubuntu:31100 m31101| Tue Nov 27 17:49:38 [rsSync] replSet initial sync drop all databases m31101| Tue Nov 27 17:49:38 [rsSync] dropAllDatabasesExceptLocal 3 m31101| Tue Nov 27 17:49:38 [rsSync] removeJournalFiles m31101| Tue Nov 27 17:49:38 [rsSync] removeJournalFiles m31101| Tue Nov 27 17:49:38 [rsSync] replSet initial sync clone all databases m31101| Tue Nov 27 17:49:38 [rsSync] replSet initial sync cloning db: foo20 m31101| Tue Nov 27 17:49:38 [FileAllocator] allocating new datafile /data/db/test-rs0-1/foo20.ns, filling with zeroes... m31101| Tue Nov 27 17:49:38 [FileAllocator] done allocating datafile /data/db/test-rs0-1/foo20.ns, size: 16MB, took 0.382 secs m31101| Tue Nov 27 17:49:38 [FileAllocator] allocating new datafile /data/db/test-rs0-1/foo20.0, filling with zeroes... m31101| Tue Nov 27 17:49:38 [conn19] end connection 127.0.0.1:35212 (10 connections now open) m31101| Tue Nov 27 17:49:38 [initandlisten] connection accepted from 127.0.0.1:35257 #21 (11 connections now open) m31101| Tue Nov 27 17:49:38 [FileAllocator] done allocating datafile /data/db/test-rs0-1/foo20.0, size: 16MB, took 0.312 secs m31101| Tue Nov 27 17:49:38 [rsSync] build index foo20.bar20 { _id: 1 } m31101| Tue Nov 27 17:49:38 [rsSync] fastBuildIndex dupsToDrop:0 m31101| Tue Nov 27 17:49:38 [rsSync] build index done. scanned 0 total records. 0 secs m31101| Tue Nov 27 17:49:38 [rsSync] replSet initial sync cloning db: admin m31101| Tue Nov 27 17:49:38 [FileAllocator] allocating new datafile /data/db/test-rs0-1/admin.ns, filling with zeroes... m31101| Tue Nov 27 17:49:39 [FileAllocator] done allocating datafile /data/db/test-rs0-1/admin.ns, size: 16MB, took 0.307 secs m31101| Tue Nov 27 17:49:39 [FileAllocator] allocating new datafile /data/db/test-rs0-1/admin.0, filling with zeroes... m31101| Tue Nov 27 17:49:39 [FileAllocator] done allocating datafile /data/db/test-rs0-1/admin.0, size: 16MB, took 0.303 secs m31101| Tue Nov 27 17:49:40 [rsSync] build index admin.foo { _id: 1 } m31101| Tue Nov 27 17:49:40 [rsSync] fastBuildIndex dupsToDrop:0 m31101| Tue Nov 27 17:49:40 [rsSync] build index done. scanned 1 total records. 0 secs m31101| Tue Nov 27 17:49:40 [rsSync] replSet initial sync data copy, starting syncup m31101| Tue Nov 27 17:49:40 [rsSync] oplog sync 1 of 3 m31101| Tue Nov 27 17:49:40 [rsSync] local.oplog.rs Assertion failure !loc.isNull() src/mongo/db/pdfile.cpp 1636 m31101| 0xaffd31 0xac65cd 0x85ce03 0x81e23b 0x9a78f6 0x9879c0 0x988ee8 0x98a027 0x9a5518 0x9a55ba 0x9a5878 0xb45ba9 0x7fd6454dce9a 0x7fd6447f24bd m31101| mongod-2.2(_ZN5mongo15printStackTraceERSo+0x21) [0xaffd31] m31101| mongod-2.2(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) [0xac65cd] m31101| mongod-2.2() [0x85ce03] m31101| mongod-2.2(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27b) [0x81e23b] m31101| mongod-2.2(_ZN5mongo7replset11InitialSync16oplogApplicationERKNS_7BSONObjES4_+0xe6) [0x9a78f6] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl30_syncDoInitialSync_applyToHeadERNS_7replset8SyncTailEPNS_11OplogReaderEPKNS_6MemberERKNS_7BSONObjERS9_+0x370) [0x9879c0] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl18_syncDoInitialSyncEv+0x8d8) [0x988ee8] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl17syncDoInitialSyncEv+0x37) [0x98a027] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl11_syncThreadEv+0x68) [0x9a5518] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x9a55ba] m31101| mongod-2.2(_ZN5mongo15startSyncThreadEv+0xa8) [0x9a5878] m31101| mongod-2.2() [0xb45ba9] m31101| /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fd6454dce9a] m31101| /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fd6447f24bd] m31101| Tue Nov 27 17:49:40 [rsSync] replSet initial sync failed during oplog application phase m31101| Tue Nov 27 17:49:40 [rsSync] replSet cleaning up [1] m31101| Tue Nov 27 17:49:40 [rsSync] replSet cleaning up [2] m31101| Tue Nov 27 17:49:40 [rsSync] replSet initial sync failed will try again m31101| Tue Nov 27 17:49:46 [rsSync] replSet initial sync pending m31101| Tue Nov 27 17:49:46 [rsSync] replSet syncing to: ubuntu:31100 m31101| Tue Nov 27 17:49:46 [rsSync] replSet initial sync drop all databases m31101| Tue Nov 27 17:49:46 [rsSync] dropAllDatabasesExceptLocal 3 m31101| Tue Nov 27 17:49:46 [rsSync] removeJournalFiles m31101| Tue Nov 27 17:49:46 [rsSync] removeJournalFiles m31101| Tue Nov 27 17:49:46 [rsSync] replSet initial sync clone all databases m31101| Tue Nov 27 17:49:46 [rsSync] replSet initial sync cloning db: foo20 m31101| Tue Nov 27 17:49:46 [FileAllocator] allocating new datafile /data/db/test-rs0-1/foo20.ns, filling with zeroes... m31101| Tue Nov 27 17:49:46 [FileAllocator] done allocating datafile /data/db/test-rs0-1/foo20.ns, size: 16MB, took 0.33 secs m31101| Tue Nov 27 17:49:46 [FileAllocator] allocating new datafile /data/db/test-rs0-1/foo20.0, filling with zeroes... m31101| Tue Nov 27 17:49:46 [FileAllocator] done allocating datafile /data/db/test-rs0-1/foo20.0, size: 16MB, took 0.311 secs m31101| Tue Nov 27 17:49:46 [rsSync] build index foo20.bar20 { _id: 1 } m31101| Tue Nov 27 17:49:46 [rsSync] fastBuildIndex dupsToDrop:0 m31101| Tue Nov 27 17:49:46 [rsSync] build index done. scanned 0 total records. 0 secs m31101| Tue Nov 27 17:49:46 [rsSync] replSet initial sync cloning db: admin m31101| Tue Nov 27 17:49:46 [FileAllocator] allocating new datafile /data/db/test-rs0-1/admin.ns, filling with zeroes... m31101| Tue Nov 27 17:49:47 [FileAllocator] done allocating datafile /data/db/test-rs0-1/admin.ns, size: 16MB, took 0.307 secs m31101| Tue Nov 27 17:49:47 [FileAllocator] allocating new datafile /data/db/test-rs0-1/admin.0, filling with zeroes... m31101| Tue Nov 27 17:49:47 [FileAllocator] done allocating datafile /data/db/test-rs0-1/admin.0, size: 16MB, took 0.303 secs m31101| Tue Nov 27 17:49:47 [rsSync] build index admin.foo { _id: 1 } m31101| Tue Nov 27 17:49:47 [rsSync] fastBuildIndex dupsToDrop:0 m31101| Tue Nov 27 17:49:47 [rsSync] build index done. scanned 1 total records. 0 secs m31101| Tue Nov 27 17:49:47 [rsSync] replSet initial sync data copy, starting syncup m31101| Tue Nov 27 17:49:47 [rsSync] oplog sync 1 of 3 m31101| Tue Nov 27 17:49:47 [rsSync] local.oplog.rs Assertion failure !loc.isNull() src/mongo/db/pdfile.cpp 1636 m31101| 0xaffd31 0xac65cd 0x85ce03 0x81e23b 0x9a78f6 0x9879c0 0x988ee8 0x98a027 0x9a5518 0x9a55ba 0x9a5878 0xb45ba9 0x7fd6454dce9a 0x7fd6447f24bd m31101| mongod-2.2(_ZN5mongo15printStackTraceERSo+0x21) [0xaffd31] m31101| mongod-2.2(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) [0xac65cd] m31101| mongod-2.2() [0x85ce03] m31101| mongod-2.2(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27b) [0x81e23b] m31101| mongod-2.2(_ZN5mongo7replset11InitialSync16oplogApplicationERKNS_7BSONObjES4_+0xe6) [0x9a78f6] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl30_syncDoInitialSync_applyToHeadERNS_7replset8SyncTailEPNS_11OplogReaderEPKNS_6MemberERKNS_7BSONObjERS9_+0x370) [0x9879c0] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl18_syncDoInitialSyncEv+0x8d8) [0x988ee8] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl17syncDoInitialSyncEv+0x37) [0x98a027] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl11_syncThreadEv+0x68) [0x9a5518] m31101| mongod-2.2(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x9a55ba] m31101| mongod-2.2(_ZN5mongo15startSyncThreadEv+0xa8) [0x9a5878] m31101| mongod-2.2() [0xb45ba9] m31101| /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fd6454dce9a] m31101| /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fd6447f24bd] m31101| Tue Nov 27 17:49:47 [rsSync] replSet initial sync failed during oplog application phase m31101| Tue Nov 27 17:49:47 [rsSync] replSet cleaning up [1] m31101| Tue Nov 27 17:49:47 [rsSync] replSet cleaning up [2] m31101| Tue Nov 27 17:49:47 [rsSync] replSet initial sync failed will try again m31101| Tue Nov 27 17:49:53 [rsSync] replSet initial sync pending m31101| Tue Nov 27 17:49:53 [rsSync] replSet syncing to: ubuntu:31100 m31101| Tue Nov 27 17:49:53 [rsSync] replSet initial sync drop all databases m31101| Tue Nov 27 17:49:53 [rsSync] dropAllDatabasesExceptLocal 3 m31101| Tue Nov 27 17:49:53 [rsSync] removeJournalFiles m31101| Tue Nov 27 17:49:53 [rsSync] removeJournalFiles m31101| Tue Nov 27 17:49:53 [rsSync] replSet initial sync clone all databases m31101| Tue Nov 27 17:49:53 [rsSync] replSet initial sync cloning db: foo20 m31101| Tue Nov 27 17:49:53 [FileAllocator] allocating new datafile /data/db/test-rs0-1/foo20.ns, filling with zeroes... m31101| Tue Nov 27 17:49:54 [FileAllocator] done allocating datafile /data/db/test-rs0-1/foo20.ns, size: 16MB, took 0.356 secs m31101| Tue Nov 27 17:49:54 [FileAllocator] allocating new datafile /data/db/test-rs0-1/foo20.0, filling with zeroes... m31101| Tue Nov 27 17:49:54 [FileAllocator] done allocating datafile /data/db/test-rs0-1/foo20.0, size: 16MB, took 0.32 secs m31101| Tue Nov 27 17:49:54 [rsSync] build index foo20.bar20 { _id: 1 } m31101| Tue Nov 27 17:49:54 [rsSync] fastBuildIndex dupsToDrop:0 m31101| Tue Nov 27 17:49:54 [rsSync] build index done. scanned 0 total records. 0 secs m31101| Tue Nov 27 17:49:54 [rsSync] replSet initial sync cloning db: admin m31101| Tue Nov 27 17:49:54 [FileAllocator] allocating new datafile /data/db/test-rs0-1/admin.ns, filling with zeroes... m31101| Tue Nov 27 17:49:54 [FileAllocator] done allocating datafile /data/db/test-rs0-1/admin.ns, size: 16MB, took 0.353 secs m31101| Tue Nov 27 17:49:54 [FileAllocator] allocating new datafile /data/db/test-rs0-1/admin.0, filling with zeroes...