Mon Jun 24 04:27:28.028 [rsSync] fastBuildIndex dupsToDrop:0 Mon Jun 24 04:27:28.028 [rsSync] build index done. scanned 310 total records. 0.002 secs Mon Jun 24 04:27:28.566 [conn3518] end connection 217.70.34.135:58157 (1 connection now open) Mon Jun 24 04:27:28.627 [initandlisten] connection accepted from 217.70.34.135:58160 #3521 (2 connections now open) Mon Jun 24 04:27:28.682 [conn3521] authenticate db: local { authenticate: 1, nonce: "ca1a9ef5f5489ca7", user: "__system", key: "1f7804cfffc68ac5b5c299ac585fb2fc" } Mon Jun 24 04:27:34.156 [conn3520] end connection 217.70.33.64:52456 (1 connection now open) Mon Jun 24 04:27:34.212 [initandlisten] connection accepted from 217.70.33.64:52488 #3522 (2 connections now open) Mon Jun 24 04:27:34.267 [conn3522] authenticate db: local { authenticate: 1, nonce: "e8184579f7a303e3", user: "__system", key: "8ce13707ea8bd86c825185bc6f63db62" } Mon Jun 24 04:27:54.953 [rsSync] build index jerome5.UserMessage { _id: 1 } Mon Jun 24 04:27:54.962 [rsSync] fastBuildIndex dupsToDrop:0 Mon Jun 24 04:27:54.963 [rsSync] build index done. scanned 1712 total records. 0.009 secs Mon Jun 24 04:27:55.017 [rsSync] build index jerome5.ShopifyShop { _id: 1 } Mon Jun 24 04:27:55.018 [rsSync] fastBuildIndex dupsToDrop:0 Mon Jun 24 04:27:55.018 [rsSync] build index done. scanned 1 total records. 0 secs Mon Jun 24 04:27:59.567 [conn3521] end connection 217.70.34.135:58160 (1 connection now open) Mon Jun 24 04:27:59.626 [initandlisten] connection accepted from 217.70.34.135:58164 #3523 (2 connections now open) Mon Jun 24 04:27:59.674 [conn3523] authenticate db: local { authenticate: 1, nonce: "c5e2720d54bfc923", user: "__system", key: "6065ce8f6971787938bb14ed4fada9b3" } Mon Jun 24 04:28:00.997 [rsSync] build index jerome5.PathAlias { _id: 1 } Mon Jun 24 04:28:01.069 [rsSync] fastBuildIndex dupsToDrop:0 Mon Jun 24 04:28:01.071 [rsSync] build index done. scanned 55016 total records. 0.073 secs Mon Jun 24 04:28:01.071 [rsSync] replSet initial sync cloning db: admin Mon Jun 24 04:28:01.129 [FileAllocator] allocating new datafile /var/lib/mongo/admin.ns, filling with zeroes... Mon Jun 24 04:28:01.129 [FileAllocator] done allocating datafile /var/lib/mongo/admin.ns, size: 16MB, took 0 secs Mon Jun 24 04:28:01.129 [FileAllocator] allocating new datafile /var/lib/mongo/admin.0, filling with zeroes... Mon Jun 24 04:28:01.130 [FileAllocator] done allocating datafile /var/lib/mongo/admin.0, size: 64MB, took 0 secs Mon Jun 24 04:28:01.130 [FileAllocator] allocating new datafile /var/lib/mongo/admin.1, filling with zeroes... Mon Jun 24 04:28:01.131 [FileAllocator] done allocating datafile /var/lib/mongo/admin.1, size: 128MB, took 0 secs Mon Jun 24 04:28:01.134 [rsSync] build index admin.system.users { user: 1, userSource: 1 } Mon Jun 24 04:28:01.135 [rsSync] build index done. scanned 0 total records. 0.001 secs Mon Jun 24 04:28:01.190 [rsSync] build index admin.system.users { _id: 1 } Mon Jun 24 04:28:01.191 [rsSync] fastBuildIndex dupsToDrop:0 Mon Jun 24 04:28:01.191 [rsSync] build index done. scanned 4 total records. 0.001 secs Mon Jun 24 04:28:01.246 [rsSync] build index admin.Example { _id: 1 } Mon Jun 24 04:28:01.248 [rsSync] fastBuildIndex dupsToDrop:0 Mon Jun 24 04:28:01.248 [rsSync] build index done. scanned 0 total records. 0.001 secs Mon Jun 24 04:28:01.248 [rsSync] replSet initial sync cloning db: tmtest Mon Jun 24 04:28:01.302 [rsSync] replSet initial sync cloning db: system Mon Jun 24 04:28:01.357 [rsSync] replSet initial sync cloning db: config Mon Jun 24 04:28:01.412 [rsSync] replSet initial sync cloning db: * Mon Jun 24 04:28:01.466 [rsSync] replSet initial sync cloning db: sprawk Mon Jun 24 04:28:01.521 [rsSync] replSet initial sync cloning db: upload Mon Jun 24 04:28:01.576 [rsSync] replSet initial sync data copy, starting syncup Mon Jun 24 04:28:01.576 [rsSync] oplog sync 1 of 3 Mon Jun 24 04:28:01.889 [rsBackgroundSync] replSet syncing to: sture.transmachina.net:27017 Mon Jun 24 04:28:05.639 [conn3522] command admin.$cmd command: { replSetHeartbeat: "sprawk", v: 1, pv: 1, checkEmpty: false, from: "oden.transmachina.net:27017" } ntoreturn:1 keyUpdates:0 reslen:206 2519ms Mon Jun 24 04:28:05.639 [conn3523] command admin.$cmd command: { replSetHeartbeat: "sprawk", v: 1, pv: 1, checkEmpty: false, from: "sture.transmachina.net:27017" } ntoreturn:1 keyUpdates:0 reslen:206 1820ms Mon Jun 24 04:28:05.811 [rsSyncNotifier] replset setting oplog notifier to sture.transmachina.net:27017 Mon Jun 24 04:28:07.696 [conn3522] end connection 217.70.33.64:52488 (1 connection now open) Mon Jun 24 04:28:07.750 [initandlisten] connection accepted from 217.70.33.64:52517 #3525 (2 connections now open) Mon Jun 24 04:28:07.805 [conn3525] authenticate db: local { authenticate: 1, nonce: "c17591d4494720c0", user: "__system", key: "584bc0246cc7a1e8b51bf2d077fb3489" } Mon Jun 24 04:28:10.650 [repl writer worker 1] replication update of non-mod failed: { ts: Timestamp 1371998024000|180, h: 2476669463202080495, v: 2, op: "u", ns: "jerome5.Example", o2: { _id: ObjectId('4e7ed5c2e4b02a2f79287aad') }, o: { $set: { pc: 7 } } } Mon Jun 24 04:28:10.650 [repl writer worker 1] replication info adding missing object Mon Jun 24 04:28:10.929 [repl writer worker 1] replication missing object not found on source. presumably deleted later in oplog Mon Jun 24 04:28:10.929 [repl writer worker 1] replication o2: { _id: ObjectId('4e7ed5c2e4b02a2f79287aad') } Mon Jun 24 04:28:10.929 [repl writer worker 1] replication o firstfield: $set Mon Jun 24 04:28:11.461 [repl writer worker 1] replication update of non-mod failed: { ts: Timestamp 1371998104000|197, h: -2171996614525419282, v: 2, op: "u", ns: "jerome5.Example", o2: { _id: ObjectId('4e7ed5c2e4b02a2f79287aad') }, o: { $set: { pc: 8 } } } Mon Jun 24 04:28:11.461 [repl writer worker 1] replication info adding missing object Mon Jun 24 04:28:11.734 [repl writer worker 1] replication missing object not found on source. presumably deleted later in oplog Mon Jun 24 04:28:11.734 [repl writer worker 1] replication o2: { _id: ObjectId('4e7ed5c2e4b02a2f79287aad') } Mon Jun 24 04:28:11.734 [repl writer worker 1] replication o firstfield: $set Mon Jun 24 04:28:12.000 [rsSync] replSet initialSyncOplogApplication applied 5611 operations, synced to Jun 23 16:36:38:10 Mon Jun 24 04:28:13.086 [repl writer worker 1] ERROR: exception: failing update: objects in a capped ns cannot grow on: { ts: Timestamp 1371998402000|9, h: 6117456418327665188, v: 2, op: "i", ns: "jerome5.TranslationResult", o: { _id: ObjectId('51c708c2e4b00a8dfc3e931c'), prep: "", postp: "", sourceText: "Sign up as a Sprawk customer or freelancer at http://www.sprawk.com/en/signupChoose", sourceTextLc: "sign up as a sprawk customer or freelancer at http://www.sprawk.com/en/signupchoose", sth: -3109906834912301281, sth2: 0, stlen: 135, sl: "eng", destText: "Sign up as a Sprawk customer or freelancer at http://www.sprawk.com/en/signupChoose", dl: "nld", sr: 0.0, tt: 1, b: false, mime: "text/html", start: new Date(1371998402694), g: ObjectId('4defe4887112ede06261f72f'), translator: "transmachina.jerome.translate.HtmlTranslator", sourceUrl: "http://www.transmachina.com/node/96", suhost: "www.transmachina.com", suhosth: -680836827, sup: "/node/96", suh: 7017488501028237330, subResultsSize: 0, fromCache: false, et: 0, status: -1, mb: "MT", note: "translateViaBestExample failed Failed to translate in MemoryTranslate.translateSentence", server: "alpha", same: false, cacheSuffix: 5 } } Mon Jun 24 04:28:13.086 [repl writer worker 1] Fatal Assertion 16361 0xdd2331 0xd92323 0xc22f84 0xd9fe71 0xe1aad9 0x7f602cd44851 0x7f602c0ea90d /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdd2331] /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xa3) [0xd92323] /usr/bin/mongod(_ZN5mongo7replset21multiInitialSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x174) [0xc22f84] /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) [0xd9fe71] /usr/bin/mongod() [0xe1aad9] /lib64/libpthread.so.0(+0x7851) [0x7f602cd44851] /lib64/libc.so.6(clone+0x6d) [0x7f602c0ea90d] Mon Jun 24 04:28:13.096 [repl writer worker 1] ***aborting after fassert() failure Mon Jun 24 04:28:13.096 Got signal: 6 (Aborted). Mon Jun 24 04:28:13.103 Backtrace: 0xdd2331 0x6cfb19 0x7f602c034920 0x7f602c0348a5 0x7f602c036085 0xd9235e 0xc22f84 0xd9fe71 0xe1aad9 0x7f602cd44851 0x7f602c0ea90d /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdd2331] /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x6cfb19] /lib64/libc.so.6(+0x32920) [0x7f602c034920] /lib64/libc.so.6(gsignal+0x35) [0x7f602c0348a5] /lib64/libc.so.6(abort+0x175) [0x7f602c036085] /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xde) [0xd9235e] /usr/bin/mongod(_ZN5mongo7replset21multiInitialSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x174) [0xc22f84] /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) [0xd9fe71] /usr/bin/mongod() [0xe1aad9] /lib64/libpthread.so.0(+0x7851) [0x7f602cd44851] /lib64/libc.so.6(clone+0x6d) [0x7f602c0ea90d] ***** SERVER RESTARTED ***** Mon Jun 24 04:29:01.345 [initandlisten] MongoDB starting : pid=5980 port=27017 dbpath=/var/lib/mongo 64-bit host=haga Mon Jun 24 04:29:01.345 [initandlisten] db version v2.4.4 Mon Jun 24 04:29:01.345 [initandlisten] git version: 4ec1fb96702c9d4c57b1e06dd34eb73a16e407d2 Mon Jun 24 04:29:01.345 [initandlisten] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49 Mon Jun 24 04:29:01.345 [initandlisten] allocator: tcmalloc Mon Jun 24 04:29:01.345 [initandlisten] options: { config: "/etc/mongod.conf", dbpath: "/var/lib/mongo", fork: "true", keyFile: "/var/lib/mongo/sprawk.key", logappend: "true", logpath: "/var/log/mongo/mongod.log", oplogSize: 28610, pidfilepath: "/var/run/mongodb/mongod.pid", replSet: "sprawk" } Mon Jun 24 04:29:01.347 [initandlisten] journal dir=/var/lib/mongo/journal Mon Jun 24 04:29:01.347 [initandlisten] recover begin Mon Jun 24 04:29:01.361 [initandlisten] recover lsn: 42705740 Mon Jun 24 04:29:01.361 [initandlisten] recover /var/lib/mongo/journal/j._21 Mon Jun 24 04:29:01.378 [initandlisten] recover skipping application of section seq:40922186 < lsn:42705740 Mon Jun 24 04:29:01.473 [initandlisten] recover skipping application of section seq:40981582 < lsn:42705740 Mon Jun 24 04:29:01.528 [initandlisten] recover skipping application of section seq:41040983 < lsn:42705740 Mon Jun 24 04:29:01.571 [initandlisten] recover skipping application of section seq:41100366 < lsn:42705740 Mon Jun 24 04:29:01.628 [initandlisten] recover skipping application of section seq:41159767 < lsn:42705740 Mon Jun 24 04:29:01.673 [initandlisten] recover skipping application of section seq:41219167 < lsn:42705740 Mon Jun 24 04:29:01.709 [initandlisten] recover skipping application of section seq:41278553 < lsn:42705740 Mon Jun 24 04:29:01.760 [initandlisten] recover skipping application of section seq:41337946 < lsn:42705740 Mon Jun 24 04:29:01.832 [initandlisten] recover skipping application of section seq:41397356 < lsn:42705740 Mon Jun 24 04:29:01.912 [initandlisten] recover skipping application of section more... Mon Jun 24 04:29:04.295 [initandlisten] recover create file /var/lib/mongo/admin.ns 16MB Mon Jun 24 04:29:04.312 [initandlisten] recover create file /var/lib/mongo/admin.0 64MB Mon Jun 24 04:29:04.401 [initandlisten] recover cleaning up Mon Jun 24 04:29:04.401 [initandlisten] removeJournalFiles Mon Jun 24 04:29:04.428 [initandlisten] recover done Mon Jun 24 04:29:04.671 [websvr] admin web console waiting for connections on port 28017 Mon Jun 24 04:29:04.671 [initandlisten] waiting for connections on port 27017 Mon Jun 24 04:29:04.731 [initandlisten] connection accepted from 217.70.33.64:52663 #1 (1 connection now open) Mon Jun 24 04:29:05.099 [rsStart] replSet I am sprawk.member0.mongolayer.com:27017 Mon Jun 24 04:29:05.099 [rsStart] replSet STARTUP2 Mon Jun 24 04:29:05.100 [rsSync] replSet initial sync pending Mon Jun 24 04:29:05.101 [rsSync] replSet initial sync need a member to be primary or secondary to do our initial sync Mon Jun 24 04:29:05.291 [rsHealthPoll] replset info sture.transmachina.net:27017 thinks that we are down