Thu Nov 21 08:32:24.681 [initandlisten] MongoDB starting : pid=6564 port=27017 dbpath=/var/lib/mongo 64-bit host=haga Thu Nov 21 08:32:24.681 [initandlisten] db version v2.4.8 Thu Nov 21 08:32:24.681 [initandlisten] git version: a350fc38922fbda2cec8d5dd842237b904eafc14 Thu Nov 21 08:32:24.681 [initandlisten] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49 Thu Nov 21 08:32:24.681 [initandlisten] allocator: tcmalloc Thu Nov 21 08:32:24.681 [initandlisten] options: { auth: "true", config: "/etc/mongod.conf", dbpath: "/var/lib/mongo", fork: "true", keyFile: "/var/lib/mongo/sprawk.key", logappend: "true", logpath: "/var/log/mongo/mongod.log", oplogSize: 28610, pidfilepath: "/var/run/mongodb/mongod.pid", replSet: "sprawk" } Thu Nov 21 08:32:24.712 [initandlisten] journal dir=/var/lib/mongo/journal Thu Nov 21 08:32:24.712 [initandlisten] recover begin Thu Nov 21 08:32:24.720 [initandlisten] recover lsn: 0 Thu Nov 21 08:32:24.721 [initandlisten] recover /var/lib/mongo/journal/j._0 Thu Nov 21 08:32:24.885 [initandlisten] recover cleaning up Thu Nov 21 08:32:24.885 [initandlisten] removeJournalFiles Thu Nov 21 08:32:24.886 [initandlisten] recover done Thu Nov 21 08:32:25.741 [websvr] admin web console waiting for connections on port 28017 Thu Nov 21 08:32:25.741 [initandlisten] couldn't unlink socket file /tmp/mongodb-27017.sockerrno:1 Operation not permitted skipping Thu Nov 21 08:32:25.741 [initandlisten] waiting for connections on port 27017 Thu Nov 21 08:32:26.008 [rsStart] replSet I am n1-rs1.mongo.sprawk.com:27017 Thu Nov 21 08:32:26.008 [rsStart] replSet STARTUP2 Thu Nov 21 08:32:26.013 [rsHealthPoll] replset info arb-rs1.mongo.sprawk.com:27017 thinks that we are down Thu Nov 21 08:32:26.013 [rsHealthPoll] replSet member arb-rs1.mongo.sprawk.com:27017 is up Thu Nov 21 08:32:26.013 [rsHealthPoll] replSet member arb-rs1.mongo.sprawk.com:27017 is now in state ARBITER Thu Nov 21 08:32:26.141 [rsHealthPoll] replset info n2-rs1.mongo.sprawk.com:27017 thinks that we are down Thu Nov 21 08:32:26.141 [rsHealthPoll] replSet member n2-rs1.mongo.sprawk.com:27017 is up Thu Nov 21 08:32:26.141 [rsHealthPoll] replSet member n2-rs1.mongo.sprawk.com:27017 is now in state PRIMARY Thu Nov 21 08:32:26.500 [initandlisten] connection accepted from 217.70.33.122:49201 #1 (1 connection now open) Thu Nov 21 08:32:26.566 [initandlisten] connection accepted from 217.70.33.122:49202 #2 (2 connections now open) Thu Nov 21 08:32:26.601 [conn2] authenticate db: local { authenticate: 1, nonce: "c7852e8928b70be", user: "__system", key: "890db9e4b3ab1ee931dbca42d3e9160b" } Thu Nov 21 08:32:26.862 [initandlisten] connection accepted from 217.70.34.135:59810 #3 (3 connections now open) Thu Nov 21 08:32:26.895 [initandlisten] connection accepted from 217.70.34.135:59811 #4 (4 connections now open) Thu Nov 21 08:32:26.897 [conn3] end connection 217.70.34.135:59810 (3 connections now open) Thu Nov 21 08:32:26.928 [conn4] authenticate db: local { authenticate: 1, nonce: "26fb5b27edf1f718", user: "__system", key: "46e5381dc147d330f18ca94a867cce16" } Thu Nov 21 08:32:26.993 [conn4] authenticate db: local { authenticate: 1, nonce: "8cdda649cff607f3", user: "__system", key: "5d39cff1cf8f551abfe45e44f9c0335c" } Thu Nov 21 08:32:27.010 [rsSync] replSet still syncing, not yet to minValid optime 528d6b34:2 Thu Nov 21 08:32:27.100 [initandlisten] connection accepted from 88.198.175.66:43436 #5 (4 connections now open) Thu Nov 21 08:32:27.103 [conn5] end connection 88.198.175.66:43436 (3 connections now open) Thu Nov 21 08:32:27.104 [initandlisten] connection accepted from 88.198.175.66:43437 #6 (4 connections now open) Thu Nov 21 08:32:27.105 [conn6] authenticate db: local { authenticate: 1, nonce: "f655a5df707aed76", user: "__system", key: "117a9d9c1d850cd2af90e5ffdb5d0dcc" } Thu Nov 21 08:32:27.106 [conn6] authenticate db: local { authenticate: 1, nonce: "69ecc934c31e5e82", user: "__system", key: "08a71d9841ab5b1edd9193aad651cc6a" } Thu Nov 21 08:32:28.085 [initandlisten] connection accepted from 217.70.33.122:49203 #7 (5 connections now open) Thu Nov 21 08:32:28.118 [conn7] authenticate db: local { authenticate: 1, nonce: "97f6201a0eae72d9", user: "__system", key: "603bb6bbf021a165c12dc3c82711bee0" } Thu Nov 21 08:32:28.401 [initandlisten] connection accepted from 217.70.33.116:34821 #8 (6 connections now open) Thu Nov 21 08:32:28.434 [conn8] authenticate db: local { authenticate: 1, nonce: "6955db3e3c9df580", user: "__system", key: "cd672f5263267905783a7bbd64b4ad6b" } Thu Nov 21 08:32:28.509 [initandlisten] connection accepted from 217.70.33.116:34822 #9 (7 connections now open) Thu Nov 21 08:32:28.575 [initandlisten] connection accepted from 217.70.33.116:34823 #10 (8 connections now open) Thu Nov 21 08:32:28.608 [conn10] authenticate db: local { authenticate: 1, nonce: "57d04e033086ca7", user: "__system", key: "1cd75b65a0be8d81f585346649d6d35c" } Thu Nov 21 08:32:28.854 [initandlisten] connection accepted from 144.76.98.109:47769 #11 (9 connections now open) Thu Nov 21 08:32:28.859 [conn11] authenticate db: admin { authenticate: 1, user: "mmsagent", nonce: "cee14278ed585da7", key: "e6ededabeccb0aa9a20492863be7ae39" } Thu Nov 21 08:32:30.034 [initandlisten] connection accepted from 46.4.84.213:37297 #12 (10 connections now open) Thu Nov 21 08:32:30.035 [initandlisten] connection accepted from 46.4.84.213:37298 #13 (11 connections now open) Thu Nov 21 08:32:30.036 [conn13] authenticate db: local { authenticate: 1, nonce: "e39cd2d2223441d7", user: "__system", key: "2b2ed0912e6b7d322825879918fd9084" } Thu Nov 21 08:32:31.010 [rsBackgroundSync] replSet syncing to: n2-rs1.mongo.sprawk.com:27017 Thu Nov 21 08:32:31.011 [rsSync] replSet still syncing, not yet to minValid optime 528d6b34:2 Thu Nov 21 08:32:32.394 [initandlisten] connection accepted from 144.76.98.109:47773 #14 (12 connections now open) Thu Nov 21 08:32:32.397 [conn14] authenticate db: admin { authenticate: 1, user: "mmsagent", nonce: "10dc96b16502449e", key: "fa5c52f7c24374471cf0e95cd1faeded" } Thu Nov 21 08:32:32.746 [rsSyncNotifier] replset setting oplog notifier to n2-rs1.mongo.sprawk.com:27017 Thu Nov 21 08:32:32.747 [conn14] command admin.$cmd command: { authenticate: 1, user: "mmsagent", nonce: "10dc96b16502449e", key: "fa5c52f7c24374471cf0e95cd1faeded" } ntoreturn:1 keyUpdates:0 locks(micros) r:356 reslen:74 350ms Thu Nov 21 08:32:32.783 [initandlisten] connection accepted from 217.70.34.135:59813 #15 (13 connections now open) Thu Nov 21 08:32:32.849 [initandlisten] connection accepted from 217.70.34.135:59814 #16 (14 connections now open) Thu Nov 21 08:32:32.881 [conn16] authenticate db: local { authenticate: 1, nonce: "477efdc6888e1224", user: "__system", key: "7931bfd76cd8018c1929cba2acbab4c5" } Thu Nov 21 08:32:32.885 [conn14] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:28 reslen:3268 129ms Thu Nov 21 08:32:33.030 [conn14] assertion 13436 not master or secondary; cannot currently read from this replSet member ns:config.settings query:{} Thu Nov 21 08:32:33.033 [conn14] end connection 144.76.98.109:47773 (13 connections now open) Thu Nov 21 08:32:33.034 [initandlisten] connection accepted from 144.76.98.109:47774 #17 (14 connections now open) Thu Nov 21 08:32:33.036 [conn17] authenticate db: admin { authenticate: 1, user: "mmsagent", nonce: "47f0acb41f76cf6e", key: "c956f3c5a19325a02e3fdfa8972f8844" } Thu Nov 21 08:32:33.218 [repl writer worker 3] jerome5.Example caught assertion update index jerome5.Example.$indicesSize_1 10287 btree: key+recloc already in index _id: ObjectId('4e27622cc9b253f335eb327a') Thu Nov 21 08:32:34.531 [repl writer worker 5] jerome5.Unit caught assertion update index jerome5.Unit.$lc_1_wpm_1 10287 btree: key+recloc already in index _id: ObjectId('4eda12f6e4b0c2e5b8943dc8') Thu Nov 21 08:32:34.531 [repl writer worker 5] jerome5.Unit caught assertion update index jerome5.Unit.$lc_1_f_-1 10287 btree: key+recloc already in index _id: ObjectId('4eda12f6e4b0c2e5b8943dc8') Thu Nov 21 08:32:34.714 [repl writer worker 5] jerome5.Unit caught assertion update index jerome5.Unit.$lc_1_wpm_1 10287 btree: key+recloc already in index _id: ObjectId('4ee06cb9e4b07190c187e6ec') Thu Nov 21 08:32:34.736 [repl writer worker 5] jerome5.Unit caught assertion update index jerome5.Unit.$lc_1_f_-1 10287 btree: key+recloc already in index _id: ObjectId('4ee06cb9e4b07190c187e6ec') Thu Nov 21 08:32:34.783 [repl writer worker 5] jerome5.Unit caught assertion update index jerome5.Unit.$lc_1_wpm_1 10287 btree: key+recloc already in index _id: ObjectId('4ee3ab2be4b07455887ebc54') Thu Nov 21 08:32:34.783 [repl writer worker 5] jerome5.Unit caught assertion update index jerome5.Unit.$lc_1_f_-1 10287 btree: key+recloc already in index _id: ObjectId('4ee3ab2be4b07455887ebc54') Thu Nov 21 08:32:34.836 [repl writer worker 5] jerome5.Unit caught assertion update index jerome5.Unit.$lc_1_wpm_1 10287 btree: key+recloc already in index _id: ObjectId('4ee1ca33e4b00341c9d6842a') Thu Nov 21 08:32:34.836 [repl writer worker 5] jerome5.Unit caught assertion update index jerome5.Unit.$lc_1_f_-1 10287 btree: key+recloc already in index _id: ObjectId('4ee1ca33e4b00341c9d6842a') Thu Nov 21 08:32:34.907 [repl writer worker 5] jerome5.Unit caught assertion update index jerome5.Unit.$lc_1_f_-1 10287 btree: key+recloc already in index _id: ObjectId('4ee1ccf0e4b00341c9d724d0') Thu Nov 21 08:32:34.990 [repl writer worker 5] jerome5.Unit caught assertion update index jerome5.Unit.$lc_1_wpm_1 10287 btree: key+recloc already in index _id: ObjectId('4ee1c877e4b00341c9d63489') Thu Nov 21 08:32:34.999 [repl writer worker 5] jerome5.Unit caught assertion update index jerome5.Unit.$lc_1_f_-1 10287 btree: key+recloc already in index _id: ObjectId('4ee1c877e4b00341c9d63489') Thu Nov 21 08:32:35.066 [repl writer worker 4] ERROR: writer worker caught exception: btree: key+recloc already in index on: { ts: Timestamp 1384999721000|6, h: 402201434552044898, v: 2, op: "i", ns: "jerome5.ExtendedLog", o: { _id: ObjectId('528d6b29e4b0adfc422870db'), u: ObjectId('4df3712e71123b7848e81894'), g: ObjectId('4defe4887112ede06261f72d'), title: "REINDEXED_SENTENCE", l: "info", properties: { EXAMPLE_TOSTRING: "De, der mener, det indoeuropæiske urhjem befandt sig i Lilleasien, vil være tilbøjelig til at hævde, at sproget kom til Grækenland direkte derfra...", LANGUAGE: "dan", EXAMPLE_ID: "4e6b9950e4b0c4359109055f", TEXT: "De, der mener, det indoeuropæiske urhjem befandt sig i Lilleasien, vil være tilbøjelig til at hævde, at sproget kom til Grækenland direkte derfra...", FIRST_TIME: "false", TYPE: "PLAIN", SERVER: "alpha", INDEX_TEXTS: "med, der, til, det, derfra, direkte derfra, sproget, Lilleasien, direkte, mener, de, indoeuropæiske urhjem befandt, hævde, vil, kom, sammen, indoeur..." } } } Thu Nov 21 08:32:35.066 [repl writer worker 4] Fatal Assertion 16360 Thu Nov 21 08:32:35.071 [repl writer worker 4] Thu Nov 21 08:32:35.071 Got signal: 6 (Aborted). Thu Nov 21 08:32:35.075 Backtrace: