***** SERVER RESTARTED ***** Thu Mar 21 21:02:36.569 [initandlisten] MongoDB starting : pid=11985 port=27017 dbpath=/var/lib/mongodb 64-bit host=db20 Thu Mar 21 21:02:36.569 [initandlisten] db version v2.4.0 Thu Mar 21 21:02:36.569 [initandlisten] git version: ce2d666c04b4a80af58e8bbb3388b0680e8cfeb6 Thu Mar 21 21:02:36.569 [initandlisten] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49 Thu Mar 21 21:02:36.569 [initandlisten] allocator: tcmalloc Thu Mar 21 21:02:36.569 [initandlisten] options: { config: "/etc/mongodb.conf", dbpath: "/var/lib/mongodb", logappend: "true", logpath: "/var/log/mongodb/mongodb.log", noauth: "true", replSet: "nbset", rest: "true", slowms: 100 } Thu Mar 21 21:02:36.579 [initandlisten] journal dir=/var/lib/mongodb/journal Thu Mar 21 21:02:36.579 [initandlisten] recover : no journal files present, no recovery needed Thu Mar 21 21:02:36.666 [websvr] admin web console waiting for connections on port 28017 Thu Mar 21 21:02:36.666 [initandlisten] waiting for connections on port 27017 Thu Mar 21 21:02:36.714 [rsStart] replSet I am db20:27017 Thu Mar 21 21:02:36.714 [rsStart] replSet STARTUP2 Thu Mar 21 21:02:36.715 [rsSync] replSet initial sync pending Thu Mar 21 21:02:36.715 [rsSync] replSet initial sync need a member to be primary or secondary to do our initial sync Thu Mar 21 21:02:36.736 [rsHealthPoll] replSet member db04:27017 is up Thu Mar 21 21:02:36.736 [rsHealthPoll] replSet member db04:27017 is now in state PRIMARY Thu Mar 21 21:02:36.736 [rsMgr] replset msgReceivedNewConfig version: version: 335598 Thu Mar 21 21:02:36.736 [rsMgr] replSet info saving a newer config version to local.system.replset Thu Mar 21 21:02:36.736 [rsMgr] replSet saveConfigLocally done Thu Mar 21 21:02:36.736 [rsMgr] replSet REMOVED Thu Mar 21 21:02:36.736 [rsMgr] replSet info self not present in the repl set configuration: Thu Mar 21 21:02:36.736 [rsMgr] { _id: "nbset", version: 335598, members: [ { _id: 8, host: "db04:27017", priority: 3.0 }, { _id: 2, host: "db05:27017" } ] } Thu Mar 21 21:02:36.737 [rsMgr] replSet info Couldn't load config yet. Sleeping 20sec and will try again. Thu Mar 21 21:02:38.330 [initandlisten] connection accepted from 127.0.0.1:45306 #1 (1 connection now open) Thu Mar 21 21:02:41.216 [conn9] end connection 199.15.252.109:41359 (1 connection now open) Thu Mar 21 21:02:51.066 [conn10] end connection 199.15.249.98:36317 (1 connection now open) Thu Mar 21 21:02:52.716 [rsSync] replSet initial sync pending Thu Mar 21 21:02:56.737 [rsMgr] trying to contact db04:27017 Thu Mar 21 21:02:56.737 [rsMgr] trying to contact db05:27017 Thu Mar 21 21:02:56.781 [rsMgr] replSet I am db20:27017 Thu Mar 21 21:02:56.781 [rsMgr] replSet got config version 335599 from a remote, saving locally Thu Mar 21 21:02:56.781 [rsMgr] replSet info saving a newer config version to local.system.replset Thu Mar 21 21:02:56.782 [rsMgr] replSet saveConfigLocally done Thu Mar 21 21:02:56.783 [conn1] attempting to sync from db05:27017, but its latest opTime is 0 and ours is 1363810090 so this may not work Thu Mar 21 21:02:56.783 [rsSync] replSet syncing to: db05:27017 by request Thu Mar 21 21:02:56.783 [conn1] command admin.$cmd command: { replSetSyncFrom: "db05:27017" } ntoreturn:1 keyUpdates:0 reslen:135 10586ms Thu Mar 21 21:02:56.803 [rsHealthPoll] replSet member db05:27017 is up Thu Mar 21 21:02:56.804 [rsHealthPoll] replSet member db04:27017 is up Thu Mar 21 21:02:56.804 [rsHealthPoll] replSet member db05:27017 is now in state SECONDARY Thu Mar 21 21:02:56.804 [rsHealthPoll] replSet member db04:27017 is now in state PRIMARY Thu Mar 21 21:02:56.815 [rsSync] replSet initial sync drop all databases Thu Mar 21 21:02:56.815 [rsSync] dropAllDatabasesExceptLocal 3 Thu Mar 21 21:02:56.819 [rsSync] removeJournalFiles Thu Mar 21 21:02:56.826 [rsSync] removeJournalFiles Thu Mar 21 21:02:57.015 [rsSync] replSet initial sync clone all databases Thu Mar 21 21:02:57.041 [rsSync] replSet initial sync cloning db: admin Thu Mar 21 21:02:57.074 [FileAllocator] allocating new datafile /var/lib/mongodb/admin.ns, filling with zeroes... Thu Mar 21 21:02:57.098 [FileAllocator] done allocating datafile /var/lib/mongodb/admin.ns, size: 16MB, took 0.023 secs Thu Mar 21 21:02:57.098 [FileAllocator] allocating new datafile /var/lib/mongodb/admin.0, filling with zeroes... Thu Mar 21 21:02:57.100 [FileAllocator] done allocating datafile /var/lib/mongodb/admin.0, size: 64MB, took 0.001 secs Thu Mar 21 21:02:57.100 [FileAllocator] allocating new datafile /var/lib/mongodb/admin.1, filling with zeroes... Thu Mar 21 21:02:57.102 [FileAllocator] done allocating datafile /var/lib/mongodb/admin.1, size: 128MB, took 0.001 secs Thu Mar 21 21:02:57.105 [rsSync] build index admin.system.users { user: 1, userSource: 1 } Thu Mar 21 21:02:57.106 [rsSync] build index done. scanned 0 total records. 0.001 secs Thu Mar 21 21:02:57.172 [rsSync] build index admin.system.users { _id: 1 } Thu Mar 21 21:02:57.174 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 21:02:57.174 [rsSync] build index done. scanned 1 total records. 0.001 secs Thu Mar 21 21:02:57.174 [rsSync] replSet initial sync cloning db: nbanalytics Thu Mar 21 21:02:57.198 [FileAllocator] allocating new datafile /var/lib/mongodb/nbanalytics.ns, filling with zeroes... Thu Mar 21 21:02:57.200 [FileAllocator] done allocating datafile /var/lib/mongodb/nbanalytics.ns, size: 16MB, took 0.001 secs Thu Mar 21 21:02:57.200 [FileAllocator] allocating new datafile /var/lib/mongodb/nbanalytics.0, filling with zeroes... Thu Mar 21 21:02:57.201 [FileAllocator] done allocating datafile /var/lib/mongodb/nbanalytics.0, size: 64MB, took 0.001 secs Thu Mar 21 21:02:57.202 [FileAllocator] allocating new datafile /var/lib/mongodb/nbanalytics.1, filling with zeroes... Thu Mar 21 21:02:57.205 [FileAllocator] done allocating datafile /var/lib/mongodb/nbanalytics.1, size: 128MB, took 0.002 secs Thu Mar 21 21:03:01.783 [rsBackgroundSync] replSet syncing to: db05:27017 Thu Mar 21 21:03:11.495 [rsSyncNotifier] replset setting oplog notifier to db05:27017 Thu Mar 21 21:03:13.368 [FileAllocator] allocating new datafile /var/lib/mongodb/nbanalytics.2, filling with zeroes... Thu Mar 21 21:03:13.371 [FileAllocator] done allocating datafile /var/lib/mongodb/nbanalytics.2, size: 256MB, took 0.002 secs Thu Mar 21 21:03:26.202 [FileAllocator] allocating new datafile /var/lib/mongodb/nbanalytics.3, filling with zeroes... Thu Mar 21 21:03:26.205 [FileAllocator] done allocating datafile /var/lib/mongodb/nbanalytics.3, size: 512MB, took 0.002 secs Thu Mar 21 21:03:42.543 [rsSync] build index nbanalytics.feed_fetches { _id: 1 } Thu Mar 21 21:03:48.495 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 21:03:48.504 [rsSync] build index done. scanned 1222105 total records. 5.96 secs Thu Mar 21 21:04:01.745 [FileAllocator] allocating new datafile /var/lib/mongodb/nbanalytics.4, filling with zeroes... Thu Mar 21 21:04:01.747 [FileAllocator] done allocating datafile /var/lib/mongodb/nbanalytics.4, size: 1024MB, took 0.001 secs Thu Mar 21 21:04:05.396 [rsSync] build index nbanalytics.page_loads { _id: 1 } Thu Mar 21 21:04:07.722 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 21:04:07.767 [rsSync] build index done. scanned 836966 total records. 2.37 secs Thu Mar 21 21:04:07.767 [rsSync] replSet initial sync cloning db: newsblur Thu Mar 21 21:04:08.010 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.ns, filling with zeroes... Thu Mar 21 21:04:08.011 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.ns, size: 16MB, took 0.001 secs Thu Mar 21 21:04:08.012 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.0, filling with zeroes... Thu Mar 21 21:04:08.013 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.0, size: 64MB, took 0.001 secs Thu Mar 21 21:04:08.014 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.1, filling with zeroes... Thu Mar 21 21:04:08.015 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.1, size: 128MB, took 0 secs Thu Mar 21 21:04:13.366 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.2, filling with zeroes... Thu Mar 21 21:04:13.368 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.2, size: 256MB, took 0.001 secs Thu Mar 21 21:04:20.995 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.3, filling with zeroes... Thu Mar 21 21:04:20.998 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.3, size: 512MB, took 0.002 secs Thu Mar 21 21:04:33.389 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.4, filling with zeroes... Thu Mar 21 21:04:33.392 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.4, size: 1024MB, took 0.002 secs Thu Mar 21 21:05:08.245 [rsSync] clone newsblur.stories 812543 Thu Mar 21 21:05:09.006 [rsSync] 820146 objects cloned so far from collection newsblur.stories Thu Mar 21 21:05:11.415 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.5, filling with zeroes... Thu Mar 21 21:05:11.419 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.5, size: 2047MB, took 0.003 secs Thu Mar 21 21:05:46.294 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._0 Thu Mar 21 21:05:46.294 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._1 Thu Mar 21 21:06:07.666 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.6, filling with zeroes... Thu Mar 21 21:06:07.669 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.6, size: 2047MB, took 0.002 secs Thu Mar 21 21:06:08.090 [rsSync] clone newsblur.stories 1314431 Thu Mar 21 21:06:10.117 [rsSync] 1335385 objects cloned so far from collection newsblur.stories Thu Mar 21 21:07:08.008 [rsSync] clone newsblur.stories 1923967 Thu Mar 21 21:07:12.241 [rsSync] 1952986 objects cloned so far from collection newsblur.stories Thu Mar 21 21:07:43.744 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.7, filling with zeroes... Thu Mar 21 21:07:43.747 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.7, size: 2047MB, took 0.003 secs Thu Mar 21 21:08:08.166 [rsSync] clone newsblur.stories 2536319 Thu Mar 21 21:08:13.099 [rsSync] 2597892 objects cloned so far from collection newsblur.stories Thu Mar 21 21:08:57.111 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.8, filling with zeroes... Thu Mar 21 21:08:57.114 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.8, size: 2047MB, took 0.002 secs Thu Mar 21 21:09:08.006 [rsSync] clone newsblur.stories 3161343 Thu Mar 21 21:09:14.067 [rsSync] 3210032 objects cloned so far from collection newsblur.stories Thu Mar 21 21:09:25.911 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._2 Thu Mar 21 21:10:06.384 [conn1145] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:63 reslen:3198 640ms Thu Mar 21 21:10:07.804 [conn1154] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:47 reslen:3198 104ms Thu Mar 21 21:10:08.082 [rsSync] clone newsblur.stories 3687167 Thu Mar 21 21:10:15.109 [rsSync] 3742924 objects cloned so far from collection newsblur.stories Thu Mar 21 21:10:40.501 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.9, filling with zeroes... Thu Mar 21 21:10:40.505 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.9, size: 2047MB, took 0.004 secs Thu Mar 21 21:11:08.079 [rsSync] clone newsblur.stories 4191231 Thu Mar 21 21:11:16.141 [rsSync] 4266545 objects cloned so far from collection newsblur.stories Thu Mar 21 21:11:17.798 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._3 Thu Mar 21 21:12:08.073 [rsSync] clone newsblur.stories 4763647 Thu Mar 21 21:12:17.060 [rsSync] 4819884 objects cloned so far from collection newsblur.stories Thu Mar 21 21:13:04.825 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.10, filling with zeroes... Thu Mar 21 21:13:04.829 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.10, size: 2047MB, took 0.004 secs Thu Mar 21 21:13:08.009 [rsSync] clone newsblur.stories 5290495 Thu Mar 21 21:13:08.420 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._4 Thu Mar 21 21:13:18.122 [rsSync] 5394432 objects cloned so far from collection newsblur.stories Thu Mar 21 21:14:08.194 [rsSync] clone newsblur.stories 5887871 Thu Mar 21 21:14:19.033 [rsSync] 5996285 objects cloned so far from collection newsblur.stories Thu Mar 21 21:14:55.940 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._5 Thu Mar 21 21:15:08.096 [rsSync] clone newsblur.stories 6450559 Thu Mar 21 21:15:20.195 [rsSync] 6569817 objects cloned so far from collection newsblur.stories Thu Mar 21 21:15:47.426 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.11, filling with zeroes... Thu Mar 21 21:15:47.432 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.11, size: 2047MB, took 0.005 secs Thu Mar 21 21:16:08.012 [rsSync] clone newsblur.stories 7011071 Thu Mar 21 21:16:21.114 [rsSync] 7142079 objects cloned so far from collection newsblur.stories Thu Mar 21 21:16:42.611 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._6 Thu Mar 21 21:16:42.622 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._7 Thu Mar 21 21:17:08.042 [rsSync] clone newsblur.stories 7622911 Thu Mar 21 21:17:22.006 [rsSync] 7792065 objects cloned so far from collection newsblur.stories Thu Mar 21 21:18:08.024 [rsSync] clone newsblur.stories 8154111 Thu Mar 21 21:18:23.135 [rsSync] 8313400 objects cloned so far from collection newsblur.stories Thu Mar 21 21:18:30.914 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.12, filling with zeroes... Thu Mar 21 21:18:30.918 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.12, size: 2047MB, took 0.003 secs Thu Mar 21 21:19:08.207 [rsSync] clone newsblur.stories 8773375 Thu Mar 21 21:19:24.038 [rsSync] 8898616 objects cloned so far from collection newsblur.stories Thu Mar 21 21:20:08.005 [rsSync] clone newsblur.stories 9332607 Thu Mar 21 21:20:08.379 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._8 Thu Mar 21 21:20:25.254 [rsSync] 9492087 objects cloned so far from collection newsblur.stories Thu Mar 21 21:21:08.005 [rsSync] clone newsblur.stories 9852031 Thu Mar 21 21:21:09.109 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.13, filling with zeroes... Thu Mar 21 21:21:09.113 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.13, size: 2047MB, took 0.004 secs Thu Mar 21 21:21:27.767 [rsSync] 10000087 objects cloned so far from collection newsblur.stories Thu Mar 21 21:21:52.400 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._9 Thu Mar 21 21:22:08.168 [rsSync] clone newsblur.stories 10409343 Thu Mar 21 21:22:28.164 [rsSync] 10634198 objects cloned so far from collection newsblur.stories Thu Mar 21 21:23:11.937 [rsSync] clone newsblur.stories 11000063 Thu Mar 21 21:23:29.183 [rsSync] 11158581 objects cloned so far from collection newsblur.stories Thu Mar 21 21:23:33.192 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._10 Thu Mar 21 21:23:49.688 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.14, filling with zeroes... Thu Mar 21 21:23:49.693 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.14, size: 2047MB, took 0.004 secs Thu Mar 21 21:24:11.059 [rsSync] clone newsblur.stories 11527807 Thu Mar 21 21:24:30.135 [rsSync] 11697061 objects cloned so far from collection newsblur.stories Thu Mar 21 21:25:10.290 [conn3458] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 4577, after repl: 4577, at end: 4577 } Thu Mar 21 21:25:10.290 [conn3458] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:30 reslen:3538 4649ms Thu Mar 21 21:25:11.228 [rsSync] clone newsblur.stories 12011775 Thu Mar 21 21:25:23.227 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._11 Thu Mar 21 21:25:31.196 [rsSync] 12238462 objects cloned so far from collection newsblur.stories Thu Mar 21 21:26:11.919 [rsSync] clone newsblur.stories 12673791 Thu Mar 21 21:26:32.005 [rsSync] 12822535 objects cloned so far from collection newsblur.stories Thu Mar 21 21:26:34.582 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.15, filling with zeroes... Thu Mar 21 21:26:34.587 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.15, size: 2047MB, took 0.004 secs Thu Mar 21 21:27:08.497 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._12 Thu Mar 21 21:27:11.005 [rsSync] clone newsblur.stories 13104383 Thu Mar 21 21:27:33.005 [rsSync] 13282529 objects cloned so far from collection newsblur.stories Thu Mar 21 21:28:11.027 [rsSync] clone newsblur.stories 13641983 Thu Mar 21 21:28:34.169 [rsSync] 13808563 objects cloned so far from collection newsblur.stories Thu Mar 21 21:29:02.142 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._13 Thu Mar 21 21:29:11.010 [rsSync] clone newsblur.stories 14074111 Thu Mar 21 21:29:34.420 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.16, filling with zeroes... Thu Mar 21 21:29:34.425 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.16, size: 2047MB, took 0.004 secs Thu Mar 21 21:29:35.206 [rsSync] 14277955 objects cloned so far from collection newsblur.stories Thu Mar 21 21:30:06.632 [conn4240] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:31 reslen:3198 431ms Thu Mar 21 21:30:11.270 [rsSync] clone newsblur.stories 14598271 Thu Mar 21 21:30:36.094 [rsSync] 14777446 objects cloned so far from collection newsblur.stories Thu Mar 21 21:30:43.887 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._14 Thu Mar 21 21:30:43.898 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._15 Thu Mar 21 21:31:11.264 [rsSync] clone newsblur.stories 15011583 Thu Mar 21 21:31:37.489 [rsSync] 15133935 objects cloned so far from collection newsblur.stories Thu Mar 21 21:32:11.232 [rsSync] clone newsblur.stories 15394431 Thu Mar 21 21:32:35.306 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.17, filling with zeroes... Thu Mar 21 21:32:35.311 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.17, size: 2047MB, took 0.004 secs Thu Mar 21 21:32:38.256 [rsSync] 15592532 objects cloned so far from collection newsblur.stories Thu Mar 21 21:32:42.338 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._16 Thu Mar 21 21:33:11.573 [rsSync] clone newsblur.stories 15835263 Thu Mar 21 21:33:41.548 [rsSync] 16000075 objects cloned so far from collection newsblur.stories Thu Mar 21 21:34:11.008 [rsSync] clone newsblur.stories 16207359 Thu Mar 21 21:34:42.139 [rsSync] 16476452 objects cloned so far from collection newsblur.stories Thu Mar 21 21:35:06.635 [conn5012] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:41 reslen:3198 127ms Thu Mar 21 21:35:11.203 [rsSync] clone newsblur.stories 16682495 Thu Mar 21 21:35:29.064 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.18, filling with zeroes... Thu Mar 21 21:35:29.068 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.18, size: 2047MB, took 0.004 secs Thu Mar 21 21:35:43.127 [rsSync] 16935049 objects cloned so far from collection newsblur.stories Thu Mar 21 21:36:11.225 [rsSync] clone newsblur.stories 17137023 Thu Mar 21 21:36:23.162 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._17 Thu Mar 21 21:36:44.108 [rsSync] 17404568 objects cloned so far from collection newsblur.stories Thu Mar 21 21:37:11.161 [rsSync] clone newsblur.stories 17626495 Thu Mar 21 21:37:45.109 [rsSync] 17887930 objects cloned so far from collection newsblur.stories Thu Mar 21 21:38:11.136 [rsSync] clone newsblur.stories 18100863 Thu Mar 21 21:38:12.800 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._18 Thu Mar 21 21:38:23.539 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.19, filling with zeroes... Thu Mar 21 21:38:23.544 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.19, size: 2047MB, took 0.004 secs Thu Mar 21 21:38:46.004 [rsSync] 18385643 objects cloned so far from collection newsblur.stories Thu Mar 21 21:39:11.062 [rsSync] clone newsblur.stories 18626175 Thu Mar 21 21:39:47.100 [rsSync] 18947364 objects cloned so far from collection newsblur.stories Thu Mar 21 21:39:51.333 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._19 Thu Mar 21 21:40:11.073 [rsSync] clone newsblur.stories 19097471 Thu Mar 21 21:40:48.006 [rsSync] 19405961 objects cloned so far from collection newsblur.stories Thu Mar 21 21:41:04.440 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.20, filling with zeroes... Thu Mar 21 21:41:04.445 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.20, size: 2047MB, took 0.005 secs Thu Mar 21 21:41:11.034 [rsSync] clone newsblur.stories 19627903 Thu Mar 21 21:41:38.738 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._20 Thu Mar 21 21:41:38.755 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._21 Thu Mar 21 21:41:49.057 [rsSync] 19946854 objects cloned so far from collection newsblur.stories Thu Mar 21 21:42:11.165 [rsSync] clone newsblur.stories 20086015 Thu Mar 21 21:42:50.186 [rsSync] 20377892 objects cloned so far from collection newsblur.stories Thu Mar 21 21:43:11.204 [rsSync] clone newsblur.stories 20497919 Thu Mar 21 21:43:51.163 [rsSync] 20863413 objects cloned so far from collection newsblur.stories Thu Mar 21 21:44:11.184 [rsSync] clone newsblur.stories 21024511 Thu Mar 21 21:44:17.489 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.21, filling with zeroes... Thu Mar 21 21:44:17.498 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.21, size: 2047MB, took 0.009 secs Thu Mar 21 21:44:52.060 [rsSync] 21457138 objects cloned so far from collection newsblur.stories Thu Mar 21 21:45:11.016 [rsSync] clone newsblur.stories 21579135 Thu Mar 21 21:45:40.043 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._22 Thu Mar 21 21:45:40.057 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._23 Thu Mar 21 21:45:53.214 [rsSync] 21947866 objects cloned so far from collection newsblur.stories Thu Mar 21 21:46:11.036 [rsSync] clone newsblur.stories 22021247 Thu Mar 21 21:46:54.321 [rsSync] 22326961 objects cloned so far from collection newsblur.stories Thu Mar 21 21:47:05.129 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.22, filling with zeroes... Thu Mar 21 21:47:05.136 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.22, size: 2047MB, took 0.006 secs Thu Mar 21 21:47:11.152 [rsSync] clone newsblur.stories 22468991 Thu Mar 21 21:47:55.138 [rsSync] 22711644 objects cloned so far from collection newsblur.stories Thu Mar 21 21:48:11.007 [rsSync] clone newsblur.stories 22831103 Thu Mar 21 21:48:56.197 [rsSync] 23066355 objects cloned so far from collection newsblur.stories Thu Mar 21 21:49:06.288 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._24 Thu Mar 21 21:49:11.066 [rsSync] clone newsblur.stories 23186943 Thu Mar 21 21:49:52.656 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.23, filling with zeroes... Thu Mar 21 21:49:52.667 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.23, size: 2047MB, took 0.01 secs Thu Mar 21 21:49:57.004 [rsSync] 23477200 objects cloned so far from collection newsblur.stories Thu Mar 21 21:50:11.278 [rsSync] clone newsblur.stories 23589119 Thu Mar 21 21:50:48.081 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._25 Thu Mar 21 21:50:58.259 [rsSync] 23969071 objects cloned so far from collection newsblur.stories Thu Mar 21 21:51:11.093 [rsSync] clone newsblur.stories 24026111 Thu Mar 21 21:51:59.159 [rsSync] 24361882 objects cloned so far from collection newsblur.stories Thu Mar 21 21:52:11.008 [rsSync] clone newsblur.stories 24455551 Thu Mar 21 21:52:28.063 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._26 Thu Mar 21 21:52:32.909 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.24, filling with zeroes... Thu Mar 21 21:52:32.916 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.24, size: 2047MB, took 0.007 secs Thu Mar 21 21:53:00.005 [rsSync] 24834830 objects cloned so far from collection newsblur.stories Thu Mar 21 21:53:11.073 [rsSync] clone newsblur.stories 24886271 Thu Mar 21 21:54:01.007 [rsSync] 25229038 objects cloned so far from collection newsblur.stories Thu Mar 21 21:54:11.020 [rsSync] clone newsblur.stories 25306239 Thu Mar 21 21:54:14.025 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._27 Thu Mar 21 21:55:02.149 [rsSync] 25617531 objects cloned so far from collection newsblur.stories Thu Mar 21 21:55:11.006 [rsSync] clone newsblur.stories 25692543 Thu Mar 21 21:55:19.268 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.25, filling with zeroes... Thu Mar 21 21:55:19.273 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.25, size: 2047MB, took 0.004 secs Thu Mar 21 21:55:49.001 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._28 Thu Mar 21 21:56:03.939 [rsSync] 26000055 objects cloned so far from collection newsblur.stories Thu Mar 21 21:56:11.115 [rsSync] clone newsblur.stories 26040191 Thu Mar 21 21:57:04.169 [rsSync] 26316285 objects cloned so far from collection newsblur.stories Thu Mar 21 21:57:11.060 [rsSync] clone newsblur.stories 26363519 Thu Mar 21 21:57:34.169 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._29 Thu Mar 21 21:58:00.402 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.26, filling with zeroes... Thu Mar 21 21:58:00.409 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.26, size: 2047MB, took 0.007 secs Thu Mar 21 21:58:05.042 [rsSync] 26744148 objects cloned so far from collection newsblur.stories Thu Mar 21 21:58:11.006 [rsSync] clone newsblur.stories 26777471 Thu Mar 21 21:59:06.093 [rsSync] 27104447 objects cloned so far from collection newsblur.stories Thu Mar 21 21:59:08.759 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._30 Thu Mar 21 21:59:11.341 [rsSync] clone newsblur.stories 27118847 Thu Mar 21 22:00:07.301 [rsSync] 27403786 objects cloned so far from collection newsblur.stories Thu Mar 21 22:00:11.191 [rsSync] clone newsblur.stories 27425791 Thu Mar 21 22:00:48.240 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.27, filling with zeroes... Thu Mar 21 22:00:48.251 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.27, size: 2047MB, took 0.01 secs Thu Mar 21 22:00:51.393 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._31 Thu Mar 21 22:01:08.105 [rsSync] 27715571 objects cloned so far from collection newsblur.stories Thu Mar 21 22:01:11.214 [rsSync] clone newsblur.stories 27734271 Thu Mar 21 22:02:09.023 [rsSync] 27905944 objects cloned so far from collection newsblur.stories Thu Mar 21 22:02:11.062 [rsSync] clone newsblur.stories 27915263 Thu Mar 21 22:03:07.304 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._32 Thu Mar 21 22:03:11.022 [rsSync] 28202743 objects cloned so far from collection newsblur.stories Thu Mar 21 22:03:11.023 [rsSync] clone newsblur.stories 28202751 Thu Mar 21 22:04:11.299 [rsSync] clone newsblur.stories 28452479 Thu Mar 21 22:04:12.804 [rsSync] 28455727 objects cloned so far from collection newsblur.stories Thu Mar 21 22:04:27.511 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.28, filling with zeroes... Thu Mar 21 22:04:27.518 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.28, size: 2047MB, took 0.006 secs Thu Mar 21 22:05:07.672 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._33 Thu Mar 21 22:05:11.281 [rsSync] clone newsblur.stories 28714239 Thu Mar 21 22:05:13.260 [rsSync] 28718363 objects cloned so far from collection newsblur.stories Thu Mar 21 22:06:35.517 [rsSync] 28758495 objects cloned so far from collection newsblur.stories Thu Mar 21 22:06:35.518 [rsSync] clone newsblur.stories 28758527 Thu Mar 21 22:07:40.281 [rsSync] 28771703 objects cloned so far from collection newsblur.stories Thu Mar 21 22:07:40.281 [rsSync] clone newsblur.stories 28771711 Thu Mar 21 22:08:40.405 [rsSync] clone newsblur.stories 28948991 Thu Mar 21 22:08:41.149 [rsSync] 28952678 objects cloned so far from collection newsblur.stories Thu Mar 21 22:09:40.469 [rsSync] clone newsblur.stories 29203071 Thu Mar 21 22:09:42.470 [rsSync] 29204646 objects cloned so far from collection newsblur.stories Thu Mar 21 22:09:45.798 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._34 Thu Mar 21 22:09:45.812 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._35 Thu Mar 21 22:10:37.078 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.29, filling with zeroes... Thu Mar 21 22:10:37.084 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.29, size: 2047MB, took 0.005 secs Thu Mar 21 22:10:40.739 [rsSync] clone newsblur.stories 29292031 Thu Mar 21 22:10:44.994 [rsSync] 29294943 objects cloned so far from collection newsblur.stories Thu Mar 21 22:11:40.002 [rsSync] clone newsblur.stories 29379967 Thu Mar 21 22:11:45.589 [rsSync] 29389812 objects cloned so far from collection newsblur.stories Thu Mar 21 22:12:51.201 [rsSync] 29448613 objects cloned so far from collection newsblur.stories Thu Mar 21 22:12:51.204 [rsSync] clone newsblur.stories 29448703 Thu Mar 21 22:13:54.050 [rsSync] 29456614 objects cloned so far from collection newsblur.stories Thu Mar 21 22:13:54.051 [rsSync] clone newsblur.stories 29456639 Thu Mar 21 22:14:54.038 [rsSync] clone newsblur.stories 29586815 Thu Mar 21 22:14:55.173 [rsSync] 29595425 objects cloned so far from collection newsblur.stories Thu Mar 21 22:15:54.708 [rsSync] clone newsblur.stories 29751295 Thu Mar 21 22:15:56.026 [rsSync] 29752143 objects cloned so far from collection newsblur.stories Thu Mar 21 22:16:31.393 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._36 Thu Mar 21 22:16:42.571 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.30, filling with zeroes... Thu Mar 21 22:16:42.577 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.30, size: 2047MB, took 0.005 secs Thu Mar 21 22:16:54.235 [rsSync] clone newsblur.stories 29890047 Thu Mar 21 22:16:57.031 [rsSync] 29897431 objects cloned so far from collection newsblur.stories Thu Mar 21 22:17:54.002 [rsSync] clone newsblur.stories 30098175 Thu Mar 21 22:17:58.399 [rsSync] 30136318 objects cloned so far from collection newsblur.stories Thu Mar 21 22:18:28.871 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._37 Thu Mar 21 22:18:54.078 [rsSync] clone newsblur.stories 30329727 Thu Mar 21 22:18:59.091 [rsSync] 30361362 objects cloned so far from collection newsblur.stories Thu Mar 21 22:19:47.128 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.31, filling with zeroes... Thu Mar 21 22:19:47.136 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.31, size: 2047MB, took 0.007 secs Thu Mar 21 22:19:54.093 [rsSync] clone newsblur.stories 30733951 Thu Mar 21 22:20:00.025 [rsSync] 30770429 objects cloned so far from collection newsblur.stories Thu Mar 21 22:20:08.522 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._38 Thu Mar 21 22:20:54.078 [rsSync] clone newsblur.stories 31131647 Thu Mar 21 22:21:01.035 [rsSync] 31166669 objects cloned so far from collection newsblur.stories Thu Mar 21 22:21:53.029 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._39 Thu Mar 21 22:21:54.055 [rsSync] clone newsblur.stories 31479039 Thu Mar 21 22:22:02.213 [rsSync] 31530397 objects cloned so far from collection newsblur.stories Thu Mar 21 22:22:28.014 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.32, filling with zeroes... Thu Mar 21 22:22:28.023 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.32, size: 2047MB, took 0.009 secs Thu Mar 21 22:22:54.037 [rsSync] clone newsblur.stories 31862399 Thu Mar 21 22:23:03.110 [rsSync] 31967404 objects cloned so far from collection newsblur.stories Thu Mar 21 22:23:44.505 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._40 Thu Mar 21 22:23:44.517 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._41 Thu Mar 21 22:23:54.001 [rsSync] clone newsblur.stories 32232703 Thu Mar 21 22:24:04.065 [rsSync] 32307764 objects cloned so far from collection newsblur.stories Thu Mar 21 22:24:54.126 [rsSync] clone newsblur.stories 32660095 Thu Mar 21 22:25:05.316 [rsSync] 32743374 objects cloned so far from collection newsblur.stories Thu Mar 21 22:25:20.941 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.33, filling with zeroes... Thu Mar 21 22:25:20.957 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.33, size: 2047MB, took 0.015 secs Thu Mar 21 22:25:54.001 [rsSync] clone newsblur.stories 33031167 Thu Mar 21 22:26:06.052 [rsSync] 33145837 objects cloned so far from collection newsblur.stories Thu Mar 21 22:26:54.208 [rsSync] clone newsblur.stories 33412095 Thu Mar 21 22:27:07.023 [rsSync] 33492293 objects cloned so far from collection newsblur.stories Thu Mar 21 22:27:21.238 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._42 Thu Mar 21 22:27:54.007 [rsSync] clone newsblur.stories 33785471 Thu Mar 21 22:28:08.000 [rsSync] 33875198 objects cloned so far from collection newsblur.stories Thu Mar 21 22:28:17.920 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.34, filling with zeroes... Thu Mar 21 22:28:17.926 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.34, size: 2047MB, took 0.005 secs Thu Mar 21 22:28:54.033 [rsSync] clone newsblur.stories 34092159 Thu Mar 21 22:29:02.344 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._43 Thu Mar 21 22:29:09.054 [rsSync] 34218225 objects cloned so far from collection newsblur.stories Thu Mar 21 22:29:54.129 [rsSync] clone newsblur.stories 34467071 Thu Mar 21 22:30:10.052 [rsSync] 34554394 objects cloned so far from collection newsblur.stories Thu Mar 21 22:30:47.507 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._44 Thu Mar 21 22:30:54.058 [rsSync] clone newsblur.stories 34822527 Thu Mar 21 22:31:11.008 [rsSync] 34936537 objects cloned so far from collection newsblur.stories Thu Mar 21 22:31:11.573 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.35, filling with zeroes... Thu Mar 21 22:31:11.580 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.35, size: 2047MB, took 0.005 secs Thu Mar 21 22:31:54.042 [rsSync] clone newsblur.stories 35173631 Thu Mar 21 22:32:13.492 [rsSync] 35263308 objects cloned so far from collection newsblur.stories Thu Mar 21 22:32:47.350 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._45 Thu Mar 21 22:32:54.005 [rsSync] clone newsblur.stories 35552895 Thu Mar 21 22:33:14.007 [rsSync] 35691044 objects cloned so far from collection newsblur.stories Thu Mar 21 22:33:54.050 [rsSync] clone newsblur.stories 35970559 Thu Mar 21 22:34:08.242 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.36, filling with zeroes... Thu Mar 21 22:34:08.250 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.36, size: 2047MB, took 0.007 secs Thu Mar 21 22:34:15.222 [rsSync] 36079410 objects cloned so far from collection newsblur.stories Thu Mar 21 22:34:21.952 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._46 Thu Mar 21 22:34:54.057 [rsSync] clone newsblur.stories 36298111 Thu Mar 21 22:35:16.205 [rsSync] 36351571 objects cloned so far from collection newsblur.stories Thu Mar 21 22:35:54.151 [rsSync] clone newsblur.stories 36591615 Thu Mar 21 22:36:17.094 [rsSync] 36748446 objects cloned so far from collection newsblur.stories Thu Mar 21 22:36:19.630 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._47 Thu Mar 21 22:36:54.005 [rsSync] clone newsblur.stories 36963455 Thu Mar 21 22:37:17.272 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.37, filling with zeroes... Thu Mar 21 22:37:17.277 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.37, size: 2047MB, took 0.005 secs Thu Mar 21 22:37:18.173 [rsSync] 37075852 objects cloned so far from collection newsblur.stories Thu Mar 21 22:37:54.035 [rsSync] clone newsblur.stories 37278975 Thu Mar 21 22:38:20.568 [rsSync] 37395003 objects cloned so far from collection newsblur.stories Thu Mar 21 22:38:32.144 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._48 Thu Mar 21 22:38:54.004 [rsSync] clone newsblur.stories 37646079 Thu Mar 21 22:39:20.873 [rsSync] build index newsblur.stories { _id: 1 } Thu Mar 21 22:39:30.004 [rsSync] Index: (1/3) External Sort Progress: 1243800/37850249 3% Thu Mar 21 22:39:40.004 [rsSync] Index: (1/3) External Sort Progress: 2492500/37850249 6% Thu Mar 21 22:39:50.005 [rsSync] Index: (1/3) External Sort Progress: 3774400/37850249 9% Thu Mar 21 22:40:01.132 [rsSync] Index: (1/3) External Sort Progress: 5000000/37850249 13% Thu Mar 21 22:40:11.004 [rsSync] Index: (1/3) External Sort Progress: 6147600/37850249 16% Thu Mar 21 22:40:21.004 [rsSync] Index: (1/3) External Sort Progress: 7378600/37850249 19% Thu Mar 21 22:40:31.004 [rsSync] Index: (1/3) External Sort Progress: 8635300/37850249 22% Thu Mar 21 22:40:41.004 [rsSync] Index: (1/3) External Sort Progress: 9849000/37850249 26% Thu Mar 21 22:40:51.748 [rsSync] Index: (1/3) External Sort Progress: 11000000/37850249 29% Thu Mar 21 22:41:01.005 [rsSync] Index: (1/3) External Sort Progress: 12080900/37850249 31% Thu Mar 21 22:41:11.004 [rsSync] Index: (1/3) External Sort Progress: 13235300/37850249 34% Thu Mar 21 22:41:21.004 [rsSync] Index: (1/3) External Sort Progress: 14365300/37850249 37% Thu Mar 21 22:41:31.005 [rsSync] Index: (1/3) External Sort Progress: 15393600/37850249 40% Thu Mar 21 22:41:41.006 [rsSync] Index: (1/3) External Sort Progress: 15982400/37850249 42% Thu Mar 21 22:41:51.004 [rsSync] Index: (1/3) External Sort Progress: 16618100/37850249 43% Thu Mar 21 22:42:01.005 [rsSync] Index: (1/3) External Sort Progress: 17196100/37850249 45% Thu Mar 21 22:42:11.004 [rsSync] Index: (1/3) External Sort Progress: 17912500/37850249 47% Thu Mar 21 22:42:21.004 [rsSync] Index: (1/3) External Sort Progress: 18542700/37850249 48% Thu Mar 21 22:42:31.005 [rsSync] Index: (1/3) External Sort Progress: 19459000/37850249 51% Thu Mar 21 22:42:41.004 [rsSync] Index: (1/3) External Sort Progress: 20523700/37850249 54% Thu Mar 21 22:42:51.004 [rsSync] Index: (1/3) External Sort Progress: 21660900/37850249 57% Thu Mar 21 22:43:01.005 [rsSync] Index: (1/3) External Sort Progress: 22563100/37850249 59% Thu Mar 21 22:43:11.004 [rsSync] Index: (1/3) External Sort Progress: 23321900/37850249 61% Thu Mar 21 22:43:21.004 [rsSync] Index: (1/3) External Sort Progress: 24146300/37850249 63% Thu Mar 21 22:43:32.075 [rsSync] Index: (1/3) External Sort Progress: 25000000/37850249 66% Thu Mar 21 22:43:42.004 [rsSync] Index: (1/3) External Sort Progress: 25996000/37850249 68% Thu Mar 21 22:43:52.004 [rsSync] Index: (1/3) External Sort Progress: 26735900/37850249 70% Thu Mar 21 22:44:02.005 [rsSync] Index: (1/3) External Sort Progress: 27438200/37850249 72% Thu Mar 21 22:44:12.005 [rsSync] Index: (1/3) External Sort Progress: 28132100/37850249 74% Thu Mar 21 22:44:22.004 [rsSync] Index: (1/3) External Sort Progress: 28870200/37850249 76% Thu Mar 21 22:44:32.005 [rsSync] Index: (1/3) External Sort Progress: 29363100/37850249 77% Thu Mar 21 22:44:42.004 [rsSync] Index: (1/3) External Sort Progress: 29863900/37850249 78% Thu Mar 21 22:44:52.004 [rsSync] Index: (1/3) External Sort Progress: 30415600/37850249 80% Thu Mar 21 22:45:02.004 [rsSync] Index: (1/3) External Sort Progress: 31282900/37850249 82% Thu Mar 21 22:45:13.018 [rsSync] Index: (1/3) External Sort Progress: 32000000/37850249 84% Thu Mar 21 22:45:23.004 [rsSync] Index: (1/3) External Sort Progress: 32957500/37850249 87% Thu Mar 21 22:45:33.005 [rsSync] Index: (1/3) External Sort Progress: 33755900/37850249 89% Thu Mar 21 22:45:43.005 [rsSync] Index: (1/3) External Sort Progress: 34439400/37850249 90% Thu Mar 21 22:45:53.004 [rsSync] Index: (1/3) External Sort Progress: 35127400/37850249 92% Thu Mar 21 22:46:03.998 [rsSync] Index: (1/3) External Sort Progress: 36000000/37850249 95% Thu Mar 21 22:46:13.005 [rsSync] Index: (1/3) External Sort Progress: 36806400/37850249 97% Thu Mar 21 22:46:23.004 [rsSync] Index: (1/3) External Sort Progress: 37631000/37850249 99% Thu Mar 21 22:46:26.939 [rsSync] external sort used : 38 files in 426 secs Thu Mar 21 22:46:36.004 [rsSync] Index: (2/3) BTree Bottom Up Progress: 1725800/37850249 4% Thu Mar 21 22:46:46.004 [rsSync] Index: (2/3) BTree Bottom Up Progress: 3687300/37850249 9% Thu Mar 21 22:46:56.004 [rsSync] Index: (2/3) BTree Bottom Up Progress: 5645000/37850249 14% Thu Mar 21 22:47:06.004 [rsSync] Index: (2/3) BTree Bottom Up Progress: 7429300/37850249 19% Thu Mar 21 22:47:16.004 [rsSync] Index: (2/3) BTree Bottom Up Progress: 9363300/37850249 24% Thu Mar 21 22:47:26.004 [rsSync] Index: (2/3) BTree Bottom Up Progress: 11309700/37850249 29% Thu Mar 21 22:47:36.004 [rsSync] Index: (2/3) BTree Bottom Up Progress: 13235000/37850249 34% Thu Mar 21 22:47:46.004 [rsSync] Index: (2/3) BTree Bottom Up Progress: 15051200/37850249 39% Thu Mar 21 22:47:56.004 [rsSync] Index: (2/3) BTree Bottom Up Progress: 16945300/37850249 44% Thu Mar 21 22:48:06.005 [rsSync] Index: (2/3) BTree Bottom Up Progress: 18851400/37850249 49% Thu Mar 21 22:48:16.192 [rsSync] Index: (2/3) BTree Bottom Up Progress: 20707400/37850249 54% Thu Mar 21 22:48:26.004 [rsSync] Index: (2/3) BTree Bottom Up Progress: 22564400/37850249 59% Thu Mar 21 22:48:36.004 [rsSync] Index: (2/3) BTree Bottom Up Progress: 24399300/37850249 64% Thu Mar 21 22:48:46.005 [rsSync] Index: (2/3) BTree Bottom Up Progress: 26199600/37850249 69% Thu Mar 21 22:48:56.005 [rsSync] Index: (2/3) BTree Bottom Up Progress: 28044700/37850249 74% Thu Mar 21 22:49:06.004 [rsSync] Index: (2/3) BTree Bottom Up Progress: 29806200/37850249 78% Thu Mar 21 22:49:16.004 [rsSync] Index: (2/3) BTree Bottom Up Progress: 31654300/37850249 83% Thu Mar 21 22:49:26.004 [rsSync] Index: (2/3) BTree Bottom Up Progress: 33528900/37850249 88% Thu Mar 21 22:49:36.004 [rsSync] Index: (2/3) BTree Bottom Up Progress: 35322300/37850249 93% Thu Mar 21 22:49:46.004 [rsSync] Index: (2/3) BTree Bottom Up Progress: 37251800/37850249 98% Thu Mar 21 22:49:48.737 [rsSync] done building bottom layer, going to commit Thu Mar 21 22:49:49.744 [rsSync] old journal file will be removed: /var/lib/mongodb/journal/j._49 Thu Mar 21 22:50:01.394 [rsSync] fastBuildIndex dupsToDrop:1416 Thu Mar 21 22:50:01.691 [rsSync] build index done. scanned 37850249 total records. 640.817 secs Thu Mar 21 22:50:01.695 [conn15365] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:46069] Thu Mar 21 22:50:02.479 [conn15216] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 482167, after repl: 482167, at end: 482167 } Thu Mar 21 22:50:02.479 [websvr] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 589267, after repl: 589267, at end: 589267 } Thu Mar 21 22:50:02.479 [conn15188] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 493677, after repl: 493677, at end: 493677 } Thu Mar 21 22:50:02.479 [conn15216] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:41 reslen:3538 489330ms Thu Mar 21 22:50:02.479 [conn15188] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:37 reslen:3538 501248ms Thu Mar 21 22:50:02.479 [conn15216] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:46060] Thu Mar 21 22:50:02.479 [conn15188] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:46058] Thu Mar 21 22:50:02.480 [conn15301] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 446226, after repl: 446226, at end: 446226 } Thu Mar 21 22:50:02.480 [conn15301] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:40 reslen:3538 452541ms Thu Mar 21 22:50:02.480 [conn15301] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:46065] Thu Mar 21 22:50:02.480 [conn15913] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 198616, after repl: 198616, at end: 198616 } Thu Mar 21 22:50:02.480 [conn15913] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:35 reslen:3538 201477ms Thu Mar 21 22:50:02.481 [conn15247] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 469794, after repl: 469794, at end: 469794 } Thu Mar 21 22:50:02.481 [conn15977] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 174649, after repl: 174649, at end: 174649 } Thu Mar 21 22:50:02.481 [conn15247] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:43 reslen:3538 476660ms Thu Mar 21 22:50:02.483 [conn16075] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 138677, after repl: 138677, at end: 138687 } Thu Mar 21 22:50:02.483 [conn16075] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:86 reslen:3538 140797ms Thu Mar 21 22:50:02.481 [conn15977] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:48 reslen:3538 177194ms Thu Mar 21 22:50:02.483 [conn15332] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 434214, after repl: 434214, at end: 434224 } Thu Mar 21 22:50:02.484 [conn15946] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 186659, after repl: 186659, at end: 186669 } Thu Mar 21 22:50:02.484 [conn15946] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:222 reslen:3538 189370ms Thu Mar 21 22:50:02.484 [websvr] Socket http response send() errno:32 Broken pipe 127.0.0.1:48216 Thu Mar 21 22:50:02.483 [conn15247] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:46061] Thu Mar 21 22:50:02.483 [conn15332] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:181 reslen:3538 440368ms Thu Mar 21 22:50:02.485 [conn15332] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:46067] Thu Mar 21 22:50:02.483 [conn16044] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 150664, after repl: 150664, at end: 150674 } Thu Mar 21 22:50:02.485 [conn16044] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:138 reslen:3538 152932ms Thu Mar 21 22:50:02.488 [websvr] Socket http response send() errno:32 Broken pipe 127.0.0.1:48220 Thu Mar 21 22:50:02.495 [conn15392] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:46070] Thu Mar 21 22:50:02.495 [conn15275] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:46063] Thu Mar 21 22:50:02.496 [websvr] Socket http response send() errno:32 Broken pipe 127.0.0.1:48224 Thu Mar 21 22:50:05.279 [rsSync] build index newsblur.classifier_title { _id: 1 } Thu Mar 21 22:50:05.361 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 22:50:05.363 [rsSync] build index done. scanned 25541 total records. 0.083 secs Thu Mar 21 22:50:07.395 [websvr] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 1477, after repl: 1477, at end: 1477 } Thu Mar 21 22:50:10.685 [rsSync] build index newsblur.classifier_tag { _id: 1 } Thu Mar 21 22:50:11.215 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 22:50:11.226 [rsSync] build index done. scanned 213503 total records. 0.54 secs Thu Mar 21 22:50:11.227 [websvr] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 1220, after repl: 1220, at end: 1220 } Thu Mar 21 22:51:18.058 [rsSync] 550564 objects cloned so far from collection newsblur.userstories Thu Mar 21 22:51:18.060 [rsSync] clone newsblur.userstories 550655 Thu Mar 21 22:52:19.808 [rsSync] 804437 objects cloned so far from collection newsblur.userstories Thu Mar 21 22:52:19.809 [rsSync] clone newsblur.userstories 804479 Thu Mar 21 22:53:35.633 [rsSync] 888638 objects cloned so far from collection newsblur.userstories Thu Mar 21 22:53:35.634 [rsSync] clone newsblur.userstories 888703 Thu Mar 21 22:54:47.509 [rsSync] clone newsblur.userstories 1120639 Thu Mar 21 22:54:47.510 [rsSync] 1120667 objects cloned so far from collection newsblur.userstories Thu Mar 21 22:55:47.383 [rsSync] clone newsblur.userstories 1163135 Thu Mar 21 22:55:50.089 [rsSync] 1184929 objects cloned so far from collection newsblur.userstories Thu Mar 21 22:56:48.043 [rsSync] clone newsblur.userstories 1823999 Thu Mar 21 22:56:53.270 [rsSync] 1843932 objects cloned so far from collection newsblur.userstories Thu Mar 21 22:57:48.006 [rsSync] clone newsblur.userstories 2392959 Thu Mar 21 22:57:54.004 [rsSync] 2562371 objects cloned so far from collection newsblur.userstories Thu Mar 21 23:00:14.720 [rsSync] clone newsblur.userstories 2762879 Thu Mar 21 23:00:14.721 [rsSync] 2762904 objects cloned so far from collection newsblur.userstories Thu Mar 21 23:01:14.103 [rsSync] clone newsblur.userstories 3000063 Thu Mar 21 23:01:17.617 [rsSync] 3015253 objects cloned so far from collection newsblur.userstories Thu Mar 21 23:02:39.290 [rsSync] clone newsblur.userstories 3265279 Thu Mar 21 23:02:39.291 [rsSync] 3265316 objects cloned so far from collection newsblur.userstories Thu Mar 21 23:03:39.008 [rsSync] clone newsblur.userstories 3421183 Thu Mar 21 23:03:40.361 [rsSync] 3460388 objects cloned so far from collection newsblur.userstories Thu Mar 21 23:04:39.011 [rsSync] clone newsblur.userstories 4322431 Thu Mar 21 23:04:41.006 [rsSync] 4375423 objects cloned so far from collection newsblur.userstories Thu Mar 21 23:05:08.840 [websvr] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 4631, after repl: 4631, at end: 4631 } Thu Mar 21 23:05:26.466 [conn18671] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 1, after globalLock: 1, after indexCounters: 1, after locks: 1, after network: 1, after opcounters: 1, after opcountersRepl: 1, after recordStats: 14524, after repl: 14524, at end: 14524 } Thu Mar 21 23:05:26.467 [conn18671] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:38 reslen:3538 14712ms Thu Mar 21 23:05:26.902 [conn18708] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:51 reslen:3198 114ms Thu Mar 21 23:05:27.322 [conn18710] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:41 reslen:3198 116ms Thu Mar 21 23:05:27.365 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.38, filling with zeroes... Thu Mar 21 23:05:27.368 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.38, size: 2047MB, took 0.003 secs Thu Mar 21 23:05:39.609 [rsSync] clone newsblur.userstories 5226239 Thu Mar 21 23:05:42.669 [rsSync] 5265947 objects cloned so far from collection newsblur.userstories Thu Mar 21 23:06:00.820 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._50 Thu Mar 21 23:06:52.638 [rsSync] 6000007 objects cloned so far from collection newsblur.userstories Thu Mar 21 23:06:52.643 [rsSync] clone newsblur.userstories 6000127 Thu Mar 21 23:07:54.939 [rsSync] 6603892 objects cloned so far from collection newsblur.userstories Thu Mar 21 23:07:54.940 [rsSync] clone newsblur.userstories 6603903 Thu Mar 21 23:09:06.749 [rsSync] 7083952 objects cloned so far from collection newsblur.userstories Thu Mar 21 23:09:06.751 [rsSync] clone newsblur.userstories 7084031 Thu Mar 21 23:10:11.082 [rsSync] 7163073 objects cloned so far from collection newsblur.userstories Thu Mar 21 23:10:11.084 [rsSync] clone newsblur.userstories 7163135 Thu Mar 21 23:11:33.124 [rsSync] clone newsblur.userstories 7504895 Thu Mar 21 23:11:33.126 [rsSync] 7504957 objects cloned so far from collection newsblur.userstories Thu Mar 21 23:12:38.125 [rsSync] 7526674 objects cloned so far from collection newsblur.userstories Thu Mar 21 23:12:38.128 [rsSync] clone newsblur.userstories 7526783 Thu Mar 21 23:13:52.911 [rsSync] 7548137 objects cloned so far from collection newsblur.userstories Thu Mar 21 23:13:52.912 [rsSync] clone newsblur.userstories 7548159 Thu Mar 21 23:14:54.411 [rsSync] clone newsblur.userstories 7590783 Thu Mar 21 23:14:54.412 [rsSync] 7590809 objects cloned so far from collection newsblur.userstories Thu Mar 21 23:15:54.005 [rsSync] clone newsblur.userstories 8219391 Thu Mar 21 23:15:55.005 [rsSync] 8258194 objects cloned so far from collection newsblur.userstories Thu Mar 21 23:16:54.015 [rsSync] clone newsblur.userstories 9491583 Thu Mar 21 23:16:56.007 [rsSync] 9558420 objects cloned so far from collection newsblur.userstories Thu Mar 21 23:17:54.004 [rsSync] clone newsblur.userstories 10191743 Thu Mar 21 23:17:57.006 [rsSync] 10294512 objects cloned so far from collection newsblur.userstories Thu Mar 21 23:17:58.827 [rsSync] build index newsblur.userstories { _id: 1 } Thu Mar 21 23:18:08.629 [rsSync] Index: (1/3) External Sort Progress: 4000000/10354884 38% Thu Mar 21 23:18:18.171 [rsSync] Index: (1/3) External Sort Progress: 8000000/10354884 77% Thu Mar 21 23:18:23.804 [rsSync] external sort used : 11 files in 24 secs Thu Mar 21 23:18:33.004 [rsSync] Index: (2/3) BTree Bottom Up Progress: 3099700/10354884 29% Thu Mar 21 23:18:43.004 [rsSync] Index: (2/3) BTree Bottom Up Progress: 6412900/10354884 61% Thu Mar 21 23:18:53.004 [rsSync] Index: (2/3) BTree Bottom Up Progress: 10195600/10354884 98% Thu Mar 21 23:18:53.764 [rsSync] done building bottom layer, going to commit Thu Mar 21 23:18:56.251 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 23:18:56.314 [rsSync] build index done. scanned 10354884 total records. 57.486 secs Thu Mar 21 23:19:29.113 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._51 Thu Mar 21 23:19:58.100 [rsSync] clone newsblur.starred_stories 258687 Thu Mar 21 23:19:58.470 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.39, filling with zeroes... Thu Mar 21 23:19:58.474 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.39, size: 2047MB, took 0.003 secs Thu Mar 21 23:19:59.260 [rsSync] 262462 objects cloned so far from collection newsblur.starred_stories Thu Mar 21 23:20:31.683 [rsSync] build index newsblur.starred_stories { _id: 1 } Thu Mar 21 23:20:33.119 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 23:20:33.140 [rsSync] build index done. scanned 410872 total records. 1.456 secs Thu Mar 21 23:20:34.817 [rsSync] build index newsblur.classifier_author { _id: 1 } Thu Mar 21 23:20:34.868 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 23:20:34.870 [rsSync] build index done. scanned 23270 total records. 0.052 secs Thu Mar 21 23:20:36.258 [rsSync] build index newsblur.classifier_feed { _id: 1 } Thu Mar 21 23:20:36.354 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 23:20:36.356 [rsSync] build index done. scanned 41530 total records. 0.097 secs Thu Mar 21 23:20:36.357 [rsSync] build index newsblur.system.users { user: 1, userSource: 1 } Thu Mar 21 23:20:36.357 [rsSync] build index done. scanned 0 total records. 0 secs Thu Mar 21 23:20:36.434 [rsSync] build index newsblur.system.users { _id: 1 } Thu Mar 21 23:20:36.436 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 23:20:36.436 [rsSync] build index done. scanned 1 total records. 0.001 secs Thu Mar 21 23:20:36.517 [rsSync] build index newsblur.statistics { _id: 1 } Thu Mar 21 23:20:36.519 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 23:20:36.520 [rsSync] build index done. scanned 17 total records. 0.002 secs Thu Mar 21 23:21:36.004 [rsSync] clone newsblur.feed_icons 585215 Thu Mar 21 23:21:37.346 [rsSync] 591444 objects cloned so far from collection newsblur.feed_icons Thu Mar 21 23:21:43.585 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._52 Thu Mar 21 23:21:43.596 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._53 Thu Mar 21 23:22:36.141 [rsSync] clone newsblur.feed_icons 893567 Thu Mar 21 23:22:38.264 [rsSync] 906658 objects cloned so far from collection newsblur.feed_icons Thu Mar 21 23:23:25.026 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.40, filling with zeroes... Thu Mar 21 23:23:25.031 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.40, size: 2047MB, took 0.004 secs Thu Mar 21 23:23:36.169 [rsSync] clone newsblur.feed_icons 1070463 Thu Mar 21 23:23:39.405 [rsSync] 1086363 objects cloned so far from collection newsblur.feed_icons Thu Mar 21 23:24:36.408 [rsSync] clone newsblur.feed_icons 1206527 Thu Mar 21 23:24:40.301 [rsSync] 1214633 objects cloned so far from collection newsblur.feed_icons Thu Mar 21 23:25:36.113 [rsSync] clone newsblur.feed_icons 1403263 Thu Mar 21 23:25:41.558 [rsSync] 1423675 objects cloned so far from collection newsblur.feed_icons Thu Mar 21 23:26:36.349 [rsSync] clone newsblur.feed_icons 1571071 Thu Mar 21 23:26:42.119 [rsSync] 1604523 objects cloned so far from collection newsblur.feed_icons Thu Mar 21 23:26:45.020 [rsSync] build index newsblur.feed_icons { _id: 1 } Thu Mar 21 23:26:54.154 [rsSync] fastBuildIndex dupsToDrop:3 Thu Mar 21 23:26:54.163 [rsSync] build index done. scanned 1622136 total records. 9.142 secs Thu Mar 21 23:26:54.475 [rsSync] build index newsblur.feedback { _id: 1 } Thu Mar 21 23:26:54.476 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 23:26:54.476 [rsSync] build index done. scanned 20 total records. 0 secs Thu Mar 21 23:26:54.496 [rsSync] build index newsblur.inline { _id: 1 } Thu Mar 21 23:26:54.497 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 23:26:54.497 [rsSync] build index done. scanned 16 total records. 0 secs Thu Mar 21 23:27:55.336 [rsSync] clone newsblur.feed_fetch_history 1765759 Thu Mar 21 23:27:55.338 [rsSync] 1765812 objects cloned so far from collection newsblur.feed_fetch_history Thu Mar 21 23:27:56.051 [rsSync] build index newsblur.feed_fetch_history { _id: 1 } Thu Mar 21 23:28:04.470 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 23:28:04.482 [rsSync] build index done. scanned 1811237 total records. 8.43 secs Thu Mar 21 23:28:50.440 [rsSync] build index newsblur.page_fetch_history { _id: 1 } Thu Mar 21 23:28:55.695 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 23:28:55.703 [rsSync] build index done. scanned 1097016 total records. 5.262 secs Thu Mar 21 23:29:00.477 [rsSync] build index newsblur.social_profile { _id: 1 } Thu Mar 21 23:29:00.656 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 23:29:00.660 [rsSync] build index done. scanned 68182 total records. 0.182 secs Thu Mar 21 23:29:17.326 [rsSync] build index newsblur.shared_stories { _id: 1 } Thu Mar 21 23:29:17.477 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 23:29:17.481 [rsSync] build index done. scanned 46343 total records. 0.154 secs Thu Mar 21 23:29:19.825 [rsSync] build index newsblur.social_subscription { _id: 1 } Thu Mar 21 23:29:19.995 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 23:29:19.999 [rsSync] build index done. scanned 66864 total records. 0.173 secs Thu Mar 21 23:29:30.394 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.41, filling with zeroes... Thu Mar 21 23:29:30.398 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.41, size: 2047MB, took 0.003 secs Thu Mar 21 23:29:36.593 [rsSync] build index newsblur.social_services { _id: 1 } Thu Mar 21 23:29:36.825 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 23:29:36.831 [rsSync] build index done. scanned 61712 total records. 0.237 secs Thu Mar 21 23:29:37.011 [rsSync] build index newsblur.social_invites { _id: 1 } Thu Mar 21 23:29:37.016 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 23:29:37.016 [rsSync] build index done. scanned 734 total records. 0.005 secs Thu Mar 21 23:29:41.068 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._54 Thu Mar 21 23:29:41.079 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._55 Thu Mar 21 23:30:15.888 [conn22379] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 6108, after repl: 6108, at end: 6108 } Thu Mar 21 23:30:15.888 [conn22379] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:41 reslen:3538 6204ms Thu Mar 21 23:30:16.608 [conn22395] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:38 reslen:3198 416ms Thu Mar 21 23:30:37.189 [rsSync] clone newsblur.feed_push_history 3495423 Thu Mar 21 23:30:38.004 [rsSync] 3578027 objects cloned so far from collection newsblur.feed_push_history Thu Mar 21 23:30:50.388 [rsSync] build index newsblur.feed_push_history { _id: 1 } Thu Mar 21 23:30:59.962 [rsSync] external sort used : 5 files in 9 secs Thu Mar 21 23:31:09.004 [rsSync] Index: (2/3) BTree Bottom Up Progress: 3905500/4208125 92% Thu Mar 21 23:31:09.628 [rsSync] done building bottom layer, going to commit Thu Mar 21 23:31:11.204 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 23:31:11.233 [rsSync] build index done. scanned 4208125 total records. 20.844 secs Thu Mar 21 23:31:12.600 [rsSync] build index newsblur.interactions { _id: 1 } Thu Mar 21 23:31:12.672 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 23:31:12.674 [rsSync] build index done. scanned 28919 total records. 0.073 secs Thu Mar 21 23:31:24.545 [rsSync] build index newsblur.activities { _id: 1 } Thu Mar 21 23:31:25.458 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 23:31:25.477 [rsSync] build index done. scanned 365687 total records. 0.931 secs Thu Mar 21 23:31:27.380 [rsSync] build index newsblur.sent_emails { _id: 1 } Thu Mar 21 23:31:27.418 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 23:31:27.419 [rsSync] build index done. scanned 15566 total records. 0.039 secs Thu Mar 21 23:32:23.904 [rsSync] build index newsblur.uploaded_opml { _id: 1 } Thu Mar 21 23:32:23.984 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 23:32:23.986 [rsSync] build index done. scanned 18531 total records. 0.081 secs Thu Mar 21 23:32:24.009 [rsSync] build index newsblur.category { _id: 1 } Thu Mar 21 23:32:24.010 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 23:32:24.011 [rsSync] build index done. scanned 9 total records. 0 secs Thu Mar 21 23:32:24.290 [rsSync] build index newsblur.category_site { _id: 1 } Thu Mar 21 23:32:24.291 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 23:32:24.291 [rsSync] build index done. scanned 52 total records. 0 secs Thu Mar 21 23:32:24.606 [rsSync] build index newsblur.follow_request { _id: 1 } Thu Mar 21 23:32:24.607 [rsSync] fastBuildIndex dupsToDrop:0 Thu Mar 21 23:32:24.607 [rsSync] build index done. scanned 55 total records. 0 secs Thu Mar 21 23:32:24.607 [rsSync] replSet initial sync data copy, starting syncup Thu Mar 21 23:32:24.607 [rsSync] oplog sync 1 of 3 Thu Mar 21 23:32:24.633 [rsSync] replSet error possible failover clock skew issue? 514b7580:62 Thu Mar 21 23:32:24.633 [repl writer worker 1] build index newsblur.stories { story_feed_id: 1, story_date: -1 } Thu Mar 21 23:32:34.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 1351800/37848833 3% Thu Mar 21 23:32:44.005 [repl writer worker 1] Index: (1/3) External Sort Progress: 2518900/37848833 6% Thu Mar 21 23:32:54.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 3767400/37848833 9% Thu Mar 21 23:33:04.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 4979000/37848833 13% Thu Mar 21 23:33:15.016 [repl writer worker 1] Index: (1/3) External Sort Progress: 6000000/37848833 15% Thu Mar 21 23:33:25.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 7267800/37848833 19% Thu Mar 21 23:33:35.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 8524500/37848833 22% Thu Mar 21 23:33:45.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 9610800/37848833 25% Thu Mar 21 23:33:55.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 10580100/37848833 27% Thu Mar 21 23:34:05.010 [repl writer worker 1] Index: (1/3) External Sort Progress: 11501200/37848833 30% Thu Mar 21 23:34:15.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 11936400/37848833 31% Thu Mar 21 23:34:25.010 [repl writer worker 1] Index: (1/3) External Sort Progress: 12268900/37848833 32% Thu Mar 21 23:34:35.005 [repl writer worker 1] Index: (1/3) External Sort Progress: 12641600/37848833 33% Thu Mar 21 23:34:45.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 13057000/37848833 34% Thu Mar 21 23:34:55.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 13979000/37848833 36% Thu Mar 21 23:35:05.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 14714400/37848833 38% Thu Mar 21 23:35:15.005 [repl writer worker 1] Index: (1/3) External Sort Progress: 15469300/37848833 40% Thu Mar 21 23:35:25.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 16267200/37848833 42% Thu Mar 21 23:35:35.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 17083400/37848833 45% Thu Mar 21 23:35:45.527 [repl writer worker 1] Index: (1/3) External Sort Progress: 18000000/37848833 47% Thu Mar 21 23:35:56.980 [repl writer worker 1] Index: (1/3) External Sort Progress: 19000000/37848833 50% Thu Mar 21 23:36:08.109 [repl writer worker 1] Index: (1/3) External Sort Progress: 20000000/37848833 52% Thu Mar 21 23:36:19.253 [repl writer worker 1] Index: (1/3) External Sort Progress: 21000000/37848833 55% Thu Mar 21 23:36:30.261 [repl writer worker 1] Index: (1/3) External Sort Progress: 22000000/37848833 58% Thu Mar 21 23:36:40.005 [repl writer worker 1] Index: (1/3) External Sort Progress: 22865500/37848833 60% Thu Mar 21 23:36:50.005 [repl writer worker 1] Index: (1/3) External Sort Progress: 23561400/37848833 62% Thu Mar 21 23:37:00.005 [repl writer worker 1] Index: (1/3) External Sort Progress: 24213300/37848833 63% Thu Mar 21 23:37:10.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 24892900/37848833 65% Thu Mar 21 23:37:20.005 [repl writer worker 1] Index: (1/3) External Sort Progress: 25328900/37848833 66% Thu Mar 21 23:37:30.007 [repl writer worker 1] Index: (1/3) External Sort Progress: 25955100/37848833 68% Thu Mar 21 23:37:40.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 26434100/37848833 69% Thu Mar 21 23:37:50.005 [repl writer worker 1] Index: (1/3) External Sort Progress: 27046100/37848833 71% Thu Mar 21 23:38:00.005 [repl writer worker 1] Index: (1/3) External Sort Progress: 27769900/37848833 73% Thu Mar 21 23:38:10.006 [repl writer worker 1] Index: (1/3) External Sort Progress: 28328800/37848833 74% Thu Mar 21 23:38:22.058 [repl writer worker 1] Index: (1/3) External Sort Progress: 29000000/37848833 76% Thu Mar 21 23:38:32.007 [repl writer worker 1] Index: (1/3) External Sort Progress: 29702000/37848833 78% Thu Mar 21 23:38:42.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 30021200/37848833 79% Thu Mar 21 23:38:52.005 [repl writer worker 1] Index: (1/3) External Sort Progress: 30824800/37848833 81% Thu Mar 21 23:39:02.005 [repl writer worker 1] Index: (1/3) External Sort Progress: 31491000/37848833 83% Thu Mar 21 23:39:12.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 32151000/37848833 84% Thu Mar 21 23:39:24.450 [repl writer worker 1] Index: (1/3) External Sort Progress: 33000000/37848833 87% Thu Mar 21 23:39:34.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 33698500/37848833 89% Thu Mar 21 23:39:44.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 34107500/37848833 90% Thu Mar 21 23:39:54.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 34738500/37848833 91% Thu Mar 21 23:40:04.005 [repl writer worker 1] Index: (1/3) External Sort Progress: 35257800/37848833 93% Thu Mar 21 23:40:14.005 [repl writer worker 1] Index: (1/3) External Sort Progress: 35916000/37848833 94% Thu Mar 21 23:40:24.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 36614400/37848833 96% Thu Mar 21 23:40:34.005 [repl writer worker 1] Index: (1/3) External Sort Progress: 37206800/37848833 98% Thu Mar 21 23:40:43.562 [repl writer worker 1] external sort used : 38 files in 498 secs Thu Mar 21 23:40:53.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 1821100/37848833 4% Thu Mar 21 23:41:03.306 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 3798600/37848833 10% Thu Mar 21 23:41:13.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 5776600/37848833 15% Thu Mar 21 23:41:23.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 7609600/37848833 20% Thu Mar 21 23:41:33.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 9624000/37848833 25% Thu Mar 21 23:41:43.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 11541800/37848833 30% Thu Mar 21 23:41:53.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 13517800/37848833 35% Thu Mar 21 23:42:01.842 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.42, filling with zeroes... Thu Mar 21 23:42:01.846 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.42, size: 2047MB, took 0.003 secs Thu Mar 21 23:42:03.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 15371500/37848833 40% Thu Mar 21 23:42:13.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 17278000/37848833 45% Thu Mar 21 23:42:23.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 19190300/37848833 50% Thu Mar 21 23:42:33.005 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 21094900/37848833 55% Thu Mar 21 23:42:43.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 22851100/37848833 60% Thu Mar 21 23:42:53.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 24762800/37848833 65% Thu Mar 21 23:43:03.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 26581300/37848833 70% Thu Mar 21 23:43:13.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 28404700/37848833 75% Thu Mar 21 23:43:23.310 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 30128200/37848833 79% Thu Mar 21 23:43:33.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 31941700/37848833 84% Thu Mar 21 23:43:43.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 33802800/37848833 89% Thu Mar 21 23:43:53.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 35609400/37848833 94% Thu Mar 21 23:44:03.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 37531600/37848833 99% Thu Mar 21 23:44:04.899 [repl writer worker 1] done building bottom layer, going to commit Thu Mar 21 23:44:17.852 [repl writer worker 1] build index done. scanned 37848833 total records. 713.218 secs Thu Mar 21 23:44:17.866 [websvr] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 546945, after repl: 546945, at end: 546945 } Thu Mar 21 23:44:17.867 [websvr] Socket http response send() errno:32 Broken pipe 127.0.0.1:48630 Thu Mar 21 23:44:17.867 [conn23378] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 439446, after repl: 439446, at end: 439446 } Thu Mar 21 23:44:17.867 [conn23378] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:125 reslen:3538 445129ms Thu Mar 21 23:44:17.867 [conn23352] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 451273, after repl: 451273, at end: 451273 } Thu Mar 21 23:44:17.867 [conn23352] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:79 reslen:3538 457106ms Thu Mar 21 23:44:17.868 [conn23405] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 20, after globalLock: 20, after indexCounters: 20, after locks: 20, after network: 20, after opcounters: 20, after opcountersRepl: 20, after recordStats: 427283, after repl: 427283, at end: 427283 } Thu Mar 21 23:44:17.868 [conn23405] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:29 reslen:3538 432664ms Thu Mar 21 23:44:17.869 [conn23471] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 403476, after repl: 403476, at end: 403486 } Thu Mar 21 23:44:17.869 [conn23471] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 reslen:3538 408547ms Thu Mar 21 23:44:17.869 [conn24148] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 130778, after repl: 130778, at end: 130788 } Thu Mar 21 23:44:17.869 [conn24148] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:96 reslen:3538 132344ms Thu Mar 21 23:44:17.869 [conn24202] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 106773, after repl: 106773, at end: 106783 } Thu Mar 21 23:44:17.870 [conn24202] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:82 reslen:3538 108062ms Thu Mar 21 23:44:17.870 [conn24085] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 154743, after repl: 154743, at end: 154753 } Thu Mar 21 23:44:17.870 [conn24116] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 142799, after repl: 142799, at end: 142809 } Thu Mar 21 23:44:17.870 [conn24085] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:112 reslen:3538 156593ms Thu Mar 21 23:44:17.870 [conn24116] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:134 reslen:3538 144510ms Thu Mar 21 23:44:17.870 [conn24235] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 94801, after repl: 94801, at end: 94811 } Thu Mar 21 23:44:17.870 [conn24235] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:30 reslen:3538 95946ms Thu Mar 21 23:44:17.871 [conn23405] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:46475] Thu Mar 21 23:44:17.871 [conn23530] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:46483] Thu Mar 21 23:44:17.871 [conn23378] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:46474] Thu Mar 21 23:44:17.871 [conn23352] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:46473] Thu Mar 21 23:44:17.871 [conn23504] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 391502, after repl: 391502, at end: 391512 } Thu Mar 21 23:44:17.873 [conn23504] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:824 reslen:3538 396419ms Thu Mar 21 23:44:17.873 [conn23504] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:46482] Thu Mar 21 23:44:17.871 [conn23471] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:46479] Thu Mar 21 23:44:17.873 [websvr] Socket http response send() errno:32 Broken pipe 127.0.0.1:48633 Thu Mar 21 23:44:17.878 [websvr] Socket http response send() errno:32 Broken pipe 127.0.0.1:48636 Thu Mar 21 23:44:17.881 [websvr] Socket http response send() errno:32 Broken pipe 127.0.0.1:48638 Thu Mar 21 23:44:17.937 [conn23552] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:46484] Thu Mar 21 23:44:17.938 [conn23440] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:46478] Thu Mar 21 23:44:18.616 [repl writer worker 1] build index newsblur.activities { user_id: 1, date: -1 } Thu Mar 21 23:44:20.148 [repl writer worker 1] build index done. scanned 365687 total records. 1.532 secs Thu Mar 21 23:44:20.260 [repl writer worker 1] build index newsblur.classifier_tag { user_id: 1, feed_id: 1 } Thu Mar 21 23:44:21.085 [repl writer worker 1] build index done. scanned 213503 total records. 0.824 secs Thu Mar 21 23:44:21.145 [repl writer worker 1] build index newsblur.classifier_tag { feed_id: 1 } Thu Mar 21 23:44:21.797 [repl writer worker 1] build index done. scanned 213503 total records. 0.651 secs Thu Mar 21 23:44:21.895 [repl writer worker 1] build index newsblur.social_profile { follower_user_ids: 1 } Thu Mar 21 23:44:22.318 [repl writer worker 1] build index done. scanned 68182 total records. 0.423 secs Thu Mar 21 23:44:22.370 [rsSyncNotifier] repl: old cursor isDead, will initiate a new one Thu Mar 21 23:44:22.386 [repl writer worker 1] build index newsblur.feed_push_history { feed_id: 1 } Thu Mar 21 23:44:32.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 2666000/4208125 63% Thu Mar 21 23:44:41.941 [repl writer worker 1] external sort used : 5 files in 19 secs Thu Mar 21 23:44:51.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 3987400/4208125 94% Thu Mar 21 23:44:51.434 [repl writer worker 1] done building bottom layer, going to commit Thu Mar 21 23:44:52.097 [repl writer worker 1] build index done. scanned 4208125 total records. 29.71 secs Thu Mar 21 23:44:52.392 [repl writer worker 1] build index newsblur.activities { category: 1 } Thu Mar 21 23:44:53.576 [repl writer worker 1] build index done. scanned 365687 total records. 1.183 secs Thu Mar 21 23:44:53.656 [repl writer worker 1] build index newsblur.classifier_feed { social_user_id: 1 } Thu Mar 21 23:44:53.770 [repl writer worker 1] build index done. scanned 41530 total records. 0.113 secs Thu Mar 21 23:44:53.791 [rsSync] warning replset primary is an older version than we are; upgrade recommended Thu Mar 21 23:44:53.792 [repl writer worker 1] build index newsblur.classifier_tag { user_id: 1, social_user_id: 1 } Thu Mar 21 23:44:54.593 [repl writer worker 1] build index done. scanned 213503 total records. 0.801 secs Thu Mar 21 23:44:54.641 [repl writer worker 1] build index newsblur.classifier_author { feed_id: 1 } Thu Mar 21 23:44:54.716 [repl writer worker 1] build index done. scanned 23270 total records. 0.074 secs Thu Mar 21 23:44:54.725 [repl writer worker 1] build index newsblur.social_profile { following_user_ids: 1 } Thu Mar 21 23:44:55.101 [repl writer worker 1] build index done. scanned 68182 total records. 0.375 secs Thu Mar 21 23:44:55.157 [repl writer worker 1] build index newsblur.userstories { feed_id: 1, read_date: -1 } Thu Mar 21 23:45:05.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 2066700/10354884 19% Thu Mar 21 23:45:15.676 [repl writer worker 1] Index: (1/3) External Sort Progress: 4000000/10354884 38% Thu Mar 21 23:45:22.388 [rsSyncNotifier] Socket recv() timeout 199.15.249.99:27017 Thu Mar 21 23:45:22.388 [rsSyncNotifier] SocketException: remote: 199.15.249.99:27017 error: 9001 socket exception [3] server [199.15.249.99:27017] Thu Mar 21 23:45:22.388 [rsSyncNotifier] DBClientCursor::init call() failed Thu Mar 21 23:45:26.564 [repl writer worker 1] Index: (1/3) External Sort Progress: 6000000/10354884 57% Thu Mar 21 23:45:37.459 [repl writer worker 1] Index: (1/3) External Sort Progress: 8000000/10354884 77% Thu Mar 21 23:45:48.399 [repl writer worker 1] Index: (1/3) External Sort Progress: 10000000/10354884 96% Thu Mar 21 23:45:50.394 [repl writer worker 1] external sort used : 11 files in 55 secs Thu Mar 21 23:46:00.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 3195200/10354884 30% Thu Mar 21 23:46:10.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 6481400/10354884 62% Thu Mar 21 23:46:20.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 9888400/10354884 95% Thu Mar 21 23:46:21.556 [repl writer worker 1] done building bottom layer, going to commit Thu Mar 21 23:46:24.119 [repl writer worker 1] build index done. scanned 10354884 total records. 88.962 secs Thu Mar 21 23:46:24.121 [websvr] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 79350, after repl: 79350, at end: 79350 } Thu Mar 21 23:46:24.593 [journal] old journal file will be removed: /var/lib/mongodb/journal/j._56 Thu Mar 21 23:46:24.651 [repl writer worker 1] build index newsblur.interactions { user_id: 1, date: -1 } Thu Mar 21 23:46:24.654 [rsSyncNotifier] replset tracking exception: exception: 9001 socket exception [FAILED_STATE] for db05:27017 Thu Mar 21 23:46:24.794 [repl writer worker 1] build index done. scanned 28919 total records. 0.142 secs Thu Mar 21 23:46:24.822 [repl writer worker 1] build index newsblur.classifier_title { user_id: 1, feed_id: 1 } Thu Mar 21 23:46:24.932 [repl writer worker 1] build index done. scanned 25541 total records. 0.109 secs Thu Mar 21 23:46:24.959 [repl writer worker 1] build index newsblur.statistics { key: 1 } Thu Mar 21 23:46:24.960 [repl writer worker 1] build index done. scanned 17 total records. 0 secs Thu Mar 21 23:46:24.961 [repl writer worker 1] build index newsblur.userstories { user_id: 1, feed_id: 1, story_id: 1 } Thu Mar 21 23:46:34.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 1508000/10354884 14% Thu Mar 21 23:46:48.071 [repl writer worker 1] Index: (1/3) External Sort Progress: 2947900/10354884 28% Thu Mar 21 23:46:58.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 4504200/10354884 43% Thu Mar 21 23:47:12.150 [repl writer worker 1] Index: (1/3) External Sort Progress: 5852200/10354884 56% Thu Mar 21 23:47:22.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 7221800/10354884 69% Thu Mar 21 23:47:32.004 [repl writer worker 1] Index: (1/3) External Sort Progress: 8684900/10354884 83% Thu Mar 21 23:47:44.395 [repl writer worker 1] Index: (1/3) External Sort Progress: 9687700/10354884 93% Thu Mar 21 23:47:50.050 [repl writer worker 1] external sort used : 11 files in 85 secs Thu Mar 21 23:48:00.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 2027100/10354884 19% Thu Mar 21 23:48:10.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 4050900/10354884 39% Thu Mar 21 23:48:20.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 6049600/10354884 58% Thu Mar 21 23:48:27.689 [FileAllocator] allocating new datafile /var/lib/mongodb/newsblur.43, filling with zeroes... Thu Mar 21 23:48:27.694 [FileAllocator] done allocating datafile /var/lib/mongodb/newsblur.43, size: 2047MB, took 0.004 secs Thu Mar 21 23:48:30.004 [repl writer worker 1] Index: (2/3) BTree Bottom Up Progress: 7903300/10354884 76% Thu Mar 21 23:48:39.569 [repl writer worker 1] ERROR: exception: E11000 duplicate key error index: newsblur.userstories.$user_id_1_feed_id_1_story_id_1 dup key: { : 110143, : 2012, : "http://feeds.gawker.com/io9/455883497" } on: { ts: Timestamp 1363810094000|51, h: 8083352424645111176, op: "i", ns: "newsblur.system.indexes", o: { unique: true, name: "user_id_1_feed_id_1_story_id_1", key: { user_id: 1, feed_id: 1, story_id: 1 }, ns: "newsblur.userstories", background: false } } Thu Mar 21 23:48:39.569 [repl writer worker 1] Fatal Assertion 16361 0xdcae01 0xd8ab83 0xc1ecb6 0xd986b1 0xe13709 0x7f7e43b18e9a 0x7f7e42e2bcbd /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdcae01] /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xa3) [0xd8ab83] /usr/bin/mongod(_ZN5mongo7replset21multiInitialSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x166) [0xc1ecb6] /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) [0xd986b1] /usr/bin/mongod() [0xe13709] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7f7e43b18e9a] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f7e42e2bcbd] Thu Mar 21 23:48:39.576 [repl writer worker 1] ***aborting after fassert() failure Thu Mar 21 23:48:39.576 Got signal: 6 (Aborted). Thu Mar 21 23:48:39.579 Backtrace: 0xdcae01 0x6ce879 0x7f7e42d6e4a0 0x7f7e42d6e425 0x7f7e42d71b8b 0xd8abbe 0xc1ecb6 0xd986b1 0xe13709 0x7f7e43b18e9a 0x7f7e42e2bcbd /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdcae01] /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x6ce879] /lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x7f7e42d6e4a0] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7f7e42d6e425] /lib/x86_64-linux-gnu/libc.so.6(abort+0x17b) [0x7f7e42d71b8b] /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xde) [0xd8abbe] /usr/bin/mongod(_ZN5mongo7replset21multiInitialSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x166) [0xc1ecb6] /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) [0xd986b1] /usr/bin/mongod() [0xe13709] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7f7e43b18e9a] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f7e42e2bcbd]